Listen
Links
Data Scientist (n.): Person who is better at statistics than any software engineer and better at software engineering than any statistician.
— Josh Wills (@josh_wills) May 3, 2012
Subscribe
Transcript
[00:05] Tim: This is episode 1 of the Dads of Toddlers podcast. I’m Tim Hopper. I’m a senior data scientist in the cybersecurity space, and I’m here with 2 friends from the internet, Oscar Boykin and Josh Wills. The 2 of you were talking a couple of weeks ago on Twitter about Oscar’s thread on build systems, and I made a joke about you guys being on a podcast, and one thing led to another, and here we are.
[00:42] Oscar: Yeah, thanks for putting this together. I think it’ll be a fun chat.
[00:45] Josh: It’s an important lesson. Don’t make jokes on the internet. Look what’ll happen to you. Very severe consequences.
[00:53] Tim: You’re a great example of that, Josh.
[00:55] Josh: Thank you.
[00:57] Tim: Both of you have recently left your gainful employment for greater things. Josh, you have a background at Google and Cloudera and Slack most recently. And Oscar, you were at Twitter and then Stripe. And you’re both considering what’s next at the moment. Is that right?
[01:16] Oscar: Yeah. Do you want to introduce yourself a little bit, Josh, and talk about like where you got to this point? You know, you’re— and then I can kind of share some of my thinking as well?
[01:26] Josh: Yeah, sure, man. I don’t know, like, the dark and twisted path that has led me down here. So yeah, my name is Josh Wills. I’ve been an unemployed software engineer data science person for about 3 months now. Slack for 4 years before that, and I used to run data engineering there, and then I worked on Slack search infrastructure, and then it’s services infrastructure stuff kind of generally. I was director of data science at Cloudera, where I founded a Apache project called Apache Crunch, is a data pipeline library. And then I was at Google before that, and I worked on ad auction stuff, and I worked on A/B testing stuff, and other data infrastructure-y, machine learning-y things. So yeah, that’s pretty much me. And I guess I will have to get a job again someday, but I don’t really want one, and I’m going to try to avoid that for as long as I can.
[02:18] Tim: Yeah.
[02:19] Josh: Oscar.
[02:19] Oscar: We have some similarities in the background. My first professional job out of grad school was as a professor at the University of Florida. It turns out being a professor is not all it’s cracked up to be. There’s a lot of fundraising and so on. That’s its own podcast, I guess. But I decided it would be pretty fun to get into industry and leave academia. I joined Twitter in 2011, and that was a pretty exciting time to be getting into data. I was one of the early members of the ads team. There’s a great quote about all these mathematicians and physicists working on getting people to click on ads. I was one of those people.
And yeah, while I was there, I worked on a system that we built at Twitter that kind of helped us make larger-scale data infrastructure problems easier on Hadoop. And that’s how I met Josh. I had a project called Scalding, which was very similar to Crunch in many ways. Actually, there were a few of these things. I mean, actually, there was another one called Scoobi, and actually Spark was being kicked around. The first versions of Spark were being kicked around at about the same time. I think that’s how we met. Then I was at Twitter for about 5 years, and then I was at Stripe for 4, pretty much doing very similar things. At Stripe, it was more focused on, by that point in time, machine learning infrastructure, so directly productionizing our machine learning models for card fraud, for protecting against merchant fraud, etc.
But mostly on the infrastructure side, very little work training the models at Stripe. Yeah, and I left and just trying to enjoy. I happen to live on the island of Maui and everyone assumes that my life is all surfing and sunshine and rainbows. But I was actually behind a computer most of the time, so I’ve been enjoying the last month and a half or so of really enjoying Maui.
[04:25] Josh: That’s so great. I’m so happy for you. That’s wonderful.
[04:28] Tim: One of the topics we had talked about having you guys discuss is just how things have changed in maybe the last 10 years of your time in these tech startup worlds. I met both of you, I think 2012, 2013 era when data science and being a data scientist was the hottest thing in the world. And big data was something still people talked about. And I don’t know if anyone talks about big data anymore.
[04:58] Josh: I certainly have not.
[04:59] Tim: How have we gotten from that to here?
[05:03] Josh: A lot of Python. A lot of Python, I think, really was the key. Just a lot of Python glue. There’s so much more Python now.
[05:13] Oscar: Yeah.
[05:14] Josh: Yeah. Go ahead. Go ahead, man. What do you think? I don’t know.
[05:15] Oscar: Well, I mean, I think, no, that’s an interesting point, but there’s like, yeah, there’s a lot of layers to it. I do. I mean, there is the interesting social layer of the kind of trend nature of like, you know, the words that people use, like big data, data scientist, even the term, like, what was it? DJ Patil, I think, is credited with coining that term. And then Josh, I think, is maybe one of the— are you credited with— did you think of the joke, the statistician programmer joke. You know the one I’m talking about, Josh?
[05:44] Josh: I do. The famous tweet, a data scientist is someone who’s better at statistics than any software engineer and better at software engineering than any statistician.
[05:53] Tim: That’s your doctoral dissertation, isn’t it, Josh?
[05:56] Josh: It is. I actually literally, I probably have more citations for that tweet in the academic literature than Oscar has for his PhD dissertation.
[06:06] Oscar: I think it’s probably accurate. That’s probably accurate.
[06:09] Josh: Yeah, I was trying to be clever one day back in 2012, and that tweet, which will go on my tombstone, was the result. But back then, I don’t know how you two felt, but you could almost know everybody who had this weird job title. There weren’t that many people, broadly speaking, who did it, and we all sort of found each other. Now, obviously, it’s huge. You couldn’t possibly keep track of all the data scientists and all the variations on the title these days. It’s gone crazy. It’s just blown up like you wouldn’t believe.
[06:49] Oscar: One thing that’s changed a lot is during those early days, you know, if you look at all sorts of, you know, My view is that at an up-and-coming company, which was Twitter, had all these aspirations, but there were— if you look at each of the companies that we would talk to that we considered ourselves our peers or looked up to, all of them had this glued-together infrastructure of several pieces, and that was the price of admission, setting all that up and probably having a pretty competent team do that.
[07:25] Tim: Yeah.
[07:25] Oscar: I think that has changed. I mean, that’s not really, you don’t really have to do those things anymore. You can download Airflow, you can use Spark, you can use like, you know, there’s a lot of things you can just kind of download and install now that really weren’t possible, you know, at that time. And I think it’s actually made the job kind of interesting because there are a lot of data engineers who got into it, you know, just like a startup grows and the startup changes and you have a very different job from, you know, year to year. This whole industry has changed in a way that you have, I think, have a pretty different job from year to year.
[07:59] Josh: I feel like everyone I talk to these days, like everyone is doing a job that’s kind of like one level below whatever their job description is. So like if you’re, if you’re a data scientist, you’re spending most of your time doing what we would think of as data engineer work and like constructing pipelines. And if you’re a data engineer, you’re working on data infrastructure-y sorts of things. And what I mean by that is, at least these days, it seems like you’re running more and more applications in Kubernetes clusters. And then if you’re a data infrastructure engineer, your job is actually running the Kubernetes cluster. And then if you’re a data SRE or data ops person, your job is putting out the fires in the Kubernetes cluster. I don’t know. I hear what you’re saying. S3 and Airflow and Spark, all the visualization stuff, We have so much more stuff now, but it still feels like everyone’s doing a job that’s one level below whatever their job description is. Was that your experience at Stripe? Did you all see that as well?
[09:03] Oscar: Yeah, I would definitely agree with that. I think that’s a very accurate take. The only thing I would say, the only color I would want to add to it is that at least you have these Lego pieces that maybe it’s a crappy set of Legos and you don’t like all of them and it’s like maybe they’re too small to build the thing you want to build, but you kind of know, Here’s a set of things you can try to put together. And most people would think that you’re going to put them— you’re going to put together your org and infrastructure out of like, you know, N out of K of these pieces, right? Whereas in 2012, 2013 even, you know, you would just start writing code basically. It’s like you would probably have to assemble almost everything. Like maybe, you know, Twitter wasn’t on the cloud. You didn’t have cloud infrastructure.
You know, we had a data center, things like this. So it was like everything, there was, So, I think it’s not a lot of fun, honestly, if you’re an engineer, if you like to do it, because it’s like you’re expected, like, here are these tools to play with. Actually, they all kind of suck, and you’re gonna glue them together in some way and grunge things together. And whatever your level is, it feels like you’re dealing with some grungy, half-baked tool, kind of as you described.
[10:13] Josh: Yeah, very much so. Although, it’s funny. I mean, I think it’s like, I don’t know. I don’t think we idealize the past. I remember hanging out— I don’t want to throw too much shade here. I’m always throwing shade in my podcast, but I remember hanging out with Dropbox’s data team once upon a time. I was going in, I’m thinking, all right, we’re going to have a cool data science conversation, talk about awesome streaming tech. And honestly, we spent the entire time about keeping the goddamn NameNode of their Hadoop cluster up and running. Like, that was the all-consuming problem they had was just keeping like the name node alive.
[10:49] Oscar: Right, right, right.
[10:50] Josh: Like 80% of the time. And it’s like, you know, now there’s like S3 and there’s GCS and like you just, you just don’t worry about it. Like it just works, you know? I mean, except for the one day where it doesn’t and then it’s like tech industry snow day. Um, but you know, so it’s like, I don’t know, I, I guess I felt this way at every company I’ve worked at too, where like you’re always so consumed by the problems that are giving you pain today, and you immediately forget about all the problems that you had yesterday that you’ve solved already. And so, like, S3 is great, and we don’t have these— no one has to go debug a NameNode anymore. It just isn’t required. And so, we’ve, like, instantly forgotten that pain and moved on to new sources of pain or something like that.
[11:35] Oscar: Yeah.
[11:36] Josh: There’s a pithy tweet in there. I’m going to try to figure out what it is.
[11:39] Oscar: Yeah, I guess, I mean, humans, that’s how humans work, right? If we were satisfied with any accomplishments, we’d still be sitting around a campfire, right? So, it’s like you have to immediately forget your successes and go on to the parts that are causing you the most pain. Yeah, I mean, I also wouldn’t wanna say that things are, I mean, even S3 has a lot of good aspects. It has some negative aspects as well. I don’t know why, but I have had a really hard time, I feel like, getting any colleagues really interested in the problem of consistency, like, at least, HDFS is a file system, you know? Yeah, yeah, yeah. S3 is not, but people are like, ah, fuck it, who cares? What if we pretend it is a file system?
[12:17] Josh: Yeah, exactly. It’s so true, right? Well, there’s the, you know, at least at Slack, we used EMRFS, which was like that DynamoDB thing that Netflix used that Amazon turned into like a product. That was like our consistency solution.
[12:31] Oscar: Yeah, and I guess that worked well, huh?
[12:35] Josh: It worked pretty good, honestly. Like, it did the job. It solved— like, we had consistency problems, and then, you know, we didn’t have consistency problems. We did have, like— there was one time where I think it took like 8 hours for a file to show up or something like that. But, you know, aside from that, it was great.
[12:50] Tim: One of the things I think about quite a bit is, you know, all 3 of us are kind of math nerds to some degree at heart. Josh and I have operations research backgrounds, and Oscar, a physics background. And the big promise when I was in grad school was like, there’s not been a better time to be a mathematician. And I was like, oh yes, finally, I’ve, you know, my interests and skills in math are gonna be in big demand in industry. And the reality is the last 7 years have felt more like a fight to actually find the opportunities to use math versus do a lot of these types of things that you guys are talking about, which are— fundamentally more like just engineering challenges. It seems like those—
[13:33] Josh: I don’t know about that, man. I mean, if you can use topological sorting and constraint programming systems to figure out whether it’s possible to create a set of Python dependencies for your machine learning system that exists, that’s exciting.
[13:51] Tim: Anaconda already hired Michael Grant, who has a PhD at Stanford, and that’s to work on that problem.
[13:57] Josh: Ah, that’s too bad. That’s a shame. Okay, there’s only one mathematician. The rest of us suffer. That’s right.
[14:06] Tim: Well, actually, I do think that’s one of the interesting things is with math, like, you— the implementation type things that are really interesting, once you implement them, you don’t have to do it again, and then you can use them. Like, we use all these machine learning libraries that have a lot of interesting math going on underneath, but they become commodity— the math becomes commoditized, right?
[14:23] Oscar: I think that’s true, though. The algorithms are the easiest thing to commoditize, right? If you can put them into code, it works. I don’t know what Josh is going to say. I’ll yield to him, but I imagine you might say that the difference is the intuition and good sense of when to use what. That’s not very easy to commoditize. Some of it is taste or just awareness of the map of the space of tools that exist. So when you see a problem, you know what to apply. A mathematician as a directory of knowledge is very difficult to commoditize.
[15:07] Josh: Oscar, I’m gonna say one of, like, my, my favorite of your projects, which was, like, the monoid library that you created. Uh, I think that was derived from Scalding and stuff like that. Um, yeah, it’s very, you know, general idea of, like, a semigroup and a monoid as a way of representing associative computations that comes up all the time in data processing. Like, what is data processing if not a long series of associative operations? And I liked the overarching framework, the organization of addition and hyperlog-log counter and all this kind of stuff falling underneath one library.
And the thing I would say that I think where I think math is still useful, first and foremost, one is the law of leaky abstractions, which is that even if the algorithm is written for you, you still have to understand how the thing works because you have to know when is it appropriate and when is it going to fail. Even if you’re using the HyperLogLog estimator or some other sketch, what is the error bound going to be on the parameters you’re setting and what is the right trade-off for the problem you’re choosing? I don’t know. I still love nerding out on that stuff. Even if the algorithm is implemented in a particularly elegant way as Oscar did in that library, I still dig this stuff, and I like the framing of it.
I think that’s sort of the sort of like, I’m like a closet Haskell programmer, even though I mostly make fun of functional programming, I still deeply admire the elegance of the abstractions and stuff like that. That’s the mathematician in me.
[16:36] Oscar: There’s also a hiring piece to this puzzle. Stripe is a great place to work and attracts a lot of applicants, so getting applicants isn’t difficult. But at a growing company, you struggle to hire someone if you’re worried they don’t have the engineering expertise or experience to solve their own problems. That gets back to Josh’s joke about being a better programmer than any statistician. If you hire someone who’s really strong in math but really weak at getting everything done, a lot of these companies favor a certain amount of independence. There isn’t going to be someone to help you. In many cases, you have to do a lot of the work to get your idea into production.
And so you could say, why don’t we train like an LSTM on this data and use the following features to train a model to predict X, Y, or Z? That’s relatively straightforward to say. And if I gave you a CSV with a sample of all the data, you could probably do it in a day if you’re experienced with it. Maybe you’d have a pretty clear idea as to whether or not the model was good, but actually getting that model to be usable by Stripe as an entity, that’s going to be probably months of work to integrate with all of the systems, to monitor, is the model correct? Is the model still correct? Has the input features, have they changed? Et cetera and so forth. And that’s the challenge, I think, with, back to your point, Tim, about it being a great time to be a mathematician. I think that’s right.
But I think that industry isn’t quite ready to just slot in mathematicians without fairly sophisticated engineering expertise.
[18:33] Josh: So Oscar, I’m going to see you on that. I’m going to raise you. I think the problem I felt like we ran into at Slack over and over again, and again, another problem I see in people I talk to these days, is really the lack of generalizable infrastructure. And, you know, generalized, like we talk about monoids as a generalizable abstraction for this idea of an associative operation. I see a lot of data products, machine learning models and otherwise, that are really being built in this super, super, super specific kind of one-off way and not a lot of focus on generalizable, reusable feature computations, reusable ways of ensuring that you don’t run into data leakage problems and stuff like that. So, I mean, I hear you in the sense that, like, you know, we— you’re exactly right.
Like, pure math is not— pure math alone is not going to, like, get the job done when it comes to having impact on the business and getting products shipped out the door. But at the same time, I think that instinct for, like, designing things in a way where, like, generalization is, like, a thing. You know, it’s so funny when I think about, like, writing Crunch, like, way back in the day. So the genesis of Crunch was, I mean, it was similar to like a lot of other people. I worked at Google, and Google wrote that paper about FlumeJava, which was like their high-level MapReduce API, and it was great because it really did a great job of explaining like these 4 fundamental primitive operations that would go into constructing data pipelines that could be used to describe them.
Like, they did a great job of creating both a tool and a library that was useful for actually solving real problems, but in a way where you could extract the principles from it and stuff like that and make them useful in lots of different contexts. And I feel like that’s what I’m not seeing enough of these days. And that’s what kind of bums me out. Everything feels very, very one-off and not generic enough. I don’t know. I feel like we’ve almost gone too hard in that direction.
[20:33] Oscar: Well, Josh, what do you make of this? I’m sure you’ve heard the term differentiable programming, which tries to generalize the notion of deep learning and say: we’re going to parameterize some function. I don’t know what all these parameters are, but it’s differentiable. Then I have some loss function I’m trying to minimize, and I just go to town. I might say that’s pretty much the antithesis of writing generic libraries, right? You come up with some particular architecture that you imagine will be useful for this particular dataset. Then you train it on some particular data, and it’s probably not transferable to a very different domain.
[21:26] Josh: Hmm. That is a super interesting question. That is, let me give that some thought. So yes, I have heard the term, and yes, I have seen things that way. I’ve seen people talk about this kind of problem. I think Andrej Karpathy wrote a great— he wrote an essay called Programming 2.0, which is essentially this, which is that we’re not going to program on code, we’re going to program on data, and we’ll program on our data and our differentiable functions, and we will generate customized solutions for all the things. That’s such a good question. I guess, like, I feel like I’m like, for whatever reason, I’m gonna get into biology, uh, when I answer this. Like, so dolphins are mammals, right? But dolphins evolved fins in the same way that fish have evolved fins, right?
So even though, like, dolphins and other fish diverged from each other, you know, from a genetic perspective, like millions of years ago, they still re-arrive at the same general principle solution through an evolutionary process. And so I guess I think it’s fascinating to apply these differential programming techniques to things like vision and things like, say, natural language processing, where we don’t really understand. We don’t really have good generalizable principles that we can use to build things. But what we discover Like, I, I love, I love the field, the subfield of, uh, you’ve got to see BERTology. Like, this is like linguists who study BERT, the natural language, like the, the kind of transformer model.
[23:01] Oscar: I don’t, yeah, I don’t know people who study that, but yeah, I know the, I know the model that you’re talking about.
[23:04] Josh: Yep. Yeah. So there’s the model, like there’s a whole subdiscipline of linguistics called BERTology, which is just like linguists studying BERT and trying to like understand how it works and kind of tease it apart. Because I, I think that’s, I think that’s true. I think that if we apply that technique to lots of different problems, I strongly suspect that we would see the same kind of patterns occurring over and over again. And that learning in those patterns and understanding those patterns would be incredibly— I mean, Oscar, it wasn’t like when you were creating the monoid library, the bijection stuff and all that, you weren’t up in your ivory tower. You were solving real problems.
[23:39] Oscar: Right.
[23:39] Josh: And because you were solving problems in the context of the person who had this background that you had, You saw the generalizable pattern that fell out from them. And that’s fantastic. That’s the way it’s supposed to work. That’s how MapReduce was born. That’s great. But I don’t want to dunk on it too much. But I do kind of feel like we’ve reached the point with some of this ML stuff where there are some generalizable patterns. And we do need to write them down and get them out for everyone to use. And maybe I should be doing that instead of just playing with my son all day. I’m not sure. Anyway.
[24:09] Oscar: I hope he doesn’t hear this recording later.
[24:12] Josh: I think I gave— like, I did another podcast where I talked about how, uh, it took my wife and I 4 days to come up with his name. So I’m looking forward to obviously the internet no longer existing in about 5 or 6 years so he never finds out about this stuff. Anyway, if you ever find any— if you ever— it’s been a matter of time, he’s gonna find a bunch of my talks on YouTube where like it’s Dad saying fuck this and fuck that, and that’s gonna be tough to explain too. So anyway. I’ve got a lot of internet debt built up already.
[24:40] Tim: I think we can only hope that our children will one day find our tech talks from the 2010s and devour them.
[24:47] Josh: My dad was a newscaster, and I’ve actually found video of him from like the ’70s on YouTube and stuff like that. And that’s just such a trip for me to like see my dad.
[24:56] Oscar: Oh, that’s amazing.
[24:57] Josh: On TV when he’s like younger than I am now. It’s fantastic. I’m just hoping that— honestly, we just need to get self-driving cars done. In the next 12 years, that’s all I’m living for, self-driving cars. So never have to learn how to drive.
[25:09] Oscar: Well, you know, I mean, I don’t know if we want to, we want to really turn on the old man yelling at the sky thing. I think that’s a good jumping off point because a lot of the folks that I knew who I worked with on ads and data at Twitter went on to do self-driving cars at Uber and Lyft. And I remember them recruiting me in 2016 saying, oh, like, we’re definitely, we’re going to have a product in 4 years. And there was so much groupthink around this that everybody was so sure of that. And I, I would say that one of my real kind of like, I became maybe, you know, you know, cynicism I think is just generally bad, but I think I’ve become a lot more cynical in the last, you know, several years that I feel like a lot of the industry is just like overly trend-driven.
And there’s just like, there’s not a lot of thinking for, there’s not a lot of independent thought. It’s like, you know, Why are self-driving cars coming in 4 years? Because actually, you know, I’ve heard 5 other people who tell me that, and I think they’re smart. And it’s just like this big loop, right? That like those— each of those 4 people are pointing to like a graph. There’s a graph of 5 nodes, and each of the nodes has repeated one thing that they thought kind of half-heartedly, and then they all believe it very strongly because they’ve got 4— they’re connected to 4 other nodes. that say like, yeah, self-driving cars are going to be here in 4 years.
[26:27] Josh: Yeah.
[26:27] Oscar: And I feel like infrastructurally, I see that with technology choices all the time where it’s like, why should you use X? Well, I don’t know. Everybody, that’s the thing to use now. And I get really worried that that causes these really rapid fits and starts of like, well, okay, people can wake up tomorrow and realize actually that’s not the hip thing anymore. Now the hip thing is thing Y. And, uh, it’s just trend chasing, and it’s not really— you know, it’s replacing a lot of like rigorous thought on like, you know, actually evaluating technologies like in and truthfully reporting them. You know, it’s like, I, I feel like you— very many conferences are advertising advertisements for some product, right? It’s like, hey, you should use X. It’s not like, like a real measurement. It’s not a science paper.
It’s not a like, you know, even a, you know, IEEE industrial-type conference, it’s like, I’m gonna try to sell you on using X, Y, and Z. And I’m gonna like, you know, lowball or, you know, hide the faults. I’m going to like oversell the successes. I don’t know. That’s, I don’t know what made me spun out, but you’re kind of feeling that there’s lack of generalizability, kind of relates to me. Like the industry just feels very kind of much trend-driven right now. And it’s like, it feels kind of dangerous to progress.
[27:43] Josh: Yeah. I mean, I’m going to see your angry old man and raise you angrier old man. Blockchain? Really? Blockchain? The whole— I don’t know. There’s sort of these epochs of startups, right? There was the early web stuff, and then there was the social web stuff, and then we had the whole mobile thing, and then we had the blockchain thing. And it’s just an absolute waste of a generation of technologists. And I realize, I mean, obviously, Tim, I hope no one ever listens to this podcast because I’m just going to get such angry tweets at this. Like, you’ve got to be kidding me with this blockchain stuff. It was an absolute— it was like, ah, this is like the chat roulette of technology cycles or whatever. It just drives me crazy.
[28:31] Oscar: Yeah. I mean, I totally agree with you. I mean, that strikes me as a solution-looking for a problem, um, and which I don’t really think that we have a lot of those in the data and ML space, fortunately, where it’s like there’s something you legitimately don’t know what you want to use like a technology for, but like somehow it’s still popular. I don’t, I don’t think we have too many of those, fortunately.
[28:55] Josh: Maybe that’s, maybe that’s our own bias. Maybe it’s possible that there’s like blockchain applications that I am just like, like I just, they elude me. I am just not sophisticated enough to see like what the point of this is. and the fact that I inhabit the data ML space, everything’s like, oh yeah, that kind of makes sense. I don’t know. I had this thought, this idea around the analytics value chain. Can I— and I sound very business school-y when I say that, but I think of it as when I’m doing end-to-end data processing stuff, I need some way of ingesting my data. I need somewhere to store it. I need some way of doing computation on it, and I need some way of visualizing and communicating the results.
And so that could be like, I have a Kafka cluster that feeds data into S3, and I spin up, you know, like a Hadoop cluster or whatever to do processing on it, or I use Snowflake or whatever. I have some sort of compute solution, and then I’m visualizing this, uh, using like Looker or Tableau or any of these kind of different systems. Um, and what I’ve seen now in like in the data space is like the, the sort of the innovation and the startups or whatever are kind of moving to like a meta level on top of that value chain. Like it’s a lot more, people are a lot more focused on like data catalog solutions.
[30:10] Oscar: Yeah.
[30:11] Josh: Or metadata stuff. Did you guys have anything at Stripe that you use for this stuff? Like what, like what did, what have you seen here?
[30:19] Oscar: Uh, no, we didn’t have any good data catalog stuff at all, really, to be honest. So, uh, so I’ve seen 2 really different models. Twitter’s model was, it kind of like, it was more like service-oriented architecture. It wasn’t really mono or microservices, but the side effect, really nice side effect of Twitter’s model was that we just wrote, we had thrift services or, you know, the kids today might know those as like gRPC type stuff, where all you have to do to like satisfy your data scientist to like a decent degree is take the request, the serialized request object that comes in, and the response object that goes out, and you have a schema for them, so that’s relatively nice to work with, and we write them onto a Kafka queue with timestamps, and that’s all you’ve got.
So, that’s actually pretty nice, because it’s the real history of what really, really happened. Stripe took a different model. It’s like a financial type company, so it’s like, it’s got, and so financial data is just like orders of magnitude smaller than social data, so it was always kind of addicted, I would say, maybe, that has a negative connotation, but really liked having full data dumps of all the history of just snapshots of the database. So it was more of a database snapshot type approach. And when I’ve spoken to peers, it seems like people are in both boats. If you’re forced to do the log-based approach, if you ask any data engineer, they love a log-based approach. Log the data as it happens, we recompute views of that, If we need to rewrite the log, we can do it, et cetera.
Almost no data engineer I ever talked to is like, no, no, no, what you need is snapshots of databases that are mutating all the time.
[32:03] Josh: Yeah, that’s fair. That’s fair. Agreed.
[32:06] Oscar: Yeah. So, I just mentioned that. So, back to your point about data catalogs, in the world where you have snapshots of the data, you kind of, like, it’s a little bit more clear what your catalogs are because it’s exactly your database tables. That you already have. So it’s a little ugly because you’re kind of exposing internal structure of the actual app and systems that you’re building to your data scientist, where like, it’s— I think it’s nicer to say, okay, I’m intentionally logging X, Y, and Z according— like, these are semantic things, not like happenstance of data structures that I’m like jamming into a database. I think it’s a more intentional thing to do explicit logging. But what if you— the upside is that the catalog is kind of in some sense handled for you. It’s like a messy catalog, but you have it.
Twitter, on the other hand, did have to build the thing that you’re talking about, a data catalog. People use Hive Metastore for a lot of these use cases. We also built some things on top of that to integrate with more of the systems we had built. Maybe in retrospect, we should have just gone all in on Metastore. I think most people probably think they should have. But anyway, it was a weak coupling to Metastore with an additional service on the side that had extra data. We called this a DAL, Data Abstraction Layer.
[33:22] Josh: That is fascinating. That’s fascinating. Slack, I think, classically had both, like, both the log-based system for the services or whatever you would call them. But they weren’t really like microservices. They were like macroservices by and large. But they all had logs. And then we had the database backup snapshots for, yes, exactly that, for financial reporting reasons and other sort of use cases. and that kind of thing. Heavily, heavily relied on the Hive metastore as the metadata source of truth. Yeah. That’s just— anyway, that’s super interesting. This must be like why I just never ran into this problem. I don’t think— I just didn’t have— we just didn’t have that many different microservices where keeping track of all of them actually was like a problem above and beyond like what you could derive from like just looking at the Thrift schema or the corresponding Hive table for the very obviously named services log definition. That’s super interesting. That’s pretty awesome.
[34:20] Oscar: Well, I think where it really gets interesting for data scientists is the next layer up. You’ve probably read that Google— there’s this famous— I don’t know, the paper isn’t— basically, the title is the best part of the paper almost, but I mean, you can kind of imagine what the rest of it says. But I don’t know, something like machine learning or data science is the high-interest credit card of the technical debt or something.
[34:42] Josh: The D. Skelly paper, Machine Learning: The High-Interest Credit Card of Technical Debt. Yes, excellent paper.
[34:49] Oscar: Right. And it kind of details a lot of the problems that people who’ve worked in the space kind of see, especially one that really stood out with me is that when you start to share artifacts, be it like models or derived feature sets or et cetera, like that coupling across your systems is is like, what systems track that? So most people don’t have tools that really track that. And at scale, it gets really challenging. You’ve got some Team A over here, Team B’s consuming something they wrote, and then Team A decides they don’t want it anymore and they stop making it. And now Team B has an incident. And I saw many, many examples of this in my time at Twitter and at Stripe. And so I think that really the interesting parts of data catalogs is not kind of like the source data or the event data, the database dumps, or even logs. But it’s like, how do you describe and share and talk about how production-ready various datasets are to share across these data teams? That, I think, is still a very open area. Maybe there’s a great tool out there, but I don’t know about it.
[35:53] Josh: That’s a great one. I remember Google had one where the search team changed, like there was a language encoding for every search. It was the 2-letter thing like EN, EF, DE, whatever, right? And then the search team changed it one day to a 4-letter encoding instead of a 2-letter one, and it broke the ad click prediction model fairly catastrophically.
[36:14] Tim: Right, right.
[36:15] Josh: None of those features worked anymore because they were all trained on 2-letter strings and a 4-letter string came in. And that was exactly it. No one told anyone. Why would they? How could they possibly know where this stuff was used? Oh, that’s fantastic. And yes, I am right there with you. I have no idea if anyone is working. I’m sure someone must be working on this problem, but I have not seen a good, keeping with my current theme, appropriately generalizable solution to this problem. But I would be, I would love to see one.
[36:40] Oscar: So, so that brings me around to actually back to like the, the, the point of the conversation that got us going here. I’ve long had this idea that the solution is something like a build system for your data that you describe, okay, here this. So, you know, build systems, you know, if you’re familiar with ones that are like Buck or Bazel or Pants or things like this, they’re— or Make. A lot of people might be familiar with Make. It’s like, if you want to produce this thing, it depends on these K inputs, and then you run this program in a certain way and that spits this thing out. And one, in some sense, like, you know, that is a data build system. What’s the— what do I mean by data build system?
I think one thing different with data is that a lot of these calculations are extremely expensive, like $100,000 or $1 million to train some of these models, and in some cases more. That changes how eagerly you want to invalidate previous artifacts. That’s one challenge for a data build system: the cost of producing artifacts. The next thing is that data build systems usually care about time. My roll-up of all my analytics for today, if it’s an all-time history, might depend on yesterday’s all-time history plus the deltas that came in today. So I want a way to talk about time that most build systems for computer programs don’t have.
You check out the repo and it’s like a fully visible graph. There’s nothing else coming. Future commits will go, but they throw out everything to some degree. There’s caching, etc. There’s some greatness here. But I feel like if we could get, as an industry, a way to integrate our notions of building things with our workflows and our permission systems, like, this is how you produce that language set or that search log you’re talking about, and now, Well, if I change my schema, that’s gonna invalidate something that this build tool kind of detects that your system over there has declared that it depends on it. And we can kind of see if there’s no hidden dependencies, you know? And I think it’s these hidden dependencies that have really caused a lot of these problems.
That was kind of the genesis of the kind of back and forth we were having. And I still am interested in that problem in that space. And I think a lot of people are stabbing at it vaguely, but I don’t, I haven’t seen any really solve it really, really well. I don’t know, like Pachyderm was doing things like this. Airflow, to some degree, you could say is marching in this direction. I know a friend who’s got a Conducto, I think it’s going to be launching, that’s also in this space. But I don’t know, I think that was kind of the— attacking that problem was kind of, to me, top of mind after my time at Twitter and Stripe.
[39:30] Josh: I guess I’m broadly interested in this question as well. I tried to build something, it was a relatively limited thing, but I tried to build something like this at Slack. Slack had the problem of, like, we hired Facebook data scientists who were primarily Hive users and Twitter data engineers who were primarily, like, Scalding users. And the question was, like, how exactly can we integrate these 2 teams together so that they can kind of collaborate on a common codebase when one of them really wants to use Hive and one of them really wants to to use Scalding.
And so my terrible compromise was to bring everything into Spark, which had obviously SQL bits and Scala bits, and try to create a framework for expressing dependencies between the calculations of various columns across tables in a way where, yes, you could track the lineage of a given column and stuff through the build and execution tool at first. This, it turned out, like, I tried this and it just didn’t work for like 18 different reasons. One, I was obviously like, I was at least nominally director of the team and had like other stuff to do besides just working on the tool full-time. And I just fundamentally couldn’t get anyone excited about it.
But like the folks at Fishtown who were doing dbt, like the data build tool, like SQL-based systems, they basically built the thing that I wanted at Slack for representing these relationships and these dependencies. Again, just for the context of SQL, which is not the full machine learning Python, Scala, et cetera, et cetera workflow that folks use today. But they did a really, really good job at representing this stuff. I ended up actually investing in their company because they built something that I had wanted for a long time, just correctly. But yeah, I mean, they have this notion of some computations are incremental and some things can be left as views and some things need to be materialized and stuff like that.
But yeah, getting to the point, I think, where you could just express these costs as a higher— without having to explicitly say, yes, materialize this, yes, incrementally do this, where the system could be smart enough to figure out. Like, this is sort of my desired state of the world. This is how long I want everything to take. This is how quick I want updates to occur. Like, I mean, it’s clichéd at this point, right? But like Kubernetes style, like declared in specification, this is what I want to exist. These are the rules that happen. Go do it. Like, go make it happen. That kind of thing. That’s the dream.
[42:05] Oscar: No, it sounds cool. I haven’t had a chance to dig into that. I saw a really nice blog post by Dmitriy Ryaboy, and he called out dbt. It looked interesting, so that’s cool to hear.
[42:20] Josh: It spoke to me in the sense of, like, with this theme of, you know, basically trying to create, like, it’s funny, it’s like they’re trying to create a new kind of role in companies, like akin to data scientists once upon a time. They’re calling it analytics engineer. Which I think is like, it’s reflecting this idea. I guess what I would be curious for y’all’s perspective on this, it seems like data scientists do a lot more software engineering these days than I feel like data scientists did maybe. I’m not sure that’s true. I think I see data scientists doing more and more software engineering kind of stuff, like across the board.
And I think it’s like, so like way back, like at Google back in the day, In the same way, if you were at Twitter and you wanted to analyze the logs, you need to learn Scala so you can write a Scalding pipeline, and that creates this relatively narrow funnel or whatever for people who want to become data scientists. Google was the same thing. If you wanted to analyze the logs, you needed to learn this terrible programming language called Sawzall that Rob Pike inflicted on Google before he went on to inflict Go on the rest of the world. But again, it was a very high bar. It was like you had to— it was a real programming language. You had to really program to do it. And then when data scientists started taking off, the title became sexy and cool, and then it was like anyone could be a data scientist.
And then nowadays, I kind of see the software engineering aspect of it creeping up more and more. Like, I’m a data scientist, and here I am writing a custom resource for a Kubernetes cluster. In some ways, it’s gotten even more and more software engineering once again. I don’t know if I’m trying to make a point with this.
[44:07] Tim: Interesting.
[44:08] Oscar: I wouldn’t say that I observed that latter movement because when you were at Slack, I was at Stripe. I would say that I did not really see that as much. One of the ways that we organized, we had a pretty hands-on machine learning infrastructure team. That tried to do all those kinds of things that you talked about. So it’s like, we, we, and we had a different problem, but it’s like, you know, here are some rails that you can run your trains on, and it goes to this station and that station, and that’s pretty much it. And if you, if you want to ride those rails, it’s probably going to be really straightforward, you know. Um, you want to go to a town that’s not on the, on the railroad line, you’re, you know, good luck. Good luck and Godspeed. And like, probably Almost nothing got shipped that wasn’t on those rails because of the problems you’re talking about.
[44:55] Josh: But I think that’s the difference. I think at Slack, production machine learning primarily meant search ranking. And so that meant we had to do, like the search ranking team, which again nominally would be working on search ranking, had to do everything, like build up the services infrastructure on the JVM. And, oh, you guys want to use the TensorFlow stuff? Great, here you go. Go build yourself a TensorFlow cluster. And oh, you want to run that in production? Okay, you need to figure out how to productionize the gRPC TensorFlow service. So there was, like, when you only have one machine learning use case at a company, the team that works on it is going to be responsible for building all that infrastructure as well as the, oh yeah, sure, the machine learning model at some point in the future, once you’ve shaved all these yaks and all that kind of good stuff.
[45:41] Oscar: Well, you know, the way that you say that now, it kind of occurs to me, I wonder how you would think about this. My experience with machine learning, actually, I’ve never been on machine learning or worked on machine learning at companies where it was related to, super directly related to the customer, like to some like experience. So like search ranking at Slack or Google, you know, it’s a human, it’s the user is gonna type something in, they want like a good experience. So I worked on like ads at Twitter. So like, it’s basically like, oh, here’s this website. And then there’s an API call that we’re gonna hit at this point. And you’re gonna tell us, you know, stitch in here. And for us, like, we did, you know, you know, the users weren’t demanding the ads.
And so we could actually really cordon off, like, here’s a model, and this model just needs to predict as accurately as possible, um, what the click probability is. And then we use that to do ranking of the ads in, in all the standard ways. And at Stripe, it was somewhat similar. It’s like, here’s a credit card charge, should we block it before we ever hand it over to the, uh, to the, the the banks and the processors, you know, no one really— you know, that’s not like a part of anyone’s experience. In fact, Stripe’s customers are merchants. They’re not actually even the buyers at that point in time. So we could actually super clearly delineate where that call is, like, here’s the part for machine learning magic, right? And so the people who worked on those problems really just worked on models all day long, in most cases.
In most cases, there are some experience aspects to Stripe’s product radar for merchants to visualize what are their block rates, et cetera, and so forth. But the modeling part of it, at that moment, there’s no experience there. There’s no human that’s a customer of Stripe. So I wonder if that plays a role in the difference between what it’s like at Slack, where it’s a big part of your experience for your users.
[47:36] Josh: It’s kind of, I think the funny thing for me about working on search at Slack was it made me appreciate kind of how easy Google has it, like as a search problem in some ways. I realize that’s kind of like blasphemy, right? But all of, like basically all of Google’s stuff that they search is public. And like, you know, if you type in a search query, I can type in the same search query and it’s totally fine if I get to see all the results and stuff like that.
[48:01] Oscar: Right.
[48:02] Josh: That’s not true. It’s not even remotely true. It’s not even true for people who are on the same team. Everyone has a unique search experience. And in fact, it would be disastrous for anyone to see other people’s search results. That would be among the worst possible things that could happen to Slack. And so that set a fairly— the ability— the fact that you cannot look at the corpus that you’re searching over sets a pretty high bar in terms of how good you can ever make the user experience, broadly speaking.
[48:30] Tim: Oh, wow.
[48:31] Oscar: That’s— yeah.
[48:31] Josh: I mean, at Google, I would say to— I mean, for what this is like, it’s hard to be like sort of like rah-rah and Pollyanna-ish about advertising, but I’m gonna do my best. Like, when I was there, the team I worked on was Ads Quality. It was akin to the Search Quality team on the search side. And our goal, honestly, swear to God, lived every day, at least once upon a time, that we wanted people to come to Google for the ads. We wanted the ads to be so good that you would come to Google for the ads. And so that meant that even for the machine learning models, where we would do a major machine learning model release, we had a human evaluation team. We would actually run the ads that would be shown for queries in front of humans who would tell us which machine learning model was better.
It was like a UX thing, just like they would for search results. which again, back in the day, were not a machine learning model. They were hand-tuned. Like, they were hand-tuned, every change. It was kind of a joke. It was like one of the worst bits of code at Google, unsurprisingly, was like the function that did search ranking. It was like 20,000 lines long in one file called the A-scorer. It was just like an absolute disaster. It was so funny. This guy came up with this really elegant hierarchy for plugging in these composable scorers and stuff, and it just never got used. There was one scorer, and it was called the A scorer, and it was just an absolute disaster.
[49:56] Oscar: That’s it?
[49:58] Josh: It was all this beautiful, elegant code surrounding this deep, dark, ugly secret that actually contains the whole brains of the operation. But yeah, it was like, yeah, the UX thing was a really big deal. And you’re exactly right. I think if I was working on fraud or something like that, would be refreshing, I think, to not have to worry about these concerns. But I think too, it’s like, you know, the core of Slack’s business is really chat and chat and channels and stuff. Channels, organization, that’s really the core of their business. Search is a piece of that, but like at Stripe and at Google too, like the fraud, like getting good at fraud, that’s the difference between like success and failure as a business. That’s a really big deal. That’s like not a joke. Google Search, like this is all you do. If you don’t do this good, then like you’re not a very good product.
[50:45] Oscar: Yeah, it also kind of hits me. One of the things you’re talking about here kind of talks to me about the diversity of use cases for machine learning and how different, kind of like how sure you, how clean your labels are, I guess. Because I realized, like, you know, you started this discussion with the problem of Slack where it’s— I don’t know what optimization function you were trying to optimize, you know. Yeah, I guess maybe some of the standard ones for like, you know, but you could— but the fact that you couldn’t inspect them, then over to Google where you could inspect them. But then the problem, you know, there are people out there who are dealing with problems like Stripe where it’s like, it’s totally unambiguous. Like, eventually we will have a label for every charge as to whether or not it was disputed, right? There’s like, like total ground truth situation. And your job is simply, um, well, one of the jobs, um, is, uh, to, to predict that, that bit of information that’s coming from the future.
[51:35] Josh: Totally. I would love that. That sounds fantastic.
[51:38] Oscar: Yeah, it’s really well-defined, right?
[51:41] Josh: And so let me tell you how great it is. Like, I don’t know, it’s an important thing for your career. Like, work on easy problems. Like, like, my thing is like, I don’t know, Google did it right. Like, work on easy, easy lucrative problems. I think I would love— I think at every company, I think data is very useful for growth. I think that’s true at every company. I think at every company Data is very useful for performance, like understanding performance, quality, reliability. Data is incredibly useful. But I think the companies— like the company I want to work at next, whenever I work again, is I want a place that has that kind of core, that has a core machine learning model, that really has a machine learning model at the absolute core of the business, kind of linking— ideally one that links the growth system and the performance system together like that. I just love that. And I super took it for granted having it at Google. And yeah, that’s my number one filter for next code. That’s what I want to do next.
[52:38] Oscar: You really think you’re employable, Josh?
[52:41] Josh: I don’t. I’ll be honest with you. I honestly, like a lot of things, really, I think, especially doing these— it was so funny. I did Software Engineering Daily, which you guys should definitely do sometime. And I threw shade at Mongo. MongoDB, like right off the bat, which again, Stripe runs on MongoDB. So I guess I can’t throw that much shade here. And then I got a recruiter call to come like be the product manager for search at MongoDB. And I was like, ah, it’s not— they’re probably not going to be happy with me. I don’t think that’s going to be a good pair, a good match. No, I may not be employable. I may not be employable. You may well be right.
[53:21] Tim: I don’t think there’s ever going to be any competition again for a Software Engineering Daily guest after you, Josh. You set the bar.
[53:27] Josh: Is that right? So I don’t, I don’t listen to podcasts. So I had no idea so many people listened to this podcast. It was like so funny.
[53:37] Tim: Yeah, it’s huge. I think I can’t keep up with it. It’s amazing, but I can’t keep up with it.
[53:42] Josh: Yeah, absolutely. Yeah. I don’t do anything. I don’t do cereal. I just, I’m like often, I just listen to music and read books.
[53:48] Tim: I don’t even understand the genre We’ve already talked a lot about this, but what’s— this industry has changed so much in the last 10 years. Where’s it going to be 10 years from now?
[53:59] Josh: I think, you know, so I’ll say my nerdy thing. There’s so much Python right now. I’m going to close where I left off. There’s so much Python right now. But I am seeing, like, I’m seeing still this impedance mismatch between a lot of production systems that are written in Go. or Rust or C or whatever, and machine learning systems developed in Python. And there’s the same impedance mismatch in the data structure, within the data systems, which are still primarily JVM-based, and these Python layers on top of them. And I want to really— my fondest wish is that in the next 10 years we solve this. I actually feel like it’s kind of unfortunate. Actually, I’m going to throw another blast from the thing out there. I actually think Scala has become kind of a great data science language. over the past few years.
There’s actually a bunch of really good libraries and stuff for it now. And it’s sort of like, I guess, again, I get it. It’s like GPUs and all this kind of stuff is very important. But it’s like everyone is doing everything in Python, both because they like Python and it’s easy to use. And I think because they’re afraid that at any moment they might have to fit a TensorFlow or a PyTorch model to solve a problem. You know what I mean? Like how everyone in San Francisco walked around wearing camping gear or hiking gear as if we might just go scale Mount Everest. Everyone is writing in Python because just in case we need to fit a deep learning model, it’s okay, we’re in Python. I guess I don’t know what it would take.
Maybe this is a nightmarish version of Python 4, but what would it take for there to be a Python-centric systems language or some systems language that’s designed to interoperate with Python that we could build everything in so we could have just one language for stuff. And it’s kind of like, I mean, if the language has to be Python, then like FML, let’s make it Python. But yeah, I just feel like this impedance mismatch is just kind of killing us. And that’s the one thing I would like to see different in 10 years. That’s my fondest wish.
[56:00] Tim: So that’s more of a dream than a prediction, I think.
[56:02] Josh: But what’s the joke? It’s like the best way to predict the future is to invent it, right? And I’m So let’s invent it. Let’s just do it, right?
[56:09] Tim: You’ve heard it here first, folks. Josh Wills is going to write the next language for data science.
[56:15] Josh: I was actually kind of hoping that someone on the call would hear this and be like, oh yeah, that’s a great idea.
[56:19] Tim: Yeah, I know a good candidate for that.
[56:21] Oscar: Yeah, unfortunately too much of my life has already been being like nerd sniped into like— that’s like, that’s basically Avi Bryant’s role in my life. Nerd sniping. me into a multi-year technical project. I would love to say that in 10 years from now, I don’t know, it’s like, you know, like where we talked about, we’re old men here. I remember coding before people felt like they had to have, certainly, they might not even have a unit test, they might not have a test suite, they certainly probably weren’t measuring code coverage. Maybe they were using a really crappy version control system. Those are things that are actually progress that we’ve really gotten past. Git maybe is ugly to use, but it’s solid.
You’re doing full snapshots of your code, you can go check back out, you can do git bisect, these things work, they’re continuing to get better. So, that was a real step up, and it’s a qualitative change. People do testing now. God, I hope that they do. It seems like type systems are spreading. You’re seeing like TypeScript, you’re seeing Python 3 with type hints. So things kind of get a little bit better. So I would like to believe that these integration problems that we have now between like, you know, the things like I want to go train a model to do X, but actually you have to solve this massive integration pain and write your Kubernetes YAML and everything else, that those are the things that we can actually get solved in a much nicer way. And that the relative fraction, Let me just say, how will things be?
I think the relative fraction of time that you’re doing modeling and you’re doing analysis and like the sciencey part of the data science, I hope can be a larger fraction than it is today in 10 years. That’s my hope. And I would bet that it will happen. That would be my bet. So it’ll happen by some of the problems we talked about just being solved and people rolling up their sleeves. Maybe even one of them is what Josh mentioned.
[58:16] Tim: Yeah.
[58:16] Oscar: But I think we’ll, we’ll actually look back and say that it was just like ridiculous how primitive it was when, you know, in 2011 up to even 2020, you know.
[58:25] Josh: So yeah, I would say amen, amen to that. That’d be fantastic. I love it.
[58:29] Tim: Excellent, guys. Thank you very much for coming on the first and hopefully not only episode of this podcast.
[58:38] Josh: Thanks so much, Tim. This was great. Good to see you, Oscar. Good to see you as always.
[58:42] Oscar: Thanks for, thanks for hosting and putting it together, Tim. Good to, good to talk to you, Josh.