Listen
Links
Subscribe
Transcript
[00:11] Tim: Welcome to episode 4 of the Into the Hopper podcast. In this episode, I talk with Adam Laiacano, staff machine learning engineer at Spotify. Adam studied electrical engineering and worked on atomic clocks before turning to a career in data science about 10 years ago.
[00:26] Tim: Hi, Adam.
[00:27] Tim: Prior to Spotify, he’s worked at places like Tumblr and Stripe. Today I wanted to talk with Adam about how his career has progressed from an atomic clock engineer to leading many machine learning efforts at Spotify, the world’s most popular audio streaming subscription company. Hello, Adam.
[00:47] Adam: Hey, Tim. Thanks for having me. I appreciate it.
[00:48] Tim: Yeah, I’m excited about doing this. I like— it seems with most of my guests, I’ve known you through Twitter for a long time and we’ve met up a couple times in person and you’ve been a big influence on me and in watching your career and talking with you about data science and my career over the years. I think I was a grad student when I came to New York for a conference or something and I met you and you gave me a tour around the Tumblr office in 2011 or ‘12, which would have been probably my first time in a startup office, which was kind of eye-opening for a grad student from North Carolina.
[01:23] Adam: Yeah, that was my first time in a startup office when I worked there as well. It was a radically new thing for me to work at a place like that. So I’m sure we’ll talk about that too.
[01:32] Tim: Rows and rows of desks, like kind of open office. And I remember you had some kind of sparkling water machine, which I thought was pretty cool.
[01:39] Adam: Yeah, I miss those.
[01:42] Tim: You studied electrical engineering. What got you into electrical engineering and going that route?
[01:48] Adam: So I actually started college as a physics major. I went to Northeastern University. And it was a great school, I loved it. And one thing about Northeastern is they have this co-op program where you go— it’s a 5-year program, and by the end of the 5 years you get your normal 4-year degree, but they also get you placed in jobs, and so you have 2 full years of work experience, so you kind of earn an extra year in those 5. And I got a co-op job at this place that designs atomic clocks. And my first time was just twiddling numbers and writing down optical intensity levels of just these different laser settings that we were working with. But I realized that what I liked about it was the building the stuff, not necessarily the pushing the envelope on the understanding of the universe that I was learning about in physics classes. And so I switched to electrical engineering so that I could learn how to, you know, actually take these ideas and turn them into a thing that you can use or hold.
[03:00] Tim: I started in physics also, but I ended up in math for basically the opposite reasons. I didn’t want to do anything with physical objects.
[03:07] Adam: Yeah, I would love to see how many people go in one direction or the other, or just stick with physics all the way through their career.
[03:16] Tim: Yeah. I didn’t even like trying to understand the universe. I wanted all these theories and imaginary things. That’s what I enjoyed. So then, did you continue working at the company where you had started as a part of the co-op?
[03:32] Adam: I did, yeah. I worked there through college and then I worked for 2 or 3 years after that. I think I was there for a total of like 7 or 8 years. And it was awesome working there full-time. It was really interesting because this place was like total Bell Labs style. You’ve got your lab notebook that you initial and date every day, and when you fill the notebook at the end, you file it away in some locker in case we need to file a patent, which we rarely ever did. And they were like, oh, well, we can do stuff with microcontrollers now, right? And so when I joined, my boss had recently started and he came from the MathWorks, and so he was building all these crazy Simulink things where you press a button, it programs an FPGA. And he was like, well, we don’t need that for these particular things, but I started programming all these microcontrollers and just got to learn how to do low-level C and dropping into a little bit of assembly stuff. And it was so much fun. And then kind of seeing that all the way through. I mean, there’s so many components to these atomic clocks that we can talk about or not, but—
[04:44] Tim: What were the applications of the clocks?
[04:47] Adam: Yeah, so there’s a handful of different kinds. There are atomic clocks in all of the GPS satellites that basically anytime you want very, very specific synchronization between things, atomic clocks will be the best for something like that, especially in the longer term. And you can actually get any normal GPS receiver, and it’s a 1 pulse per second signal that is sort of noisy and inconsistent from second to second, but over the course of 3 months, the number of seconds will be exactly the number that you would expect. It doesn’t drift over a very long term.
[05:26] Tim: Okay.
[05:27] Adam: So there’s a lot of applications for that or other sorts of high-frequency communication things. What I was building were very, very low-power atomic clocks, so they ran on half a watt of power versus 40 watts or more usually. And that was for things like underwater GPS networks since our radio frequency doesn’t travel underwater, so they were using these sonar-based underwater GPS things for autonomous underwater vehicles. which was crazy, but I was mostly building prototypes for those sorts of things and never got to go out to the naval yards and deploy them or anything.
[06:11] Tim: So as interesting as it’d be to go into the details of that, I think more interesting for the sake of this podcast to talk more about your career as it progressed from there. I think when I met you, you were working with a professor or professors at Columbia Business School doing data munging and scraping and different things like that. How’d you end up there?
[06:32] Adam: So I went to grad school for a year and I had, for a number of reasons, a not great experience. I think I just didn’t— my expectations were not aligned with reality. I thought I would just take some classes and whatever and was never really part of a research group and never really fully committed to like a particular field or subject or whatever and was feeling kind of bummed about it. And I saw this position that opened up at Columbia Business School to do what basically the 2010 or ‘12 version of data science, which was working with professors to do on their research. And some of it was like gathering data and joining datasets and doing these sort of fuzzy merges to combine datasets together. And some of it was, you know, I wrote this paper and it has, you know, a bunch of code and I would like to be able to run simulations using— or a bunch of equations rather— and I’d like to run some simulations using these things, but like, I don’t know how to code it, or I did and it’s really slow. So I just sort of did a lot of that sort of stuff. My title was research computing specialist. Which is very funny. I’m 100% sure they would call it a data scientist now.
[07:57] Tim: Yeah, of course.
[07:58] Adam: You know, it seemed like an opportunity to take some more classes. You know, I took classes in data mining and statistics and things like that. And then also actually like kind of get that research group vibe that I, that I like missed out on in grad school. And so I sort of like made the move and have since learned that like this program I kind of want to give a plug for it actually because it’s called a pre-doc. That is a thing that I think is more popular in economics and finance, but a lot of the kids that I worked with were straight out of undergrad and I was a few years older. And you go and you work with professors on papers, like co-author papers or work with them and kind of do research and then get a fantastic letter of recommendation. And a lot of the people that I worked with then are now junior faculty at NYU Stern or Harvard Business School or Wharton or Northwestern, I think someone teaches at. And so, it’s like all of these extremely difficult-to-get positions. They’re very smart people, but it was like a great springboard for a lot of them. So any undergrad economists listening to this, look into pre-doc programs.
[09:12] Tim: Yeah, that’s neat. I didn’t really realize that was a thing, but that sounds like a very smart opportunity. Was that time when you were working there a big learning time for you? Meaning, were you picking up a lot of new things as you went, or was it more using skills you had already built in earlier days?
[09:32] Adam: No, that was— it was actually a really great time. So this was— I was doing this 2009 through like 2011. It was a 2-year gig. You know, data science was sort of becoming a thing. Drew Conway drew his famous Venn diagram in that time, things like that. And it seemed like everyone was sort of like, there is this thing that is data science. We don’t really know what it is, but maybe visualization is cool. Hadley Wickham published his split-apply-combine paper around that time, which was the beginning of Plyr. And I think Wes gave the first talk about Pandas. Wes McKinney gave the first talk about Pandas sort of around that time. So a lot of really cool new tools and ideas were in the kind of meetup scene of New York City, and I was just sort of trying to absorb as much of this as I can.
And one nice thing in this program that I was in was I kind of had to do the full spectrum of what we think of as data science, like a little bit of modeling, a little bit of simulation, a lot of programming, just sort of implementing stuff, a lot of visualizations and those sorts of things. And it was like each professor had different needs. And so when you talk about this amorphous blob that is a data science job description, I was kind of covering all of them at the time, which was really fantastic. I was very lucky to have been in that opportunity.
[11:09] Tim: Yeah, I didn’t do that kind of thing, but I feel similarly of some of the things I worked on in grad school and in a summer research program I got to do during undergrad. A lot of those kinds of skills that I didn’t really get in the classroom, but are very practical day-to-day skills even now. Things like, you know, using the command line effectively and being able to do some quick data munging and generate a plot pretty quickly is still a very powerful skill. I mean, I think maybe those are being taught more now, but I only ever did those types of things in some crappy lab software or in Excel or something like that. In school, I mean.
[11:53] Adam: Yeah. And I’m sure that you probably had a similar thing to me where you’re like, okay, this is the specific problem that I have, but what’s a cooler way of solving it? Like, I used to write, like, I had to, like, solve problems for my, like, Circuits 101 class in the beginning of college. And I was like, well, I bet I could program something in MATLAB that will, like, just give me all of these resistor values of this weird network. And, like, you know, that’s sort of like, oh, I bet I could make a, I don’t know, it’s sort of like that XKCD problem where someone asks them to hand them the salt and then 45 minutes later they’ve almost finished designing a device for handing arbitrary objects across the table. You know, do those sorts of things. And I feel like that curiosity is super valuable in data science, because you kind of have to be able to figure these things out and kind of come up with a new way of doing, whatever this particular idea is, because nobody else in your company knows what a data scientist is.
[12:51] Tim: Yeah, absolutely. I completely agree. I mean, I think even just in modern software engineering and the challenges of using open source software requires a lot of persistence and just being willing to go out there and try some things and see if they work and being able to power through. You get some cryptic error and you have to decide, is this worth going down the path to try to figure this out or do I give it up at this point? Like, I think a lot of those things are, for me anyway, and it sounds like for you, very comparable to the little, you know, here are 2 messy datasets in a CSV file that a professor needs, figure out how to do a join on them. It’s a lot of comparable problem solving that I think is hard to teach in a classroom, and people are becoming more aware of needing to teach those types of things in a classroom. But it’s hard to teach that until I think you maybe start to need it more when you have some motivation for learning it.
[13:53] Adam: Yeah, for sure. I’m super curious of how data science curricula handle that with people who are graduating with degrees in data science these days. I don’t know. I wonder how they teach that sort of intuition and that interest in it because you studied math and physics and I studied electrical engineering. And as I talk to all of the people that I know who got into data science around the time that we did, they studied social sciences or psychology or economics or whatever because there was no data science. And we just sort of like, you know, had this— we’re like, oh, I bet I could take the things that I learned in my program and like build a trending thing on Tumblr or whatever with it. And it’s like, I feel like that being able to learn skills in a particular domain and adapt them sort of broadly is like a super valuable quality of a data scientist. And I think we used to self-select into that and I hope I’m sure the folks who are applying to data science programs are self-selecting in a way, but I hope that there’s like someone’s figured out a way to teach that ability to pivot, I guess. No pun intended.
[15:07] Tim: Yeah. So that leads us actually nicely to your next job, which was at Tumblr, which you joined in August of 2011 according to your LinkedIn. How did that opening happen?
[15:18] Adam: I DM’d somebody on Twitter. And then 3 days later, I started working there or something like that, yeah.
[15:27] Tim: Those were the glory days of data science Twitter when kind of everybody could follow everybody and you could interact with anybody.
[15:34] Adam: Yeah, I’m not sure. So yeah, I mean, I did message someone who had Tumblr in their Twitter profile. You know, I went in and I interviewed and I didn’t know what to expect and I don’t think they knew what to expect, but they, I don’t know, maybe they just took a flyer on me and it seemed like I might be able to figure something out useful. One thing I made sure I did was have like— I don’t know if it’s still sort of in vogue to have a portfolio of data science projects. I would go all day, every day when I was working at Columbia, it was like, oh, we need to download this stuff from FRED or from all these S-1 filings or from wherever. And I was like, oh, I can just get data and make visualizations or try to do some clustering or whatever. And so it was like political data or baseball attendance data and things like that.
I just ran all these scrapers for fun and put a little blog together of just little projects. And I like to think that that was sort of like part of why they thought I would be a good employee. I like remember getting there on day 1 and our VP of engineering ran me and the one other new hire through like how the Tumblr stack works and my eyes glazed over. Like I didn’t know like what a, you know, an MVC stack or like framework is and I didn’t know what caching was and I didn’t know any of these like web things. But like was able to figure it out, I guess. And they just like gave me enough rope to figure it out, which was great.
[17:18] Tim: You know, I still encourage people to do portfolio type things like that. I gave a talk at a— we had a PyData conference here in the Research Triangle a few years ago, and I gave a talk basically trying to motivate early career and students type people to share the types of things they’re learning and working on. And that was actually an important thing in my early career. So I was in grad school in this time, 2009 to 2012, and I was on Twitter and seeing people like you and John Miles White and Drew Conway, other grad students and other people who were getting their names out there because they were publicizing the things that they’re working on, on blogs or whatever. And I was like, yeah, I’m doing interesting things in grad school, just nobody knows about it because it’s me alone in my office working on it.
And I mean, you know, now that I think about it, that’s a very generalizable skill in itself, as we’ve both learned in industry. You in many ways have to be your own advocate and your own marketer internally on projects you’re working on and things, proclaiming that to the rest of the company, not in an obnoxious way, but in a helpful way, is an incredibly valuable skill. And I think that’s worth starting early.
[18:48] Adam: Yeah, I completely agree. There’s a line, I guess, probably somewhere between advocating for yourself and fully playing politics, but You know, like maybe the difference is like where your best interests are. But like, you know, if you’ve got an idea that you think is very valuable and you can show the value in it, being able to show that value is certainly a useful skill to have for engineers and for really everybody. And whether that’s presentations or talks or prototypes, or whatever, like being able to do those sorts of things is super useful. And like if you’re starting early, then like being able to do that on just a blog is like a great place to start.
[19:37] Tim: Nowadays it’s not a blog, you make a Streamlit app I think is the way to go.
[19:42] Adam: I guess so, yeah. Yeah, you make a Streamlit app, but then you still have to get it out there somewhere, which I guess is like a Medium post with some highly controversial title.
[19:52] Tim: Yeah. How I deployed Kubernetes on my laptop to deploy a Streamlit app.
[19:56] Adam: Right, exactly.
[19:57] Tim: Were you the first data scientist hired at Tumblr?
[20:04] Adam: Yeah. Yeah, I was. And I’m not really sure what made them decide that they needed a data scientist.
[20:11] Tim: That was my next question.
[20:12] Adam: So yeah, I think with every— so it’s a social network, which means like there’s 2 things that are super important. One is some sort of recommendation or discovery, whether that’s discovery of new blogs for a person or whether that’s trending and things like that. Whether you surface trends externally or just have them internally, it’s good to know what people are doing on your network. It’s a pretty useful thing. And then the other one, of course, is always going to be spam or some sort of malicious activity and getting that off the network. So I started and it was like, fully gung-ho on, I learned about matrix factorization and I can do a singular value decomposition on this huge interaction matrix and start doing some recommendations that way. And quickly realized that you can’t just load all of Tumblr into NumPy on my dev box.
And so I had to start learning other things, big data things, learned about what Hadoop is. Never actually did that matrix factorization problem, but ultimately, I think a lot of the times my concern with things is making the worst experience better. The sort of spam problem, which Tumblr wasn’t exactly overridden with spam for most of the time that I was there. But just getting it out of places where people go to find good stuff was like— I was like, this is a problem that needs to be solved, and went out solving that. And so I did a lot of binary classification problems. And I know you’ve worked in this cybersecurity and detection of bad actors space for a long time, so certainly know a lot more about it than I do, I bet, at this point. But I just sort of like got into those sorts of things at Tumblr and it was fun. It was a fun time.
But someone was like, hey, would you take a look at this thing? That was like no OKRs, no planning, and just sort of got to figure it out.
[22:27] Tim: That’s neat. And what were some of the tools that were used at that point for big data and computational challenges?
[22:35] Adam: So the most important thing is if you’re going to analyze data is to write stuff down. And so Matt Hackett, who was the VP of engineering there, was like, hey, we’ve got these databases with Tumblr blog posts, but there are all of these actions that people are taking, page views and liking a thing and unliking a thing as 2 different immutable events. And so they set up some pretty good logging where you can just record certain actions and it would record a number of common fields like the person and the time and the machine information or whatever. And so they were using Scribe, which is a thing at Facebook that they use for logging events and then dumping all of those into HDFS.
And so They had this Hadoop cluster that was set up that basically was a gigantic bucket that they were just filling with stuff and not really looking into, but they had the idea to start with that. So when I got there, we had this Hadoop cluster and I had no idea what I was doing. So I started MapReduce 101, right? get a Java thing and you have these mappers and you have these reducers. And I quickly realized there has to be a better way. And around this time, I think a lot of data folks were like, well, we can do these streaming operations on Hadoop, which it’s not the native access to MapReduce, but you can write Python at least. And Last.fm made a thing called Mr. Job, and Spotify made Luigi that did a lot of sort of streaming processing at the time, and started figuring that out.
And then I discovered Pig, which, have you ever used Pig?
[24:40] Tim: Apache Pig? Only a tiny bit.
[24:42] Adam: It’s fantastic.
[24:45] Tim: It has its fans, I know.
[24:47] Adam: Yeah. It was great at the time, but then you can write— I think it was called Pig Latin was the name of the language, but you start to see the beginnings of common modern data engineering patterns in that with group bys and functions that get applied to the group by, the bag or whatever it was called. you could start doing these aggregations on groups of data and it would scale really well and it was really nice. And you start to very quickly learn about key skew when you do these group by things. And so we started building a lot of tooling around that. I guess that was the sort of engineering side of it. We used a lot of those for getting our recommendation pipelines together. I think a lot of our stuff was written in Pig.
And then we also set up Hive, which is like a SQL interface for business intelligence stuff, and had a lot of dashboards and things like that. So pretty much all of the engineers at the company or analysts at the company at this point were firing off Hive queries and clogging up the whole cluster for everybody. So those were the sort of two things, were Pig and Hive for a bit. And then I’m not sure what year it was that Scalding got released. Oscar, who was on your first episode, and the folks at Twitter released Scalding, and that was game-changing. Because one of the things about both Pig and Hive, if you want to do any custom functions, you have to write a UDF, which means now all of a sudden, instead of SQL or this other Pig language, you have to start writing Java and bundling those jars with all the rest of your stuff.
And it just got to be a huge pain in the butt. But with Scalding, you just write Scala code, and if you want some special function, you just write a function. And so it was huge. And I’ve been using that there, and we built a lot of sort of data pipelines and things with Scalding. And even we’re doing trend detection and stuff like that in that. A lot of the recommendation stuff got moved to that. And then I guess the next couple of jobs after that was using Scalding for a long time until now we use Scio at Spotify, which is a similar API over Dataflow. We don’t need to jump that far ahead, but yeah, so those were really the tools. But then for detection of fraud and things like that, we used Vowpal Wabbit, which came from Yahoo Research because it could train these logistic regressions on a stream of data.
So you could pass through all the data and you didn’t have to worry about fitting it all in memory, and you would just sort of learn observation by observation. Which was really awesome.
[27:57] Tim: That’s cool. I mean, that’s interesting to me for a number of reasons. One is, you know, how much the tooling has changed even in a short 8 to 9 years since then, in that people aren’t using Hive much, people aren’t using Pig much, and things have moved on, which is kind of its own story. But I guess one other question about that is, was that your first introduction to Scala, which I know you’ve done a lot now in recent years?
[28:27] Adam: Tumblr is like your classic LAMP stack, or it was at the time. I’m not sure now. So it’s a PHP application, and they were interested in kind of breaking pieces of it out into microservices, so really starting to get into a more kind of microservice thing rather than one monolithic PHP app for Tumblr.com. And so the folks on the engineering team were like, we think we’re going to use this Finagle thing also from Twitter. And so Scala was around the company. And so when it came time for me to learn it for data purposes, it was like other people at least were using it day to day. They were using it for completely different applications, but I could at least ask questions about the language and learn how to do things. And then, yeah, I mean, I feel like it’s such a nice language to use for data pipelines. You’ve got Spark and Scalding and all these other things that all at least are, I would say Scala first, I think Spark is. I’ve never actually used Spark just because.
[29:47] Tim: One of the other interesting things to me, and you and I were talking about this a little bit the other day, was it seems to me from talking to you that you were really enabled to ship a lot of things that you were working on at Tumblr, which is a challenge that, you know, getting things out the door, and I think sometimes especially in data science projects, is a challenge that companies face. And there’s this trade-off between, you know, the just free-for-all and, you know, anyone can, can start to add things. And then other times you can have gatekeeping through approvals required or just the technologies that required you have to check all these boxes before you’re even allowed to, to ship something. Or, I mean, on the other hand, Some companies are so mature that they make it very easy to ship things because they have so much advanced tooling in place that’s really focused on engineers and data scientists being able to ship things, which is— that’s a whole can of worms there.
[30:56] Adam: Yeah.
[30:57] Tim: But I’m kind of interested in, like, from my perspective, that was a— in talking to you, a very very good job for you, and you were able to do a lot for the company as well. I guess, do you have any kind of summary thoughts about what enabled you to contribute so much to the product through your data science work in just a few years’ time that you were there?
[31:30] Adam: I think one thing is There has to be a clear objective that is valuable to the company. We also did some stuff that were like, it was around the, I guess, probably 2012 election that we’re looking for trends in what people were saying on Tumblr. And this was shortly after the infamous OkCupid blog that like, I can’t believe it’s been 10 or 11 years or something since that thing was live. But it’s a fascinating blog where they just dug through all of the dating trends on OkCupid and wrote about them. And I think that helped get a lot of people, including me, into data science. But like, I like it dug up— it like, there was a lot of hype about OkCupid because of that. But it’s hard to say that like that sort of data science-y, tell a story with data thing is going to be super valuable to every company. So Right.
You know, one thing at Tumblr where they were just like, spam’s a problem, like people are complaining about it. Um, you know, Tumblr got acquired by Yahoo not long, like while I was there, so certainly someone must have said something about, you know, hey, what about this problem? And they were like, we’ll clean it up. And so, um, having that important need, uh, for something that really can only be solved by some sort of data science or machine learning application, whether that’s because of scale or something else, like recommendations or whatever. Having that need is absolutely critical.
And then sort of giving folks the space to figure it out and what is the right way of doing it and just sort of having the trust. in the team or the person in some cases to find the spam or build the clustering algorithm that is needed for whatever business-critical application or do accurate forecasting for so many people. It’s the need and then the sort of space and the flexibility at a small company are critical, I think. Things are totally different once you get to, I don’t know how many thousands of people work at Spotify, 5,000 or something. You can’t just be like, hey, go figure it out.
[33:58] Tim: Right.
[33:59] Adam: But yeah, I think one of the things that we were talking about was this sort of, there’s like MLOps as a field that is all about sort of data tracing and reproducibility and making sure that your models are all evaluated properly and against the same datasets. And that is all super interesting and necessary, certainly more necessary at places like when I worked at Stripe and there were dollars involved and not just teenagers’ websites. No offense to the teenagers of Tumblr. But it was a little lower risk and getting any sort of model out there however you could was more valuable than some perfectly analyzed, robust, scalable, reproducible system. At Spotify where I work now, that is very important and we have the scale and the ability to do that. But one fear that I have is that people are like, oh, well, I can’t do a machine learning application because I read this book on ML systems or I read these blog posts about MLOps platforms and like, if I’m not detecting the training/serving skew of my dataset, I have to do that before I can even put a model into place. And I think the answer is you probably don’t need to do that most of the time.
[35:30] Tim: Yeah, absolutely. Well, your time at Tumblr only brings us up to 2004, so we have 6 more years of your career to hit. So from Tumblr, you joined a company that was somewhat short-lived, from what I understand, called— and that was on LinkedIn, you call it Sum, but that was what was known as Project Florida most of the time. Is that correct?
[35:55] Adam: Yeah, that’s right.
[35:57] Tim: So what was that company about?
[36:00] Adam: So we were building our— it was a hardware and software thing, and it was like a bunch of folks from New York City startups, and probably the best team I’ve ever worked with, just wonderful people all around and just super brilliant. And this idea was— a lot of it kind of lives in what the Apple Watch is now, actually. We were building a heart rate monitor that you wear on your wrist, and we can sort of We were able to get it to identify when you’re stressed and identify activity and see how your heart rate recovers from a certain amount of activity and could tell when you fell asleep and woke up and could analyze your sleep. And this was all from accelerometer and heart rate data. And the sort of goal was, and I think a lot of the inspiration for the company was, just because you’re have a low BMI doesn’t mean you’re necessarily healthy.
I think a lot of people right now in the world are just sort of stressed because of whatever coronavirus-induced situation they may be in in their daily lives. And while you may look fit, you might have other detectable, preventable sort of health conditions. And if we could sort of help identify them and steer people towards maybe a better lifestyle, whether that’s food, exercise, stress, sleep, something.
[37:35] Tim: Sure.
[37:37] Adam: We were trying to do that. One thing I learned there is that hardware is very, very hard. And so we built some prototypes and our software was working pretty well for detecting all of these things. And we’re working on the product of how do we nudge people to do things and had some really fascinating machine learning ethics questions. Do you ask someone if they’re pregnant or not? Or do you ask someone if they are or were or used to be a smoker? And do you trust their answers?
[38:16] Tim: Yeah.
[38:19] Adam: And a lot of other really interesting things of do we want to use certain signals in our models or not? Because these are highly personal, about a person sometimes. And if we really need to ask it, then we have to justify it. Otherwise, let’s not dig too deep into these people’s lives if we don’t have to.
[38:44] Tim: So what types of things were you in particular doing there? Building models or what?
[38:55] Adam: So I was working, I guess, a lot on streaming infrastructure for like— I guess it was more sort of on the model inference side. We had folks who— I guess I did less of the modeling there and less and less as my career goes on probably. But everything was built on these streams of heart rate data. And so we had to aggregate them up and extract the features that we needed for the model in order to make predictions. I forget a lot of the real details of what we were doing. We were using Apache SAM.
[39:31] Tim: Yeah.
[39:32] Adam: There were a lot of fun engineering challenges there with connectivity issues and time series arriving out of order and things like that, just restrictions that we got from the hardware itself. And it was so like figuring out a lot of those sorts of things was like super fun to work on.
[39:53] Tim: And that company didn’t make it, which is probably a story for another time. But from there you joined Stripe, which is a company people are pretty familiar with, as a machine learning engineer. What kind of work were you doing at Stripe?
[40:10] Adam: So Stripe had, when I was there, it was the primary sort of machine learning application was fraud detection. It was similar, I guess, in concept to the work I did at Tumblr, but of course much more mature in many ways and much more high stakes in many ways. And it was— yeah, I was only there for a year. It was a brief time, but it was fun. It was super eye-opening. It was like there’s a lot of really smart people thinking about very, very big problems. And it sort of goes back to, I guess, what I said earlier about wanting the sort of worst-case experience to be less terrible. So if people are having fewer fraudulent transactions at their business or whatever, it’s like the goal, I guess, that I was thinking towards.
[41:05] Tim: Stripe is somewhat known for their, I guess, kind of thought leadership and presentations and blogging and various things. how they do things and they make a big point of being a company that does things with excellence. Did you have takeaways from working there that have been valuable to you in shaping your own thinking about tackling machine learning problems or about systems or teams, organization, anything like that?
[41:38] Adam: I’ve worked remotely in like Stripe has a pretty large remote community and probably much more so now than 4 or 5 years ago when I was there, which meant that a lot of things were overcommunicated. And I’ll say that in a good way. Lots of meeting notes that got shared out and things like that. But folks, I think, there also think a lot about the reason behind why they do a certain thing and can sort of draw up a system or draw up a sort of organization or structure or model or whatever it is, and they’ll kind of think it through. And then before building things, kind of take a step back and be like, does this make sense? And then if yes, then go forward. Everything is, like you said, sort of principled. Maybe that’s not a good word, I don’t know.
But I feel like it’s a very academic, like, let’s think of the long history of this and where we’re going sort of mentality. I think it’s helped me now. I don’t know if you want to jump all the way to the current, but I’m working on a platform team at Spotify building machine learning tools, and it’s sort of like, Well, let’s think of the whole system as a complete piece and not just focus on our one independent little part of it that we’re building right now.
[43:12] Tim: Yeah. Well, I mean, I did want to move on to Spotify and hear about how your career has progressed there. And now you’ve been there over 4 years and are a staff engineer at this point and getting to do more Big picture thing and set technical direction for machine learning, which isn’t what, if I’m correct, isn’t what you joined Spotify to do. So you’ve been able to take that kind of higher and higher level view as well as helping put things into place to make machine learning more effective across the company, which I think is really fascinating. And an interesting culmination of— not that this is the end for you, but that it seems like a lot of different things have worked towards you getting to do what you’re doing now. So all that to say, I’m interested to hear how your role at Spotify has progressed and developed from, I think, being very hands-on with some of the some somewhat high-profile models at Spotify in the early days to now this platform team.
[44:27] Adam: Yeah, I joined Spotify working on the team that was— so Discover Weekly was, I think, about a year old when I joined, which is a playlist of 30 songs that you haven’t heard, at least on the platform, that updates every week. And it was like, it’s probably still like the flagship like one of the flagship recommendation products. I would say that the homepage is probably now. But, you know, I joined to work on that. And, you know, this was again sort of one of those things that someone like had an idea for and like there was a platform or there was sort of a way of like making these playlists and serving these playlists using the existing infrastructure from like Spotify. And they were like, well, what if we make a new playlist every week and update it for everybody?
And We’re fiddling around with this model and we think that this is going to be really cool. And they made it for employees and some of them loved it and some of them hated it. And it sort of goes back to this giving people the flexibility to play around with machine learning ideas. And as long as there’s a business need, like we need to be good at recommendations, then Having a broad directive but letting people fill in the details with the tools that they know and the things that they kind of are best at is really great. And so Discover Weekly sort of came out of that. And I remember hearing our CEO one time talking about— somebody asked him a thing that had changed his mind about working at Spotify, and he said that he was really skeptical of Discover Weekly. He thought it was a stupid idea. And kind of came around on it.
And I think Spotify works very much in that way of broad directives from above and then specific implementations from the ground up, which is fantastic. And so—
[46:27] Tim: I used to love Discover Weekly before I had a 3-year-old who can talk to my Amazon Echos and request music, and now my Discover Weekly is totally shot.
[46:38] Adam: I’m about a year away from being in that position. Hopefully we’ll come up with a solid solution.
[46:44] Tim: You start to have several significant principal components in your model, I think. I’m looking and I have songs from Winnie the Pooh on mine at the moment, so.
[46:54] Adam: Oh, great. Good stuff. Maybe, hey, maybe you’ll love it. I guess I worked on a couple of big projects at Spotify. The first one was Discover Weekly, which really was less of an improving the algorithm thing and more of a cloud migration thing. We were migrating all of our stuff to GCP at the time, and it was just like a— I mean, it was a software rewrite problem that are both fun and annoying. And I’m sure a lot of people have been parts of migrations to or from clouds. Maybe not everybody, but you either love it or hate it, and I kind of embraced it and really liked it. And so it was a lot of fun. But from, I don’t know, after working on that for a while, the homepage of Spotify, which is, you know, we’re like, we need to really personalize this.
And there was a huge effort put into that, and a lot of people moved on to that project, including me. And so it was like figuring out what exactly this is going to be. There was a lot of like really fun product development stuff that we were doing at the time. From an engineering perspective, one thing that we realized was that we needed more content to rank in the ranking model that we were building, and we needed to be able to iterate on this model quickly with maybe 4 to 6 machine learning engineers poking and prodding at it. Adding new features was an important thing. And so I worked with a couple of other folks on a system that would— and this is sort of exactly what I said that you might not need a few minutes ago, but it was making sure that it would be easy to add features to a machine learning model.
Those features would be available at serving time as well as training and sort of cutting as many steps out of it as we could. And I think at this point, my, like, I want the worst experience to be less bad started to focus on the sort of internal developer experience rather than the customer experience necessarily. Because it was hard dealing with both the scale of data that we have and just sort of the scale of the organization. There are a lot of— I think there’s over 100 machine learning engineers at Spotify. People do things— if there’s not a clear way that you have to do things or that it’s most advantageous to do things, then you’re just going to do them however you know, and that gets to be unwieldy.
And so while working on this thing from home, we were like, well, maybe we can start to expand some of these ideas and build tools that go to the whole company. And so I started working on prototypes for how we would do a feature store and how we would organize our machine learning experiments and things like that, that ultimately after a 6-month paternity leave and some other work, I finally landed on the team that’s responsible for actually doing that. And then I guess along the way in there, putting together a staff promotion packet was part of it. And that seemed to work out, which was great.
[50:18] Tim: So what does that look like for you in the day-to-day? And I’m sure no 2 days are the same, but what does it look like from the perspective of being a staff engineer as well as being on this platform team, not working on modeling projects and things, but on tools for others? How are you spending your time?
[50:43] Adam: I’ve been in a lot of sort of— you get to be in a lot of planning meetings for better or worse and status update meetings and things like that.
[50:52] Tim: Right.
[50:52] Adam: But my role on this team is someone who has been doing machine learning stuff at the company and knows kind of the processes and the tools and how our machine learning tools fit with our data tools because there’s a very mature and robust sort of data engineering stack, much more so than the machine learning one is still today. And knowing where and how those things should merge and what should be separate, and also, what are the pain points that we need to be solving for? So I’m sort of a bit of a developer advocate for the other machine learning engineers to say, these are the tools we’re building for the platform, but there’s this particular pain point that comes first. And so sort of having this holistic view of the experience, the team I’m on is called Machine Learning UX.
It’s the user experience of machine learning engineers at Spotify. And it’s sort of tying the whole thing together. And I really like it. So some of that is writing code to build, you know, model tracking stuff that’s part of the rest of our software infrastructure system. It’s called Backstage. It’s a thing that we’ve open sourced recently.
And then also all of the libraries and things that fit into the machine learning developer workflow and toolkit to actually provide all of this information so that things are both useful to the ML engineers and researchers, but also— useful to us on the infrastructure team so that we can make it easier to go from some model idea that you had to deploying that to a QA thing that you can build Streamlit apps for or whatever and make predictions, or easily move from this idea generation phase into shipping things to production, retraining regularly and monitoring and all of those sorts of things. it’s a lot of stuff, and I guess I’ve been able to work more on some of it, but it’s nice to sort of be aware and sort of involved in the whole holistic view of the whole thing that we’re offering as a platform. I feel very lucky to have that spot.
[53:13] Tim: Yeah. And I mean, you are fortunate to be at a company that is big enough to need something like that. I think a lot of people hear of things like that, and that sounds nice to do, but that’s not actually realistic at a lot of places.
[53:30] Adam: Yeah. And the last customer-facing thing that I worked on was a small project that only a subset of users would face. And the amount of traffic that this particular thing gets is probably don’t know, 3 orders of magnitude lower than the homepage of Spotify. And we get this benefit where it is— Spotify’s homepage is, I mean, it’s gotten so much better in the last couple years. I absolutely love it. I’m trying not to sort of toot the company’s horn too much, but I think it’s really a great product and you find kind of what you want. And it’s taken a lot of investment to get there. but we have hundreds of millions of users who use that, and so it’s worth the investment there.
And then you get the benefit that these smaller projects, the marginal cost of spinning up another smaller little machine learning project is so much smaller because you’ve got this sort of cornerstone thing like search, right? You have to be able to have search and personalized search and things like that. So the infrastructure for these gigantic projects makes it easy to sort of spin up other smaller ones. Like the marginal cost of optimizing little itty-bitty things becomes so small. So it’s really great.
[54:56] Tim: So to throw you maybe a hard question, I’m curious as we wrap up if you, from this experience, have advice for Adam joining Tumblr in 2011, as the first data scientist, tooling less mature, team obviously significantly different. A lot of people are still in that exact same situation, maybe not using Pig, but they’re coming into a problem and trying to identify problems that need solved, trying to identify if they can solve them with machine learning, trying to figure out— tools and technology to tackle that. Do you have any advice that you wish you could go back and give your younger self?
[55:45] Adam: I think especially for people coming out of school and maybe especially grad school or PhD programs where your goal has been to be an expert in a thing, and that it’s learn enough of the software and the systems and the organization around you to unblock yourself from getting the thing that you want to do done. So if you have some model that predicts recommendations offline, then your goal should be, well, how do I get these things served? Well, whether you find a person or you learn what it would take to, I don’t know, dump those recommendations into some sort of data store and get into the code base and fetch them back out. That’s a pretty specific example, but whatever it may be where you’re like, this is the limit of— you can’t just throw a model over the wall as they sometimes say.
You have to be able to get through and at least know enough about what is difficult and what is going to be easy and how you can kind of clear your own path so that the thing that you’re most interested in or the most knowledgeable about, can actually see the light of day and get built and shipped. And usually that means learning— well, I mean, a lot of times it means doing more of the stuff outside of your expertise than the expertise itself. Becoming more of a generalist is going to require learning a lot more of the things that you haven’t been studying or doing for the last 5 years. I guess I’m very pro-generalist and just sort of learn those other skills. And I’m sure you’ve done similar things. I mean, you’re the person that I go to to ask about Python packaging and things like that, which isn’t a math problem.
[57:44] Tim: Yeah, we’re in very different companies in terms of product and scale and different things, but we followed, I think, a similar path in a lot of ways to where I’m doing the same kind of thing, trying to enable data scientists and machine learning engineers to do their jobs more effectively. And that came not really from me setting out wanting to do that, but from seeing my own pain points and wanting to reduce them in the future that I’ve moved away from the math and modeling that I am, you know, that’s what motivated me in a lot of ways to get into this industry. I really love that stuff. But just through my own exploration and own needs, my career has developed in sort of a similar direction. And I think that’s— most people’s careers are going to develop in this, probably anything in technology and a lot of other things in the world. People’s careers are going to develop and change in ways you don’t expect, and you just have to embrace that.
[58:51] Adam: I think in a lot of ways having a strong, like, punk rock DIY spirit is useful. Like, you can’t just wait on and depend on anybody else. You have to get out there and figure out how to do the thing you want to do, which is great.
[59:05] Tim: Yeah, excellent. All right, Adam, thank you very much for coming on the Into the Hopper podcast. I think this was a great discussion, and hopefully we’ll be able to come talk about some other things more in depth in the future.
[59:17] Adam: Awesome. Thank you so much for having me.
[59:18] Tim: Thank you.