LinkedIn Data Scientist Mock Interview + Feedback with Ex-DoorDash & Spotify Data Scientist!

first couple things that i can think of related to engagement to on a news feed is um click-through rate so the likelihood of a user just clicking on a piece of content [Music] hi everyone this is jay from interview query uh today i'm joined by jeff uh who works at doordash jeff uh i'd love for you to do a quick uh introduction uh on kind of like how you got into data science yeah definitely so i've been in data science for about three years uh before data science i was working in technology consulting at the time i was actually playing a lot of poker on the side so i found it really interesting was spending my friday nights playing poker i realized it wasn't really a good long-term uh strategy since i was trying to like make enough money to like live off of it but i really like math and statistics behind it behind it so that's what drove me towards data science and uh yeah i've been in data science for the last three years started off at a edtech company called dataquest then moved on to the machine learning team at doordash and uh yeah happy to have you to uh happy to be here awesome cool um and then i guess is there anything you could talk about about uh like machine learning at doordash uh in terms of the stuff that you've worked on that's uh been pretty cool or interesting yeah definitely so um yeah i was one of the first ml data science hires at doordash so i worked on a little bit of everything on all sides of the business i worked a little bit on our personalization recommendation algorithm helped work a bit on marketing segmentation and marketing attribution and i've also done some work on our pay model how do we optimally pay drivers and then uh recently i've been working a lot on demand forecasting gotcha cool yeah um awesome so i guess for the first question i think it might be kind of similar it's more machine learning based um and so let's say you work as like a data scientist on the uh linkedin um engagement team right um and let's say that we uh have like a obviously we have like a news feed ranking algorithm right so that when you log in uh you see like a general news feed of um stuff that might be interested to you so the first question is how would you actually measure the success of the news feed ranking out okay taking some notes here yep definitely okay cool so i think before i start diving in i just want to make sure i really understood the problem uh i'm going to be a data i am a data scientist on linkedin's engagement team and we're working on the news feed ranking algorithm and then we want to measure the success of a new news feed algorithm and the first step will be to just kind of come up with some metrics that we think can evaluate how effective this news feed algorithm is am i getting it right yep cool okay cool so you mind if i take a second i'm just gonna kind of brainstorm to myself so then we can't walk through the solutions for sure okay cool so i think um just kind of preliminary i uh first couple things that i can think of related to engagement to on a news feed is um click-through rate so the likelihood of a user just clicking on a piece of content however i think a better measure on like how engaged and how relevant the content could be for a user is if they actually share content because that means that they're much more likely they like it enough to say hey i think this is valuable i think that you should also look at this as well and the third one i can think of is related to comments so um yeah comments is a huge indicator of how engaged they are and i think the next one is how many times they post as well it's a number of posts i think uh yeah so i would say those are the four different metric uh four different components that i might want to look at um it's still a little bit vague right now would you want me to break this down into more of a defined metric or uh do we want to keep uh moving moving forward yeah so let's say that um we uh want to like uh tweak the model i guess a little bit um and so um i guess how would we uh know and let's say that um some of the metrics that you're tracking like these four metrics some of them are going up and then some of them are actually going down uh so how would you approach like thinking about which ones are more important uh for uh the team okay so yeah thinking out loud here i'm thinking that there are two different ways i can approach it so the the first way i can approach it is actually from a business perspective so what what metric has a say like a strong correlation or a strong indicator of a positive impact for a business so in linkedin's case they pro likely make money through different advertisements shown uh they likely make money through their recruiter platform and they likely make money from companies posting uh jobs on their website so um i think one perspective we can actually go with is the business perspective uh the second one we can go with was just um basically very focused on the user so we want to make this the best user experience possible so i think that depending on what the business's priorities are that would indicate what the metric is so i think if we're going to go from a business perspective so if linkedin makes money on ads then ideally if if advertisers will make money through impressions and click-through rates then if we're going to base this off business maybe ctr is something that we might want to optimize for in that case okay if we if we want to optimize for the user experience uh ideally users who love the content i think that let's sharing could be one however my guess uh this is not validated by data i is that most people don't share that much content on linkedin only like a select few people do share content so there's a chance if we train a model on that or improve the ranking algorithm there we might just have a very sparse data set i think another one could be uh comments i think that's just a much stronger indicator of how it like that that the comment is relevant to that user compared to say ctr you can have like a very big uh i don't know like very juicy headline to get somebody to click so i think if we're gonna go with user experience i might say think about i might say comments would be one a likelihood of commenting on say a post or a link shared if we're going to focus on user experience so um yeah but i think what i would do here was i'd first talk to the product manager talk to leadership make sure we're aligned on hey what is the goal what goal are we trying to achieve with the business okay um so let's say that the pm uh comes back to you and says that uh one of their biggest goals or one of the things biggest problems that they've seen is that people will be on linkedin um when they're searching for a job but after they they're like done with a job they like stop using linkedin uh and so how do we then kind of like reapproach this uh and look at uh if there are any other metrics or things that we should take and consider from that got it so so basically people you want to use linkedin a lot when they're looking for jobs but then once they get a job they just stop using it as much is that right yeah so they have like a general longer term um let's say like engagement issue then right uh and so um potentially like can we think of like kind of new goals or like new metrics uh or just new strategies um in terms of like optimizing which metrics to basically uh improve the um the longer like retention curve got it okay so ideally here if we were to improve long-term retention so right now the problem is that users will drop off because so it sounds like from a user perspective linkedin really does a good job of solving that pain point of trying to help them find a job so then i'm so trying to think of this from the user what other pain points might a user have out once they do find a job so i think that kind of brainstorming out loud here i think that one if i were an employee once i found a job through linkedin i would definitely want to continue improving my skills so actually so what comes to mind there can be like linkedin learning is there like a really strong like education platform for me to improve my skills uh another thing i can think of from the user perspective is if i was a hiring manager that just got hired i want to find great candidates through linkedin even if i'm not looking for a job that's going to be a huge one so is there is is there a really high quality product that can help me find high quality candidates candidates cool so um yeah what else can i think of i think that in addition to that um yeah i think other things i can think of depending on the role uh if you're like an account manager you're definitely going to be using linkedin a lot to say build out sales leads so uh like linkedin sales navigator that's another one i can think of and then i've noticed recently that linkedin has been really big on just like content production so you can people can write articles they can share links or write write articles to like build their brand uh just within their career so building a brand building okay cool okay so kind of okay these are like a good amount of there's like kind of a wide variety of things that i've kind of mapped out here so now we want to translate these into some say metrics that that we want some metrics that can measure this side of the product that would make long-term improved retention um okay so and let's say that we want to like tie it back to a news feed as well right so um even if we have these products we want to put them into the news feed to improve like longer form uh retention right over time so um and say we're going back to like the core problem of measuring success of like the ranking specifically um think of like potentially more metrics now like um that would then be more focused on like the longer term because i think a lot of the like ctr right optimizes for ads but it's um very much so like once you click through it then like who knows if it was like click bait and then you just go off linkedin right um same with like uh you know comments and then a number of posts right so um can we like i guess think more on that form of uh like kind of like specifically recommendation engine for the news feed and then also metrics that will then indicate that a user might come back in like 30 60 or like 90 days okay cool yeah so that definitely makes sense yeah a lot of the metrics were very short-term focused so how can we actually optimize for uh like say just more long-term engagement with users so uh if like say we have a very high quality like say linkedin learning product it's very likely that a user like if i was learning something new i would probably want to log in at least a couple times per week or maybe yeah like weekly on a certain day so a metric that might come to mind would be say uh like like weekly sessions or like monthly active monthly active sessions right from a user perspective if we want to go even more high level monthly active users and then weekly active users and then um or we can do like say a monthly yeah monthly retention so does the user who logs in for one month come back the next month yep um yeah so i would say those would be the ones that come to mind all right cool uh and then uh i guess in terms of um could you think of like uh potentially more around um let's say uh like let's say we want like a graph to basically um showcase uh like retention and engagement um or like a dashboard effectively um so could you think about like a way that we could what kind of like graph we could have that would then uh showcase um how news feed ranking then relates to um like engagement and like we could see actually like a number going up or down so a graph that would show the news feed ranking in relation to engagement yeah so basically like a pm wants to be able to look at like a graph like every single day or like a dashboard um and basically be able to then understand if like we're helping like the engagement uh number um basically uh increase or like decrease or if people are engaging more with uh with the news feed or not um can you think of like maybe uh like a couple metrics or like a graph that we could present uh essentially got it okay so like how can like a graph that might be able to tie how what we're doing to the news feed to a uh to long-term engagement so yeah i think the first thing that would come to mind would be like say a cohort based retention graph okay so have individual cohorts on each week and then we would just show each subsequent week uh how many people were retained for the previous week or we can it will depend on how we want to define the time period it can be like a weekly based cohort retention or a monthly basis cohort retention and then we can maybe for each cohort track exactly which news feed algorithm that we wanted to show them and then maybe show the different retention curves across these different variants of the newsfeed algorithm gotcha cool so i think you did a pretty good job um and i think uh for the first part like doing the short term metrics are definitely necessary um i think i was like kind of pushing you more towards some of the more engagement type metrics on linkedin um specifically around uh this one is like a combination of recommendation engine metrics plus like engagement metrics um and so you got it mostly with the uh kind of like cohort retention uh and then thinking of like kind of like new products that should be um kind of like integrated i think i also might have like phrased that kind of strangely in terms of like longer form retention stuff um i think a couple of like recommendation engine metrics that are pretty useful to like keep in mind are like um looking at a lot of these metrics cohorted by like ranking on the news feed so if you have like if you imagine like a recommendation engine has to sort through like you know like i don't know hundreds of thousands piece of content and then place them all in order uh if we look at these metrics by like their um specific like rankings so if we look at like one what's placed in like slot one the ctr for slot one what's the ctr for slot two ctr for slot three four and then compare that across like different ranking algorithms then that helps us like understand um if people are like engaging uh within like the most recent stuff right um also the other thing is like how many um you know like pieces of items do they actually see right so if you only like you know you load the page and then only like one or two things show up and then you exit obviously that means that the news feed ranking algorithm sucked right because otherwise you just keep on scrolling uh so just like getting like the a couple of these like um kind of like uh algorithms are like uh or like metrics are like mean precision so getting like the number of times that they like uh a user like saw before they exited so they're just like the average number of like piece of content they consumed or like viewed uh so stuff like that um is like helpful and then like the number of uh like think like pieces of content that they interacted with out of the ones that they actually saw um so i think in terms of like the general engagement metrics you got uh most of those but i think next time it'd be good to like tie them into the um like the recommendation algorithm uh specifically like uh like kind of uh content in terms of um how it like surfaces uh stuff because uh that way then you can compare like the recommendation engines against each other as well um in terms of yeah okay got it yeah i was thinking of it more from the business perspective but uh yeah getting like kind of deeper into specifically how the recommendation engine was specifically ranking different pieces of content and then tying that into say like hey how many pieces of content did the user see against like or how many pieces of content did the user have to scroll through until they clicked through to whatever content uh yeah that definitely makes a lot of sense yeah cool yeah and uh yeah that's true i think a lot of these are also pretty open-ended um so it's like a lot of the times depends on like how the interview wants to take it too because um in any one of those moments they could just ask you about you know improving the longevity of success of like different products outside of just um news feed too as well um

Channel: Data Science Jay

Views: 24,500

Keywords: linkedin data science interview, machine learning interview questions, data science interview questions, interview query, facebook data science interview, data scientist interview, machine learning engineer interview questions, doordash data scientist, data analyst interview questions, ml interview questions, modeling interview questions, data science interview questions and answers, data science preparation, data science jay, product analyst interview

Id: dPSzIiW1x8s

Length: 18min 21sec (1101 seconds)

Published: Wed Sep 09 2020