Data Science Business Case Study: Netflix Pricing

[Music] hi so for this case question we're gonna go through a more product business style question asked by netflix so the question is let's say you work at netflix as a data scientist on their pricing team netflix is a subscription business and so they charge a monthly price for the consumer to pay given this price they've decided how you determine if the price of the netflix subscription is actually the deciding factor for a consumer to buy okay great thank you for the question we're talk a little bit about the problem like approach how to structure those problems so the very first thing you guys probably hear a lot about is asking clarifying questions so in this case because you're asking for a netflix subscription service so a few relevant clarifying questions i can think of is we talk about subscription right but right now if i'm correct netflix currently have three subscription tiers the basis standard and the premium so as an interview would you like me to focus on one specific tier or for all three tiers for example let's focus on all three tiers oh three yes okay sounds good let's start with one and then focus and then yeah i think yeah we can start with one and then maybe we can maybe talk about standard package and making we can elaborate time permits okay to the other tiers as well and then my second question is i'm aware that netflix is currently available across different countries in the world but because of copyright and stuff there is a geographic difference in terms of what content users are seeing right for example in korea they might be seeing like more career copyright content which will be different from us so for the purposes question is it okay i just let's say focus on the u.s yeah that's the us okay u.s only okay so let's talk about standard package and us only let's just take a little pasta the reason why clarify is really important is because it helps you simplify the question right because what jay had asked was really vague and broad you always want to answer an easier question than a more complex question so this will help you narrow the problem down so definitely ask good and relevant clarifying questions and then the very next thing i would do is specify like the goal or getting more context around this question so for jay i understand you want to know whether price is the deciding factor could you also talk about more around like what's the motivation behind it is netflix thinking about changing prices in the near future maybe you are seeing a lot of competition during covey like we're worried about other factors so i think two aspects let's say that we want to know if we can let's say increase prices and not lose customers whatever that means and then also we're seeing growing competition from other services such as disney plus hulu apple tv etc yeah so this question basically i'm trying to get a sense of gold from jay but also at the same time like getting more context around why we're solving this problem what's the ultimate goal right we're trying to reach so going back to the goal again i think you talk a little bit about the competition and price so in terms of goal is red netflix goal ultimately to like gain more market share or is the ultimate goal just to maybe increase the total revenue or is it around like maybe around engagements gotcha so i think as an interviewer i'd be just curious to think what yeah i thought would be like the best goal between this yeah because they are good yeah also thank you for throwing the fall back to me because usually as an interviewer that's they want to see your thought process as i what i did is i offer like three options like revenue market share and then engagement as an interviewee i think i can provide one solution and then strategy to do that is you can pick the solution you're like comfortable with like you've done a lot about maybe like revenue model in the past so on the technical perspective strategically pick the topic you think you're more likely to excel at but also give a reasonable response yeah that's yeah so you don't want to pick okay let's go with let's say revenue because i think it's important that to the interviewer it's not a very good reasoning like you need to have evidence of some sort of why you pick one over the other so for example in this case maybe since you you talked about competition and all that i know previously netflix has never sort of reduced their price in the past so we're thinking about increasing the price but we're also worried about like the competitions and like maybe losing subscribers so maybe i think ultimately revenue is the top line because revenue ultimately decides on the profit margin of how much netflix is making if let's say we're we're seeing a lot of competitions but we wanna as they increase our catalog of the type of shows we're offering that means more cost to netflix that also means we need to increase the price in order to group that difference so that part i've explained maybe it's a little bit on the longer side but just maybe give a one-liner on why revenue is important in this case and why you pick that one over the other no that's a good example i like that so after you've clarified the goal the next thing you want to go into is talk about your structure and approach so very important tip for all these interviews is having a logical way of explaining your answer so what i would do is i would maybe take a few seconds just say straight up like jay is it okay for me to take a few minutes or take a few seconds writing down my thought process before i proceed so usually the interview is completely okay with it let's talk about how i would approach this problem so basically we want to reiterate the question again we care about whether price is a determination factor right so there are a couple ways we can go about there are three approaches i like to take the first of all is to do some sort of retrospective analysis looking at historical data and then have an indication the relationship between price and then maybe the subscription rate or renewal rate or the churn rate or one over the other and then the second is i want to think about maybe doing some type of user service to get a direct response of when a user decides to turn the service like out of the top five answers which one did they pick the most so that would be like a more causal type of relationship and then finally we can also think about some kind of a b experiment now the a b experiment might be tricky in a netflix case just because know offering different user a different price might be disturbing especially in the same country right we talk about u.s only but there might be some trick we can do for example maybe offer like a one-month free right that could be an indication of whether they are price sensitive or not or we can do some kind of country test like finding a country that's similar to the us maybe canada right provided the fact they offer similar content so we have to think about all these confounding factors so these are basically three areas i like to explore and then the next i will talk about like how do i run this analysis so in terms of how to run analysis i talk about like data first and then i talk about model and finally we'll talk about like how to validate so around the data basically i think as an interviewee you can talk about how do you collect the data what type of data would you collect and then over a how long period of time right so for the first retrospective analysis we already addressed it's like we want to look at historical data so benefits have definitely increased their price in the past i actually have most recently i just read about like in october they just increased their price for the standard and premium package for one or two dollars respectively so um i think a plus here for all the interviewees out there like that's why doing the previous research is so important on the team you'll be interviewing for so if you can mention like some of these news that you read in the articles or like you've read in the media blogs and all that that's definitely a huge pass because it just provided you like you are interested in this field and then my second tip is i also use the product netflix you might have to pay or i don't know if they still have the trial going on right now but um pretty much all the tech software products is free so definitely download it and try it out because sometimes you might even run into problems right like maybe there are a particular area that they can possibly fix and by by not using the product you wouldn't figure out that part and then that could be a plus if you bring up to the interview you know you telling me a solution i haven't even thought about so that's another tip here so after i talked about the three i think i talked about the observational study and talked about how you would collect the data and then how you would compare the model so going back to the price again another tip is you laid out a structure you have three problem areas and then you go back to the first one and then we might derail like your interviewer depending on their interest like for example jay might be particularly interested in retrospective right so try to be also time conscious of what you spend in each area if you talk about all three you might want to cover all three and if you talk about like validation and also maybe trade-offs between all these three approaches sometimes as a team we might not have the time and effort to run all these three analysis right maybe this first one is what i should do so try to be time conscious on how you spend on each because if you don't end up having the time to cover all of them then it might be a problem of you know not hitting the points where the interviewer is looking for but be flexible too because the interviewer might have a very strong preference over one area over the other then they're okay for you to spend more time so for example i already spent a lot of time talking about observation in this case j then i could prompt back to the interviewer i think survey and also a b test are also really important do you have a preface of me talking one over the other because time permits i also want to get into the validation part so i would say that i care less about the survey and more about the actual analyses that can be done given the existing data given the existing data okay so you want me to talk a little bit more on how to collect the historical data and how to analyze existing data yes okay yeah so a really great tip uh even though you offer three analysis but then by asking jay like what you should narrow down you just have to focus on one and then make that case really really good so going back to the analysis like the data we should collect from here i think there are three sort of dimensions of data we should collect obviously price is one of them right we talked about increasing price in the past but then maybe the interview also care about like if price is not the most important like what other factors could play a really important factor maybe let's say the user didn't watch the show over the last 28 days they're more likely to churn or like maybe they're very interested in a particular topic of show like let's say i'm interested in competitions right whatever baking competitions went up but netflix stopped offering the great biggest baking show was the only show and then yeah the season ended they stopped offering other baking related shows then i also lose interest so to summarize i think i would look for like three key metric areas so one is like around consumption measure like a time span and all that i talked about and then two around interaction right like the people ever liked or read a video did they share with their friends and also around the time metric is around tenure and how long they have been on the platform and what's their life cycle and finally we can also think about demographic or what i call attitudinal so maybe this person is living a certain zip code there's some socio-economical situation maybe like they're more price sensitive to a price increase depends on the physical address of where they live so those are demographic data you can collect so after i talk about like collecting all these features make sure you try to categorize them oftentimes some candidates just talk about some like random ideas here and there like oh i think about this feature or i think about some other features but like there's no logical connection of how you thought about these three features so the interview can get lost so try to categorize them sometimes it can be hard so you're very initial like two three minutes you took to jot down your thoughts are really important because that helps you to categorize them and that makes it really clear to the interviewer what you're looking for so let me take a pause here i know we talk about data collection all that next time i want to talk about modeling and talk about what type of model or retrospective analysis we should do in order to get some sort of indication of whether price is important before you take a pause oh sure okay do you think taking multiple pauses throughout is better than taking one big one in the beginning or like how do you think about that yeah um i think it depends on the preference like personally speaking i'm okay with them taking a longer pause in the initial and then i prefer like the rest of the interviewer just kind of like as a conversation back and forth although i've seen candidates like stuck during the interview and needing the time to take a pause i think it's okay as long as you're not taking too many of them like let's say okay you know i need another minute to write down my thoughts like if you do more than two or like more than three i think three would probably be the max like that would be probably too much i would say try to take you know as much as time you can't in the beginning because the interviewer also wanted to be engaged right like if you take pauses in between the interview like they might get bored like the flow kind of got broken right in between so like i think the flow is also really important so some people might not mind for me personally i'm okay with them taking long pause initially because i also want you to talk out loud sometimes like that awkward silence can be a problem if it's too often yes yeah modeling i won't talk too much detail on the technicals i think as the interviewee you can give couple of options so to me it's a pretty like obvious chair model or renew model we're thinking about so think about like binary like what your response variable should be right what's your why what you're trying to optimize for like turn 0 1 and what your x's should be so we talk a lot about these features already so have some you know model in the back of your mind for this case it could be like the virus analysis some type of like deep learning regression so have these model options in the back of your mind and talk through like why would be rallying in this case so i won't go into too much detail on the technicals for this part okay and then finally i think after doing all these analysis we'll probably figure out do some kind of ranking right so price being one of the factors but there are maybe a hundred other factors that i talked about our features talk about and then maybe we can rank them based on our model result and then price turned out to be it could turns out to be like the top one or it could turns out maybe the top three so based on the analysis result i can give you an answer of what whether price is the determination factor or not yeah okay yeah so that's pretty much as far as what the structure goes i think for the most of the interviewer depends on the time right depends on how much you set explaining all these models and stuff you might have extra time for in this case we have maybe like 10 more minutes to talk about other considerations so i think an important tip here is what i'm looking for as an interviewer is also whether the candidate is being creative so creativity is very very important all these things i talked about maybe it's sort of standard like if i've done a lot of interviews like all these kind of games can think about future collection model and all that but then i'm looking for always looking for that moonshot idea so being creative i think it can be hard to teach i think it comes down to really comes down to your experience and then when i say experience is about like readings and research you've done in the past because coming up with something brand new on the spot like you're already really nervous during an interview right can be really hard like you probably just want to finish the interview with no hiccups like that's your goal but then if you can do some research and readings ahead of time have some ideas in the back of your pocket that will really help so for the netflix case i would do beside the research i do i talked about is another tip is like you can think about what problems netflix could be facing so for example we talk about competition like that's already addressed but the other one is i've read about like family sharing could be a potential problem because as you're aware right like netflix a lot up to four accounts depending on the tier but the maximum is six but then a lot of the millennials and these days are sometimes like 10 or like 20 or 50 i i don't know the extreme cases can be sharing one account which is actually violating the netflix sort of policy so you can if you read about certain like you can talk about like how netflix is trying to address this problem obviously it has to relate it to the pricing or whatever like the topic you guys are addressing like not talk about like some random idea that's unrelated just because you want to say that idea that's also no good try to be relevant to the question you're trying to address that's why i feel like you need to have a couple of ideas in place and then another idea i thought about is when we talk about pricing right we talk about like possible city like if you increase price um obviously there will be less subscription but the total revenue or my increase like vice versa yeah so one idea is about like the maturity of the market netflix is in since we're focusing on u.s one of my assumptions like methods right now have a pretty sort of saturated marketplace and the competition is really high so instead of growing new users we'll probably just worry about like not to lose existing users to our competitors right so in order to not lose new users and by by increasing the price we might think about like how much to increase depends on this price sensitivity so for example like maybe i'm already subscribed to the service like for example like if you're subscribed to amazon prime like you're not constantly looking at how much you subscribe and then people are generally like forgetful or lazy about whether they increase one dollar or two so for a saturated market like this material market like this price increase is probably less affecting the churn rate than in a brand new market right for example you're tapping into a new market i don't know like china probably doesn't have nothing maybe one day we can tap into china for example then people are extremely price sensitive right of how much you're offering so like this mature versus less mature market could be a consideration so you can bring it up even though you picked us but you can still say okay if we were to do a completely different geographic region my analysis could be completely different so have these like good innovative ideas in the back of your pockets are really helpful and then any thoughts from your end before no i like that idea and i think having it be more real world also helps like you can bring in facts about that you know you know that it's like pretty saturated right now you're not just like spewing these like ideas of churn and retention based on like historical stuff and it's actually like related to stuff that's happening right now so it's just like actually applying it as like a real business scenario where yeah to like someone in which you might be the decision holder for like the actual company versus just like a theoretical kind of scenario where it's you don't know too much about the background or the information yeah like the context of what's happening around it yeah exactly i think yeah i totally agree with you jay on this one like just imagine you're already an analyst or data scientist on the netflix team like pricing team right let's say your manager come to you with this question like how would you answer how would you collect the data i think that really puts you at ease like you're really like emerging yourself into the business and trying to solve this problem together with the interviewer rather than just like trying to answer this interview question i think that's the best approach it will also make you more comfortable so having this mindset i think for all the candidates i think like super super important and then i wanted to share like two other tips then just throughout this interact two other taboos or tips that can help with the interview the first is around be prepared to expand on any topic you mentioned so whether that's like a particular terminology mentioned or like whether that's like a particular point you mentioned for example i talked about price elasticity earlier right maybe you talk about like collaborative filtering like if we talk about recommended system in netflix then those are slightly more complex terms and sometimes if the interviewer is familiar with this subject they will actually brill you on it or they will ask a deeper question so be prepared to answer these questions if you are actually not familiar with this topic or this turn it's better not to mention it then like mention something i often see kind of this like conventional term that they are not a hundred percent familiar with and then once i ask a deeper question i just feel like they just completely you know so not like not be able to answer it so like that's a really really big taboo make sure it's same thing as anything you put on your resume you should have the evidence to back them up like that's the same thing so uh make sure you are ready of everything you say and how well you answer the follow-up question to me is actually more important than like how well your initial thought process is because you're directly addressing what the what's pressing on the interviewer's mind okay so that's one other tip and then i guess my final tip is you also want to draw your current expertise into the equation what i mean by that is for example i mentioned earlier the team i'm currently with is stadia at google we also have a subscription model it's slightly different from the netflix subscription model because we also have a free tier meaning that like people can sign up for free and then purchase the games on a one-off basis if you can possibly draw from your current existing experience to provide some ideas to the subscription that netflix has like that's a big plus so for example since we talk about survey right maybe netflix right now i'm sure they're running service but maybe like netflix right now is thinking about improving their survey response or improving how they design their survey maybe only like one percent people fill out the survey when they turn or like the survey pop up there's some problem with it so i can offer some advice on how we design the survey as stadia or like when we pick the multiple choice right maybe the ranking of the choice matter like if let's say we have other like usually people were just lazy right they just pick other like what's the reason you unsubscribe blah blah blah right price maybe not enough content and i don't care about 4k whatever and then you have other and then people just like oftentimes just like pick others just they want to skip to and close this window quickly so like that's our techniques we can learn to like or provide value to netflix i'm sure netflix is already thought about but it's good to bring it up and like draw from your existing experience and how you guys address it in your current team so this not only shows like if i were to hire you jay or someone else right i feel like you're bringing extra value to my team provided you already worked on this problem in the past and you already worked on this problem probably in the past and then you're bringing these innovative ideas drawing from your existing experience so usually interviewer would prefer someone if you're a direct hire not a generic hire probably doesn't matter if a direct hire they prefer someone has the experience over someone has no experience right also the domain expertise also matters so that's a little tip you might not get lucky in this case i talk about subscription right but think about other areas you can draw the thoughts connect the thoughts and then try to bring what you're working on your current team as a add-on value to this team you're interviewing for awesome no that's great and that's super helpful in general and i've heard of adding your own expertise and experience is very very helpful because you're showcasing that you have that knowledge and that you've thought about it very deeply before right yeah and sometimes the interviewer might not like ask you for it like specifically so try to take the initiative and mention it when it comes to the right spot and also don't make it too rigid i talked this about like rigid framework a lot like sometimes people feel like they have to talk they prepared this and they have to talk about it so they just saw that idea in very like awkwardly in certain spots that just make the interviewer feel like you want to save for the sake of saying it so finding the right spot to say it i think that's also comes with experience and practice as well that's something to be aware of awesome thanks jing and i'll talk to you later yeah thank you thanks jay take care

Channel: Data Science Jay

Views: 14,136

Rating: 4.8328981 out of 5

Keywords: data science jay, interview query, data science, data scientist, data science mock interview, analytics interview, data science interview, data analyst, product analyst interview, product analyst data science interview, product analyst data science, data science product, data science product interview, data science product questions, netflix data science interview, netflix business product interview, business product analyst, business product analysis

Id: 7VFoLDN3apI

Length: 26min 12sec (1572 seconds)

Published: Mon Dec 07 2020