How to Think Product Analytics in PM Interviews by Amazon Sr PM, Vivek Pandey

so like I said thinking product analytics and what I consider product analytics is essentially I mean it's not it would be a subset of analytics in general right if you're looking at a B testing if you're looking at basically any kind of statistics we applied that we apply strategy and we look at product from that perspective and that's what we're going to talk about today so I know there are three questions in your mind right now one is who is this guy why should we listen to him does he have dog pictures and when you talk to a dog owner that's the risk you run anyway so yeah these are my dogs Oh what are my dogs Obie and Wayne that's important that was you on that later so just give you a quick rundown of where I've been I have been heavily technical person early in my career I was a programmer with IBM I was an engineer with Sony worked on a Playstation for a bit I worked in Sony Pictures and did my MBA at um go blue how many anybody from Michigan here yes go blue I was at Groupon I ran an analytics team there so which is kind of where I really got into analytics and then I've been at Amazon for almost four years now I am part of the prime video team that is the streaming service at Amazon anybody use it do you guys watch prime video perfect keep doing more of that all right so what will we talk about today so analytics in product management of course I mean when you think about statistics and analytics it's a very broad field but what does that mean from a product management perspective we will talk about that what does the so when you get questions from the interviewer what are they really looking to get out of you sound like it's more than just statistically rigorous more than the accuracy that you can bring to the table there's so we'll talk a little bit more about that the meat of it is going to be on the second part where we are talking about analytics why the structure of the questions and there are what should you be thinking about when you're asked one and finally we will get a little bit more technical about how to design funnels how do you look at a be tests what they can and cannot do sound good what will we not talk about very quickly Fermi problems strategy questions how are we designing products all that know we are going to be focused on analytics cool so part one analytics as a part of product management let's see so that's another picture of vane I call it little Wayne but yeah so what do you think your product manager does and I don't mean like you know terms you hear or he's the CEO of the product or she's the CEO of the product I mean well yeah kind of but they don't go in the morning have the coffee like I'm gonna be CEO today no what do they actually do once they get to work and as far as I'm concerned it's essentially one of these four things that they're doing on a day to day basis their I da ting they're thinking about how do I make a new product how do i improve the product i have what new features i can add that kind of thing prioritizing of all the things that i've already thought of which are the ones that i want to actually work on persuading now that's kind of like always a little bit the understated part of the job but as a product manager you're spending like half your day talking to your leadership about why you're thinking of a good idea you have to convince your engineering teams or the UX designer whoever because that companies as you know are very democratic you're very responsible for your own time so you have to go convince other people to come work for whatever idea you have so how do you do all that with data of course and finally the execution bit is yes do you thought up an idea you convinced everybody now you gotta go get your done so that's where you go and actually build and launch products track and manage everything and so when you think about this what is the product manager doing on a daily basis as they are thinking of change they are creating change and then they manage it continuously so but that would be chaos right like if you're constantly changing things that's very chaotic and the way you bring order to all this chaos is through analytics through data you have to think of how do you prioritize with data how do you convince somebody I mean there's some people that I just I'm going to go for a moonshot idea until you can show the data show them what you can actually do so whenever the interviewer is asking you questions this is what they're really trying to gauge can you identify what data is relevant if I go and say hey I'm gonna run Facebook events cool but how are you gonna run it what data do you think would you look at if you were to run a feature like Facebook events for instance can you correctly interpret the data so if something is up or down do you know how to make sense of it how do you go about and identify interpreting it can you use the data to persuade others and that is that is where the statistical rigor comes in because a lot of people are from a tech background they're from a heavy math background if and from a business background as well so you have to make sure that the data you're presenting is persuasive and are you realistic about what data can be collected like some things such as user sentiment for instance are very very difficult to accurately gauge and if you base your whole case what I'm gonna measure that then yeah that's probably not gonna happen so are you certain that you can I you can't collect the data that you want to and sometimes it just may be an engineering challenge for instance you will be surprised how many gaps products are shipped with and the data that you think will be readily available just may not be available when you're in the job so you have to kind of think of those things when you're answering these and lastly and I think that is probably the most important part is even if you can identify all the data in the world do you know which ones you can actually control as a product manager if my whole thing is based on say I'm a harmony calm my whole thing is based on how many people are single I can't really control how many people are single even though there's no way to do that so which of the data elements can you actually use as a lever to run your product so in my mind at least when I'm interviewing these are the things that I'm looking for so let's see now let's actually get into how do you think about the questions themselves right i group all product analysis into these four buckets you make I mean you guys are all smart he may come up with one that hey this that doesn't fall in here but think about it it probably will but in my day to day job like I would say almost every analysis that I come across would either be sizing the opportunity should we launch this product product in Hong Kong what is how many people are going to use it that kind of thing or should we just launched this feature how are we going to cite an opportunity prioritization like okay fine I have these three features now which one do I want to work on this year because resources are always limited you only have X engineers or X UX designers to work on thing success definition how are you going to use the data how are you going to measure your product and how are you going to define what success means so I launched a product I run for a year how do I know it's doing well how hard what am i comparing it against that kind of thing and lastly when you are lucky enough you have a product launched things will always get messed up what is going on with that you launch it day three nobody is using it at all like how do you go about diagnosing what went on with it and I think what do you think about this so I think Adam Nash put it really well as a product manager you are really responsible for two things a you have to know what game you're playing and B you have to know how you keep score so and I think all of this falls under that what game are you playing how do you keep score and that's where analytics comes in so if you look at example questions for something like this this is the opportunity sizing when you hear the interview saying should we launch something should we launch this new service in wherever what they're really asking is can you identify first what is the size of the opportunity of course there's more to that question but the basic thing is is it worth pursuing if it's going to make me $800,000 great but is it worth good Amazon pursue $800,000 what Google do that so the first thing is you have to even understand how big an idea it is that you have the other ones like so you may think that's more like a Fermi problem like how many Italian restaurants are in Seattle or how many X is in Y kind of a thing but when you think about it there are only as many restaurants in Seattle as the demand is right I mean that's like a very competitive market restaurants are closed open whatever so if you can accurately size the demand if you can accurately say how many people are stored so that's basically just an opportunity sizing problem prioritization that always comes up at the end usually at the end of a brainstorming session you say how how many ideas you can think off of improving YouTube you rattle off like ten things and then say okay which one would you do first that's like okay how do I do that so what are you using to measure everything how do you go about you know talking about prioritization and like of all the features we've discussed which ones would you build that kind of thing this comes very often in success definition if you had to pick just one metric what would you pick and it's usually I I don't like that question at all it's like asking what what would you look at when you're driving your car would you just look at the speedometer no I don't know things to look at but that does get asked very often so it's a good thing to have in mind how you're gonna answer that and they're not really like looking for one metric everybody knows you can't just pick one metric but how do you the process is more important than these and lastly diagnosing issues ad revenues dropped by experts said how to do that why a product page we use down something of that nature that something has changed how are you going to identify what has changed so these are things we will talk about opportunity sizing first and the three buckets that you see down below they have you can actually just do take one approach to answer any of those questions and we'll talk about that in the second part of this so first let's talk about opportunity sizing so again I'm using e-commerce terminology you can replace first-years here with whatever other variable that may fit in but there are five things you have to really do and try to break it down as simply as possible one is you have to clarify the scope so if I ask you something like okay how many restaurants should I open like okay in what city are you going to open them how what is the source of revenue are you renting your space what not whatever you would come up with that then you identify the base population and what I mean by mean by the base population is and we'll talk about these more but it's basically how many people have access to the service not how many people will use the service but how many even have access to it like Seattle Airport for instance like everybody has access to it but are there is does everybody use it now and that's where you get to the addressable segments of all the people in your base population how many are you even going after then you purchase frequency is how many times are they going to use your service and lastly how much will they pay each time that they use it and we'll run through an example so let's think about this question a should we launch a bicycle based food delivery service in Seattle and I actually did get asked this question one time maybe in the and keeping in mind of the way I'm describing this it there's no perfect answer to this right nobody knows whether he should start a bicycle based food delivery service but what the interviewer is looking for is are you going through all the right steps in your mind are you looking at it in a logical framework where you can actually come up with an answer had you enough data at that time so let's see what did we say we're gonna do first two parts to the answer what is the size of the opportunity we'll talk about that right now but that wouldn't complete the answer right if I could say okay this was like a 10 million dollar thing so would I do it but the other part that falls more under the bucket of strategy and designers find this a 10 million dollar mark and how much of that can I actually even capture how many other people are doing it what are the unit economics if I did it would I be profitable in the short term in the long term however and does this align with the company's strategy if you're YouTube then it does not like no I wouldn't do it even if it was profitable because that's not what I'm doing will you find enough writers or delivery people and all of that comes into more of the design of the service rather than how big is the opportunity so let's talk about this force it'll clarify the scope I mean in this case it's really simple we the interviewer gave it to you you're just talking about Seattle but the other things that you can clarify is how are the sources of revenue are you selling the food or are you just selling the delivery as a service what are you actually doing and when asking for clarifying questions it's always best not to ask for data directly don't go around asking like okay how many people are we talking about or whatever just make assumptions and ask if those assumptions are corrected I find that to be a lot more helpful for me to look at look at what the interviewee is thinking about they go about saying I would ask something like I'm assuming it's Seattle is that correct I'd never like I mean of course that makes sense and you're assuming right and of course I mean the strength of your assumptions also gets tested I can't go out saying I believe there are 1 billion people in Seattle I assume no that's not right at all so you have to make meaningful assumptions of God will do clarify that so let's let's say identifying the base population so for something like this what we are saying is the base population when I talk about a base population is who has access to your service it's not like who is actually going to use it but you're going to base it off of something right it can be it can be an industry it could be a geographical area a demographic that you're targeting and in this case we are saying what's your unit of measure as well right because you are I could say that if I'm going after say a service like concert tickets is per person every person is going to the ticket so my unit of measure really is how many people are in my base population if I was selling televisions it's probably one per household is how I would think about it if I was this is b2b company I would be thinking about how many businesses in my industry are there so that's how you're thinking about a base population of this is the entire population that I can serve with my service so in this case we are going after households because it's not like every person in the household is going to order food separately usually you order for the household if you'll deliver if something is coming in so I think that - skipped over once I know okay it is base pop so base population to this so I'd say number of households I we are serving Seattle I just take 800 K people in Seattle about two and a half individuals per household that's my base population is 320,000 households it's always okay to round up or down just to make sure that your calculations are easy enough to make on the fly nobody is looking for accuracy in these interviews to be honest I mean you have to be in the ballpark of course but you don't have to be like precise to the decimal point which is very different from a consulting interview how many of you have gone through case interviews for consulting so you guys know what I'm talking about over there you have to ask for data you have to be super precise you like they you're judged on your map not as much in a product interview we're looking more for the strength of your assumptions the strength of the logic and which is why I also encourage people not to I keep asking for data like you do in the consulting interview you're basically making assumptions that make sense so you go with that now this is where you get to become creative now you have based population studies say okay this is how many people are going to have access to my service now you start whittling out how many people are actually going to use your service and they are the target they're your target segments and you can use basically any idea here really to come up with a theory how many people can even afford to eat out how many people are going to be in the area that is serviceable like you can come up with X number of factors to whittle down your base population down to a logical addressable segment for instance if I was going to start a charter plane company from out of Seattle I would my base population would be everybody in Seattle I guess but if I'm thinking about addressable segments I probably would not be a target customer for it I can't drop twenty thousand on a flight so I look after who can actually talk twenty thousand on a flight how many of them are even traveling on a frequency good enough and that way I whittle it down to an addressable segment and how many of those are there so let's see what fraction of the base population is addressable and like I said no longer right answer here's how I would do it I would say we are delivering by a bicycle so speed is a constraint you can only deliver X miles on a bicycle so I would just look at places where restaurants and people are concentrated so I say I'll start with downtown Ballard Green Lake and Capitol Hill and solve your hey my areas not here I'm sorry three so for that I thought of you could add more if you want but I get the general idea I said these are the areas I could think of where it would work and let's assume 50% of the households here order again all the time so well just like for all these 30,000 households are now my target segment that number may be widely off I don't know doesn't really matter but as long as my logic is strong enough I say this is how I'm going about calculating it the interviewer knows that given enough time here you go on to Google all these numbers and come up with a really good answer but you know what questions to ask and that is important so we're going down to like okay 30,000 households and this is where you start with the next part which is how do you look at the purchase frequency so I have 30,000 people who are going to be ordering all this time but how many times does a person order out I yeah who knows right but you have to pick a reasonable unit of measure if you're looking at a weekly or a monthly basis if I'm thinking about if I was expedia.com thinking about a vacation product how many times people do take a vacation maybe twice a year three maybe so I would go at an annual cadence and go ahead after that number and I'm looking at eating out monthly seems about right there so I can think of how many times I eat out a month kind of extrapolate from that see how many other people would be doing the same what assumptions are you making same way you have to be strong about that if you're assuming it's five times a month then why do you think it's five times a month and it can be anything you can say I looked at industry data just 5 times a month or I eat five times a month people in my age group eat five times a month and that's very that's not a very good answer because it's very anecdotal but you don't have any basis for really making an assumption there so you start with something as long as it makes sense you how many you thought you watch called they're weird of truthiness so as long as it's true thene of it makes sense as long as the logic holds the numbers can change I mean they know you can go out and find the numbers but your logic has to be rock-solid so let's see we have that how many and the purchase frequency and for physical products I mean this is just a delivery service but if you were talking about toasters how many toasters does a person buy I mean even if you knew that okay we are going to sell ten people have toasters how many of them are buying replacement toasters so you have to always think about the replacement purchase as well when you're thinking about a physical product or something that has a set lifespan so in our case it's just a service there is I mean yeah there's a replacement service it's like how many times are using in a month so I'm just gonna pick a number say three times on average is how many people are eating out a month now 30,000 households was my target segment they are eating out three times a month ordering in three times a month maybe they're eating out more often but I actually looked up this number an average person eats out about six times a month and I'm gonna say okay three people three times they're just ordering it instead of going out because they're whatever they want to watch a show so we have three times so 90 times per month 90,000 times per month is how many times people will use my service so lastly what does that mean in terms of revenue so you may not even have to get to this stage I mean the interior could be very he could just basically just ask you directly how many people do you think will use this and you're done at that point but it's always when you're thinking about the size of an opportunity if there is a revenue associated with it and there always is you wouldn't be running like things just like that anyway so it's always good to bring it home with a revenue number at the end so what is the unit price typically the interviewer will also not give you a price for delivery or whatever number you're looking at but you can again make an assumption based on based on industry standards if other products exist so in this case who are competitors you can have think about ubereats amazon restaurants post mates door - whatever we know how much they charge so you can say we have to price it accordingly or somewhere in that ballpark if it does something completely new to the world that you have never used before so teleportation device how would you price it what's the next best thing to a teleportation device it's and you would take a flight and get to some place but then if you're saving for hours what is that four hours worth in time and then you can extrapolate from that and say this is how I would price with a levitation device so if it is if it's not new to the world based it on whatever competitors are doing whatever other people are doing you can base it on cost as well if you're really in growth mode you don't care about profits right now you just want to recover your costs if it's completely new to the world then you look at what is the next best option that this person can use who's using my product and what is the value I'm adding over that product and that you can then extract I mean ideally you want to extract all that value but you really can't so it's it's going to be somewhere between the next best and whatever value you are adding so based on that let's see we're just gonna say five dollars per delivery sounds reasonable we that is what everybody host charges that's what I'm gonna charge and people will pay for it because it's a bicycle it's environmentally friendly whatever that's how it market it so okay so we do $5 so we bring it together we said total households 320 we are only going to address 30,000 households because of XY and Z they're buying three times a month from us $5.00 per delivery and then we are going to make about 5.4 million dollars annually a so good money but so that's your opportunity size right I mean we just went through all the steps we came up with the number but does that doesn't complete your answer you're not going to complete this answer in this session by the way they were just looking at the analytics piece of this the next part that we are not talking about is so you come up with the opportunity size you like five and a half million that's that's awesome but should we do it and that's where the strategy piece of product management kicks in right who are should we do it and that's an easier question to answer because you have you know in this space how many competitors are there what are their benefits like ubereats can just pick up from one place deliver to another they don't have a constraint on distance and you can think about what are our competitive advantage they just versus others how what fraction of the market can be meaningfully even capture that would complete the answer of course and it's not even going to be profitable like how much are you gonna pay the rider if you're only charging five dollars or in delivery that he's gonna take half an hour to make so it may not be a good idea at all so like we said we are only discussing the opportunity sizing here so to summarize five steps or you can basically I mean my man there are very few things that you can't size using this because like I said I'm using purchase here you could use whatever else like if it was how many people are going to use events it's kind of a similar flow but instead of purchasing they're clicking on something or doing some action that I'm just calling purchase over here in my example so of course I mean other things you always know about a seasonality regional differences got to talk about those as well so the next part of course is all three of these prioritization success metrics and issues diagnosis like I said you can have one framework to deal with all these all these disparate questions and we will talk about that so questions around ease we talked about those already all the ideas which one would you implement if you have to pick one metric or something dropped 20% how you gonna do this so they all have one common theme and that is how well do you know or understand a funnel how many of you are familiar with the funnel no that's so yeah you're halfway there already so what do you really have to have so basically I mean I'm just describing a very simple funnel let's think about an e-commerce website you have a hundred percent everybody who's entering the other site seventy percent of those are going to an item detail page looking at the item reading it up twenty percent are going to add it to cart seven percent I'm going to check out and three percent are completing the purchase I mean I totally made up these numbers but they are close enough for most retail sites so that's of course one funnel what I'm looking for as an interviewer is like like I said before do you understand the game we are playing and this is how you understand the game you're playing if you were an e-commerce site these are the rules this is how a person will navigate your site and do you understand that at the onset the second part of course is do you understand all the inputs and outputs that go with this so your total site visitors what are the inputs to that your marketing your search engine optimization how many times you come up on a search result your seasonality like of course if it's 4th of July then Expedia is going to do much better prime day Amazon is going to do better Christmas time everybody does well so like do you understand that as well the number so that some of that you can control seasonality of course you cannot control and those are things that you have to kind of know intuitively and as you practice more you get there or the detail page how many people are visiting it that can now that completely depends upon you a lot of it depends upon you for instance how many items are you showing on the home page how many items you even have in your catalog first of all relevance are you showing the right thanks to the customer price UX quality I won't go through all of these but to give you an idea everything has inputs and how quickly and easily can you identify those inputs now one way to practice for a product management interview is just think up any service think of like Facebook events think of Google Calendar can you draw a funnel for that and can you understand the inputs and outputs for each step in the funnel and if you can and the quicker you can do that the the better that interview goes because right at the onset you have a framework of how you're thinking about it that you understand the rules of whatever game you're playing that day so those are inputs and outputs yeah some of these you obviously cannot control like product ratings and reviews if you were doing that that would be that could probably not be the most ethical thing to be doing but yeah there are levers here that you cannot actually control so what we do as product managers for the most part as we design features that will impact one or more of these inputs and we try to raise each step in the funnel whatever that funnel may be here we are talking about an e-commerce site but if you are talking about Facebook events for instance how many people our discovery would be the first at the top of the funnel how many people are discovering it how many people are clicking on it how many people are creating an event what not that kind of thing so you have to think about first how quickly can you design the funnel how quickly can you identify what inputs are relevant and what features can you actually build to impact any of these inputs for instance relevance is a good one I mean you'd it's hard to do because it is basically you could probably use a machine learning algorithm but if you go into that then if you don't understand washing learning very well that's probably not the first one I would pick to go into Oh price you can play with price but that's not also a very good fun to pick to you still want to be profitable you and if you're a market like Amazon you can't always control the price oh you can too but for the most part you can't always control the price profitable yet you'd say the rest there's a lot of leeway and the same you can think about Facebook events as well the example I was giving the other example that I was giving you can say as a product manager you could say okay you know what I'll do is I'll plaster or Facebook events all over my website and that way a lot of people will see it everybody would click on it but what does that do to the company strategy what is that to the customer experience somebody coming on Facebook so when you're thinking about the inputs what features are you building for these inputs and how does that kind of integrate with the rest of your website with a strategy of the company and where it's going so that's on basically how do you what we do as with these inputs you talk about prioritizing let's think about you always have to go down for the impact right this is where you'd get asked okay you've you described all these features that are going to control all these livers which one would you implement so the first say for example I'm saying I come up with two features one is I'm going to have a lot more items than I have right now I'm going to like how my icons will be smaller I can show more to the people or whatever have my relevance is better however I do it I'm going to now increase the item detail page views from 70% to 80% in the second case I remove friction at the checkout step maybe there was a security thing that I'm saying okay instead of re-entering your credit card now you can like just use the card on file or you only you don't have to re-enter something or I take away something that improves this to 70 so that's 14 this is 17 awesome right 17% more but the core the point that's missing from this slide is that the goal though as a company right now is revenue my goal or as I'm and just wanting to improve growth so the first thing is like of course you have to understand what you are impacting before you come up with that so the steps really are what are the features how much are they going to impact the metric that matters and I think that's one that you have to talk about the interviewer and kind of get a sense off of what metric is the most important in this stage of the company like for an early stage company growth would probably be the most important one revenue would be for more mature companies you're looking at profitability more revenue expanding to other things they may not care as much well I mean everybody cares about growth but they may have be at a point where something else is more important to them so identify what it's what is the metric that you're optimizing for which ones of your features that which one of your features that you describe hit that metric the most and that's how you would frame the the answer to what feature would I prioritize of all the ones that I just described and the same thing helps you with troubleshooting as well right if I say that okay oh my my revenue is down and usually the interviewer is going to ask you a metric that is really down the funnel I can come up with my revenue is down 20% how do you diagnose that and what the questions that will get you there is a first of all you have to understand the funnel steps right with how did you even get to your revenue in the first place how many people came to the site how many people looked at it how many people bought something how much did they pay for it that's your revenue so as long as you can identify all the steps that are leading to that metric like then you can just backtrack and say okay where's the dip do we have that our total users dropped so I generally start going down the top of the funnel then our total users drop though okay so everybody's still coming the same as they were pageviews drop if yes then I know that something is going on over there which is preventing users from hitting the detailed pages so once you have identified the right step in the funnel then it becomes easier right because you can have another funnel also another thing I wanted to mention is the funnel varies by your platform the funnel can vary by segments say in this case I'm saying 50% of my views were on mobile the other 50 were on web and maybe they behave differently it is easier a lot more people check out on the web than they do on mobile for instance or what people would act differently so once you know the step in the funnel you can also see which or which platform who got impacted the most or which user segments got impacted the most and that's where you can keep digging down until you can identify those I had one of the more interesting analyses I had done was a Groupon where we had a dip that was we went through all these steps and you basically just couldn't identify where it was going on it just seemed across the board but this was everything was on mobile anyway so we didn't know where it was and it came down to screen size if somebody was using an iPad they had a better experience than somebody who was using like a smaller phone than that so it or it's like the so I mean there are things of that can come down to that level of detail in an interview it's probably not going to happen but knowing what to look for does get you a long way there now the third one this is kind of my favorite topic actually you're a B tests how many of you here run a be tests are familiar with them on a regular basis oh very cool funny people very good so you have a lot of context on this for those who are not do a very quick primer a bee tests basically are you just create a feature his control bees treatment I say I was going to launch a product I would split this room into half the this half only gets to use the feature that use the website the way it was before this other half sees the feature and then for after a certain number of time I will compare the metrics between the two and see if it did better or worse so basically just four steps to it once you have designed your test you have to select a metric what are you measuring between these two groups okay I'm going to measure your performance sources there but what is it I'm measuring how many times did you come to product school how many times you didn't or like if I was looking at product school it was base it would be attendance I guess but I Brian is looking with me like that's a good good now that but if I was looking at basically this room it would be a different metric than if then I was if I was looking at the website for instance so we select a test metric calculate a sample size depending on your metric you will need to know how many people should be in both groups to get a statistically rigorous result and this is I mean this group is too small for like running hooray let me test it would just be very anecdotal but you'd be talking in like thousands or millions even depending on the metric that you pick and we're in the funnel your metric lies then you run the test and then basically at the end of it once you have the right right sample size you compare the results so we're going to talk about a lot of mistakes that people make when running a B tests one is a lot of people pick the wrong metric at a company I was we we were getting I was running the analytics team and we're getting a lot of push from the business to use profit as a metric saying that every time that we are running a test we should see if one is more profitable than the other because you could be selling more but if we are just selling you just reduce the price to $1 the other group is going to do really well but that's not what we want but the problem is that financial metrics are really hard to measure with an a/b test the key is variance the higher the variance the more noisy your metric is and the bigger sample size you need for that so it can actually I mean yeah if in that example if you're looking at just a financial metric like profit the variance is so high because one user could be buying a pack of toothpicks the other one is buying a plasma TV on Amazon and the variance was already so high I can just look at those two and say hey the super group is doing better or they just made like ten thousand dollars but so that's key over there and I mean literally you can take years the other part to remember is very in the funnel is your metric if I'm just looking at the checkout page for instance then even if like a million people are heading my websites every day off that may be what like thirty thousand are hitting the checkout page so it will take me a lot longer to get to sample size if I'm changing something downstream does that make sense so here so the further up your metric is the faster you'll get to sample size the further down it is the longer you'll need depending on how many people actually get to that feature so I like picking really low variance metrics such as orders per user I mean doesn't matter how many people whether somebody's bank like the toothpicks or the plasma TV they're just buying one thing so did they convert would be my metric I can have another one but a revenue would be the hard one to measure but I can like measure that differently the most features will not affect the price of it until you're really playing with that the second one is a lot of people misunderstand statistical significance those of you working with a/b test you'll get the question a lot about okay when will this experiment get to sister SS statistically significant like I don't know maybe it'll never get to significance you don't run a test until it gets to significance if you let it run for infinity everything will get to significance eventually because at some point though it you're measuring such minor changes that you will start seeing significant results but what you the way you do it is you select a sample size you run the experiment your sample size you stop at that point either it is different or it's not and that that's when you can actually tell so you can use the mm Miller's a be testing blog as the resource I use a lot that can show you how to pick sample sizes it has a lot of background on this on what you can and cannot test here's the next one is peeking at the results and this happens a lot as well you start running a test and you say hey you know what let me look at that now how's my group do it my future doing versus the other versus the control group and don't do that not until you hit sample size and to illustrate that we ran I think this was about a hundred experiments just random ones and you'll see over here almost 60% of them and you can't tell from this blur over here but 60% of them showed statistical significance at some point or the other right after we launched the experiment and that's just noise you really can't make any decisions from that but a lot of people want to get early data because you want to do things faster and the end of it once you did get to sample size there was only one experiment that was actually statistically significant so patience is key which is why a B tests are hard to sell to management because you like I mean it takes a lot of time and a good well-designed a/b experiments are really hard to find to be honest so and the last one is ignoring seasonality and I think this would be true for basically any analytics question that you're answering on a high-traffic website like if I need a sample size of even a couple of million I'll probably reach it in a few hours but that does not tell me how people would behave on a Sunday or a Monday or whenever else so when you're running a test or you're thinking about a metric you also want to understand the seasonality you want to make sure that you're running everything long enough and yeah that's that's gonna key as well yeah well lastly there are situations when you truly cannot use an a/b test and say for content based services like Netflix or Amazon Prime video you can't really show a customer a certain set of movies and not show it to another you really need to do that legal issues you can't really play with closed captioning in the u.s. you have to show closed captioning to everybody when you're streaming a video so there are there are issues where you really cannot truly hide a feature from people and what you do there is a pre/post analysis you basically just launch the feature see how things were doing before it launched and then you measure half after how it launched and that's a very messy way to do it not I mean kind of a last resort thing but it happens a lot more than you would think because things are always changing a B tests are take too long people's lives to pre-imposed but the things to watch out for there are that in your post there will be a lot other features affecting it like for instance if there is a lot of marketing going on that is outside of your feature and that's impacting traffic to your site that will show up as a spike in the in the post period but that has nothing to do with your feature for instance or so it's very hard to take away external impacts unless you're doing everything at the same time with an a/b test but I mean you're left with the tools here left will have you don't have them available if you don't have better ones available so that was essentially all on a be tests and to close I will just go through this one how many of you have heard this you can't manage it you can show us just very few people all right did you know that Drucker actually did not say this Deming said this and so if you can't measure it you can't manage it and one more twist in the story Deming didn't even say this Deming said this it is wrong to suppose that if you can't measure it you can't manage it it's a costly myth that's the exact opposite of the quote that everybody says in every meeting if you can't measure it you can't manage it like you can't measure everything anyway and what Deming meant was not that you should check away data it's basically you cannot measure everything you should be as data-driven as possible but beyond that you have to basically just manage the things you can without the data and to be honest I mean product management is basically you are running a business it's a lot about risk taking you can't really wait for all the data you have to use a law here and one intuition you have to use your judgment which is why I keep emphasizing on the logic and the approach is more important than the actual data itself as you can get the numbers I mean but yeah but yeah that is that was basically all I had and promise this is the last dog picture thanks everyone [Applause] [Music] yes yes yes absolutely yes I am yep I'm glad you caught it because as I was saying it was going through my mind and then I just forgot to mention that yes it would I would catch that and say okay so you just remove the security figure what do you think is going this is that going to do and there's going to improve a or not improve but increased fraud it's gonna improve that metric yeah so that would definitely be a good follow-up question there and so yeah you have to think about the consequences of those changes as well and what other parts of the business that were impacted I think we kind of touched about that on the events example like I can plaster my feature all over the website but that has the effect of really that's not my customers yes sure so so when you think about that feature you are changing a menu you're not really changing the categories that people go to unless you're doing that and if you're am funneling more people to say my cosmetics instead of electronics then that would change things of course and that then you would catch that so the way I like to do that is break it down so if you're looking at profit it is what orders per user is one thing you're looking at the new price per order or something of that along those lines so you would track that but I don't think that would be ideal your test metric and I think you would catch it more in the design phase itself like if your metric is going to affect another team or another group that is actually servicing the high value items I think that would get caught more in the business strategy phase of things than all the way downstream to actually measuring the change itself but but it's a good point I mean you could have impacts that are wider than that but if you have more orders than you you probably look at you could segment the a/b test by your categories as well and say that for this thing that I have was the impact consistent across all my categories including the high value once so that way you could possibly look at if you're getting still getting the same amount of orders between the two and you haven't really changed anything with the pricing so you'd still if the effect is the same between the two groups that you're looking at your you're still fine right so you could probably chop it down by all the organic reason look at the test results that way yes yes sure so it was we're basically exhausted he's at that point so we were looking at we caught that when we were looking by platform actually so everything was fine except for the OS was fine too but when you break down OS by device that is when we started noticing that an iPad was doing fine but we had issues with an i/o over the phone with an iPhone 6 and lo actually even more with an iPhone 5 and then we started running correlations between screen size and the drop itself so it's it's a hard one to figure out that's why I use it as an example because you can get down to that level most analyses are not to that level you can typically figure out what went wrong and 60 to 70 percent of the time is just a technical issue somebody forgot to deploy it on one platform all together and then that's messing everything up yes yes the content representation we try we play with that a whole lot all the time so you would see that the content itself is a little bit trickier to play with because you can't show the same one movie to one person a slightly different movie to another I mean you can test with that but if you get caught that's just bad PR all over so it's probably not worth the cost to do that yeah yes let's see I mean traffic how do you increase traffic all the time it's sorry yeah referrals yeah referrals that would be a good one to look at how many people are referring others do it so I think when you look at the of the AAA RM metrics the acquisition adoption retention and I'm forgetting the M now but anyway it's basically monetization so if you look at the acquisition metrics I think the acquisition would be the most important one for a for a growing startup and then going down that file yeah so that's I guess that also shows you how important the a a RM thing is outside of your interviewing once you've been doing it for a while you probably just get focused on the one that you're doing yes yeah pros and cons of a service a product I think that would fall more under the design and strategy piece of it rather than just analytics you can I mean you can if you're comparing features you can probably use the metrics develop an approach if you're looking at your service versus somebody else's service you can't really compare that I mean you can compare that with analytics saying okay this is where we customers that would be one thing where we can affect more or if we have like a shorter in our case if we can reduce the delivery time versus somebody else then that's a pro but if you can't get enough drivers for our service that's a con so you can still pick the metrics that you'd use to compare one service against the other but for the most part that you do that as more as a strategy question than just using analytics yes [Music] yeah yeah I use hmm so we do have custom tools I however I use other tools I do I use SPSS a whole lot IBM SPSS I used tableau that's one of my favorite tools and just regular SQL stuff so I think I mean if you can reuse redshift of course a whole lot so any if you know how to create a redshift that's always a good skilled at least in the Amazon interviews I use a skill studio and I have an Oracle - left does that but there's a I mean just querying right chapter there's tons of tools out there yes well that's a very broad question right I think it goes back to like what game are you playing so if you were looking at depends on the product really there's a the metrics that would differentiate a product then I think in that case you're looking at the industry all together and what are the metrics that the industry follows like if I was looking at online advertising then I would look at what the funnel looks like for online advertising and then compared that across Google and Facebook and whoever else is doing it and go that way so I think it depends upon the industry itself if you can draw the fun of a photo industry and then you can compare across the funnel for different competitors and see which one is doing better does that give direction I guess not sure if I'm understanding the question correctly saying how they treat their different segments or face yeah I see oh I'm sure they do I mean that's the whole business is personalization right like the way your mom sees the feed is very different from how I see it like my dad just sees political stuff I imagine because that's all he shares but I don't see any of that so they're already doing that but I guess because it just puts you in that bubble you can't really see what they're doing at the others but personalization is probably key to everything they're doing now - yeah although it's actually yes um it is a lot of it is basically I say this but you know how you just get all the members of your team together start writing out hide here so on post-its and you start putting them up on the wall and for I da ting it's usually we use that as a team exercise more than just like a solo exercise I mean of course you can by yourself you can come up with a lot of good relevant ideas for your product but in general it is to do a whole lot with what other parts of the company are doing as well so to give you an example and I think I can talk about it without getting in trouble I mean it's open like so we launched prime in many different countries the 2-day shipping service and that may have a lot of impact on my business because prime video is offered with prime in every country right so if I say that I'm going to just launch prime video in X country then I probably have to also deal with prime so the ideation piece I would have to include a lot of other teams and see what other ideas we can get in typically what we do is we'll get the prime video team together we will come up with like 13 different ideas across the board and see which ones make sense which ones we can actually do and which ones will have more impact so I don't know if it directly answers your question but the ideation part is really how creative you are and that's one of the key things you are doing as a product manager so it's there's very few frameworks you can actually use to get really creative other than just knowing your business and getting creative I guess hmm well yes yes all the time mm-hmm yeah so conversion rate is one of the biggest ones like basically if you're looking at orders per user the in the example I was giving that's essentially a conversion rate yeah how many people are converting and how many are placing orders so that's the metric I would measure against control yes lots of things I think um well it's software engineering I mean there's opportunity to get creative but the product you are you kind of owned the product with software engineering you're working on a lot of other people's ideas you can of course be creative of how you write the code but with product you are actually coming up with the ideas you own the product and then you work with a lot of different people to make it work so I think key for me was it was a lot more creative at least in my for me not saying that and generator it's a lot more creative and there's more ownership of the product altogether and I think that's that was probably key for me yes I'm sure I mean so when you think about it funnels are restrictive but they give you a framework of how to think about it if you think about how the users users are interacting your funnel is not always going down right here one step and then a person may go back to the homepage so your funnel is more like loops rather than a straight funnel but the way I like to think about these things is you have to have a simplified framework we can all keep adding complexity to it later on right once you have a base funnel you are in this case you want the people to go down that route and go down the happy path there are complexities your more people go down to the homepage and then go down in Factor we did that Groupon we filed a patent for a Markov chain analysis where depending on how the user got to the page they are on what is the probability that they'll go to the next page and then we try to divert them to the page that is most profitable for us so there's a lot of different ways you can think about it you're exactly right files are restrictive but they give you a framework and then you can add complexity to that framework but you have to start with us but the right simplistic framework and then you can keep you know making it more complex oh yes we do so oh yeah so I can't get too much into that but we use a lot of different varieties the way we test yeah they do they do but I mean at the NFL like for a lot of features a B tests are good enough for the job at hand I mean you can design a lot of different tests but for most things you can get away with an a/b test especially with the field that I'm and with streaming it's basically you're just changing visual elements for the most part and you can just a B test most of them you

Channel: Product School

Views: 55,915

Keywords: Product, Product Manager, Product Management, Product School, Data Analytics, Coding for Managers, How to get a job, product manager salary, product manager resume, what is product management, what is a product manager, product management training, how to become a pm, product manager interview, machine learning, ai, Technology, Career, PM tools, software, metrics, product management basics, growth product management

Id: k87SPgq-me4

Length: 55min 56sec (3356 seconds)

Published: Tue Sep 03 2019