What Are the Basics of a #ProductManager Role by Google PM, Ankit Prasad

[Applause] this is a really quick intro on product management it's a fairly it's designed for people who are early in the role or who are trying to get into the role I'm gonna touch briefly on what the role is and what the actual work looks like one of my pet peeves when I was first starting in was everything I found about the role seemed to just be a laundry list of things like here's a laundry list of things PMS do and it's part of that is by the nature of the role PM's tend to bring structure to otherwise an unstructured office or unstructured project plan but part of that I think is just there is a way of looking at all of the work in a structured manner so I'm gonna try to go through it on how at least how I picture the role I'm also touch very briefly on the difference between smaller companies and larger companies I have worked at startup before this I was at a company that then got acquired by Microsoft before that so I've been through a few different places so I'll touch on the differences I've seen and across the different places and then at the end I'll touch very briefly on sort of the interview process so I there is a similarity across the interview process at least in the b2c role so the b2b roles tend to be a little bit different but the beat of scene roles there you'll find some commonality across it so I've interviewed at a ton of places myself and so through the process I've definitely noticed a pattern so I'll touch on that um okay so yeah so that was that so the background I'll touch on my background I'll touch on what a p.m. does here I'll touch on three different things sort of setting up a roadmap for your project how the execution process works and then how do you want to do the analytics and iterate through your project and then I'll touch on getting a PM so are the skills companies look for what does what is Google's grading rubric some of the other places I've worked at in the past and then sort of what questions at least Google tends to ask and other places I've seen ask for each of those roles so that's me on the left this is a little bit about me I went to Duke and I don't know if any anybody here is from the East Coast but went to school in the East Coast that's in the 2010 finals when we won then CWA championships was in Indianapolis that's pretty awesome right after Duke I worked on PowerPoint an office I was one the early days of office 365 and so this was PowerPoint online or online web editor you guys if you use PowerPoint either on the client or the web you might have seen those alignment guides that come up when you're moving shapes around I put that there in fact me and my team have a patent on some of the stuff that on the logic behind how that works because when you have 200 shapes on a slide you need to figure out which ones matter and which ones don't went to business school at Harvard after that that's where I met my wife she's actually where she's sitting back there working not listening hello we have a cat cats name is mochi after that I worked at Yammer for those who have you for those of you who aren't familiar it's basically an online work collaboration tools think slack or Facebook for work now part got acquired by Microsoft not part of office work that earnest which was a student loan refinance personal loan startup in the FinTech space probably the biggest after so Phi they just got acquired by a servicer called navient that had split off from Sallie Mae if you guys are aware of that space been at Google since last year that's sort of the Hat you get the first day you join and then called nooglers they use new Googlers and then I've been on the payments team on Android pay specifically in consumer payments if you guys saw the rebrand Monday that we announced will be rebranding to Google pay and merging a bunch of things across the company and so that's kind of what I've been working on for the past year alright so let's get started so what is a p.m. actually do this is a common thing I'm sure many of you have seen I kind of hate this diagram because it's it's kind of like saying where is it if somebody asked where the Statue of Liberty was he wouldn't say it's between Manhattan the city and Governors Island Mike that's a very not helpful explanation and so but you've probably seen this a lot so I wanted to put this out there the way I think about it is that a PM owns the what and the why the specifically what problems are worth solving and what features are products should we be building to solve those problems engineering and design often ends up owning the how the boundaries are soft so you'll often you'll work across the two but generally that's at a high level at a principal level what the PM role does in a software company but that's upgrade at a high level but which PM owns what and so depending on where you are in the high in the level of the company your scope changes and the breadth of the problems are tackling changes and so this is true at large companies this is true at small companies it doesn't matter at large companies it might be the VP but at small companies it might be the founder I'll just talk through an example of how this works out for example at my last company Ernest before I moved over to Google so the founder there comes up with sort of thinks about what problem do we want to solve at a broader level what market are we trying to attack and so there the thought was student loan process is broken let's reflect me hire a team let me get some funds raise money to go solve it and a company like Google it might be the head of Google photos saying the VP saying all right photo sharing and storage is broken let me get a hundred engineers to go fix it right so at that person they're basically putting a pillar in the sand and sort of nailing the broad market we're tackling then the level before that the head of product is going to store going to say to themselves all right I've been told we are gonna tackle student loan student loan application there's a student loan process how should we do it all right well build an online application or a phone app or something else to go address and tackle this problem then below that you might you'll have the individual PM senior PM junior p.m. they'll think through all right I'm working on this online application let's build a feature to make sure people can refer their friends to this online application because we think our acquisition cost can go down that way and then a junior PM might say I'm working on this referral feature let's make sure inviting friends is easy so you can upload contacts you can do things like that and so that's how at every stage you're owning the house it's the scope that sort of changes as the level changes I'm gonna focus on this one obviously that's sort of the target of this presentation the earlier roles so we'll talk through some of the how day-to-day work ends up working in that level by the way if you have questions where something's not clear just stop me in the middle I'm happy to answer them and then we can just also do around a questions at the end so that should be alright so as a p.m. your job is shipped great feature shipped great products shipped great features and so the way I look at it this is broadly every feature just goes through the structure like there's some pre building phase there's some execution phase and then there's some phase where you launch and iterate any one feature is pretty neat along this sign the reason why if you look at a PMS calendar it's all over the map and I'll show you what mine looks like it's all over the map it's just because you'll have a multitude of projects at every stage at any point in time but basically you have these three stages and there are key deliverables on these stages so the stuff in white you're basically have a road map initially the the goal is to figure out what you're building and how you're prioritizing then in the execution phase you're gonna have a PRD and then you're gonna work with engineers and and designers to design it the exact term that your company will be different some companies have it FAQ dot some companies have a PR D some commune have something else generally conceptually the same thing then you're gonna launch I'll talk through some of what goes into actually launching and then you're gonna analyze the results of the launch and sort of iterate based on that all right so pre building let's talk about what the output of this phase is the output of this phase is a road map that's the most important thing that you as a PM should control your job as a PM you're getting paid to prioritize what features to build so your job should always be to have a clear list of here's the prioritized list of things I want to do small companies and big companies do this slightly differently big companies will prioritize individual teams or prioritize things and this will all roll up to an okay are I'll show you what that looks like at Google it's just a way of saying measurable results that are tracked across the companies startups will have a backlog that they use in backlog grooming and agile if they're doing agile so that's the output the question is how do you get there so there's a lot of inputs that go in but you're you're basically trying to figure out what features to build and how do you prioritize them so there's some vision and strategy coming in from the top from the people above you that I just mentioned and then within that you'll think through what what does the market look like what are my competitors doing the cost you'll do some customer and user research I call this the soft stuff but you're looking for qualitative data that will help guide you and bring out new feature ideas depending on what stage you are of the company this can be easier or harder I actually find this a lot easier to do if the companies earlier you get locked in into a machine when the company's harder doing it yourself so but it's it's important to get out there and do this in fact if you're in a product that is pre product market fit it's actually one of the best times in your product you can just sit down with every single user and understand what are their pinpoints where they're doing because once you are at thousands of users then everybody is just a number on this on a dashboard and so it's actually a pretty nice place to be so but there's a customer user research aspect there's the analytics hard data as you start to scale this becomes easier and easier to gather you'll head get cross-functional ass especially if you work in a larger place and they'll be BD marketing sales they'll all have ideas so you want to pull them all in and then your understanding so part of the reason you get hired is for a little bit for your gut sense and so you want to make sure that once you're in the industry you'll have a sense of pain points and things that work and things that don't work the important thing there is to validate it with the data and the research so you know you don't want you want to use that as a guide but then you always want to validate it so all of that will go in so here's a roadmap this is what my roadmap would look like at earnest I put it in a google tricks it format really doesn't matter these are real projects we did the ideas basically came from all of the sources that I just mentioned so if you look at this list there was the idea of launching a new student loan project so we did refinance the idea was we would launch a product for new student loans at this point I was doing copy experiments on an application page to improve conversion I was starting with a basic referral program and then there was a request to build a new blogging platform for our content team so that they could improve our SEO so this was an actual road map I had at some point one thing you'll notice is like I try to keep big bets and small quick wins at any point in time and I try to categorize things in those ways just found that it's nice to make sure you always have a mix of those on your road map at any point if you only go for shooting the moon you miss out on a lot of little things that you should be improving and quick wins but if you only go for quick wins you'll get into a local maxima and it's really hard to shoot for the big projects you also don't want to go for two quarters without any wins and so the quick wins sort of help balance out if you're taking big risks and nothing ends up panning out you're see you still made a lot of headway by accumulating a bunch of quick win the potential impact this is backed by user research and analytics so you would run data you run user research and you'll try to figure out all right like I think and used to a long project that can open us up to entirely new market so huge impact copy experiments medium actually copy tends to have be pretty strong in terms of one of the quick wins you can have and so medium impact potentially across all of them referrals similar thing here's you want to measure your cost here's where a basic understanding of the technical infrastructure of higher product works helps a lot like for it to fit you should be able to fill out this column at a high level without having to run to engineering for every single row ideally you you always want to validate it with engineering and and if you're not sure then you want to have a conversation with them but this allows you to when you have when this lift gets really wrong long it allows you to scrape out a bunch of things really quickly because you know instantly oh yeah the system can't support this today this will be a lot of work you combine the impact in the cost with two your ROI and come up with a priority and then you basically just draw the line somewhere right so you figure out what you can do this quarter or this period and you're like all right that's my roadmap how does this really differ between small community and become these small companies you basically just pop the next thing off your roadmap once you have engineers that's and then you work agile typically every company differs but that's essentially how it works and so where does the urgency come from the urgency comes from because well at small companies everything's always urgent like you have aggressive growth targets your China meet you have launch date for dates for big projects that investors want to see your board wants to see so you know they'll be plenty of urgency and so you're just popping stuff off the top of your queue the moment you can and putting them in the hopper um big company like Google a lot of becoming students differently this how Google does it we have okay ours who cares are basically measurable results they stand for objective key results I think the important thing is they're measurable and so this scale goes top to bottom so this is last year genuinely how payments okrs would roll up into the company so at the top level if you looked at our company okay our obviously got rid of the numbers but if you looked at our company okay our I would say something like alright the payments team should do yr and Google and payments wants to do y dot million dollars or billion dollars of gross volume of transactions we want to launch in-store payments in X number of markets the product area which would be the payments actual payments team so this is owned by the CEO of Google the product area owned by the the individual team would then own all right so we're actually gonna go and launch this product in X Y Z markets we're gonna launch online in what in Y websites we're gonna ensure that maybe some percentage of people aren't ready to paste it which is what we called it would just you have a card in file you're ready to go the idea is all of this should help this metric and then individually as a p.m. you would own features that roll-up into that and so you would go through your backlog figure out what you wanted to do and what would contribute the most and then you put it in your okay our list and make sure it just matched up tied to an okay are above you so it might be things like improving conversion of card adding are launching campaigns with stores and merchants so things like that with the goal of improving increasing transactions or gross value or whatever the top lines metric list I think all of that should hopefully I've been straightforward yes all right so execution you have your roadmap you know what features you want to build now you're writing your requirements TOC this is basically where you write down what is the feature trying to achieve and really the details of it it should be a collaborative process honestly between with you and engineering and design like you what you don't want to do is write it PRD and throw it over the fence that rarely ends up working out so you work with your engineers to understand what are the questions they really need answered you want to make sure you lay those out these are the sections of the purities I write these days it changes depending on the company different teams matter here you'll see seconds for risks and legal we're in security we're in a we're in payments those things matter so we'll have things like that high level use cases and detailed design this right here that's a meet of it that's where all of the the key sections go again this should feel like a team the important thing to remember is you your designer and your engineer you're on the same team if it feels like you're not if it feels like you're pulling teeth and you guys want to go different ways you need to step back and figure out how to align the team there are two obviously two reasons teams can not be aligned you may not be aligned on the goal or you may not be aligned on how to get there and if you need to figure out which one of those two thinks it is because if if you're not aligned on the goal then trying to align on how you to get there is a is a bad idea it just it won't work so but you start make sure you figure out if there's a mismatch where it is and then you align and then you write the PRD once you have your PR D done often gets broken down into tasks if you are in some place that's agile they'll go in some sort of backlog and my current team at Google is pretty is not so great about this we just go with quarterly okay arts but I've worked I know the teams at Google that are and I start up we used to follow agile pretty closely typically you'll have if you have a TPM or an NG they'll they might help aggressively with this in that case I just recommend get it staying involved in the stand up so you know how well how much progress you're making as things go on and you can sense when there's going to be issues earlier on but yeah we just track what's defined what's in progress what's done we've actually use JIRA but I didn't have access to joy right now so I just made up a screen giant Trello okay so you've done that you've executed you built broken down everything to tasks you worked with your UX you put it all together and that's getting close time to launch so how do you do that so first thing obviously you're gonna run across is bugs like what are what are the bugs were a lot talking this is my day right now because we're working through this big rebranding project we're pulling everything together under Google pay so what I'm working together through to figure out all right like every day what are the bugs triage um make sure you figure out what's launch blocking that's the most important thing at this point you're looking to classify bugs as launch blocking or not aside from it you'll prioritize a bunch of things but ultimately that's the decision that matters and then are all the cross-functional teams ready so for us this is legal risk customer service that sort of thing might be marketing beauty other things in your in your company and we'll do something like this so we'll have we actually use a tracker called Ariane or launch Cal goes by a couple of names within the company but basically you can pad which teams you think are involved in that project it should be involved and they'll have to check it and then other teams just get Marquez FYI and so and the feature doesn't launch until everyone's check it the last person to check it used tends to be the VP so the VP our VP won't look at it until every other box is checked and then he'll put in a final check and we'll start rolling out the feature all right so you launch it and then now it's time to measure the results so if you watch it as an X you either launch it as an experiment or you launch it as a feature sometimes not everything can be an experiment but let's say you launch it as an experiment so your step is to your next job is to analyze the results of an experiment and there you can do a whole course on this but I'll just give like a quick two minute on tariffs like as if from a philosophy level or philosophical level what do you look at at the high level so generally some stuffs gonna go down other stuffs gonna go up and you're gonna have to figure out what to do it's AI not a ton of experiments where everything is positive there's all usually trade-offs in a product make something bigger something else gets de-emphasized so you need to figure out what to do so here's an example of an actual feature we launched at Yammer so for those of you are that who are not familiar with Yammer think of it as a slack or a Facebook for a worker and this was my job two jobs ago basically an enterprise communication tool so we had groups in Yammer kind of like kind of like slot channels or Facebook groups I guess and we launched a feature that let users search for a message within the group and this is effectively what happened I changed up the numbers a little bit but generally this is what happens so these are the metrics we looked at that were on my dashboard we had one day retention which is users coming but how many users come back the following day that seemed to go up the number in brackets is p-value the lower the better it's a measure of how statistically strong your results are what's a chance it it's actually by chance you generally want that number low three person 0.03 just means that three percent chance that this result this positive result is chance you run this experiment a hundred times they're gonna get it one point five percent lift by chance even if the two things were equal so that's why you want this number to be lower but and that sort of the lift the Delta between the test and the control groups the two groups you launched it so anyway so one day retention went up 1.5 percent is actually a fairly strong boost for mature product and retention so the one day retention went up a lot days engaged which was another metric we measured sort of tied to that it was how many days did you engage in a week so that went up slightly the number of likes that we got for posts went down actually substantially messages that people wrote went up but it went but the p-value was slightly large so not hard to read that into that too much and then posting binary number of users who posted and then thread stars the difference between messages and thirst starters as thread starters is just the initial post messages includes replies that was also flat when the hot p-value is flat we just attribute it to chance and so we say it's or the p-value is large we just attributed the chance that we say this number doesn't matter it's just it's it's flat so question so how would does anybody want to venture on how they would interpret this like would you ship this would you not ship this any ideas yeah yeah why yeah it's a good answer oh so the N here is a number of people in the experiment so that's a sample size so so the equation to power calculate p-value is its complex Uub it's a statistical curve so but it's basically the number is the percent that that result is purely by chance so for example this means that if I run this experiment a hundred times with making no difference between the test and control group twelve out of the hundred times I will I will see a lift of point four percent right just out of pure statistical chance so you want that to be lower we've looked at we typically consider p-values of 0.05 and less so 5% or less good p-values of 10% or less okay 15 or less all right fine you're stretching it though lower than that we would just said all right it's purely chance if you higher sample size you can get even closer so Facebook I'm a su I'm guessing they're gonna have go shoot for really low P values retention is good yeah yeah so so that's alright answer so the comment was the retention is good so let me talk about how to look at this generally so the wrong answers more things are green than red so we should chip it you can always add more metrics to your dashboard and you'll end up with more green or red or more red or green that's not helpful at all the correct answer is trying to figure out what the global metric is so there are global metrics and there are local metrics global metrics are the metrics that most closely represents the top-level goal of the company and so well I'll give I'll Tucker cheat sheet really but like what happened to a global to your global metric and then local metrics is basically do they support what happened to the global metric if not then you need to think through what is your hypothesis for why that might have changed so a quick cheat sheet on what the global metric is companies tend to fall in two buckets they're either or typically I'm sure there are exceptions but there is typically transactional or engagement driven so if your transactional you're trying to get people through a funnel that basically means so for example at earnest we gave out loans so loan applications or the number of loans we gave out that was transactional Amazon it's gonna be transactional it's how many people make it through a funnel sure searching on Amazon is great and so you might measure some engagement metrics as local metrics but if you're not moving orders that's a problem like something that gets searches up but moves orders down that's probably not a good thing or you could be engagement driven where you want to make sure people come back to the product or spending time on it Facebook's engagement driven Yammer in this example was engagement driven and so if you're an engagement driven company in this case you're gonna measure engagement and so the top thing you're gonna look at these things like time spent on site or number of days are coming back or things like and so given that retention and engagement went up that's your symbol for why you're gonna ship this likes went down you should think to why that could be coincidence there's always a little bit of chance I pay p-values still not zero but it could also happen our hypothesis here was well we let you search so you're doing less scrolling they're doing less scrolling you might come across less random post that you might like so likes went down so we then said all right let's validate that what happened to actual searches and so then we ran a custom query because searches wasn't in our dashboard and we said all right yeah it looks like searches did go down it sorry it looks like scrolling did go down a little bit sorry scrolling what's it in our dashboard so we said yeah scrolling went down a little bit yeah that validates why likes may have gone down and so that was our hypothesis but we said hey seems like people are getting to what they want quicker and they're posting more engaging more good thing if I were right now I'm a I'm on Android pay we're gonna be Google pay and so a lot of fun metrics have been changed but like last year one of our key metrics was taps how many times do you tap at a store to make a purchase for in-store purchases so that's a transactional product and so that would be your North Star alright so yeah this is I started my presentation with this a single project my fall just nice schedule but your schedule doesn't right like this is what my schedule looks like ends up starting at like 8 a.m. 9 a.m. sometimes goes down to the meeting with my manager at 9 p.m. it's a Thursday night meetings with Singapore so a little bit all over the map at the time at least when I looked at this was a week in November I try to figure out how I spend my time roughly a day all overall was spent on figuring out the product strategy and planning a day on partner and user research and then meeting with partners time with engineering time of design metrics and analysis and then serve general coordination general coordination saab viously hired in a bigger company you're sort of the glue and so there's a lot more of that so since you have products and every say that's kind of what it ends up looking like all right so how do you interview let's talk about that so this is what this is what Google will grade you on and and Ernest - we actually had our lead p.m. was from Google so Google and Ernest we're basically them we had just taken Google's grading rubric and I put sort of the list on what we looked at well we were at Yammer Google doesn't really break it into hard skills and soft skills that's just how I look at it so I sort of put in my lens on there but on the hard side skill side we look at analytical we look at technical abilities we look at product and strategic insight and then communication creativity and culture fit for soft skills culture fit is so you can probably figure out what communication and creativity our culture fit is basically does the interviewer get along with you effectively and do you comment for Google it's also a heavy dose of does this person feel like a team player I remember when I started one of the folks I worked with at Google told me you're not gonna get ahead at Google by working really hard because there are plenty of people who work here really hard and you're gonna burn yourself out you're not gonna get ahead at Google by being really smart because we hire really smart people and I'm sure there are plenty of people smarter than you you're gonna get ahead here by trying to be a team player and so I that is I think one of the key things I took away from my first few days and it's one of the things that they look for or we look for in on the culture side duh and then I've sort of mapped those to what Yammer looks at you phrased in a lot of different ways but a lot of the same elements are there so I guess my point is there's there's a level of consistency that you'll notice across the different companies at least the consumer facing ones b2c companies all right so yeah yeah I'll come to that I have examples so here are with some questions that I would put in the hard bucket for each of these so for a product to come very common set of questions and the nature of the company actually affects which one you're more likely to be asked so if you know people at the company I would I would try to figure that out this actually does this laser pointer work okay so the the the two types of sort of design x4y which is like design an alarm clock for a blind person or some things I call it a blue sky question or how would you improve Product X and that's I sort of call that the improvement question there you're given an existing product or you're told pick your favorite app on your phone how would you improve it like how do you improve an existing product I'll sort of come through what we what how to answer those and what we're looking at I think there's a structured approach that generally makes sense everybody has their own style I'll give you mine some analytically focused ones metrics focused talking about some of the things I just talked about around metrics we ran an experiment this happened what do you do math focused we still ask I forget whether Google still ask some of those I forget whether I was asked this but we certainly ask to use a lot of earnest how many people fly out of SFO a day how many bottles of shampoo are sold every year basic estimation questions oh yeah I was at yeah I think they're still there at Google technical questions two types typically high level technical insights so for example what factors would you consider when deciding which videos to show and how to rank them on the YouTube column in the related videos tab or in section and then sequel basics like given the following tables how would you produce this other output depending on the company there may be more or less of this I think it's good to brush up on some basic sequel queries you can do that offline you can do that I know Stanford has a great online class that's where I first picked it up before I started getting into interviews where this was a thing and then it was useful in all my jobs about the soft skills all right you're not gonna get asked a question about this I mean they'll be general behavioral questions where they'll judge some of this but there won't be a specific question for this they'll try to get at this as you answer these other things so I'll just talk through these for really quickly the two product ones and the two analytical once and that's enough so it'd be the end of the presentation just because those tend to be the most common and structured from an approach standpoint so alright so the blue sky serve question like you're given some thing to to decide the best piece of advice I can give you and this is surf advice I got when I was interviewing for my first PM job if you're asked regardless of what you're asked to design think about what is the first where are the first five questions you'll ask the interviewer when you're told to design something so for example if I tell you to design a toothbrush I haven't really told you anything like who is the person who's using this toothbrush or they're using it traveling are they using it at home are they is it is a person disabled is a person not disabled like how much are they willing to spend like I haven't really told you anything about the factors that are like are you designing our time crunch like what's going on here so you need to figure out like what are the think about what are five genetic questions that you can ask right away and that's useful because when you're getting a question there might be a moment where you're where you have a I guess a deer in the headlights moments where you're not sure what to do it's interview it's high-stress if you're prepared you'll you'll get the ball rolling quickly the most important thing generally though is to fix to start with the user as a p.m. you're the voice of the user you probably heard that so start with the user figure out who is the user what are their needs and if you're not given a user you should list some options and then prioritize and quickly pick one I'm so I now give you an example of that in a second I think through ideas and features try to have at least four or five on things you could do and try to have at least a couple that are out there this is where you get your creativity points so you want to go for some some things that seem obvious and things you could improve but some things that are creative and get out there and then generally I think this is really smart like this is where your communication skills comes in you should always be structuring so anytime you give an answer no laundry list always structure whenever you can that's super important so if you're prioritizing you should say I'm prioritizing based on these two factors I think this will drive usage and this would and this will be easier to implement or something you should have some logic or structure to answering each of these sections so this is a question I got when I was interning at Microsoft back in 2009 design a music system for a car so start with clarifying questions who is a user where do they live what type of car do they have so for example you may have and let's say you're not told you answer you get those answers and then you're not really told who the user is so some options there could be a student in college it could be a working dad in the house you could be somebody driving an SUV pick-list some options pick one tell them how you're picking one um you think the markets bigger for this or you think their current car these are the people tend to own cars or drive them more or whatever have some logic pick one then feature ideas so what features should this music system for a car have where did where does it get its music from so is it from your phone is it local storage is sync with home Wi-Fi how do you control it things like that and then for each of those questions you should prioritize and then pick your aunt's option right so for example if you want to decide where does it get music from you may say all right they could get its music from your phone your local storage or sync with your house where the pros and cons of each and then based on that it'll pick some options and go forward right so and then at the end you'll have an answer of a thing you've designed basically how would you improve X so this is sort of what I call the improvement question or like pick your favorite app how would you prove it very similar to the previous one with some added complexity around it's a product that's already out there so you need to know what what's the current vision or goal of the product and then what is the top level metric based on that so once you know what the vision or goal is what is that thing that they should be trying to improve anyways and then who are the users where someone met needs and then how would you structure an experiment to improve those like when you're doing improvement you can experiment you can't it's a little harder to do that when you're designing something new entirely a quick tip here before you go into the interview there are lots of interviews where you'll get asked to pick a favorite product and improve it have ID have thought through at least three or four products ahead of time three or four products are phone apps three or four products that are web apps three or four products that are b2b apps and then sort of think to like and come up with three or four ideas for improve to improve them the important thing is if you've thought through apps and you have ideas at the interview you still need to go through the steps you shouldn't say oh yeah I love this product here are four ideas that that laundry list will not work you need to start with and say all right this is the user and this is what they need so I'll give it an example let's say I'm told pick a favorite app improve it might say all right look I've recently been using uber eats I think it's a great app it's designed to help people get food they want quickly I'm smirking because our friend back there who works on new breed's I so I might start with a user and say all right so this app is designed to help you get food you want quickly who are some people who might use it well there might be single working men and women so they get home for dinner they don't want to cook they just order it might be office workers during lunch might have a few of those pick one so let's say we pick single working men and women they care about time they care about convenience all right so what are some improvement ideas well you want to help them get their food quicker you want to make it more convenient so maybe there's a pre-selected menu of top nearby items that you can check out with one click right so it's really quick to check out quick to make and deliver because they care about time or maybe there's a one-click order option for the items that are already there then you drank and prioritize this and you pick an idea one of the things you'll notice here is again like I might come up with ideas that I've thought of at the end of the day just having done this exercise in the past but it's important to go through this you can don't just jump into ideas the sort of metrics piece on this often a follow-up on the improve X question which is all right great you thought of a great idea a one-click order of popular items on new breed's how would you test it how would you test it that's a good idea well you can obviously build a full feature one thing to think of is can you do an MVP test a Minimum Viable Product is so is there a piece you can pull out and test and see all right check your hypothesis and see if you are down the right path so for example the goal here is convenience so maybe and and speed so maybe you have a one click repeat last order button and that way you're not dealing with creating quick order menus and things like that you're just putting a single button that lets you repeat the last order that would be an example of an MVP experiment the goal is at the end of the day think about what top-line metric is ubereats is a transactional product they probably care about orders so that's probably going to be your top line metric and I think that who would you test on so would you test on users everywhere are you gonna to the extent you can you want to test on your entire population base if it's a product that's local to certain markets then you may want to test a location by location but otherwise you try to test on your full population base because you'll get to statistical significance quicker you'll have more people and tests and more people in control here's an example of an analytics math based question how many planes fly out of SFO every month I was asked this in earnest or when I was joining our nest the most important thing is lay out the complete formula first don't do a single calculation until you've laid out the full formula so you might say number of flights is number of days per month tens minutes per day times takeoffs plus run weight each minutes times number of runways at the SFO Airport and then you might make some guess for how many runways they are at the sfa Airport but lay out the formula first then do the math then gut check and say all right does that look right or does that not look right and then you stress test it and I'm surprised by how many how few candidates and interviews actually do this this is one of the reasons why laying out the formula first is super useful like you look at it and you end up with a number that say all right a hundred flights fire defesa for every month well you're like well that can't be right there's clearly something wrong so go back to this and you can say all right like which assumption did I probably screw up and then you can stress test that and you can come up with a range so you can say all right let's let's assume the runways it's not they're not two but it's four to six what does that mean that means I have a lower bound and an upper bound at 4 at 6 so the Laura got an upper bound of what I end up calculating changes and so laying out the formula helps a lot otherwise you at some point you'll get lost in the calculations and then if your stress test it won't work ideally you stress test yourself before your interviewer stress test the answer for you some resources I've found helpful going through the interview process most of these are pretty common knowledge which is really awesome because when I was interviewing for the first time none of these books were there cracking the PM interview by Gail McDowell this book I found / and by the way I have nothing I have no vested interest in any of these resources so this is things I personally found helpful this book I found very useful early on if you want to understand just what the PM role is and I found this good for the technical part so if you want to brush up on some basic technical algorithms and things like that that you four guide it's good for that decode and conquer by Lewis Lin this is good for practice questions like to just go through questions and questions in question I found Lewis's learn better for basic like Q&A stuff than Gail's book but I found Gail's book better for like the technical stuff and an overview of the PM roll and then do you definitely do a Korra medium google search for prep on the specific company not just on interview questions but see if you can find out what the head of product has spoken about in interviews and other areas you'll understand how they think through their product at a high level and what sort of metrics they look for and what their vision is and their goal is well yeah that's basically it the one of the nice things about the interview process for PMS is I think it's actually a very preferable role like if you if you do this and spend your time you can really prepare and out prepare and then beyond that it's it's a numbers game the you'll net you'll never be a perfect fit for every company there's always going to be 40% luck and then review process regardless of how well you've prepared the interviewer doesn't like you that interview was having a bad day you stumble you blank out like a lot of stuff happens so beyond that once you've prepared it's just a numbers game so that's it questions yeah that's a great question so it depends on so it depends on what you're trying to do so typically I found that changing industry and function both is hard so if you're already in tech it's easier to make it over to p.m. if assuming you're not a p.m. already if you're not in tech then it's often easier to move to a similar function in tech that similar to what you're doing and then move over to the role couple of options if you're if you go to a startup there's typically enough for everyone to do particularly good regardless of how young the company is so I've found a lot of lateral transfers in startups as well just because you end up doing if you show an interest in that sort of thing you can do a lot of projects and then and then move over or interview at a place and say you did all of these other things at that company at larger companies at Google we also do lateral transfers so there it comes down to like one of my one of my teammates who worked on the customer facing side what we call the partnerships team helps partners implement Google technical solutions just moved over and the way he did it was he was basically working me with me for a while he started doing more and more things at p.m. would typically do which was very useful for me particularly at times where I had a lot on my plate but he would work closely with partners he would write he would put up the dashboards to analyze metrics start doing things like that and then and then it was a pretty clear process from there to then interviewing for the role and then transferring and so it's there's no one of the odd things about p.m. is it's a fairly flat organization typically so if you're a company like if you are a company with 100 engineers so typically have 8 or 10 p.m. so when while there are like three levels and PM sorry on engineering site they only be one or two on the PM side and so what that means is hierarchy doesn't matter so much the scope of ownership matters so regardless of which company you joint try to find a company where you can do some of that job and then you can move over with whether within that company or outside that's typically the post I found at the easiest a Yammer for example we had 10 PMS while I was there two of them came over from marketing one of them interned in marketing but then joined full-time as a PM two of them came over from our own BD team one of the three of them were engineers and then two of them I think were full-time or wore PM's before that so don't think that adds up to 10 but roughly it was a pretty broad sex cross-section of folks yeah oh gosh so so happiness so let's talk of it I saw I actually I liked shipping stuff like the the best thing about it p.m. is shipping stuff like seeing your products in the wild and naturally shipping things is pretty awesome it's by like being able to look at something on a phone or something at a laptop or something I'm on someone's device and say yeah I made that that's that's really cool and that's really exciting the the thing that's the thing that's great not so great let's think so the process can get tiring something so and this is this is genuinely true regardless of whether the company is small or big different companies have different different pros and cons at a smaller company you'll have bigger scope of ownership obviously but you're typically perennially in a state of urgency and everything's on fire at a larger company that happens less but you have more layers to work through managing above and managing across so different things at different companies yes yeah Oh funny I was going trying to get out of p.m. and then I got to business school I was going to go into consulting I worked at Bain for a summer and a strategy consulting and then I decided I liked product way more and so I basically came back business school is one of those weird things where I think the soft it gives you a couple of things I think the soft skills it gives you is way more valuable than many of the hard skills it's kind of like I mean all of you guys went probably went to undergrad how much of your undergrad do you use today maybe a little bit if you did CS and you're now in tech but otherwise probably not that much was it useful did you learn a lot generically yeah probably so Business School I thought was useful in that same sort of way I think it was good do you look at the world and products and things holistically it is somewhat of a funnel for to get into some PM jobs at large companies people do tend to evaluate you as okay you're graduating from Business School more than okay what is it that you are just doing right now but depending on your motives I would say if you just to get into PM roll I think it's useful only to just learn and figure out like how things work I told you what my motivation was but probably different for different people yeah good question so um no I I wasn't so undergrad was electrical engineering not computer science which means I did a bunch of stuff with wires and circuit boards so a little bit of programming then did a little bit on the side afterwards too which where I picked up a little bit more but undergrad was not see yes if that's the question I think as a p.m. you need to know and if any of you are thinking about doing coding I think more useful than like hackathons and things like that what's really useful is understanding the basics like it's more useful to understand the basics of how things work so that you can piece together an architecture in your head systematically than it is to be able to hack together something because as a p.m. you're mostly doing resource allocation and thinking at a high level what makes sense you're never actually gonna be well you may be but typically you're not going to be actually writing the code so sequel you'll do it's useful to know how to do basic sequel IMR we had a very strong data science team so I ended up not doing it so much at Google there there's a little data science team that works on the business side at least on my team but otherwise it's the PM's and engineers coding sorry writing the sequel scripts themselves so it's very useful to be able to get your data yourself yeah good question so let me put it to you this way I it is true that at Google it's one of the companies that looks for more of a technical background and perhaps other companies exactly how much will end up depending on the engineer who's interviewing you that day but we it certainly looks for a little bit more of a technical background or at least a technical understanding like you need to be able to answer in a structured fashion but the whatever question comes up whether it's like how do you rank videos on YouTube like how would you think through it like what or you're writing an algorithm for or solving Sudoku like how would you analyze you're not you don't actually end up writing the code though you could write pseudocode but it things like all right now go to a row by row out in lights for this and so how do you build that out does that do you have a way a structured way of answering that that makes sense so yes I would agree that from a at a high level yes we Google probably looks for slightly more technical respecting more than say for example I've been through interviews that LinkedIn and Microsoft and Facebook and some of the other big companies a little bit more so program managers the role at Google is much more amorphous than what product managers do I think product managers is somewhat standardized at Google even though still a huge amount of variability within teams but more so than program managers program managers in some teams so we have a few and our teams they help mostly with project project management so my job is to figure out what goes on the train their job is to make sure each train ships leaves the station on time and so and I put stuff on the train and so depending on the team that could differ I've seen folks program editors who also own actual growth metrics and things like that and so it really depends on the team there's a little bit more ambiguity and if you go to our job section and you filter for program managers you'll actually see that the different the definitions or the descriptions will be varied yeah yeah so I actually haven't worked with any technical account managers on my side my understanding is that they work with with the partner too so they work with a partner to help integrate the technical solution so in fact in many cases they're more technical than product managers their job is to is to basically if you have a solution you have api's you have big partners on the other end they're trying to work against it you help them interface and you understand understand they're actually a fairly a very strong resource but and they tend to be even more technical to be honest yeah yeah that's a good question so BtoB the biggest difference is you have a big sales team or you have a sales team and the sales team is close to the user and your users come in lumps and so individual users end up mattering way more and so thing you'll often make product decisions because there's a 10 million dollar deal on the line like that's something that happens in a b2b context that doesn't happen in a b2c context so your sales team ends up being a huge impact on your roadmap and being ends up being a huge influencer of your roadmap that's probably the biggest difference your your users are your voice of the users sometimes you're working through the sales team you want to get on the customer calls yourself and understand things yourself always so you can understand what's common versus what is what is being proposed just because of this one deal but the sales team I think is probably the biggest voice and then and then on the other side you're translating that to engineering and so one of the things that actually does happen is you end up being an even stronger voice of the customer or of the business to engineers because in b2b roles unless you're building a dev tool in b2b roles you'll often find that there are less opinions of what to build everybody can look at like a loan application flow and say oh yeah that doesn't look good but it's a little harder if you're building a b2b tool yeah oh yeah so yeah that's a good question so yeah so you'd have a so like for example at earnest our business metric at the end of the day was was the dollars of loans that we gave out that then translated that the novice and then and the average interest rate on the loans that we gave out it's or the Delta between the our cost of financing and the loans we gave out and so but then effectively that would then the you end up driving that is through some change on the user side that is the metric you end up owning as a p.m. and so that's why I was focusing on the user benefit there'll be a marketing team they'll focus on cost of acquisition and things like that and so but a lot of the business metrics will then end up coming down to you as a PM in terms of a user behavior or metric that you're trying to drive cool let's take one more yeah yeah so qualitative three ways two ways to get it you either run a survey where you're not in front of the user or you sit down with a user and you talk them through the product or mocks or whatever the if you do a survey you can typically get quicker responses but you're more limited on what you can pass and it's a little harder to get a feel for what the person is doing when you're doing an interview in person you can ask a question if they react to a screen or a product or say oh this flow doesn't make sense or or I don't understand what to do here you can follow up immediately and say well how does it make you feel what's not right about it like what would you do like if you had a magic wand and you could wave it how would you have done something else instead you can have follow-up questions those are obviously more expensive not in terms of money but just in terms of your time the rule of thumb I've seen in the past is that you start getting diminishing returns after like five or six in-person interviews and then after that you want to take your learnings back iterate on the mocks or iterate on the design and then go back and put it in front of another set of people and do it again ah you'd have to give me a more concrete example typically so oh so yes that's oh yes so the question was like if you improve if you improve if you do if you create a feature that improves transactions but the rate of fraud goes up what do you do a good question there's that's that's a trade-off where you have to make a call so I'll give you an example let's say you're a PM at LinkedIn you put more ads on the home feed you'll make more money but let's say engagement drops how do you ultimately make that trade-off I asked a friend who worked on the LinkedIn feed and he basically said the head of product said we're willing to take a 5% cut this quarter for ads how much ads can you wake so at some point that basically just becomes a trade-off you have to make a call that comes down to the business at that point and that's certainly not a startup only issue that happens just as much as at big companies in fact it big companies you have more people above you who can have the same thing happen so definitely not a start-up only thing the question was what do you do at a smaller company if your CEO or big client comes to you and says they want a specific feature how do you react to it you have to figure out where like as a p.m. you need to step back and figure out how it lands in your roadmap and then you need to be able to communicate that up this was part of your job is managing up will the roadmap get influenced by the people of a view absolutely it's going to happen there's no question about it but it's your job to make sure that it fits in the broader objective data helps a lot user studies a lot and that's the sort of thing where you take back you put out new roadmap and then you try to do get some data and some user studies to validate it is it only this customer is that all the customers do all our users feel this way right and then you can take that back and do a report and decide whether or not it's worth it see if you can break it down into a small feature that you can quickly build and do run it as an a/b test also helps that's not always possible if the idea is really out there but that's how I would approach it typically

Channel: Product School

Views: 372,748

Keywords: Product, Product Manager, Product Management, Product School, Tech Startups, Data Analytics, Coding for Managers, How to get a job, get a job as a product manager” product manager job description, product manager salary, product manager resume, product manager jobs, what is product management, what is a product manager, product management training, how to become a product manager, cracking the product manager interview, product management jobs

Id: ntzB9pGsD3E

Length: 62min 54sec (3774 seconds)

Published: Wed Feb 21 2018