How to be a Good Machine Learning PM by Google Product Manager

okay so my name my name is Ruben Lozano I'm currently a product manager at Google cloud I'm overseeing the customer experience for all the online experiences before customers actually start using the cloud and I'm currently trying to build a team of product managers and before before coming to Google I was at Amazon and I was managing a team of senior product managers technical in search and I was there for around three years and most of my talk actually is going to be about my experiences actually at Amazon but I will I will I will give you some hints as well from Google as well so this is what the agenda of the day will be here is just what is ml a lot of people from from the audience already are familiar with what MLS I'm not going to take a lot of time here I just want to make sure that at least they vocabulary is understood the next part is I think the rhythm one of the most important ones is well how can you use machine learning like should you should you use machine learning for your problems or not and really to understand when it's a good use of machine learning for your specific problem that you want to solve then I'm going to go into a very high overview about the process of using machine learning to resolve a problem and then I will take a little bit more of time and really really go into details about what is the role of a product manager in machine learning and I think that this is something that through my experiences at at Amazon that was very important because sometimes people didn't really understand where the boundaries were between different different different different roles and it's important for you as if you want to become a product manager how was your role and how can you help the whole team you're always a part of the team as a product manager and unless I just give you some overview and some advice of my experiences when I've been working with product managers I weighed with machine learning and data scientist that may be helpful for some of you as you start your journey in machine learning so first what is machine learning so machine learning is just a subset of artificial intelligence and I mean artificial intelligence is just basically like when a machine is trying to mimic what I what a human is doing and machine learning uses a lot of statistics and algorithms to actually kind of mimic what a human will be doing and and a deep learning is another concept that comes a lot with machine learning and just for you to like to get the general idea just think about the deep learning as maybe multiple layers of machine learning and and that's the only thing that you really need to understand right now we're going to be focusing specifically on machine learning how these statistical concepts can help you solve can help you solve problems so this is the actual definition of machine learning the the study that gives you computers ability to learn without being explicitly program and that's the key like the key and the key to opens how many of the ORS that didn't exist before and that's why machine learning could be super super helpful because in many circumstances in the regular classic programming you need to have very specific rules about what you wanna do and you need to know most of those things before beforehand right and there are many many problems that you can solve with classic programming right you say I don't know like specifically for Amazon if you are a prime customer maybe you want to show some specific features for them if you're not a prime customer you show something different you're in a specific place in the in the world you show customers different features or you change or your customized experience based on these specific rules and these rules can be more and more complicated and you have some data to help you do that you code all the rules and then you have your answers and that's great the interesting part is when you can know when the rules are just too many and the classic example that I will be using a lot here is search I was working at Amazon in search so how do you use rules for search when you I mean you don't have thousands of inputs you may have it unlimited types of inputs of queries and words of things that people can do so how can you actually create experiences that will be adapting to how people are using this you may be able to create some rules but that may not be the a scalable way to do it so the the interesting part with machine learning is that you use machine learning when you actually have when rules are not exactly like you you don't really know the rules but you have a lot of data and you have a lot of answers and this is key because mad machine learning is not magic and something that US product managers you will you will you will see is when you go to to up to a business meeting and you're meeting with people who are not very familiar with machine learning they will they will just say machine learning as a somatic one right like we have a problem we don't know this let's use machine learning to to understand what it is and you're like I mean if you don't understand what it is like machine learning is not suddenly going to give you the the answer and I think that that's the key the key difference you can only use machine learning when you actually know what is the answers are you and I wanna get this may be a little bit confusing but we will go into more details about that the way to think about this is as a product manager always think about what is the problem that you wanna you wanna you wanna solve with machine learning the first thing that you need to know is like do we have data and do we have enough data we will go into details about what is enough data later on but if you don't have enough data machine learning is not your solution right one of the I was talking with some people from from the audience here to say that at some point I got used at Amazon we had so much data so machine learning was pretty pretty much an option in many of the circumstances in my current role at Google what I like we're looking into something completely new that no one has used before and we we don't have data so you like okay for all these experiences like how are we going to start using machine learning if we don't know if we don't have the data about like user behavior we don't really know what people are going to do well what do we want them to do so suddenly you start saying yeah yeah we have a problem but we don't have the data so maybe machine learning may not be the issue and that the solution but in some circumstances it will and after that you start thinking about world based on the data that we have and the problem that we we want to solve what is the right algorithm algorithm to use to potentially predict the answer that you want and this is where data science and scientists will start joining you to be able to figure out what is the right algorithm to be able to create the right model and at the end the right output or prediction in machine learning something that I like to do for at least for me and I hopefully for you will be helpful is how do you how do you we can use statistics as a way to kind of compare how do they how do they how does it fit with with machine learning because some of the concepts are very very similar it's just that sometimes the I would say the vocabulary that people are using it's a little bit different so I'm maybe let's let's tell me our peoples familiar with statistics here basic statistics kind of but in general when you let's say in statistic when you have your your dependent variable Y or Y right the output variable basically the thing that you're trying to predict in statistics call like dependent variable output variable in machine learning is usually called a label or a target but it's kind of the same is what you want to predict is the answer the data that will you make predictions are these independent variables in statistics that will be called features in machine learning when you're basically trying to get all your data in the right place basically reshaping the data to get the most out of it usually is called data transformation in statistics in machine learning this called feature engineering and basically when you're trying to figure out you made a model in statistics and it's very clear that's from all your statistics professor basically they are telling you don't use a lot of valuables even if you're creating a model with like 200 variables try to figure out which ones are not the ones and just select the ones that are important because it will make your model way more solid that is like the subset or variable selection in statistics and in machine learning is feature selection so my main goal here is like you get comfortable with this terms because I will start using them more when you're I'm talking about a feature feature engineering feature selection labels it's very similar to statistics as I said in there at the beginning machine learning is just using statistics to create this to mimic what humans actually do there are two main subsets of machine learning this is very basic supervised learning and unsupervised learning I'm not going to go into a lot of details here the only difference that you need to understand for this talk is that supervised learning when you are when are you looking at the data you have to label that data you actually need to recognize what data what the data means and you need to have for example if I want to in my examples for for Amazon if I want to if I want to predict or to provide a recommendation to a customer I need to have like an order and I need to understand is this organ successful or not successful and I need to label that beforehand and you bet yeah and with unsupervised learning basically you you're using data that you don't really you cannot really label and you're trying to use machine learning to actually understand the simile like the similarities between the different data a good example is when you want to like figure out what are like customer segments or cluster things that are very similar that you may want to say well I don't really know if this similar to what right like I just want to know that two things are similar but you don't really know how many things are this is confusing but it will get better and I'm telling you so the next part that I think that is the most interesting is like when you should should you use machine learning as I started at the beginning machine learning starts with data you need to have data if you don't have data choose an or solution really but if you have data that's great and the cool part about machine learning is that it's a cycle and it could be a virtual cycle if you have a little a lot of data and really good data then you can potentially do really great predictions and if you do really great predictions you will improve the customer experience and if your improve the customer experience hopefully you will get more traffic to whatever product you're building and when you get more traffic you will get even more data so that's one of the I would say pillars of using machine learning again think about data do I have the right data or not I'm based on that a lot a lot a lot of needs to happen so how do you how do you analyze a problem and and figure out if you can use machine learning or not so I usually UI I'm using this framework I use this framework a lot in my work in search to figure out do I have should I use machine learning and the first one is that it's my problem handling very complex logic and I'm really trying to figure out that is complex right if you can solve whatever problem you have with rules with if statements even if there are like 20 if statements you said really like because it's going to be a little bit I mean machine learning is is it is a complex process so really tried to figure out if you have a problem that is that is very very difficult very difficult to solve with rules the classic examples of like complex logic could be if you want to let's say I mean with search is a classic example right like if you have a lot of different keywords I mean you cannot have rules for different keywords and then figure out what are the results but that's the first one the second one is that if your problem will be scaling up really fast you may have a problem that it's pretty complex and you say well that's great I'm going to do a machine learning solution for this problem but it's a small problem like your customers are not going to be too many you're not going to be using it a lot of times is it worth to go through that process or is it going to or are you thinking about something that is going to just simply like I don't know like there's a lot of opportunity if you automate some of these problems because you may be able to like save labor for example if you have for example some people let's think about this example that I'm going to use later but at Amazon with how do you know if for example a fruit that are you sending in Amazon fresh is effective or not if you're starting the pilot with Amazon fresh you like that process is complex right like how do you know if it's good or not but are you going to build a machine learning process when you're the number of orders that you have are like a hundred per day no but if you start thinking that okay this is going to scale and actually I'm going to start not having a hundred orders what I'm going to have a thousand 2,000 you're like yeah maybe maybe it's time right so think about that think about like the the scale of the problem are is it going to be is it going to be worth it the next one is if it requires specialized personalization again this this you will see it in most of the new services that you get like with face wood with Netflix with Google with Amazon if you really want to provide a specific answer to a specific person for something that they did specifically yeah I mean the rules are going to be too complex it won't make sense until you unless you use machine learning but again if you can actually reduce the complexity and say I have five customer segments I have 10 customer segments that's easier and then the other one and I think that that's the most one of the most important is your problem going to be changing across time so if whatever problem you're trying to solve is very how do you say it's not going to be changing from today to next year you may want and it's not very complex you may not wanna use it but for example let's say that you have a problem that it's very complex it may not scale up really fast it requires specialization and you know that it could change from day one to day to the next one and again I'm using search as an example because trends can happen right so if suddenly today beyond releases the new her new album people are going to be searching for that and you want to make sure that you can adapt and you can adapt immediately so if one of your problem is is something that something is going to change suddenly and you want to make sure that you let you catch that train machine learning is a good option because if not your rules are not going to work I'm going to give you a pretty interesting example from one of my first times at Amazon where we actually had a rule for a keyword that it was ring because the way that all machine learning algorithm was back in the in the time it was not great there were like when when people were were using ring the results were not amazing and at some point we're like you know what this is this a very difficult problem to solve right now what we're going to do is like when people actually like right ring we're going to redirect you to our rings to a page for you early where you can see all the rings and we can we can optimize that experience and it's much better than there are simple search results the problem of having that rule is that when the ring doorbell started to like to become popular people were searching ring and we were redirecting them to the jewelry player place and every time that people were trying to put ring we will redirect them you like no you can put those kind of rules in search and the most important part you actually need to have answers in must as I say in machine learning you have all the data and you really need to understand what is a good result right in my case of ring I know I know what a good result will be it will be defined searching for four shirts I want to search shirts if if not I mean if I were searching for dresses I want to see dresses so you need to have like a good a good you wanna know you you need to understand your your end result you don't use machine learning for a problem that let's say right now in Google Cloud I want to understand like what's the most the best the easiest API that or the most the API that people learn faster and I'm like well I I don't know I mean I I don't know how are you going to know how is learning faster what does that mean you have to specify everything but maybe machine learning is not the right answer for something like that so I'm going to go through some simple problems like for you to understand the different problem that machine learning uses so maybe you you can relate them a little bit more the first one is ranking is just basically helping people find the most relevant thing that's a great that's a great use of machine learning so the classic example is a search for Amazon for Google your you don't have a formula is very as we were selling at the beginning does it have candles complex logic he like yeah it does does it scales yes it does scale does it change through time yes it does and so this is this is this is like every time that you want to rank something machine learning is a good example the interesting part here is that I think there's a misconception that a lot of ranking is done through formulas but the problem is that that that doesn't scale right like how do you know like if we in the case for Amazon how do you know that if it's prime like does it have the weight how much should it be if your Prime and the product is prime if the product is going to be how do you say if it has good reviews if it's cheap if it's whatever if you start using all those variables and try to create a perfect formula I mean you're doomed right it's not going to happen recommendations that's another great use of machine learning because you give users the thing that they made more they may be interested in again this is something that is used again a lot of our amazon is using recommendation from netflix is using past experiences and trying to predict what is the specific outcome based on specific variables you can use it in google calendar that is trying to predict like what room you may want to set your meeting based on like previous time that you have set your meeting or where you are so it's using multiple different variables to try to predict an outcome and saying we think that this is what you want we're not completely sure but we think so and this is actually very important for you to understand the machine and we'll go into details later that machine learning is not always perfect so if your solution requires like a hundred percent accuracy maybe machine learning is not the best but if in this case you're recommending something and it's not great it's like the world of an the next one will be classification as and this is where we were discussing a little bit about supervised and unsupervised thinkable classifications still have a supervised learning where you're trying to to use machine learning to to classify different things something that we used a lot of allama zone for machine learning classification was a catalog people I mean there are billions of products at Amazon when different vendors are putting them in the catalog they specifically say what what kind of product is this right this is a dress or what kind of dress but they may be wrong so or maybe they wrong on purpose for all the reasons that I'm not going to discuss here but you can use machine learning to go through all your data and you catalog and say is this maybe let's use the image and trying to figure out if that image actually matches the information that is in the catalog and as we were discussing before that's even though the logic may not be super difficult it's it's the scale is so big right like when you're talking about the millions or billions of products and the even if the problem is not very complex it's a good use of the machine of machine learning another one is regression and I think again this is based on statistics right machine learning is based on statistics so you want to predict the outcome or something in particular in this case it's a lot about forecasting if you want to understand like how how much a product will be sold of course you can have like a specific model that can predict that but what happens when your products keep growing and growing and growing and growing and growing and the behavior of the people as we were discussing on the time could change drastically from one day to the other one how are you going to know that your actual forecasting model are going to adapt to that if you had a forecasting model for for something and yeah maybe you are considering seasonality you're considering out of scope you may be considering different promotion for marketing but if you don't consider like there was a scandal in the media what you you you will be you will be doing as well another one is clustering and this is clustering is something that unsupervised learning will will do very well that is when you want to just cluster things that are similar to each other but you don't really know what they are so in this case a good example to visualize this is in at Google Google search when you are going going into news and you're starting getting all these news that are related to each other so it's trying to figure out like well you found something let's figure out what is close to that what is clustering that specific point in time and the other one that we discussed an alum and anomaly that is trying to figure out what is different from the rest I really like the example of the the fruit freshness because it seems to be very simple but at the same time it could be a big and that's something that machine learning can help you decide is this good it is not good another scale it's very important this one specifically right if you start thinking that a human can actually do it well for strawberries that's great then you have to do it for bananas and then you have to do it for peers and then you have to do it for everything and then like the skill set of people of do of people doing that job Maine may not scale as easy while some of these models are well at the beginning it could be very complicated or complex to create them they can scale so much faster so we talk about like a couple of problems that machine learning can use yeah in classification you're basically just trying to it's it's similar right like classification are just trying to say like is this does this match this does this match this as much this an anomaly is just like is it different from something it's it could be different depending on like some there are some how do you say there are some models that will be better used to do anomaly and others for classification because you know yeah sorry yeah that's that's a main different like the type of model that you will be using assume that all these are fresh fruit that's you do need to tell the model what is fresh and what is not you could only see like something that is different so you could use anomaly to say is it is it a strawberry or not for that one if it's not afraid you have to label it like how do you know if it's fresh or not that's another way of doing it right so in general we talk about the problems that machine learning could could solve the the framework that you could use to see like is my is my problem worth of doing machine learning we'll look at some of the different problem that you can use but let's go back to the data and the first thing that you have to think is like can you use these data right are do you have access to this to this data is it available sometimes you will have a problem where you simple simply you just don't have the data so think about that about having it and other one being accessible when you're working with all these humongous companies that's critical right you may have all the data in the world about building and customer information but like you may not have access to it and your model may not have access to it so yeah you have this great problem but you can access the data so think about it if you can use it and the most important word like should you be using this data right you may have all these data about building information about customers and where they live what have you is is there a good use to use this data and I tell people about like the creepiness factor if you start thinking that you know I don't want this to be in the news it looks kind of weird are we using this data to actually do something good for the customer but people are not going to feel okay that you're using the data maybe don't right so not not because you have the data and it's accessible and available that means that you should be using the data respect privacy and make sure that whatever they are you all through the process of all you different pipelines is actually secure you don't want that they had to ever be compromised so if you don't have a system that is secure to put your data and to create your models I mean be careful right and is it high quality and this is this is very important right I'm talking about like being relevant data you may have a lot of data that is not actually relevant for the problem that you're solving so understand that you may have very if you have fresh data that's great like it's I'm what I'm talking it like data that you can use to actually solve that specific problem you may have a lot of data for I don't know suppose that you're launching a feature for a specific product and that feature was launched three years ago and it didn't work and you can say hey well we have a lot of data how customers use it like three years ago we're going to use it the behaviors could have changed completely so if it's if it's not fresh I will really recommend not to use it the other one that is representative right really think about your problem if you're if you're trying to yeah I mean if you're trying to for example solve this is was a classic example like when Amazon was moving to different marketplaces right if you are in in the US and we're lunching in Mexico yeah we have a lot of different data from the US is it representative of how people in Mexico by things like you have to really think about like can we just reuse this data or know like what are the patterns and the other one that is a mode one of the most important one at is that it should be on bias and there's so many biases in machine learning that I will talk about that later but again like is the data that you're capturing really is really capturing what customers are doing right like suppose that you want again that you want to build a nice picture but the only data that you have is for like logged in users because for like people who unrecognised on your side you didn't capture the data and you don't have it but the feature that you're actually going to be using it's going to be for all well that that is not representative right and it's also going to be biased so again I think that this is exactly the same that I said before but the opposite just to just to make sure that we're on the same page so don't do ml if your problem can be solved by simple rules if it doesn't really adapt to new information if it's always the same this is the other part if you want to always have the correct answer machine learning is not your solution a good example again that I would tell or it's not the only solution you can use it as a step before the end solution so a good example is that at Amazon specifically Amazon needs to when you're sending a package that has a flammable product you have to put a specific label right you maybe use a machine learning algorithm to kind of figure out which are those products but if you don't put a specific label to one of this product like Amazon will be fine with thousands like hundreds of thousands of dollars so you want a hundred percent accuracy maybe machine learning is not the best and the other one like requires full interpretability are you okay with not exactly knowing why something happens and that's big right and that comes from recommendations for absurd for all this thing that people say hey this is what the machine learning model said I want to know exactly what happened you may be able to give a kind of idea of the variables of the features but you don't really understand a hundred percent why it happened so if you are doing any kind of process that you need like full traceability about everything that happened and how it influenced that decision like for legal purposes purposes what machine learning may not be the best as well and again if it's not available if it's inaccessible it could take you months to even get to that stage so really figured out like it sit in my in my timelines for my product if there are some privacy concerns if it's not secure really don't even try and again right like when you have this irrelevant data stale bias everything self with data if you don't have the right data then there's no point to go to go ahead yes well it will need to be specifically to like what do you have a specific situation yeah yeah yeah I mean for example if you want to say are you doing regression and a forecasting to produce for example how many products should I buy from phone cases right do you have number one do you have enough data about like how many for example like how many orders have you had during the last year right do you have enough information for example if you want to do the forecasting like in the whole United States do you have information for each one of the different places where you're going to have like fulfillment centers do you are you making sure that you're you're getting the data when you actually have the products maybe at some point some of the vendors were out of stock so you if you if you got the data from like for example if you get that like if you get the data only from the holiday season and you use the data for the holiday season to predict or to forecast and not the only one that you have to forecast for the whole year that's not high quality and even yeah it's just yeah you have a lot of data but it's just for a specific part so it's not representative and is some bias even though you have it off so I'll do a simple exercise just to see if you were paying attention so the first one is like let's see if you want to ml or not to ml what Apollo items should be protected by copyright laws yeah yes who says yes why yeah when it's a rule what are you going to have like when you need a hundred percent accuracy it's not it's not the best something that you can do is just to use that to simplify the problem and actually and then you do our review but for that one you have to be very careful that your model is we will look later how do you validate the success of a model if it's the precision and recall so if you recall how do you call if you recall score is low I wouldn't use it because you don't want you don't want any product that is not that that you should be copyrighted not to be there the next one which resume should be should we prioritize to interview for a candidate for the candidate pipeline [Music] yes no yes yes [Music] yeah this is when this is when things starts to be become interesting right like you can say it is a complex problem it changes over time it scales a lot so it actually has all of the qualities of being a great candidate to be for machine learning right we have all the data we may have a lot of quality data but it some is bias right like you only have the data of the candidates that who apply and who made it right you don't have all the data of people who were great that never applied you don't and that I mean that that that that's it like you have you will be making the same mistakes that you did currently so you will never get better right like the Joker Amazon it like if we do this everyone is going to be caught like everyone who joins Amazon will be called Jeff so what product should be exclusively sold to Hispanics in the US yes No who said yes no one so no no one I mean this is very tempting right because you may have all the data to actually say Oh a lot of people who feed this profile and they have this behavior and they actually buy these things like it's very tempting but it's not a discrimination right you should not do it because you never know right now you're basically making a lot of assumptions about people and you shouldn't which sellers have the greatest revenue potential yeah that's easy a lot of people will say well it is discrimination well as well and the interesting part is like how are you going to be using this data so at the Amazon at Amazon is like well you're always trying to help sellers to be better for customers and you have and you have limited resources so you cannot help every single seller so you'll you need to find a way to decide which sellers to help a little bit more and it's okay if you miss some sellers I have a high potential or you have like some sellers that maybe we're not great but that will help you a ton to be able to use your limited resources the problem itself is discrimination correct yeah but I mean but that's the case right like and I I can tell you like my experiment like people will come with ideas to say should we do this and you're like oh no we shouldn't I mean like what we can I'm like yeah we can but we shouldn't and yeah it could be a great problem I mean it's it's fit it just it's not the right thing to do where should I assume bill HQ - yes who said who says who says yes come on the issue that this is not going to be a repeatable problem so what's it what's the point of going through everything that I mean all the variables it may it may help you like what is the answer you don't have answers I mean you have Seattle right that's it so that again machine learning is not a magic wand is not going to give you an answer on data for something that you don't know you but what data do you have you have Seattle right yeah so you can use all of that to create rules and actually create a program that will based on all these rules and you just input them and it will tell you but you don't have to go through the process of creating a machine learning model because like the series I mean it's not something that it will be different if you're thinking that I'm a Starbucks and I'm going to be opening new franchises right like and I want to open new franchises like continuously I don't know if it will helpful but it like that's something that I'm saying well things are changing it's a complex problem you're growing maybe like it may be interesting to see if things change right but for a decision that is only one time maybe not and you'll also have like for several franchises you will have a lot of data from how they differ so this one is like with search query so with scope for the Amazon fresh store this is very specific yes and obviously this this is this is a combination of machine learning and and regular programming because you can make the you can make the process way easier right and I was actually in this team that built that when when fresh was starting and we were like well if you are searching for a banana or not for a banana let's say Nutella where should we put you in the regular store of Amazon on the Amazon fresh store where should we put your right and at the beginning was like well if you have our fresh membership may like we should we should put you in the in the fresh store or not you start like having well we can have certain rules that we can apply to and then your data will be a little bit more like will be better and then you just use that data to actually improve your process so you can use combination of machine learning and regular programming for for things so okay so any specific questions yeah yes it's always it's always a little bit biased right but again so for example in this in the case for for Amazon fresh something that was super interesting is that we have the fresh that amazon.com homepage where we had a lot of data about people who were buying fresh there but then we had to integrate it with all the information that we have in amazon.com so we had some data it was not perfect because one of the main things is like if you are in fresh doramas on.com you are in your mindset that you are in fresh that that amazon.com 'you wanna fresh order if you are in amazon.com you may want to be in fresh or may not want to be in fresh so it is a little bit bias so then we have to be very careful to say what kind of how do you say what are we okay like what is what is your threshold for mistakes and we experimented but that's a completely different thing right like what we have to say well we believe that we can have a better experience if we use this data with a specifically machine learning model for fresh we call it like fresh orders copying to be able to decide if you're ordering like milk or but now if you're searching for milk banana whatever we send you to the right specific spot and then we figure out is it good or not right because at some point whatever we are deciding that those other better customer experience may not be correct right so something that our Amazon and at Google and most of these companies I will do it like everything that you're doing you need to actually experiment and figure out if it's good or not so that's the the next level yeah yeah I mean not only that but for example one of the problems like we had tons of problems in that specific circumstance because not all the products that were in fresh were in the amazon.com a in amazon.com yeah we had the categories were different so even like when we wanted to scope people in a specific like search result page with specific filters that were different from fresh so we couldn't do actually the same kind of matching so then we started to say oh we can only no we can only do it with very high confidence so it was a very subset of of queries that will say for this subset of queries with this confidence level for all of these we think that experience is better but the good part of a machine learning it like it will start learning right so the model will keep getting better and better and better better and eventually eventually it will it got better right so let's do machine learning right so what do you need for machine learning there are three main things you need people of course processes you need tools and systems and actually all of the three are very important if you don't have one of them very likely may you may not be able to do machine learning even if you want and even if the problem is great so the first thing is get get the right people right there are there are many different there are many different roles in different organizations I guess that all might Texas a little bit off but the part that I think that it's important for us as product managers or people who want to be product managers to understand is just to differentiate science or math through engineer engineering it's not actually the same so think about all the people who are doing the all the research scientists data scientist supply scientist ml Sciences bending on the company they're called differently they're looking into all the math like what is the right model what is there what is the right way to look at the data and to figure out like what is the variable like what are the thresholds etc but they're not actually doing the engineering and what I talk about the engineering is how are you collecting the data how are you cleaning the data how are you going to make sure that you're processing the data how are you making sure that is that is correct if you had a model and then you have to for example you may want to update your model and all your data every day I mean a data scientist is not going to be that engineering work that's going to be an engineer who will be saying like what is the data how do you I collected how do I rank it how do I access it so think about those two there are some people who can do everything but but just just just be aware of of the difference okay so what is the process so first and the most important one is to formulate the problem and this is I'm going to go through this really fast and then I'm going to go in to review it again but with the lenses of a product manager so I think that that will be a little bit more helpful so stay with me as I'm going through this and then we can go when I go into the product management review then please ask questions so the pro as I say the problem is is the most important part we we kind of understood what are what are the things that you have to think if you if your problem is the right one for machine learning make sure that it's super clear what you wanna what you wanna solve what is the measurable goal and what do you want to predict right as I as I say again like machine learning is not magic you really need to understand like what what what exactly do you want to predict how do you know that it's going to be successful and what is your problem we're going to go through examples later the second one is again data for data just think about like the important of selecting the data first like what is the data that is available you need to understand what it is okay this is what it is then you have to figure out what is if data that you're missing right there's a lot of times where that that part is it's critical to know if you're actually going to have a good machine learning model when we were talking about the fresh example like yeah we have a lot of data that of product that we're selling in amazon.com that were nuts where we're not selling in fresh comm so we're missing a lot of data there so there are many techniques to to figure out what to do when you have missing data but first like selecting the data understanding what is the data that is missing and then discarding that's a part that a lot of people don't like to do but you have to remove as many data that is not important and people will be like well but it may like no no remove it like really remove it and because if not your model is not going to be okay right and and that's how can I say this this is sometimes a big issue when you're dealing with different business teams because I will come and say but we have all these data and you start to like I basically found all the problems and you like yeah but your data is not there and this is not okay but people say come on like you were asking me for like 5,000 records here they are like why are you like making things harder right but you have to know that you're not going to actually do a good prediction and and you need to communicate that right like and why this is not a good prediction and if after your discard things and you get like very little data and sometimes it's data that you cannot use right for privacy concerns so think about that and then the next process is like okay first is the problem go through the data do you have the right data do you understand which one is the one that you want to use and then you have to do the purpose pre-processing to really try to select the when you're getting the specific data what specific parts of the data are important right like it could be like just formatting the data just to I mean a simple as law work is making everything in lowercase could be something that you need to do and engineer you need to do like when you're collecting the data just cleaning it and sampling it we're going to go into details about how you as a product manager what role you play in those in those parts the next one is features if you remember what I talked about statistics these are the variables that you're going to be using so basically yet the variables that you're going to be using to you to make your your your your predictions so there are three main different ways to basically to this to do feature engineering the first one is this called scaling that is basically just making sure that all of your data is in the same and the same format like if you're talking about currency that's a classic example and you're doing a prediction so you wanna mix and you wanted to look in sales volumes whatever like just making sure that all your quantities are the same all the methods that you're doing are the same then the next one is decomposition a good example for this is suppose that you have a time stamp what exactly what part of the time step is important right you see the day is it the hour is it all together and try to be as again try to discard anything that is not relevant the more data that you start using that is actually irrelevant for wherever whatever you're going to be doing it's going to create more problems so decomposing and then aggregation that some of the data it's better if you're aggregated right so you already have a just to understand again all of this is already clean data right you have data that is already available that is good that is great that is high-quality but now you have to do the feature engineering that is okay now I have all this great data how am I going to actually use it so the format in the cleaning the sampling the aggregation and aggregation could be a good example will be you may have logged in you're trying to improve the customer experience but you logged in at Amazon through your laptop and your phone do you want to use those as two sessions or one session for example we like well depend on the problem that you're going to be solving sometimes you may say when I say I want to see this by user or I want to see this by session so all the data is clean it's a matter of like how are you going to be using it and the last one is actually doing in the model for the model again you have this data set it's already clean it's already good it's formatted it's sample it's perfect the first thing that you actually do is you have to divide it and you divide it in what is called training data and what is called test data the way the way that you usually think about this is that you need there's not a formula to say like what percentage of data should be training what percentage should be testing but usually you need a little bit more of training that testing so depending on the problem you may go into 70/30 or 90/10 it varies but the main point is like you use hold it all your all your all your all your data that you have you just a certain amount of that to actually train your model create your formula whatever and then after everything is done you actually use your your test data to put it again in your model and to see all did it does it predict that saying this the thing that I want them to predict you don't want to use the same data that you used to train it because obviously it's memory like you're you already know what is going to happen I said you have the data set you divided you just use the training data with all the features that you say that you're going to use you have this model training you don't need to understand this part right now that is basically the algorithm that you're going to be used yet to to predict the outcome and this is when you start saying oh it's anomaly is it a clustering this is whatever that actually will define the kind of model you will be using then you have the actual outcome when you kind of have the weights of how each one of the features will be will be valued and then you when you have all that ready then you use the test data you put it again in your model and figure out if if the output is what you were expecting or not and then you have to obviously test it so we will go into that into into into later and after you do that process then you have to productionize your model it's important that you understand that you require a deployment environment to have machine learning you have to really understand for example if you're accessing data and do you need can you do the training offline do you need to do it online again like how often are you going to be me how do you say like reviewing the data and training the data it is something that changes very fast if you want to change very fast maybe your model needs to be like updated daily or hourly so think about like do you have the right environment to actually do that kind of processing then where are you going to be storing the data and this is important we discuss security and I just want to make sure that you don't forget that and how are you going to be storing the data in terms of what type of database you're going to be using if you if you can compress some of the data because you only need like the output you just need like most of the engineering that your engineers will be able to to figure out if this is if what is the right solution but that's basically where they come in place and also this is this is very relevant like monitoring and maintenance how do you know that your model is working right because that's like we we discussed before right that you don't actually know the full interpretability of the model so if you don't really know how something is working with a formula then how do you know that things are not just gay going crazy so you need to create some kind of monitoring tool deployment or something that is going to that is going to tell you that things are not yeah that are actually okay and that's that's how can I say this like a lot of the times this step is not taken into account and and that that's a problem right because you can have a machine learning model that is just getting worse and worse and worse if you didn't do it correctly the other part is that there are I don't I don't wanna be I think that I'm only saying that not to use machine learning in this session but there are many many options where the problem is great the data is great everything is amazing but it's very costly it just doesn't make sense to actually do it because the actual outcome that you're going to get it's not worth it because maybe your model needs to be updated hourly and you need a lot of data to actually do that and it's it's just not worth it so has any other part with product management you'll really need to figure out what is a trade-off right and that's why you have to be that's why the partnership with the data scientists and the engineering team it's is key for this so now let's go into more details about okay let's go again into this ml cycle and what is a role of a product manager in the in this cycle to formulate the problem how do you think that this is like how involve product manager should be here yeah super involved right this is this this is this is where you fit what is the problem what is a measurable goal what do you want to predict and there are very different ways that you can do this I'm going to go through to two different examples when you have not USML before and maybe when you have used them or before I think about like when optimization versus just launching something so you're going through the similar problem that we discuss about the sellers that what which one are the successful sellers that are getting into Amazon like how would you define the problem so one way to define it is that they say each which the new sellers of Stax team on boards hundreds of new sellers and this group is expected to grow X percent year by year personalized coaching time however doesn't scale as such the team needed and a way to accurately predict the performance to double down on so that's a good problem right you say yeah this is we we can scale this this is going to keep growing will we need a way to do this in in a better way and as you know things will keep changing right like the problem is like if you just do some like certain specific variables you may not be able to use them like a year from now so that's the problem so what will be the the goal any takers doesn't go for the model it's part of it I think that here you're you're trying to define the problem so what what will be the goal just to know that you're doing well because you can have a model that is 90% precision right it's it solving that problem yeah I mean it's basically are you like the you you want like the goal is actually to increase the revenue right off of the couch for short note notch coach by X percent so you have a very specific business goal this is this is very business problem like what do you like what do you want to know like you have this problem and the problem is that the sellers that we don't you have limited resources that you need to identify these sellers you want to coach them and if you coach them based on this model you're they're supposed to get better right so the goal is like are these the one that we're coaching are they getting better because you actually don't know which other you don't really have how can I say this you're predicting if they're going to be good sellers or not so you don't actually have a sense of saying these are the good sellers and these are the bad sellers and these are the one that we chose you don't really know you don't really know which are the best sellers this is your best guess are you your best guess is to say from the wonder we chose that we believe that are the best one they should get like do much better oh that's another way yeah I mean that's a that's a that's a different yeah that's a different problem I guess that at this moment you're you may for example I would this will be what I what I would do right your you have your team you have like five people that are coaching five five teams and the the their success rate with the way that you were choosing them is like they increase revenues by ten percent so if it the same people that it that is doing it is just the the sellers are you're choosing you're expecting that you choose imbecile better seller so then with the coaching it should be better than ten percent right so what do you want to predict right so in this case is like the top five percent that seller so that's what you actually want what are the the net sellers and then the goal is actually the revenue right but the prediction is well which one are the sellers so that's something that for example you don't you don't you never use machine learning and you can use from from scratch another problem that you can use for machine learning and this is something that I used is for example the problem is the units per order for specific category Ramazan in the US has remained flat that means that it has in growth and engagement has declined as measured by this other metric that's kind of that I mean that's a problem in the business sense maybe you don't have here all the details but you're like well we're investing so much we have more products like but this category is just not going up so it's a problem we want up like you say yeah that's a problem you're not thinking about the solution you're not thinking about anything this is just a problem but what we want in this case or what the team wanted is to increase the unit order rate for this specific category in the you by X percent within the next X months without affecting revenue so then you're starting to put in limits right you have well this category hasn't grown a lot then we we will expect this to go so I want more people to order more products but for example something interesting here is to say suppose that your number of orders number of order I atoms per pair per order are five i want it to be six and they should like why you want to be six there are multiple business reasons right why you want to be just just just trust me that it's good that is six and not five but you don't want them to spend less money so you're also put in like some kind of boundaries to say that will be my my goal and what do you want to product but I wanna predict like well actually the solution here is about recommendation like we want to recommend you the next product to buy so what you want to predict is what is the most likely item that people may wanna buy when they already have something in their basket so as you see it's important to understand the difference about like what are you actually predicting like that is not exactly like nothing nothing about this is actually in the problem right try to try to be very good at it making sure that you understand the problem making sure that you understand what is your goal and not even think about machine learning at that moment right and then you're like okay with machine learning what kind of prediction I can do so I can create a solution to do it but it is again this is a problem the next one is to select in our prep / processing data so what do you think that is a role of a product manager here it's actually pretty high so in this one you have to be really in the weeds because if you don't have this right everything will be wrong so you need to make sure that you're like whatever data says that you're using are good ones from where they come in is the data is that right are the right data sets that you're using and are you using it for the right purposes right I say that you have to have select selecting data to train the model to test the model do you have enough about that what happens if you for some reason some of the data is you you missed it you have enough to replace it you have to go I mean if you think about the process that you have to gather that data and suppose that you go through that process and suddenly you realize that you don't have enough do you have to go again through to gather more data and it's going to basically delay everything so think about all of that as a product manager like how are you getting the data do you have enough data do you have enough data to measure success and in formatting in formatting you don't actually like a product manager you don't really do much it's I mean I'm not going to go into these details but basically you have to make sure that everything is in the same format just be involved to make sure that you're not doing like people something is not doing going completely wrong but this is pretty simple in terms of like removing or fixing missing data you do have to be a little bit more involved because then you will start right to figure out like well is this the data that is that I told you what's that I said I told you what to do but then really go and review the data and look at the data sets and actually go and see is it is it incomplete like is it noisy a good example of noisy is again like if you are for example if you are doing something with orders and you have an order that is I don't know like $50,000 like you don't want to use kind of that data to predict your model because it passes threshold so making sure that is it's not it's not nicely that is that is not bias or inconsistent pay attention actually reviewed the data I will I always went with my data scientist and say okay let me let me see the data base and then send it to me and I will start looking and making sure that that things were fine and there are many ways to do this and many ways to review if you find problems there are ways that depending on the specific problem sometimes you have to say you know what we have to delete these data like some of the day even after you selected everything you say not this data it's it's going to bias the model let's let's let's delete it right and that's that's what I was telling you it's a difficult decision but you need to make sure that you can take it if that the right one other times you say you know what yeah it's not perfect but it's like if you say well you can just substitute some of the data it's pretty easy right like you can say if some in this case if the price was empty like of course it's empty that value but it's zero right so it's very easy there are other ways that you can say well when you don't have the data about everything can you just like do an average substitution so you can actually use it that's also something that depending on the problem depending on the data you may want to do and the engineers are or the data scientists are going to ask you as a product manager like what what should I do like we have all these issues with the data should we delete it should we do a substitution should we put dummy data should we like just put whatever is most frequent should we try to actually find like what is the actual parts of the data do we have an old data base that we can relate to and gather that data so in that one you have to be you have to be involved because all of those decisions are very very relevant for your for your project then you have to also choose representative data and there are many ways to do it you can use just random data and there are pros and cons to do that because if you just random data for example for all your customers it's possible that you don't get a customer segment that you are interested tuned together or you can do stratify that you can say for all my customer segments I'm going to get like specific data all of those has pros and cons because if you use random you may be missing some kind of segment if you use stratified and the segments are not well some of the segments are like if you if your segmentation is incorrect you can also add double-counting or or have some problems so there are terms of issues seasonality that if your data we will discuss any of you only use 4x Pacific time that the data is not great if there's a specific train that you're not catching or you only caught the data with that trained leakage will mean that some of the data will actually start getting back into your model an example is sometimes you will reduce in some machine learning models to solve a specific problem because another machine learning model was not good but then this machine learning model is helping this other machine learning model and suddenly say all these machine learning models in zone is not good and you're like no actually this one fixed the issue and helped this one and without this one this wouldn't have helped so you really need to understand how you're getting your data we don't have a lot of time we have only like a couple of minutes so the the main part I want just to talk to you is that really think about biases here I don't know if you I guess that everyone has heard about the Fiasco Google when they were doing labeling of images and they were labeling gorillas for people of african-american descent so and that was because their data was biased right they didn't use representative and that's that wrong like really really wrong and the other like something so interesting that it has like unintended quotes the consequences this was something that happened at Amazon with 3-prime same day where they wanted to figure out like where should we have this service so they were looking at past orders orders to say well these are where people are buying things so these are the neighborhoods where we're going to allow these service and and that's a scalable because they can use the same model in every part of the world what happened is that specifically for Boston New York many of the big cities what it ended happening is that it was very racial like basically they say oh this is where we're going to be putting all over free prime day well we're not going to use this one because people are not buying you know from here that in makes complete sense it just happened that most of these neighborhood is it was black residents right so and it was in the middle of the series so like why you're not providing your service in the middle of the city and like well the machine learning model say that they are not going to buy right but not because you have a machine learning model that is telling you what you do is that means that you have to follow so as a product manager you really need to do to understand right so there are tons of biases there are tons of I'm sorry like we don't have enough time there are a lot of ways to actually reduce this biases in terms of like labeling there are a lot of techniques sorry I don't have time to go through feature engineering but the good party like you don't really need to be really involved in filtering engineering that's a part with the data scientists go and there are many ways to able way different models the data scientist will be going into that when you have regression you can go like root mean squared error when you have classification you can figure out if is if it's accurate or not this is where the precision and recall items come we're now going to go into that when you can read anywhere online about it the main the main concept here is that you're making prediction so you really need to figure out if the model that you want to do want to optimize for precision or recall precision is basically what proportion of the identification was actually correct and in some cases that way you want to that was that's what you want to do like this will be very good like for recommendations or or personalization like mostly I want to recommend something that is good if I miss here and they're not bad recall it like you really want to make sure that whatever you're saying is correct so this is mostly like the classically like oh there's a fire alarm you don't want do you wanna make sure that every time that there's a fire you actually predict that there's a fire and there's there's there's some trade-offs that you have to do so I don't know that we are going to have time to do this but the main I will just go with the with the with the with the takeaways trade your ml project as a partner engineers they designed this business everyone it's very important that everyone understand like why you're doing what you're doing and why data is important and why selecting data is important why biases are important people need to understand that be very clear with your problem hypothesis and success metric again think about the problem the customer problem that you want to solve the solution could be like multiple right don't don't start thinking about like what prediction I want to do it like what problem do you wanna solve and is machine learning the right one and be willing to make a lot of trade-offs in terms of time in terms of quality prediction accuracy we say a white box black balls like the you wanna really understand all of the details that are happening or not we're saying false positive or negative like recall and precision and really to figure out like why are we launching this the part is just when you're getting the data make sure that people understand what the data means make sure that you understand scientists are not software engineers machine learning creates that and be considerate right when you're when you're working with people with different skills and and what you what you bring to the table right you bring to the table this overall understanding you're trying to get people like they are designed they know what algorithm is the best one but they need to understand from you like what do you want why do you want it what problem do you need what data do you need is there the right thing but if you start saying oh then use k cluster instead of using this other model you're crossing the boundary and you won't be liked and that's it [Applause] yeah what do you mean like if it's worth to do an experiment experiment after they're in the machine learning or yeah I mean I would say that at least in my world I never had never had launched anything without no experiment but that's again that's the then at the beginning I thought old machine-learning was not good for and then I decided that I should do machine learning after I have to say that most of the times the issue is not so much of the machine learning in not gooey like there's something that is missing right so either the data is not great or we don't have access to the data or or is very very costly to do it so at some point I many times I would say this is not the right time to do machine learning but then there was something like that change to say Oh remember that problem this this this barrier is gone so then we can use it but I think that has been my experience that suddenly for example we have better processes so then creating a new model was cheaper so then suddenly like the threshold to how much money do you need to make was lower so they need like oh now it makes sense that we did machine learning and then with this arena to do it actually I've never had that situation because we specifically for me I think I was I had always been very very careful about what I put my engineers to work on so I just haven't had that issue personally we hate the bias in Bodega generate generate data yeah I I mean I actually I some of the some of the things that I would call me like I do almost a brainstorming session with people and say fine holes like fine problems this is what we we have like sometimes the issue like I I am bias every one of us are bias so I may not see some problems so then for me was all we like this is what I'm doing what do you thing and I will go with different types of people different roles people as something that I will always do ill I go with someone who is completely who couldn't care less if this project launches or not because there's not they are they are the one who will say yeah I don't yeah this doesn't make sense I don't do it [Music] yeah I never I never did deep learning like I mean myself the team did it but I just didn't I don't have experience on that you could the main problem with deep learning is a compute power it's very expensive so at the moment when I was at Amazon it was too expensive for most for my team there were other team that were doing some explorations so when I left there were some exploration but I didn't participate on that it's just a fact that it was expensive for me small about like can I help you write this I just wanna I want to make sure that you're not doing this by yourself right like do you need a second pair of eyes and usually people are very perceptive to say oh yeah yeah you I don't know right so I if you want to help go ahead right yeah so if you have any kind of model right you're discussing like you have a specific database where you're storing the data and you have to create a pipeline to basically generate the data and put it there looking at again make a prediction so you need to really understand like is it worth it like because someone will need to maintain that and then if you create a model and it's get out of date then like you need that you need you need maintenance so think about think about that sorry I mean you start with the problem right like with the customer problem that you wanna solve and then when you start trying to figure out what are the potential solutions you may think about machine learning is a potential solution and then you start looking into the data but you start with the problem delivery yeah no so like how do you I mean is that just something that like so the that was her bloomer investigation that did this and they basically when I had when they asked Amazon he like hey you completely excluded this population and they're like we didn't use like even like we didn't use any race to make these things like no you didn't but you still make that prediction so I think that you as a product manager you really need to figure out like what could I be doing wrong like what can this data how can this go wrong and the larger the company that you work for they're more careful that you have to be so I didn't use for customer segments I use some supervised learning for doing a lot of clustering specifically but not for customer segments oh yeah yeah so you will use behavior like what kind of so for example it's the way that I will and I still think about customer segmentation I think about either your intent your mission your behavior that that's how I think about customer segments I don't think about customer segments about like what's your role where do you leave no like especially because ml is still hot right now everywhere you want to be yeah seems like you would want to say no a lot yes yeah I mean that's that's great and I think for me the most important skill for a p.m. overall not just for machine learning is empathy and when I'm talking about empathy is that when you're talking with the different stakeholders you really understand what's our problem like what are the what are the pressures are the stress that they have what they want to do so then you can really empathize about like why are they pushing this so hard right like if you're talking with some of the I'm I've talked to many marketing people who would say oh this is what we wanna do and I'm like it doesn't make any sense but then I'm like why why do you wanna do that and then I understand that well we have this other business goal and I'm like oh okay I can help you with that right so really trying to go deeper not judge people and the other part that I wanna say is don't be condescending please please don't be condescending like when you are talking to so many people who have never been in this field before or like just don't say oh you don't understand anything like you didn't understand anything in like like a couple of weeks before right so we be like understand like try to put yourself in another in yeah let's try to empathize and make sure that you understand what the problem is and saying yeah when I solve the problem we can figure out what solution to do yeah oh that's a little bit more difficult and I think that it's about like let's try to figure out like together like what what information do we have like what are we gonna do and really I think that for me were like really involving them to say this is a pro Linda we have do you want to explore that solution and at some point I did have to do that like to say okay this this person really wants to do email with this it like okay let's let's let's do a blog and do two weeks and figure out if you have the right data to do it and then they will come and say oh no it's not actually good Oh learning resources good so now that I'm at Google I will I will put I will plug this I mean one of the reasons what I actually change is because Google actually had like really great resources if you are working for a smaller company to actually use a lot of their machine learning and auto email that a good example that I like I cannot leave you go through through the through the Google web site you will understand a lot of that but something that it was that is super cool is that we're talking a lot about like identification of images and classification it like what how how do I do that like how do I know that an image is an orange or an apple like what the hell like yeah I understand the concept of labeling this data but how do I do it systematically right so like right now with Google cloud with Auto ml and vision API you can just connect to that get the data label it so I don't know I that's a that's a good resource to start sure we one more it varies a lot of course as everything I would say that the one that you have to pay the most effort is in the problem statement just making sure that everyone is aligning when you have the problems statement you as a product manager you're focusing on that one you you kind of have an idea of the of the data but let's put it in this way you're not going well maybe with the new Google email you can do it very fast but if you're not doing some of those tools you're you're taking months right you're not taking weeks this is a month if not like your long process some of the some of the machine learning models need to be constantly iterated you may not have thought about some things so think long term for many of these processes as I said before you are investing our resources this is expensive and you're doing it because it's something that is going to scale and it's going to be helpful like long term you

Channel: Product School

Views: 26,984

Rating: 4.9428573 out of 5

Keywords: Product, Product Manager, Product Management, Product School, Tech Startups, Data Analytics, Coding for Managers, How to get a job, product manager salary, product manager resume, product manager jobs, what is product management, what is a product manager, product management training, how to become a product manager, cracking the product manager interview, product management jobs, google, machine learning, ai, Machine Learning Products, Technology, Career

Id: 5z1Hz-rV4zY

Length: 94min 50sec (5690 seconds)

Published: Wed Jan 02 2019