Suhail Doshi - How to Measure Your Product

we are very grateful to have sueldo she from Mixpanel who co-founded Mixpanel almost 10 years ago now and is going to talk about how to measure your product which as you heard from Gustav is really the other side of the coin of growth and everything that helps you build something that people truly want and will use so with that Suhail all right hello everyone happy to be here awesome I thought I would start off by just kind of like I sometimes try to gate try to change my presentation a little bit on the fly depending on the audience and so I thought I would just ask like a really quick question just a quick show of hands um I'm gonna define like a user's like just someone that uses your product whether it's a whether it be to be company paying free doesn't really matter like they did something in your product how many people have like zero users right now okay cool awesome and then how many people have a hundred or less well okay you have greater than zero between 0 1 and 100 okay awesome great and then how many people have like more than 10,000 okay awesome all right so we've got it's a majority of people have zero users right now cool awesome great so I'm gonna start off with just a quick about me for those who don't haven't heard of Mixpanel um I started this company called Mixpanel I think the first line of code was written in like October 2008 technically I was like 20 years old I was in my parents house in my bedroom and and you know 10 years later the company has about 300 employees we've raised a bunch of money and we have about 7,000 paying customers and we're starting to close in on around 100 million dollars in annual recurring revenues just to give you a sense of yeah 2008 to now cool awesome um if you don't know what mix phenol is we make pretty graphs like this we help you measure what people are doing inside the product so this is like you know someone inviting a colleague to their app we might help you like measure a funnel and see the conversion rate of people opening your app to like viewing an article will help you measure retention I know Gustav went over that a little bit today and I'm just showing you this just because like everything that I talked about today is is basically is something that you could probably do in Mixpanel or even any other tool there are a bunch of tools that are great to measure these things but I want you to know that if you walked out the door and you wanted to apply any of it you totally could there's there's like a freemium plan and we cater very very much to startups those are all of our early customers so feel free to do that awesome great so today what I wanted to dive into was I thought I would take a top-down approach to thinking about analytics and metrics instead of kind of coming at you with charts and visualizations and numbers and things like that I thought it would be better to kind of think about the problem that you're trying to actually solve and it's sort of ignore for a moment that you have to like do complex analysis and calculations um you will figure that out it's not hard or you'll just use tools that will make that a lot easier for you so that's one thing that I thought I would do and so I picked these three things um I tried to think pretty hard about like what the heck did I care about when I started a company like what mattered to me what was on my mind and I thought well I think the first thing is just like do people even understand what I made and mixed well I had a really hard time with that in its first 18 months we had to change a lot of things and most people do the second thing is okay great they kind of understand what it is but do they like find it relatively easy to use the product on day one and that that's kind of this never-ending mission that you'll end up taking when you start your company and when you ship your product and then they'll and the last thing is you know I kind of tried to pare it down a little bit but I thought most important thing would be then next are people coming back and using my product and I think that you'll find that like I know Gustaf talked about this and I'm going to be extremely repetitive because it's one of the most important things that early founders make a mistake with yeah so from there I thought that I would try to also I thought that it'd also be helpful for you to see it kind of broken down in a formula and so this is the formula and I think I like to think about things from first principles right like what are the things that would cause you to grow and they're pretty simple right there's just like people that visit some page or the app write basic you experience this every day and then there's like people that sign up and then of those people like who did something how many of those people did something valuable like watching a video or making a recipe or you know taking a scooter a scooter ride and then of the people like how many of those people end up coming back some period of time later and then how many people will like spread that product like tell a friend share it maybe you even have a sales team who knows right and these these five things like basically equal growth and these are your lovers and the thing that I wanted to kind of point out and I want to stress is that today I'm gonna cover the basics but I want to kind of implore on you that the basics are really important even thousand-person companies get the basics wrong in fact you'll find that if you don't get the even these thousand person companies that get it wrong they tend to overcomplicate the number of things that they should actually measure and track and it's really easy to make twenty five metrics conceive of them align them you know divide them ball by you know give each team five and then really over complicate something that is actually quite simple to do when you over complicate it what happens is is that companies even large companies have immediate paralysis even midsize large companies have this like decision making problem used to have this funny phrase at Mixpanel where I used to say that all data is like contestable you know you could like see this number go off into the right and go oh well I don't know it was like sunny outside and like and that's why people did it and people will like have these arguments no matter what because causation is really hard to discover and so what I find is that just picking like three to five things on Northstar and really simplifying it will do more good for you and I know because not only have I watched thousands of companies make this mistake at all sizes I have made this mistake I had a piece of paper that had like 25 sub metrics and it was nearly impossible to actually keep track of it and what and people end up doing is in reality what people care about is there are there are metrics that guide the team and there are metrics that help you monitor and assess if anything is going wrong and today we're gonna talk about things that guide the team okay cool great so with this formula you should be able to assess anything going wrong in the company right if you're if you have a really confusing landing page it you should be able to figure that out if your signup process is too difficult you should be able to figure that out if you've a product that people don't really value you will figure that out with just that formula it's just that simple and you can get more complicated after that right you could say okay landing page views are down all of a sudden why who is it by Clemmy look at that segmented by country maybe something happened maybe I changed something in a different country so you should be able to assess any reason why you're not growing just starting out with that and that's why I say starting simple will last a very long time so this is what it looks like right linking those two ideas together you have this funnel of people going from visits to you know eventually retaining and spreading the product and like on each one of these things you're trying to basically solve one of these three problems is it easy to understand is it easy to get started are people coming back because they find it valuable and and then it loops and the funny thing about this is even if you're like Airbnb and you're thousands and thousands of people this like this process of like optimizing each one of these steps is never-ending no matter how you're no matter what scale you're at there will always be changing something and you probably have competition so you'll have no choice so awesome and then once you are able to measure those five things and assess what's wrong you have your levers and you should be able to take action and fix any of those things put a team on it maybe you're fixing it it was really that simple awesome so we're gonna start with the first thing which is is my product easy to understand in order in order to illustrate this idea I decided that it would be somewhat sympathetic because I could show you like really awesome landing pages of like great companies or I could just show you like what it's like to start out and and I also wanted to mention that having a really bad landing page um there's this really awesome quote that PG used to have in our batch that I thought I would impart on you which is that like people basically just have their mouths like hovering on the back button and they're just waiting to say I don't know what this is confusing back and that's immediately how most companies are just like losing most of their um users from moving on and so just remember that your enemy is the back button so this is what mixed panels first front page look like in 2009 that's not what our logo looks like anymore this is a design by me original design by me and and and in all of its glory if you like actually goalie read the words there's like tons of grammatical mistakes because I was horrible at writing and for some reason in my mind I thought I know what the tagline for the company should be metrics that'll make you drool I don't I and I guess I think the reason why was because I thought these graphs were like so pretty looking compared to what was out there that I thought like that would be like the value proposition of the company and of course is like totally wrong like you're all laugh I wish I could have like made this landing page presented it to you then you don't laugh at me okay I should change that thing but I didn't have that I had zero users at the time you know and I tried to target you know small startup big company I didn't know who I was targeting application developer I like trying to target everybody like this is like every mistake that one could make on a landing page is like right here right like the signup button at least I kind of got that right thank you Endora for teaching me how to optimize buttons that slide like made it like yellow but that's about that's about maybe all that I got right right just maybe they are roughly the signup button and then you know after a lot of work a lot of hard work a lot of iteration a lot of talking to customers a lot of measurement you know eventually a couple years later it kind of turned into this right I like managed to get like the logo right a sign-up buttons like ginormous and yellow and and and what we figured out though after after a lot of time and a lot of energy was that we transform for metrics that'll make you drool to actions speak louder than pages and and you'll notice it's like extremely prominent the reason why was because the number one question that people that we would get as a company was that's cool but like how are you different than Google Analytics I don't get it right and so it turned out like this tagline for us at the front page lasted like the next five years of the company and and because it like very much differentiated us as something where we would measure engagement we did something different like Google was about pageviews we were about like what people did in your app and that resonated and then and then from there again we tried to explain like what various we try to urge people to go down the website and like look at different features and see if those things were valuable we tried to be benefits oriented and the copy tried to show customers and case studies who maybe found it valuable we've probably made like four changes after this but I thought I'd present this one just because it was the one that maybe lasted like the the copy and the general essence of what we did instead we did what we said we did kind of stayed the same largely but lots of design changes so um the real question is you know great so like how do I measure how did you measure whether people really understood whether they understood what you were doing and so I think there's like one tried and trued number and it's like very simple it's just did people even bother to sign up bothering to sign up doesn't mean that they're gonna use your products that's a whole different matter bothering to sign up is like I don't know I'm kind of interested I'll kick the tires this seems kind of interesting and cool to me and that'll tell you a lot about whether what you're doing is even something valuable for people the second thing that you can try these are just ideas trying to find the ratio of people that are going from just like hitting your page right and then there's and then doing any kind of thing after that so instead of aka instead of hitting the back button right like did they click on segmentation or funnels or attention in that case right did they bother to do that I know there are a lot of b2b companies um these days actually one quick question how many people are making like a b2b company yeah great so that's about half the room so I thought I'd give a b2b slant one thing that I discover was that people a lot of your users will click on a pricing page link at the top it's like second or third like most clicked-on link if your b2b company I think generally speaking I Jenna I think you know this is totally contestable but I find that like people clicking on your pricing page is like actually a pretty decent indicator of like okay cool you have something kind of valuable okay how much and so even something like that I think can tell you a lot and then if you're in consumer the benefit of being in consumer is that you generally have usually will have like hundreds or thousands of users such that you could actually go do an a/b test whereas in b2b it's a lot harder to achieve that and so I really I think Gustav is absolutely right that you should try to conduct as many experimentation experiments as you can and really a B test your copywriting and find out like what resonates with people but these are all things that you can do to basically figure out hmm is this easy cool so the second the second one is well is it easy to get started with my product and the thing about this is like a lot of people the thing about about these first two things is that if you don't make the first step easy if it's easy to understand a lot of people put so much energy when we're trying to build a company we try to think about like all the cool features and things that we could make that'll be really valuable for customers the problem with that is like if you don't even get step one right you don't even have a chance at any of the subsequent steps so when we think about like is it easy to get started with my product you know forgot about password is kind of a useless feature if you have like a very small percentage of people that are basically like not even bothering to get started in fact I remember there was a YC company called Zen tur which turned into like Google slides and I remember a Robby Walker during one of our one of one of our Tuesday dinners said that they had they didn't even implement the save button for like their online PowerPoint because they knew that none of their users would ever asked like hey can I save this they just even bother to do it because they wanted to see what anyone even use the PowerPoint presentation product it was like that was in that one day and part of that on me I was kind of like wow that's crazy and for a like I don't know maybe like 18 months of Mixpanel we didn't even have forgot about password because we were just so ruthlessly trying to prioritize like things that really would really matter to customers and it turned out we didn't get very many requests anyway so this is really important and here's an example of like easy to use something you use every single day that is literally one-step you just go to google you type it in and you immediately get to use the product there's no signup there's no CAPTCHAs there's none of that right a more complex example life is like Airbnb who might have like you know ten steps or something like that we worked with air me for a while helping them optimize initial user experience and you know even though Airbnb probably has like 11 pages of you know like you have to go to Airbnb and then you have to fill out this form and then you get taken to the results and then you click on those results and then you you know you click on many different results because you're trying to price compare and you're going through this very complex checkout process so they're actually many steps to booking something online but Airbnb had to kind of figure out like how do we a describe what we're doing what we offer what's useful for the world and be how do we help people get started as quickly as easily as possible and like this is Airbnb calm right now and it's you know they have this very simple like book unique homes and experiences you fill out these very basic things and you hit search and that's it that's their getting started experience it's not their entire flow but it's the first way to get started and they even try to like kind of impute something like what you might what you might be able to find something imaginative on Airbnb but the thing is is that this seems simple right this seems like something that you could just like easily make it doesn't take how hard is that right we can do that with Twitter bootstrap in no time the problem the thing that's really hard is like figuring out that that's the right thing that's really hard that takes like they probably change that you know 50 times before they figure it out like this is the right current optimal thing at this stage for the company and so it takes a lot of work in and you have to grind very hard to get to this at this point so question is like okay great so how do you how would you measure something like that well I think the really simple answer is a funnel how many people go from a landing page to signing up to doing that valuable thing like watching a video doing an Airbnb space who had been just doing a search just one search at least once you get past that measuring your funnel for the entire initial user experience there's like weird things we're like there's actually situations in games games can have like 20 steps and I've noted and I've seen hundreds of gaming companies funnels and even though they might have like 20 or 30 steps sometimes you can they actually convert very highly like they might get like 8090 percent conversion rate through all those 20 steps just the first two are like actually the ones that have the greatest drop-off and then the last thing is just like actually speed this is one that I think a lot of people don't actually do they're just like how fast how fast can someone just get started and that will tell you a lot about how complicated your experience is sometimes slow is okay it depends on the business but speed is really important does it take them like five minutes to get something figured out or does it take a minute in mixed panels case we cared a lot about like we had a complicated flow because you not only did you have to sign up but you have to be a little to write code just really complicated and that was the one downside we weren't really sure where people with Google Analytics you just copy and paste the JavaScript with mixed family had to like actually go and track a line of code and so we had this question our mind was that easy enough and so speed became really important to us how fast can someone do that and I wanted to give you a couple tips and tricks things that I have seen that companies do that are generally like not great number one you're a start-up you don't have lots of fraud problems you don't have you probably don't have a lot of people spamming your service you probably don't have a lot of like fake actors in the system trying to sign up so if you don't have those problems don't optimize for them until you do and I just want you to know that II generally email and text confirmations have an enormous drop-off like I'm talking like you get of the hundred percent of the people that come like only thirty percent will like basically go on and like click on those email confirmations or forty percent you spend a lot of time trying to optimize that and then TV make matters even worse like you have to hope to god that it isn't go to spam like it's this really horrendous experience so just be really careful you'll lose a lot of users that way it's really important to iterate on your initial user experience that will be a never-ending process you will do this forever even as the press especially as the product it becomes really critical to like have someone on it so at least one responsible person just thinking about that all the time and then the last thing is is not every company not every product is capable of doing this but to the extent that you can just let users into the product it's always better to do that in mixed fennels case it wasn't possible because like we needed to get your data so you have to sign up but we experimented with like what if someone just copied and pasted the JavaScript but they didn't sign up and we just collected their data and then after that we like asked them to sign up we did experiments like that so to the extent that you can achieve achieve this that's awesome and good examples of this are actually like Airbnb and Google where you just just try it but there are always exceptions to these rules for example like Pinterest makes you sign up and they didn't used to but they do now they're good reasons for that and then and then the last one definitely not least are people coming back to me this is probably one of the metrics that is greatly ignored by most startups and it's often the reason why I've seen companies even with millions of users actually dive I've seen probably 15 different companies that grew virally and then just died the company died and I want to demonstrate this in a graph I called this the shark fin effect and the reason why I call it the shark fin effect is because well one was because Mixpanel in in Mixpanel the default line color for your first like data point is like blue and so it always to me it always looked like a fin like in the water and it looked a lot like this and what happens is you fought you like you're slaving away and you find a way to go viral right maybe you're going super viral on Facebook or on some or Instagram or something and you're like wow I have struck gold I'm acquiring tons of users my life is awesome I'm going to be a billionaire is great right and the problem with this is is that if it happens too early but you didn't quite think about if you quite think quite thing about retention and becomes really problematic and Yulin and this hurts companies so basically what happens is the app goes viral and then eventually the rate at which you're losing users becomes so it becomes high enough that you can't acquire the new ones and it turns out that like even though we like to try to be extremely rational people it's really hard because like when the apps going viral you're just like oh my god I need to like just take advantage of this moment because what if I don't get in what if my competitor finds it so I'm just gonna optimize optimize optimize but it turns out that it ends up not being very valuable because if you're just losing all the users it's really really hard to reactivate if you talk to people that have ever done like a reactivation campaign your activation campaigns generally don't go very well you usually just like lose those users until they decide that maybe they want to give the product a second chance so it's not like like your response is like when this happens a lot of people's response is like oh we lost the users because our app wasn't valuable let's just email them again and see if they'll come back and like that's usually not a great idea so it can be really it's really important to really think through retention and you really don't want to be like this guy the shark fin effect is really really bad in fact LivingSocial is a great example a lot of people don't know that like LivingSocial was basically this app that was on facebook that had nothing to do with like like the Groupon deal cutting thing and they went super viral on I can't exactly can't remember exactly what the product was um and LivingSocial literally had to pivot the entire company to basically competing with Groupon because the app wasn't very retentive um in those days so there's some pretty basic ways to measure that people are coming back the most obvious one that I hope most people know is that there's just new users so making sure that you're tracking whether a brand new user someone who just signed up what percentage of those users will come back like a week later or thirty days later it's really important to track a longer period of time because a week later is just not a harsh enough metric and it's important to see how fast you'll end up losing users because you'll have to figure out a way to read to Rhian gate to find new ones and that becomes really really hard at some point if the number of users you have you becomes starts to become quite high and then the second one is is just daily active users there might be people that disagree with this I don't think they're too many but I now think that monthly active users has become like the new BS metric that's like very similar to like just like a number of registered users like number of registered users like a very silly metric I remember seven or eight years ago I think like I want to say it was like LinkedIn came out and said we have 200 million registered users there's like who cares how many people are using LinkedIn and I now start starting to think that like Mau is like becoming quite close to that now daily active users is really hard to maintain I mean how many products do you use every single day like if you just look at your phone and you go through all of the possible apps like how many of those do you actually use every single day so I think it's a really harsh metric and we actually found at Mixpanel so even if you're a b2b company and you might think well I don't know I'm selling to businesses so like kind of really maintain like a daily active users especially on Saturday and Sunday and we actually found that there's like a really really strong correlation with daily use and rate of churn like rate of like people like stop not no longer paying for the product and it didn't take very many users it was like one point you know for daily actives would like reduce churn to like a dollar turn by like to like I don't know like some 10 percent is incredible and so I really stress measuring a da you and then the last thing is a b2b slant again is something that I think a lot of b2b companies make an early mistake with it's actually not just a b2b company it's actually companies that do subscription so I don't know if they're any companies that would ever think that they may offer a subscription of any sort you know like if you're a meditation app or something like that um and it's just dollar its monthly dollar turns if you're charging your customers monthly many businesses charge annually but usually most startups start with monthly because an annual commitment sometimes can be hard for early adopters is just measuring revenue churn I want to give you I'm gonna give you a slight story if you have if you have if if you have revenue churn and your revenue churn is say 7% per month right you might think awesome I'm keeping 93 percent of these dollars every single month this is awesome I'm doing really really really well the problem is is it's seven percent monthly churn and if you extrapolate that out over 12 months it's actually you're actually losing fifty fifty eight percent of your revenue in a year it's really harsh and a lot of people like for some reason this like this thing is like not mathematically intuitive to most people I don't know why it wasn't even mathematically intuitive to me early on and if you think about it like you're spending all of this energy with marketing and improving the landing page and improving the getting starting flow and you're doing all of that and if you don't have retention if you're losing that seven percent every single month you have to somehow make up for that fifty eight percent that you lost going into the new year and if you start thinking about you're like wow we're making a million dollars and we're losing five hundred and eighty thousand dollars and we're starting the year like that you now you have to do that and grow on top of that and once your base of revenue gets really high it becomes really hard to keep up and you kind of have this like really long arc of like a shark fin effect basically so it's really important to look at your monthly churn and this will be the single cause of like death but the problem is it's like the worst kind of death it's like death that occurs it's really like it's really flat growth but it occurs like eventually in like four or five years if you're able to acquire users and I mentioned this not only because you know it talked to some founders about this but this is also something that like really hurt us in the early stages of Mixpanel and we really had to figure out a way to fix this and there's all kinds of ways you can use product you can use pricing there's all kinds of ways to experiment with this but this thing probably gave me the greatest number of nightmares and lack of sleep I think ever so I really want to stress that on all b2b companies here really think through this number and think about like almost graph like okay what would our rate of growth be if churn stayed constant really important cool I wanted to go over two things one I recognize that there a lot of people here that have zero users so they're like what the hell do I measure so what if I have less than 50 users well the truth is I couldn't really think of a better idea than just basically you just have to talk to your users and I thought I'd give you a mini story about at least that makes the panel what we did when we had like no users and I had that page that just said metrics you'll drool over one trick that I used is that I had like maybe 10 or 11 or 12 customers and honestly I just I just put all of them on I am like on gchat or something I don't know what the kids use these days maybe like what's that but but but I honestly just like would just badger them on like whatsapp or you know gchat or whatever and then just ask them and I remember one time we were trying to redesign our funnel UI and we used to have a vertical funnel but we had a competitor that has a horizontal funnel and it seemed like that was a better idea it seemed like that was more intuitive for customers and so I didn't really know how to like get the data cuz like how was I really gonna find out whether they liked one of those two you eyes and I didn't have the it was just me and a co-founder so like how the heck were we gonna like do an a/b test on that feature there's no real way to realistic way to do that and so all I did was I just made like a really crappy version of the UI that I thought would be better the horizontal version of a funnel I was a kind of greyscale and I mean like a really beautiful colorful version of like the vertical one which was kind of our control and then I just asked like eleven people which one's better and I want to I want to impress this thing because Gustav mentioned that it's the it's the Delta it's the difference that matters an a/b test not that this is like a true perfect AV test but like I asked eleven people and I am got feedback in a day and ten out of the eleven said horizontal funnels and then we just made it and we built it and we never looked back and it was totally the right decision in retrospect so I don't think there's any way that you can kind of skirt by talking to customers but here's the bet here's the benefit of talking to customers even though it feels maybe it's more tedious than looking at a graph you are going to get way more info and depth from talking to customers then you ever will looking at a data point in a graph no matter how much you slice and dice and segment the data no matter how mathematical you are about it you will never get as much information and into in building your own intuition than just talking to customers and it's really critical that you do it and the one thing that I wish I had done back then was I just wrote I wish I just wrote it down I wish I wrote down tons of the feedback because I think that it would have helped my co-founder would have helped employees in the company down the road really go on the journey make them feel like they went on the journey with me rather than having our roles kind of feeling like they were so split up and then the last thing that I thought I would bring up is is to impart what I kind of said at the beginning which is that I think I think that one of the biggest mistakes that people make with analytics is being really overcomplicated thinking that they need to be super sophisticated thinking that they need to track all kinds of crazy cohorts and they need to have like a dashboard with like 30 you know panes on it that all like loads you have like Mission Control I don't think that is necessary I had that it didn't work it was really hard to run the company that way and in what also happens is is that when you build your team it will be really hard you will be confusing for your team of those 30 panes like which one do I need to care about and I think that it's we need things that are humans just need to simplify things so I'd pick one Northstar metric and I think in a Northstar metric I would choose in this case is like what is a number that you're willing to bet the company on right like if that number goes south you deserve to die if that number goes up you will like you will have like made a huge dent in the universe like what is that what is that metric and I'm not saying you need to choose that metric forever but choose it for six months choose it commit some time to it if you find out that it's actually like the wrong metric which it will be probably the first time you choose is usually your metrics are wrong and then you can change it but commit again to for another six months but choose that one number and then like if what you have to do at the beginning is basically just like print it out and put it all around the office do it because people will start to be maniacally focused on it they'll show up to meetings and go yeah but that number is like down what are we gonna do and where you can get really complex is discovering why that number is down or why it's going up that's when you can get really sophisticated that's when you can slice and dice it and figure out what the retention is and things like that and measure it over a funnel but keep things really simple and then the second thing is like don't boil the ocean like pick three to five other things I think less is more and just stay there telling even large companies most large companies don't do this and it usually is very discombobulating for them for the workforce it's totally fine to have numbers that you wish to monitor but don't focus on them awesome I think that's it and then for the people that were like slightly bored because I went over the basics I thought I would help you out by just kind of having some advanced topics for later that you could read about really these are things that like I found really useful my favorite one is this one the next feature fallacy by Andrew Chen and it's just this idea that we always think that like the next thing that we make will be the thing that will change the trajectory of the company a kie like the next features right around the corner and we're gonna be huge and I would really recommend that you read that article it's really awesome one of my my favorite one so for advanced stuff these are just people worth following reading about and can give you more ideas than what I've given you today awesome thank you Jeff Kuhn a cool all right yeah I want to be fair I don't be like if like every speaker like choose them in the front yeah yeah so that's the problem with that is that um it's soo it's so yeah oh yes I think someone asked what are like good benchmarks for like those given those five steps visiting signing up you know using the product what would be good conversion rates for any one of those steps and the the chart part with benchmarks like that is that it's so dependent on the business you know in b2b like going from like a visit to a sign up like I think I got like a four to five percent conversion rate is pretty good I think I like swap notes with Stewart at slack because we kind of had similar landing pages I think I asked him you know what was your conversion rate at slack and I think we both ended up being somewhere around like four to five percent from visit to sign up but but that said like I don't know what that would be for you know something like Airbnb where it's like getting to a landing page and then you know searching for your first place that you would want a book so I think it's really dependent I know that I know that Mixpanel published it on its blog publishes lots of benchmarks so I know that I know the marketing team there that does that and they like publish all kinds of benchmarks for like what's what should your retention be if your gaming company if you're an e-commerce company if you're a b2b company things like that social company a video company things like that so I think that you could probably go there and find various benchmarks reports but it's really depending on the business yeah [Music] so when you're going through the signup process how do you see numbers vary if you're large like creative accountant a and password doing all that kind of setup versus a single like button login with like cool guy how does that like transfer do more people to sign up if they have that like super click option do you see where they will drop off because they have to use one of those so yeah are you I think the question is um if what what is what converts better like those like social buttons where you can just like quickly authenticate like Facebook Connect versus just like standard signup is that right yeah like all of these questions they require like a lot of specificity because it's actually like really unclear and actually and in that case actually don't know I truly don't know and so I think the way to answer this question is to find friends in tangential industries that like don't compete with you but basically do that and then just be like hey what did what have you what is your conversion rate have you tried something I I don't actually think a lot of those numbers are like secrets in reality right unless you I mean you can't go to your competitor and ask that but like most of these things are not like super duper secret type stuff and like you know they could either give you the information and save you you know the six months or the or they'll just like say or you try it out but it's it's just it's sort of strange to keep it a secret so I would rather give you ways that you can discover that I would just ask friends that have like have a startup or even if they work at a big company sometimes they're willing to share it yeah so question is Airbnb ease landing page was super simple and how do you balance like I mean what if a lot of people just know about Airbnb and then just know what to use Airbnb for and then that's why they can keep it kind of simple versus like having something more complex a good example of this is like Craigslist is crazy if you go to like the front page of Craigslist it's like absolutely nuts it's in like even Amazon if you go to amazon.com right like even though Amazon has like huge brand recognition you go to amazon.com and it's like whoa there's like million things going on at once right so how have these companies found the balance how have they found how to deal with this complexity and and the truth is is that I find that it's it's important to have like a guiding principle like what is it that you're trying to actually optimize for right if in the case of YouTube let's take YouTube as a good example right YouTube could just like be like Google they just like what do you want to look what do you want to watch right they could do that but they don't and so I think that the question really is is like what is YouTube what is the team at YouTube trying to optimize for are they just trying to get you to like watch a video are the is Airbnb just trying to get you to search for any kind of hotel or a hotel but Airbnb and so I think that like the team has to develop first a hypothesis of like what matter what do we think matters and then from there it's important to experiment and try to figure that out it turns out like it's it's not - again it's about the Delta of like what people will actually gravitate more towards like if you make a really simple experience but like people are like really confused because they don't even bother to do it or you or you ask them then you kind of have like new information and then I think the key is to find the minimum number of things you need to add to get someone started and you know and I think I don't think there's anything you know I'm sure Amazon has tested that front page like crazy and they found that it's like quite optimal yeah [Music] the can it depends on the product I'll give you I'll give you an example in mixed panels case oh sorry the case yeah the question was our video are like one to two minute video tutorials effective there yeah so I think that it's it's complex we did a video tutorial video and our flow for example and you know like I would say that I would say that like a lot of people didn't watch it truthfully and inner and this is what I mean by experimentation like a lot of people didn't watch that video Vittorio me actually I think eliminated it you know the first three steps but what we did find was this interesting thing that occurred as a result of looking and measuring things which was that um well we have a feature in mix panel where you can like see every action that a user takes and so I would literally just like go to every user and then just like watch like a creepy person just like sort of watching like everything that like someone would do in the product and eventually when you do that like when you do like hundreds of those and you've just watched hundreds of those you start to develop some sense of like okay you what you develop is a an idea a hypothesis and all of these things like all of these things require art and science and I think that companies that take it too far they like take take it too far to science end up getting it wrong they end up optimizing for like weird things like you could always make a button more yellow and bigger and like but that doesn't meet that doesn't make users happy eventually and so anyway to answer your question what I found was that lots of people were just clicking randomly on like all the myth side menu options and my hypothesis was well they had an integrated mix panel yet so there's nothing to look at so what the heck were you doing just like clicking on like funnels and retention like why would you do that and my hypothesis was they were just curious like what the product did what is it that they did and so we did this a B test where we like had a control with like nothing you know just the regular you know clicking around seeing like some image and then we had another one that had like videos and then we tried to see like what caused people to like do their integration their first integration what was the conversion rate and it turned out like the video like super one actually we didn't even a more simple test than that I think we just emailed people a video and said watch this video super basic didn't even bother to implement it on the website and had an amazing conversion rate and then we decided to add videos all to all those tabs where there wouldn't have been anything anyway and so that's why I want to say like I want to stress this it really depends and what matters is that you run the experiment to develop a hypothesis ie the art not the science and then measure it which is the science to find out whether the thing that you thought of was in fact right the a percent loss in revenue over a year yeah so what these are slightly two different things one is one is like user retention and then one is like dollar churn and so in the case of dollar turn like the reason I bring up dollar turn is that even like you could totally have a company where nobody uses the product and the company continues to keep paying for it right but I find that like most companies are like pretty incentivized to cut their spending and employees or incentivize because they lose it in their budget and so with respect to dollar turn you can have lot in a b2b company for example you can have a lot of use you can have you can have even one user using the product but the company could continue to you continue to spend money on it and like you might see like a forty percent retention rate like might be okay right as long as the company is like continuing to spend but losing 40 percent of your dollars in a year is really hard and not good retention so make sense so those things are not exactly linked they're not directly linked yeah yeah yeah like to give you to give you kind of a benchmark like a chat app that's like often used a lot we'll have like 60 70 percent retention like those are some of the most retentive things like something like slack would probably have like eighty percent retention it's kind of like unbelievable whereas um you know like a product like mix panel when we first started out like the first year I think we had like 30 40 percent you know retention and that's not that great but who's their first few years yeah yeah so a question is are these metrics can they be used for hardware startup yeah I don't I don't see any reason why we worked with jawbone for a long time and we work with Fitbit and most hardware does your cart hardware company have a software component to it okay so it's a hardware company with subscription model yeah yeah absolutely it may be more challenging as a hardware company to track but it's like sometimes but it's totally doable most hardware is like connected to the internet somehow so I don't see any reason why like a lot of the scooter companies for example they're like all connected to hardware and they have a software component so there's really no difference yeah so question is the first page was like trying to target like various kinds of sizes of companies and then like the second landing page of Mixpanel like in 2012 basically didn't kind of disposed of that and it had something very different it was marketing like features and stuff and how do you figure out like what to do basically from that point to the next point yeah so I think that the basic gist is that so you can clearly see like I had no idea who our customers were and then after like a few years you just you talk to users you ask them like what size of company they are if they sign up if you want to we didn't do that but we just spent a lot of time we just it was so obvious that like everyone that was signing up for make spell at that time was a start-up so for a small in our in our case would be we call that a small business I think like took like maybe no longer than six months to figure out that we were not gonna sign up a large company yeah we just didn't have the product it just wasn't there we hadn't worked on it long enough so the only people that were willing to basically try it were companies that could take the risk to try it and most of those companies didn't have anything anyway so it was even easier is it easier to target companies that had nothing and and had had we're willing to bear risk and then most of the time I would always employ this trick where I would just I would charge the money for it and then if they said no I would just give it to them for free because the feedback was more important but I always wanted to see if they would pay for it and so the only people that are willing to do that are also small businesses turns out so if for us it was just like it kind of like hit us it slapped us in the face it wasn't like this hard thing to figure out yeah yeah so we did it to be clear we weren't necessarily building things to bring new users on oh sorry the question is you guys built forgot about you didn't you just you decided to not build forgot about password how do you know when to build quality of life features versus features that just acquire new users yeah so I think that we didn't we didn't we wasn't so much that we only built features to acquire new users we built features to acquire new users and to keep existing users so we were very focused on like parts of the product that would actually make a difference right that would be clearly differentiating compared to all of the other Possible's possible options out there and like one of the things that's like not different is whether you have forgot about password right that was like in the bucket of things that like maybe don't matter and so I think the question is like really like when does it become a problem enough for you that it's worth building right like if you if you're finding that like it turns out that that the number of requests that you're getting for forgot about password is like increasing and then manually doing this process it's like annoying then start then you should just build that feature because you know that if you don't build a feature they can't login then you can't retain them and then you lose them so it becomes this like flywheel effect where like the only way to retain them is to do that and in fact even though Mixpanel didn't build forgot about password for like maybe you know 12 to 18 months we went eventually we found that people like people would try like multiple times I was looking at that stream of every action that a user would take I found that like people were like going to the login screen like five times and I was just so confused it's like why are they going to the login screen five times I was like oh I think it's because they can't login like some people don't even hit forgot about past well just like try try try and I'm sure you've all done this like with a bank or something like I'll just do it tomorrow you know you don't even bother right and so it turns out that even that happens and so we went so far even though we didn't do that I like to think that like we really made up for this later on we went so far as to like we found this like clever trick that Facebook used which was that when you failed to log in to Mixpanel like three times on your third time we just send you an email that says you know we just sent you an email just go to your email click the button you'll log in right now don't worry about forgetting it don't worry about changing your password just like login and it turned out that like that had this like great retentive effect because we were really focused on retention at that time yeah okay couple more questions yeah how do you gauge product market fit how do you measure product market fit so I think there's a lot of people that probably have already will talk about or have already talked about like you know people will say very and I think they're there all right so I'll give you a kind of I'll give you a quantitative slant versus like a qualitative slant some qualitative slants or things like when people are willing to spread you to their friends that kind of thing the quantitative version of this is really just measuring your your attention like how many people it's really measuring like da you it's really measuring the percentage of people that come back a week later or 30 days later it's the number of people that like become almost like they like need your product or they like feel like life would be a lot worse and I think that you can you can totally measure that by how they use it and so I think the quantitative measurement is basically finding like some benchmark some sense that for your industry maybe that's like thirty percent and just remember like that benchmark is probably like average and your desires to basically be above average and so if you can like significantly beat that benchmark then I think that you've like really you've you've done better than probably product market fit I think that's one thing I think another thing is like you can try to measure like overall like repetitive use like frequent use in a day so like you can make pretty harsh metrics you can be like for example with da you you could say I don't want it to just be daily active user like they did one thing I want it to be that user has to have watched five videos and then and then then and only then or do they count as a daily active users you can be particularly harsh about that metric right someone coming and like just see that's why I say ma use kind of a BS metric now because you could come and watch one video and then never come back again and you that count and that's not like a very healthy business so I think that find ways to be harsh about your North Star and I think that you'll find that that you'll find that you reach the park market fit yeah yeah I think with Mixpanel it was like 30 40 percent retention was really really horrendous and we totally didn't have product market fit yet it's like a Peter teal question what's one piece of conventional wisdom that you think is generally wrong about startups in general gosh that's a hard last question it's like everything else is like fine um gosh I think I don't know I'm not I I need to think about it but I think that one thing that has like become pretty clear to me is that I think I think after having done Mixpanel for about a decade I kind of figured out that I tried to impart it on you in in in the in the talk today which is just that it's it's much better to just like I think that we tend to complicate things as humans and we don't we tend and that comes that comes through and like everything for example like we we tend to have we tend to like make a version one of a product and then we get to like version two and we're like okay we're done we like we did it we solve the problem and then like often a founder will like kind of get distracted and then go and like work on the next product on the next thing and they haven't yet figured out like their retention is probably not great or they're operating at negative margins or like all these things that same thing even happens with numbers like we pick our three metrics then we continue to get more complex as a business and then we add the next 20 things and we all we do all of these things to like in in hopes of like feeling like we control the situation better and I just think that those are all like distractions and mistakes and I think it's really hard for the average founder to actually just be focused on something boring we tend to have like a TD or something and like we just just I decide to get very easily distracted and I think it's hard to do something do the same thing for four or five years because we don't like feel challenged anymore and I think that that's a huge mistake that founders often make and I don't know that anyone say oh that's that's that's like totally wrong Suhail but for whatever reason like every founder repeats it so there must be something I don't know something there must be some kind of wisdom that we all think this is like the right thing and I made those mistakes I made new products that I shouldn't have I met too many things over optimized things that didn't need to be optimized so yeah that's really it cool thanks [Applause] you

Channel: Y Combinator

Views: 95,317

Keywords: YC, Y Combinator, Startup School, Suhail Doshi, Mixpanel

Id: MABmQhOlmJA

Length: 59min 3sec (3543 seconds)

Published: Wed Sep 12 2018

Reddit Comments

[u/midael]

00:00: By founder of MixPanel ( I am data driven ) - heavily focused on SaaS / mobile app type of product.
03:47: Questions you need to answer in first 24 months: 1) Is my product easy to understand, 2) Is it easy to get started with my product, 3) are people coming back to my product?
05:20: ‘formula for measuring success’ for SaaS / mobile apps.
06:42 When you over complicate decision making with with too much / wrong data → decision paralysis
10:00 “Back Button is the enemy”
14:30: If yuo’re B2B company -> make separate PRICING page. This seems to be a GREAT way to see if people who landed on your site are actually interested (e.g. if they try to go find more about pricing)
14:50: With B2B → A/B testing website much harder. Don’t try to force it.
17:00: When you have few users, don’t waste time with stuff like ‘Forget password’ functionality. Just have them email you etc.
19:20: How to measure how easy your product is to get started with: ...
21:00: When you’re a startup, don’t optimize for fake actors, etc. (e.g. text / email / etc. confirmations just drive away potential users)
22:00: Your initial user experience: This is never ending cycle. It will never be done. Continuously iterate.
23:12: Are people coming back to my product/website (mostly latter for b2b). This is greatly undervalued metric.
31:00: Early on in b2b: When you lose a customer, you *really* want to hear them tell you all the painful details as of why?
31:54: When you have a few users, just put them on IM or something and stay in touch ALL the time. Dont worry about structure etc.
34:47: Pick one north star. MOnitor 3-5 metrics. LESS IS MORE. A frequent mistake is having too many metrics that you monitor.
40:00: Recommended reading: https://andrewchen.co/the-next-feature-fallacy-the-fallacy-that-the-next-new-feature-will-suddenly-make-people-use-your-product/