Gustaf Alstromer - How to Get Users and Grow

on company updates please be honest there for you and not for us and if you make them clearly crazy like you know we're never ever launching we're launching in four million years we'll get the hint so don't do that there have been a lot of questions about the graduation requirement which was that you had to submit nine out of ten weekly updates it this course was originally a ten week course and things changed around a little bit and it's actually more of a nine-week course now and so and it was a little bit difficult for some groups to get going because of our little snafu in the beginning of the course so so here's the deal eight out of nine updates is just fine but we're also going to extend the class oh actually really two weeks beyond the last lecture so that there will be plenty of time to do ten updates if you so choose and you choose to do nine out of ten but eight out of nine will be the absolute requirement for consideration for the four that the $10,000 grant so we're will be flexible but make them good updates make them real updates another reminder those weekly updates are due 11:59 p.m. every Sunday Pacific Standard Time there's been some confusing confusion about that as we've written it a number of times but please recall when that is a questions again about anything about about your moderators about your groups about your updates please send to start-up school at Y Combinator comm and my very last note is a reminder to launch if you haven't already the only way to find out if you've made something that people really is to have people really use it so get your products out there if you haven't now I'm going to introduce our first speaker who is uh my esteemed partner at Y Combinator Gustov astronomer who in 2012 was part of the group that built the growth team at Airbnb and if you know anything about Airbnb you know from 2-12 2012 for the next five years they grew an enormous amount this is something that Gustav knows better or at least as good as anyone in the entire world so I'm thrilled to have him talk today about growth thank you very much before I begin thank you so much to Steven Jeff adora bill for putting all this together so we can be her and do this today I'm going to talk about growth my background before joining YC last summer was being one of the first person on the growth team at Airbnb and then growing that team from I think we were three people in the beginning this is how many people were in 2015 we then grew two hundred and hundred and twenty people or something like that almost everything I'm talking talking about today here I owe this group that's sort of what I learned all this stuff some of the best ways to learn to work on user growth is to actually work on a growth team and I was fortunate to do that for for almost five years ago Airbnb so before I start one of the questions that might be obvious the answer might be obvious for some people but not for everyone is sort of like why should we do this why is growth important and I I'm surprised how often I actually get this questioned but it might be worth thinking about it for a second so when you make a product some people think that if you just put that product into the market people will come and people will start using the product in my experience that's not really how to work world if you start a startup or a company that's aim to be a start-up growth is important because that's actually what a start-up is if you make something that has the potential to be really big then making that really big is sort of what starts all about there's actually post that Paul Graham wrote that it's called startup equals growth where he talks about specifically why growth is so important for startups and why it's not important why not every company's to startup and why growth isn't important for every company but for startups it's it's really really important and I'll talk in the second sort of like what it means to be intentional by growth but but the first question is you kind of have to answer yes to is sort of like I have a startup I want to grow now how do we go about doing that so who's this talk for in my experience those kind of things you work on one and work on user growth started with consumer companies those were the companies that started embracing a lot of the tactics and strategies and technologies that growth teams were doing now I would argue that almost every company that sells anything online or get users online should be or could be using a lot of the skills and knowledge that I'm gonna talk about today so you don't need to be just a consumer company you don't need to be just a social network to apply these things these are applicable to a wide range of companies if you're a b2b company and are excellent at growth you have a massive advantage to other companies in your space because they are probably not going to spend as much time doing growth as you are now there's one type of companies who this is relevant for but too early and that actually might apply to most of you right now and this is what this talk is dangerous because working on growth the wrong time in your company's history can be detrimental for you it's sort of like when you are kind of off to the races a little bit too ahead of time so the most important thing which is was a lot of lectures here start school is about is how you make something people want and how you actually find product market fit if you apply a lot of the things I'm going to talk about here to a company that hasn't built something people want or haven't actually found product market fit really bad things happen you will very often then this is a very common graph grow like crazy in the beginning but then when you realize that sort of like that fuel they had isn't fueling anything it's just fueled kind of evaporates you come straight down so that's why it's dangerous and now I'll I can go into more details about that in a second of applying some of these things before you actually have product market fit so the first thing I'm talking about today is going to be around measurement around proc market fit because that's actually the most essential part for for continuing growth now do growth teams have impact or are growth team sort of like like how do you know that growth seems actually matter how do you know that that these things man I'm gonna talk about that as well like my experimentation but before I get there I want to tell your story from Facebook and this is not a story I came up with myself I found it on one of the talks from Facebook and I think it's so essential for describing the level of impact you can have if you are applying these kind of skill sets to a company so this is the timeline of Facebook from 2004 to 2015 is sort of their growth story some of this is public some of some of this wasn't the one thing Facebook had they had an excellent data science team from the early days they were really good at measuring sort of like forecasting how big face was going to be so back in 2006 2007 when Facebook had just started the girth team they had a bunch of data scientists doing this sort of forecasting how big will Facebook eventually be and they looked at all the type of metrics at facebook.com/ and and they came to conclusion that Facebook will be roughly 400 million by 2015 that was their forecast by using all the data they had now we know that wasn't really true they actually started growing really fast in 2008-2009 and this actually as a result of something that the growth team came up with and no one here had an idea what make Facebook growth so fast in 2000 2008 roughly she's in 7g a opening to high school students yes they did but that wasn't as a big of a growth newsfeed was 2005 sorry recession recession did not know 10 friends and yeah that's good actually it's sort of like the girls story all across a Michigan could have been that translations is the answer so this is surprising for many for many of you might maybe but there's actually core part of many growth teams is that you want to make your product available for as many people in the world and Facebook star is sort of hitting the limit on people who spoke English and then assume that people just because Facebook was in English and so that the content were in local language and Facebook were in English it would continue to scale that wasn't the case so they implement this new platform that automatically translated Facebook into hundreds of languages and and the growth to be accelerated now the same thing started happen again I think they call this lockdown boom they were doing very important things in 2010 2011 something very big was happening at Facebook how to make some really large changes to some extent also driven by sort of like using data to figure out these things any idea what what am what happened in 2010 it was a really big acceleration mobile I heard mobile so so actually what's happening is that the data science is a Facebook we're forecasting Facebook to be about 700 million users or something like that and because most of those people using Facebook on computers they hadn't intended this massive shift that was happening in 2010 2011 which is people getting our smart phones and Facebook switching the entire team they I even have large training classes for engineers just are learning mobile so so this was the big change to happen then and then actually one more thing happen sort of like in 2013-2014 where the Maeda forecast their Academy kind of not hitting the the ceiling of growth again but then something again happened any idea what happened so 2013-2014 Instagram and whatsapp could be good answers that's not what I'm looking for here messenger is a good answer is not what I'm looking for so Facebook actually running into the ceiling of the Internet the number of people actually the one line so this started this thing called org which was intended to get more people in line and they went to carriers and they gave basically work with carriers to get people on line to get free facebook and this actually were very important for sort of the continuous growth of Facebook now what can we learn from this well we can learn that the initial forecast of Facebook's growth was about 400 million people but Facebook today is a 2.1 2.2 billion user platform so it's a very large platform and the forecasts were wrong now the forecast wasn't wrong it just didn't intend and taken account everything great that the company was going to do so if there's anything we can learn from this is that if you're intentional about growth and you're really trying to sort of break through these forecasts in these ceilings you can grow really really fast so this applies to every company and I showed this graph to everyone that you own air B&B like Aaron leak join the growth you midden be and try to get them to think in the same way so natural adoption which is sort of like that initial forecast of how fast your part is growing without any of that work is always going to eventually slow down but if you do the things that you do as a growth team you can't sort of like continue to push push the growth of your company now I mentioned this before having a growth teams before you found parking market fit isn't really useful so let's talk about product market fit is it so the important term but it's hard to hard to so exactly define what it is so the one way that I think about measuring prime market fit is is these two things first you find the metric you can do this for yourself after this talking find the metric that represents the value that your customers get from your product and then you measure the repeat usage of that metric it sounds pretty simple and we'll see see if it works you can make this kind of a table if you want where you have the company yeah the metric that represents sort of the the value you should get from your product and then you have the ideal frequency so let's take let's take care of MB so you get value from a MB when you are booking and staying so if when you when you're actually traveling MB that's when you find that MB is really valuable it's an amazing amazing experience unfortunately people don't do that more than annually so if you would want to figure out the product market fit of it air B&B you can always ask people after the first day but you wouldn't really know until they booked again when I come back and book again so because the booking cycles for travel is so slow annal will be a good metric here let's take facebook so facebook if you come back to facebook being involuntarily come back to facebook as an active user you're doing that because you find value there's something that makes you come back to Facebook or Instagram or anything like that and the question then is how often should I come back well probably daily or hourly or something like that they start this they started looking monthly and then they went down to daily I think there might be even going more than that let's talk about Gus though what's the product value as a customer of Gus though whether you get out of using gussto well when you run your payroll which is sort of what Gus to do for your employees that's sort of the one of the best ways to sort of measure the value you get out of Gus though I know often you do that well you run pay low every other week or every month I'm not gonna go into details of all of this but for lyft it might be rides for checker it might be background checks for striping might be transactions now for your company you should figure out what is the one metric that I can measure easily every day or every time it happens and what's the ideal frequency through which that should happen if you can answer those two questions you can make a cohort analysis of your own company right now and you can start trying to figure out if you have product market fit so then you want to measure these things so you take a graph on one axis you have time and the other one you have have the metric that we just decided and from that you can try to figure out if you have a product market fit so this company is measuring this on a weekly basis the ideal use case of this company is using this product on a weekly basis so every week here there's a dot which is the percent of people that use this product on week zero the first time they used it they used it again so let's take startup school example so I joined startup school and in week one only 60 percent would come back to school and then in week four only thirty percent will come back well that would suggest the start of school we wouldn't be a very good product but that would be a good way to measure if start school is actually a good product people come back to all the content they were creating I'll take another product this is a great product this is kind of like a normal product yeah a normal product would look like because almost always you have a little bit too many people in the beginning of the product and then he will go down over time but it will sort of flatten out where you keep measuring these these events of your metrics this might sound technical but it's actually just a representation of how many people each dawn is how many people does using your product for that metric that you decided on that on that time window most good products flatten out let's look at some examples here so these are examples that based on payment retention so this is a company that you all heard of that retains 10% after one month and then 12% after 12 months it's sort of like unusual who can that be stripe is this too hard this is Shopify so shop if I have this product where a lot of people sign up not that many people continue in month two but then they kind of stick on forever so what do we know from from from from this well if if this curve would go down to zero or sort of keep going down then shop at five would be a bad product it wouldn't be something that actually found product market fit and should have been working in growth in this case they have good good product market fit and they should be working on growth they should do all they can to continue work on growth they should do something about sort of like the initial onboarding here where they lose a lot of people in the beginning but they should be working on growth here's another company 50% retention after one month and then 10% after 24 months so similar to stripe but like or similar to Shopify but they continue to lose users all the way down to even 24 months maybe there's some flattening out at the very end here but even 12 months after people start paying for this product there's they every month fewer and fewer people pay for the product is this product market fit hard to say but but it's not an obvious case that they have found product market fit so this is blue apron here's one that's pretty good so 70 percent retention after 12 months and then 30 percent after 7 years this company definitely had product market fit any ideas of which company this might be you're all using it Amazon that hurt Amazon play wasn't the right one Apple no this is Netflix so 70% of people that start paying for Netflix pay for Netflix 7 years down the line that's definitely a company with market fit let's just spend every time everything they can work in growth so raise your hand if you're if you're measuring your retention right now not that many people well there are other ways to figure out retention if you're really small you should go out and talk to users you should ask them questions such as how would you feel if you can no longer use my product and you should sort of like say as close as you can to your users this is by measuring retention like this it's very hard when you have 10 users when you have a thousand it's easier but when you have 10 users is very this is not the way to do it you can actually go and just talk to all any person so there are other ways to measure it but you sort of have to know that you have a product that's retaining otherwise you shouldn't be working on growth because you're going to be end up burning cycles and things it doesn't matter okay so a lot of people might be wondering how does growth and marketing relate to each other isn't that is it the same thing or how should I think about it historically the way you had a company 20 years ago is you had a product team that made the product and then you had a marketing team or a product marketing team the marketed the product that's nothing's used to work and a lot of the sort of like hierarchies and companies or sort of organizations are still based on this idea that you have a product team that's separate from the marketing team and these are different teams and different skill sets and and that engineers work over in the product team and then the marketing people work over the marketing team actually it's not how things work anymore so the way you should think about this is that I'm gonna come to this in this in a second is that there is three times three different types of people that can organizations that can drive growth there is what I call the product growth a growth engineering this has different names this is effectively product managers engineers data scientists and sometimes marketers that work on growth but they work on growth using technology so they're actively changing the product to drive more growth and much of this work here is about people that already arrived here product but haven't really found the value of your product yet and they're changing the product to make you grow faster commercial optimizations fall into this this group some of the growth channels actually fall into this group now there's this big other group here performance marketing which is effectively Google and Facebook marketing which is also super technical and super data-driven I would argue that these things are very very very similar so five or ten years ago you would go to a company you see these being very different groups maybe different different floors in the office now they in my opinion should sit together so engineers should sit together with performance marketers and vice versa because they actually are doing very similar things now there's this fifth button fifth thing should be number before three here which is brand marketing brand marketing is sort of like the hardest to measure the the hardest to measure a type of marketing where you're not really having a direct response you're not having someone directly giving you feedback on how that marketing perceived you kinda it can't even measure that very easily so this is not something that started should be doing in the early days my opinion start should not be getting my brand marketing entail way down the line when sort of like they hit the limits on these two things so startups should be doing these two things and there's some qualifications to which you can do this but almost everyone can do this if you're a startup you have engineers you can do product growth or growth engineering or or things like that all right let's talk about that first part Proc growth so your product is a funnel what does that mean your product has many many many steps between the first user and the person sort of like completing what they're trying to do on your product let's say I'm on an ecommerce store and I'm trying to buy something there are many many steps in that funnel and what the product growth team is working on is funnel optimization or conversion with optimization so if you look at what we did at Airbnb on the growth team many of those growth teams were we're conversion rate rate optimisation teams they're working in a specific part of the funnel so the funnel could start with SEO we could start with performance marketing it could start with referrals of morality but it would jump through a number of different steps say sign up would be a very common step or if you're an e-commerce site payment conversion or buying conversion all of these steps are things that as a conversion rate optimising part of the growth team should be working on and these are the some of the easiest things to get working out and I'm going to be a little bit high level here because to go into depth about every single area here is going to be we can go on for hours so some of the sort of like good ideas of areas to start working on for conversion realizations one of them is translation so if you have an international product it's not translated you should be thinking about that that immediately drives and drives more people to sell using a product the second thing is that occation so most of your products probably have some idea of user account so you can sign up to your product and you can come back and login to your product you'd be surprised how much how hard this is and from Airbnb I know we spent many years working on just authentication signing up and logging in it sounds so simple but it turns out that's a very fragile moment of users sort of like flowing through your product that you can always continue to make optimizations if you go to a B&B and you go to Pinterest today assume that whatever is there in terms of authentication is the most optimized version these companies spend enormous amount of time optimizing sign up conversion another big area for commercial optimization is onboarding so when I come to your product what's the first thing I experience and sort of like how do I work what can the product do to bring me towards the value of the product is quickly as possible those are things that you're working when you work on converting and then another big area here is purchase conversion sort of like when I'm about to buy something a product and actually can take the final decision around what you do there's so many things you can do here alright so then there is something called growth channels so what is it growth channels your chance is sort of how people discover your product now there is when you're a small company and I don't know how how big you guys are but let's say you have less than 50 users you shouldn't really be thinking about growth channels even if you're less than 500 users you probably shouldn't be thinking about this because it's too early but the things that you do when you're small that don't scale have the word don't scale in it for a reason they don't actually scale and there are very few companies who kind of grow big without growing on one of these scalable growth channels they're not them any platforms and the channel slash platform can be used in the same way they're not that many things that are really really big in the world on which you can build a large company so let's talk about what those those channels are so the first thing here is basically you think through the behavior of your product and let's talk about the first one here which is if the way I discover your product is a rare behavior where people use Google to find a solution and this is actually how vary a lot of products in the world are being discovered if you have something through which to answer the question around what you do once a year you go to Google probably if you're building a company trying to answer that question you should be an easy B in Google so a good example here might be buying a house you aren't buying house more than once or twice in your entire life which means when you go and buy a house you're probably going to go to Google which is not surprisingly there's something like Zillow or red fin or all of those different sites that allow you to buy homes online are entirely optimized for SEO and and sometimes paid search if you are using something every day you're not going to go back to Google every day you're gonna go straight to that product you're gonna open the app on your phone you can go to the straight to the website whatever it might be you're not gonna go to Google anymore just kind of figure out which of the different things I'm going to use and now their Google is not the only search and then there are other search engines sort of like that you can optimize for us well but Google is the one that still matters next behavior do people of my product already share the product using word-of-mouth so if that's the case then there are lots of things you can do around morality and referrals so you can grow your product by kind of accelerating that behavior from your existing users and you can either incentivize them with referrals we were you getting paid or you can do it for free and with with with using virality techniques that's having more users on my product actually improve my experience so what I mean by that well if I am building the next LinkedIn it makes sense the product is not as valuable when it's just all of us on LinkedIn but when there's a bunch of more people and companies on LinkedIn it makes sense that the value kind of increases well in that sense you should absolutely continue to do virality because every single new user is sort of opportunity to bring in more users so there's another way to think about morality and then this is that common question I asked companies in sort of the early days of YC can you make a list literally a spreadsheet of all the people in the world that would use my product let's say I sell to buyers who decides to doctors offices they sell to doctors offices well I can probably make a list of all the doctors offices in the United States or in California it wouldn't be that hard it's totally possible so I would find a way to make that list and then would go out and do sales and this is surprisingly something often something that you should start with if you can make the list if you know those people are you should go on new sales and the last one here is does my users have high LTV that's me use a hat that's does it but do I charge enough for my product for it to be valuable well then I should definitely go and do paid acquisition for example Google and Facebook I shouldn't do acquisitions or paid ads unless I actually and charging for my product all right I'm gonna go through each one of these sort of like it's not going to be possible to go into super deep detail for all of them but I'm going to go into a little bit deeper into some of them and not the others all right let's talk about referrals so I worked on a referral program at MB for very long time the way to think about referrals is sort of an engineered word-of-mouth so if people are already talking about your product referrals is a way through which you can engineer more people talking about your product one way could be just making it easier another way could be by using financial incentives so in the mb referral program we had a financial incentive where as a referre I would make $20 for every users that would sign up in travel credit and the servery I would I would get $40 as a new user of MB in travel credit and we would we would kind of like start with that principle and then try to get as many people what we call through our first phones as possible so you meant you notice I'm I use the word funnel here again so every product is a funnel and even there a full product which is a sort of a product of itself inside the RMB had its own funnel so I'm not going to go through all of these details but the way you think about something that's kind of engineering and product led is you break it down into into different steps and then you measure every single one of these steps and then you kind of measure the conversion rate to the very end the first step here is weekly active users that saw the referral program so how many uses saw their full program if I wasn't measuring this step I wouldn't know how many people did that when we started measuring this step and I've seen this with many other companies when they who have a referral program is that a small percentage of your active users see their full program well how could you be expected to use it if you can't see it so we started measuring this in the early days of a B&B turns out that there's a lot of opportunity I'm just telling people about you having your full program but then there's all these opportunities to optimize to through that funnel so there's sort of people sending invites how many they sent to the conversion rate to new users into new guests and finally to them booking their first-time nights here's another new display and this slides going to be available online afterwards you don't have to I take notes and photos we kind of separate the funnel into even more detail and we would continue to optimize this for years like this all matters a lot and we would continue to optimize for years I'm going to go through one example of sort of like one of those conversion rate optimization where did for therefore all program so here's the referral invite email so if if someone else if I invited someone one of you guys you get one of these emails and the email would say Gustav ahlstrom MIT of Jerry B&B Gustav sent you 40 dollars on your first trip on the urban bee you can book rooms Beauvoir just sign up at 25th on may 2018 and then there's a button to call accept the invitation and then there's a photo of me and my name looks pretty good now this email is a result of dozens of experiments nothing here is random everything here's for a reason let's talk about that the first one is the subject line the subject line has my name if it's sent to any of my friends that make them more likely to open it the sort of like a headline of the email have a clear value it's very very clear what this email is about you'd be surprised how many emails that don't have a clear headline and the clear values are what the email most about these email is about that I sent you $40 on your first trip on this website called Airbnb this which is sign up by May 25th 2018 is urgency you should sign up by May 25th 2018 that's the urgency that makes you actually go that increases their chance of people who see this email actually go and do it the name here except imitation it's the sense of exclusivity it's not it doesn't say sign up for reason because sign up anyone can sign up by accepting an invitation sends an idea of exclusivity and finally it has my name where I live and how long I've been a user of Airbnb with my photo that's really strong social proof that I endorse this website I endorsed this product so the way you think about this is sort of how you think about all a conversion rate optimization it's a set of optimizations pay growth this is one of the areas I'm not going to go too deep too deep into the the options here the things that matter for pay growth is that you shouldn't do it unless you have revenue too many companies are buying ads when they don't have any revenue or know how they're gonna make any revenue that's the big mistake you shouldn't be doing that if you have revenue there's a couple concepts that really matter the first one how much money am i paying for each new users that arm acquiring it's called custom acquisition cost or CAC what's the lifetime value or payback time of the users that I've acquired through paid growth so what that means is what's the longest that I can with some level of accuracy forecast how much these users will be worth if a users paying $10 part and then we can sort of have this cohort analysis that we had in beginning with the attention we can figure out exactly how much our use is going to be worth and if my customer acquisition cost is lower than my lifetime value then that's good that means you have a great payback time if if you only pay $20 for a user that means if you make $10 per month you have probably around $20 at two months payback time that's more complicated in that but it's sort of like a simplified version these are the most important concepts to keep in mind for for pay growth attribution so this matters if thinks it complicated if using both Google and Facebook if you're getting to some some level of scale you need to understand what it means to activate a new paid user to a dollar that you spent I'm not gonna go too much into detail of what this means but but this is something that you should learn if you're paying time on paper oath and finally in my opinion there are only four channels to sort of matters at scale Facebook Instagram Google and YouTube these are the very very large paid growth channels through which many companies are actually built on these days sort of like one of the sort unspoken truth in my opinion is that a lot of the free channels of growth is going down and the and the pay channels are going up a good example of that is that this hard to grow on the Facebook newsfeed unless you're paying Facebook money the same is true for Instagram it's sort of changing always it when there's a new platform they often kind of promote free usage in beginning and then they start to monetize that and that sort of was happening for both Facebook Google and Instagram today all right search engine optimization so search engine optimization or SEO some people might say well this is something of the past in my experience it is not it still matters because as long as we go back to Google to make our decisions about what we want to do in the future this matters Google is probably one of the largest websites in the world it's a big deal the one thing you should know about SEO is that there's a difference between what you see on a website and what Google see so Google can't view images Google can't view JavaScript very easily so if you go to Airbnb and you see all of this just remember that Google can't see this so if you're trying to communicate to someone who's searching for Stockholm on it on Google or apartments in Stockholm this is not what going to be delivered back to Google Google is going to see a bunch of lines of text through which you've marked up in your code and the Google have hopefully indexed if you done the right thing and if you haven't then it's kind of like your fault so they're a bunch of basics food that you can do with SEO you just have to get right the easier thing you can do is to run your website through an SEO browser and try to figure out if you only saw this would you understand what your practice is about so using clear language not in JavaScript so you describe what your product does is the most basic things you can do for for SEO there's two different areas of optimization for SEO and and this is as high level as it gets so at MMB there's a team of 20 people working on this of which 12 or 13 are engineers this is a really really big deal but but but this is sort of the high level there's two types of after my stations for SEO there are things that you can change on your website and there are other people in the world of linking to your web sites this is the two main levers that you have an SEO for the on-page optimization the right way to start is not to start with like a list make some changes the right ways to start with the strategy what am I trying to rank for what keywords exist on Google today that I might want to be the number one result for now to do that you have to do what's called a keyword analysis or keyword research you have to try to figure out what are all the things that people are searching for that might relate to my product and then hopefully they have some amount of large volume that they're searching for and then I can try to say well these keywords that have sort of a medium to high volume and they're not that much competition around these keywords those are the ones I want to rank for when you decide that that the other areas of impatient of my stations now are easier because now you know we're trying to to rank for at scale but I've seen this as small scale to SEO at this point is about SEO experimentation which means you can do the set of best practices in SEO it might take you to be a decent size company but if you want to become a massive company a really really large company that's growing through SEO you have to use experimentation to make those decisions the other thing that matter here is off page optimization who's linking to you so there are lots of tools you can use to figure out who were all the websites online the links to you what's their sort of domain Authority are they actually authoritative on the web to have something there's a huge difference if I will link to your website and if you have x link to your website it matters a lot in sort of how in google's perception of your of your of your site so one surprising way there may be is that we had a lot of press a lot of people in the world from media would come and write about Airbnb and that was surprisingly important for off page optimization a few people riding my you either impressed or in other areas that's actually a great thing if you don't have anyone writing about you well you're not gonna have them any links because the web has changed the aren't that many links anymore kind of it's not like everyone have a a website these days they'll link in to each other so they said things have changed you have to be strategic about who are you getting links from easiest way you can do is whenever you get written out up by anyone presser or whatever just ask them to link to you it matters to you last thing I'm talking about here is on growth teams so a growth team today is typically engineers data scientists designers product managers and user researchers these are not the all the people you might have in your company today but these are the company companies that you will have when you start making decisions to run growth the way you organize the growth team is there's sort of two options you either have the growth team as his own team and the rest of the practice is the product team and that's actually sort of my the the challenge here is that when you when your small company and and you're not kind of like somewhat ignorant about growth what you eat often say is like I've hired a growth engineer I hired a growth spike manager that person is going to be doing all the growth in the company that's sort of not really a recipe for success so there's kind of a fine balance here with between saying everyone is responsible for growth which doesn't work and having a small team that are responsible for all the growth you kind of have to find a balance here and the right way to find that balance it's a set very clear goal and goals and very clear dividing lines in your product so a good example here might be everyone that works on the core product which is sort of the value of your product let's say I am gusto the the running payroll for employees is the core product now everyone trying to get to that experience that's sort of like the growth area of gusto so there'll be one easy way to kind of make distinction between growth and product so how do you decide what to work on you make a simple calculation of what's the effort what's the minimal viable thing I want to try try here and sort of how big of an outcome can that be so I always try to forecast out the outcomes before you actually start doing the work because if you're forecasting something and a best-case scenario it's small you shouldn't be doing it even though it's a low effort all right I have two small short sections left of this talk the first one is called making decisions if you ask any product manager MB today what is the most important tool or learning that you learn to ever be that you'll apply to your next thing they would say experiment or experiment framework or some way of making a B test inside the company and I'm so happy that so highl is talking later because I stole like a quote from his investor deck on line this is the quote most of their world will make decisions by either guessing or using the gun they will either be lucky or wrong if you keep making decisions without using data and experimentation you will be lucky or wrong and this is a huge problem so if you if you kind of get to a scale of a and B or something even smaller than that then every decision you're gonna make that you don't use a/b testing for you won't actually know what the full true outcome of that decision might be so whatever BMB we use experimentation and a be testing for every single major decision in an tighten the entire company so this is how that tool work a and B but my talks are sort of like more about how that look like let's say in your company you decide to ship a feature and this is how many kind of measurements of Madaya metric per day and and the Wednesday here is when a ship that new feature and then I look at it two weeks later and looks like that the metric of that feature went up so that was a good thing right well it's not that easy because there's so many more different factors who knows what happened between here if you're a soccer app and the World Cup you started well that and you wouldn't know if the features have made a difference if this was just peak season or whatever you're doing this is a school app education app and this is September then you wouldn't know either so the only way you would know is what hack what's called a counterfactual basically an a/b test by running two different versions of the same feature the same site the same time you would be able to know what the true difference between making these decisions and not make me it was this might sound a bit technical but it's very very important to internalize because you will get to a point where you were successful up until that point and you think that you're so good at making these decisions and then they get harder to make and you have to have a framework to make these decisions so what we and then because this was so important MB we built something we call experiment experiment review experiment of you is when the whole growth team would meet in a room like this would go through all the features that we had recently built and before we told you which actually should have the different features in the experiment they actually won with asked the audience so I'm gonna do that with you guys here so here's a photo our experiment review we would do this every two or three weeks it's really fun but he drove home this one thing which is that making practices are really hard alright let's get started so the goal of this project was to increase the number of shares from the FMB mobile app specifically the number of shares of listings and back at that time we had two options we had the native share shoot that you'll know about and then we have this experiment share sheet which takes up the entire screen has more colors the same type of buttons sort of you can see the difference here now before I show the answer how many here think that the native share sheet led to more shares raise your hand it's behalf the room how many here thought they'd the new share sheet that one of our engineers built drove more ships raise your hand so the most many hands would do not race I'm assuming you guys think there was no difference because this is so hard you have to make a decision you have to make a decision you have to have an opinion if you don't have an opinion here you're basically saying I can't decide so this share sheet led to 40% more shares so very important in this case that we'd use experimentation because if we just gone by a gut we would have been wrong half of the room more than half of the room would have been wrong let's do one more so we send out this email to existing users of everyone b22 sort of when I would book make a booking ebon be listen up this email to the people that I listed as my code travelers and the email will look in the control like this where it would have the itinerary the address to that listing and the button on that email would say join Gustav's trip now we tried a new version where the email look a little bit different a less content and then we have a different button here called accept Gustav's trip imitation how many here the goal here was to sign up so people have to click on this button and then sign up how many here think that Joan Gustav's trip and the control led to more signups to raise your hand some people how many Herot think that except good stuff stripper mutation raise your hand that's good how many here think that there was no difference so this is a 14% increase of just changing the basically a name of the button let's try one more for simplicity so in this case now they're sharing experiments I'm kind of giving you the very easy to understand ones so the control here was sort of like a bunch of sharing options on the MMB listing with had some Twitter icons and facebook icons demon icons we got another version of that which the icons were round and then we have one that was called sort of square buttons and it was just displaying email and Facebook and then you can click on more how many here thought if the goal was sharing that the control with the icons one raise your hand some people how many a thought that the round buttons were better raise your hand some people I leave every thought the square buttons wow you guys are really good how many thought there was no difference well both of these were winners this was by far the biggest winner and these are the kind of things that we debated and because we run experiments you don't have to debate it anymore so proc decisions are really hard at scale you'll want to use experimentation a be testing to make these decisions because otherwise it'll be the loudest room it allows us voice in the room they'll make the decision so you don't want that to summary sort of my talk if you're running startup you should be thinking about growth before you start working on current act exceed be measuring your attention and knowing if people are using repeatedly using your product you should pick a metric and then pick a goal for that metric and drive that metric towards that goal very simple and then eventually but not right away you should start running a/b tests for the decisions that are hard to make in the early days decisions that hard to make as there might be obvious but the moment that not easy anymore you should be building doing this with experimentation thank you I think we have time for a couple questions yes do you have any frameworks or ways you think about trying to be honest so the question was around experimentation in SEO how do you go about doing that so what you're trying to test when you run an on-page experiment what you're trying to test is sort of the amount of organic traffic you get from Google and the sort of how your ranking shifts so Google actually are changing the rankings all the time so if you make a big change on your website on say say half a say let's call you have an for every B's case listing pages sorry searched search results pages so Stockholm London San Francisco we buy at one group and then you have New York Paris and Barcelona B another group you would change the content on the first group of pages but not the others within not too long of a time you would see more or less traffic going to either of these groups from Google sometimes there's no difference but if there is a difference you'll see more traffic give you an idea of a very small change you can make let's change let's let's for example change the title tag from air and be listings to the twenty top best listings in Barcelona one of them is going to rank better because you can be more inviting to click on and then you run that experiment towards the search engine and then nutrition will kind of tell you which one is better based on the amount of traffic and the search or anything you get yes [Music] [Music] the question was around a B testing how do I determine if it's right for me when I'm really early so a B testing is a function of change so you can have a medium to small size audience if the change is large enough it actually can be since the significant like the numbers can be significant now what you can do is you go do you can go to Google and typing a/b testing calculator and there's a kind of a form will pop up and you can type in sort of the the metric that you have and then they change and you can see how big the change have to be for you to be able to see a difference now if you're really really small it probably shouldn't shouldn't be doing maybe testing at all so I would do a B testing when you have large like enough traffic that you can see a medium to small size change within say two or three weeks so let's say small change would be a couple of percent if you can see that in a couple of Ko weeks kind of like that sort of that level you could be at that's probably sort of my general recommendation I've certainly seen a lot of companies and their early embracing a/b testing relatively early and then have that guide saw that their decision making and I don't think that's a bad idea because I think that's much it's better to start a little bit too early then to start way too late if those are the only two options of course you start at the right time yep how do you apply growth to high barrier entry market like like health insurance so I think you should separate sort of the acceleration of your growth in that market and the market itself so if it's hard to reach people that means you have to probably try to reach to reach a lot more people before you actually get sort if if I hi tell me why we mean by bytes are like a high barrier yeah yeah so if you have a risk in regulation involved I don't think that the principle of growth changes but if you're purely growing through sales for example you can apply a lot of automation and technology to how to do sales so if I'm a company and I'm selling to insurance buyers at startups I might be starting by emailing 10 people and in fact that can email 100 people or a thousand people with the same type of level of the email itself is kind of a field as personal and as direct to that to that receiver as if I send one email that is remove myself so there's certainly kind of things you can apply to your growth that allows you to reach a lot more people now the growth doesn't solve any of the risk challenges at all it doesn't solve any kind of major market issues and and be certainly had a lot of issues with legal like legal challenges in different markets and sort of growth was disconnected from that that wasn't sort of our our goal if you're in a start-up you have those those problems I don't think you should solve them solving them and solving growth are two different things and they're very separate from each other and and growth is a way to accelerate you getting to more health insurance buyers it doesn't actually say anything whether that's that's a good thing or a bad thing yeah [Music] do I have any wisdom amusing non-sustainable tactics a new market like uber so if you're doing things that don't scale you sort of have two options one is they don't scale so you should stop doing them eventually or you build sort of a playbook where you take those skills and you kind of try them in a different City now they might not eventually scale so let's say I am out manually to recruit uber drivers now that might be an unsustainable growth tactic it might be working for the first twenty drivers but eventually it's not going to have high ROI in comparison to say running ads on Facebook to recruit drivers so I think you kind of have to determine this manual thing that I'm doing in the very beginning which is doing things that don't scale which is sort of recruiting every single user to become a user of your product eventually it won't work anymore and you kind of have to find another channel or you won't be high ROI in comparison of the other channels and you'll most companies will go through this transition period where you go from say writing content manually to do engineering changing your website so that we get more search traffic for example what about doing subsidies or incentives for rides well I don't I actually think the incentives are super scalable like at MMB we still have the referral program they're sending up on hundreds of thousands of users every day almost which and large portion of them are sending up through referrals so that is the example with subsidies are actually scalable now me handing you a coupon in your hand it's not scalable but doing that through an email system or through whatsapp or through messenger or some other channel is actually scalable as long as you're not losing losing the money so you have to have a very good ROI calculation on the money that you're handing out and knowing that that's incremental money that if you didn't get get that money out those users wouldn't actually start using your product so as long as you have a good handle on that that is totally fine yes [Music] so question what's around they have two groups of users so the users and the paying users and some of them are very active and some of them are not not exactly but at least I don't want to pay how would you use growth there so one of the things that we're trying to figure out is what is the retention rate of those paying users are the actions taking around and using the product that they're paying for and what's the retention rate for the free users do that change once it starts paying and then it will look at the conversions or like how what percent of the freezes and am I converting to paid and there's many different ways to do that there's one which is having the freemium concept you can also have a free trial which means you actually don't have freemium you can just have a very limited set of free users and then they kind of have to go to paid but ultimately to the extent that you should be growing all of this comes down to how much usage you have from these the one you they want the users that your value so if the paid users are the one you value the most if if they stop using a product after three months then you have a more fundamental problem in the conversion between these two groups so I would still go back to a retention and look at usage for those people and see are they actually sticking around and using the product last question from over here yes in the back [Music] [Music] so the question is sometimes experimentation can be really hard to execute is there an ideal sort of frequency or which which experiment should I be actually be running for something like a marketplace experimentation is really hard that is correct so in that case it's easier it's harder than just setting up a simple tool online or using Mixpanel to set up your simple a/b test it's more difficult than that if you are it's kind of having a simple product funnel it's generally not that hard to set up experimentation if you have an engineer and if you have decent amount of traffic it's not that hard to set it up now there's different kind of products that make experimentation harder I think that if you're at this at this stage where you have a lot of traffic you in my opinion you don't have really an option you sort of have to invest in some level of infrastructure to use data to make decisions if you don't do that the alternative cost is you end up making a lot of bad decision or they're either long or they're lucky you're lucky because they're right or you're wrong so you don't really have an option but to run experiments and it is true that there's some type of products through which experimentation is easy if you're running a mobile app experimentation is not that difficult there are lots of tools you can use to run experimentations if you're running out of ideas for experiments well then you should go back to user research and you should look at other products that are sort of look at something like Pinterest or Airbnb or Facebook they're probably highly optimized so a lot of the things in their funnels is there for very specific reasons because of a/b testing so I don't have any better answer then if you don't do it you'll have a lot of other problems and in terms of the frequency there is a cost to setting up an experiment and that cost should be minimized as much as possible but it's sort of like you should just try to get better at it and and pick the easiest smallest thing that you can test and just test that if you have a big new features you want to test we'll just test the first part of that feature don't test all of it and see how people react to the kind of the first part of the feature I can talk for hours about this but thank you very much [Applause] you

Channel: Y Combinator

Views: 192,291

Rating: 4.9207497 out of 5

Keywords: YC, Y Combinator, Startup School, Gustaf Alstromer, Airbnb

Id: T9ikpoF2GH0

Length: 64min 14sec (3854 seconds)

Published: Wed Sep 12 2018

Reddit Comments

[u/midael]

00:00: unrelated remarks about weekly updates
04:05: Why is growth important? Really important to ask this question! Without growth there is no evolution! Without growth, competitors will eventually replace you.
06:40: Do NOT work on growth if you haven’t YET found product market fit. BAD things will happen.
13:30: Another definition of Product Market Fit: 1) identify metrics that represent value my users get from my product, 2) Measure repeat usage of that metrics
19:23: Are you measuring your retention rate currently?
19:30: If you can stomach it - go OUT and TALK to your customers and ask: What would you do IF you couldn’t anymore use our product? Answers might surprise you!
21:00: Product team with engineers & product marketing and marketing team type of structure is SO yesterday (or this is not how it works today in many successful businesses)
22:30 Startup should NOT be doing brand marketing? WHAT brand?!?
24:40: Conversion rate optimizations: Localisation, Authorization (make it *super* easy for your users to log-in. Otherwise you will start losing them, Onboarding)
26:00: If you’re selling something people only rarely need → They are going to google for it. Make sure you show up on searches. How do we know what people would google when looking for something like ours? Have we optimized at all for that?
28:00: Do your existing users share your product via word of mouth?
29:00: Do I already know who my future users are? Well, if you do, get a full list (e.g. all CIOs in California) and figure out how to reach them! It’s almost sales 101.
34:20: Example of well thought out invitation email
34:30 Do NOT do paid growth when you don’t have revenue
36:30: Attribution! You NEED to understand how users find you. And which spent $$$ brought them to you. (ALSO: run your website often through SEO browser to be reminded what search engines see about you)
37:00: SEO Is NOT dead. People go to google every day to find out about things.
39:20: With SEO you really have two things: 1) things you say on your site, and 2) sites linking to your site
41:40: When anyone writes about you, ASK them to link back to you. It matters.
43:50: Before actually doing growth activity: forecast it’s results. If small, why do it at all, and after you do it, you have something to measure against. Of course you dont always know if something small would actually work very well.