Design Testing at Netflix

hi there everyone I'm gonna start like every good presentation should start with an animated gif from Mean Girls and asked for some audience participation so how many of you here are members of Netflix good that's what I like to see a nice lot of you okay for those of you who aren't or if you're not just pretend you're not for a second do you think as a consumer you should be able to browse all the TV shows and movies before signing up hands up if you think that's the case yeah it makes sense right what you might not know is we did that once upon a time but we don't actually do that now so I'll introduce myself I'm Anna Blaylock and I'm on the product design team at Netflix and this is my little dog monkey I'll go back in time four years I just started at Netflix I was all clean and fresh face I had all these ideas about how to get new members into the service and we're brainstorming one day with the team and I have to I tell them I've got this great idea why don't we let our non-members see a whole range of wonderful content before signing up rather than making them wait till after I knew that customer service always got tons of calls from customers saying they wanted to know what the content was before signing up and I figured showing the non-members are awesome UI surely that would make them want to sign up so it turns out I wasn't the first person to have this idea I wasn't as smart as I thought but it hadn't been tested before so there was a lot of conversation about whether it was a good idea and after a while some time passed some great debate and in true Netflix style we decide to a B test the idea so for those of you who haven't done it be testing much here's a quick primer so we have our normal customer experience we'll call it the control experience first we come up with a hypothesis about how we could improve the customer experience so in this case we think that allowing non-members to browse the content before signing up will improve the amounts of signups we get and then this is where we start to design and develop different variations to explore different variables to test against the control so then once the test launches uses randomly get assigned to the different experiences either the control or any of the variations then we pick the winner the one that got the most signups and then that becomes the new control experience we roll that out to everyone and then we start thinking about the next test and the next and the next and it keeps going so I've told you about how a B testing works so how does that work with our a/b test so let's pretend there is let's pretend we a Jason so here's one of the thousands of people that would be allocated to the test he's heard about Netflix and he wants to find out more so this was a good couple of years ago he randomly gets assigned the control experience so you'll see that there's there's no navigation on the on at the top and there's no rows of content it's just very clear call to action and there's nowhere really to go other than sign up or leave then let's put ourselves in another person's shoes Emma she gets one of the many test experiences that we designed so let's say variation one so when she goes to Netflix she sees that there's this navigation and there's all this wonderful content and she can scroll down and she can scroll through rows and then she can go and check out different genres and just have a really good look around all the different content we have to offer and so obviously you can see she can just play around to our hearts content and then when she clicks on a title she can get more information she can get the synopsis she can get all the episodes that we have so after that first tense test launch with multiple variations back in 2013 like I said we it was ready for us to just sit back and let the results come in see which test cell would win and what happened the experience that had the most signups was get ready for it the control experience we were a bit disappointed but we were okay with that we knew we had so many more ideas around this testing so we decided to test multiple times with all the different executions so we did one test and then another and then another and the last one was earlier this year but each time it lost against control experience so frustrating and so we I think the reason I'm coming to share this with you is because it was a super interesting series of tests because we all assumed it would be better for the user to see the content before signing up and we were so surprised that it kept failing against the control experience so today I'm gonna quickly share some of our top learnings three of them and how it could apply to you so we learn it is so important to simplify to the appropriate choices when we compare the two experiences you see there Emma has way more choices and far less focus she can go off down various paths and get lost amongst all the different content whereas Jason has far more focus and less choice Emma's not really given the appropriate information to make the appropriate choice Jason's control experience however gives him the context he needs at the time to make a decision and his choice is very simple with very little risk we think that the low barrier to entry which is the free month and all we have to do is give us your payment details and you won't get charged outweighs the need to see all the content before signing up the next thing I want to talk to you about is this notion of not confusing the meal with the menu that might seem a little strange to you right now so what we we took for granted when we were testing all these things that we thought we were being transparent by showing all the TV shows and movies but really we were hiding some of the best bits the secret source of the product so let me put it another way here's a menu from foreign cinema one of my favorite restaurants in the city if you haven't gone there need to go there so the food looks good here right it's hardly the same as the experience of foreign cinema I mean the food is obviously delicious but then you've got the mood lighting you've got the the old films being projected it's just a really cool experience I think we can safely say that showing and telling is not the same as experiencing so in Netflix case the menu would be showing the content as we did in our tests and the meal is the experience of the product the users won't be able to weren't able to experience watching the content you know the way the when you watch Netflix it just works you just tap the the content and it plays and then the fact that you can start a show on your phone on the way to work and then do your day and then later on in the evening pick it up exactly where you left off in your living room on your TV and they weren't able to experience the personalization the way so if you look at this this is a couple of screenshots I took last night of my profile versus my husband's profile you see we're very different so you see Robert loves stand-up comedy and comedy in general and then I'm much more into my drama and my chick flicks and then a closer look you see that the content the rows are different and even the content within the rows is very different see Robert apparently isn't a warrant to TV shows through his very glad that I'm I am showing that here so in our test the users weren't able to experience that personalized con because we didn't know much about them so we had to show everybody the same content the way the free trial works means that we can ask for a very low level of commitment from the user which is again the payment payment details and then we give them a free free month in exchange for that we allow them to take full advantage of the product they get to watch the titles and we get to know them and then we can start showing them things that they're going to really love and by the end of the month they're likely to be more excited at the content and the product and then be more likely to be comfortable with pain for the experience the last thing I want to cover is that the user doesn't always know what they need I'm sure you some of you know this classic Henry Ford quote if I had asked people what they wanted they would have said faster horses the same applies for Netflix when we ask non-members if they could choose one thing to learn about before signing up the results were pretty clear you see they asked there are all these different things they wanted to know about price devices it goes on but 46% wanted to know all about the movies and TV shows so from our testing what we realize now is that the user might think they need to see the content before understanding if Netflix is right for them but really they need the experience of Netflix to see if Netflix is right for them so there were three of the things we learn and although we didn't be our control experience in any of the tests we did apply our learnings to future tests and learn every time to where we are now which you can see here so we've continued with the idea of simplifying to the appropriate choices because there's always a clear path if you have enough context and you hit the page it's very clear what you need to do just hit that button and go and then for people that do need more context there's these buckets below and these highlight the clear the biggest barriers for people and we've expanded on that functionality of these buckets from our learnings with our newer markets so we just launched in Japan about months ago and then Germany in France last year and so we're we're less known well maybe the culture is different more context is sometimes needed so now you can tap on any of the buckets and then there's I expanded information and we're not confusing the mill with the menu we don't allow for the user to get lost in the product without actually having the full Netflix experience the mill via the free trial and we do still bear in mind that the user doesn't always know what they need they think they want to know all the TV shows and movies but we've learnt that getting them to move forward into the free trial is best for them and for us but we do now have an overview of our content you can't really see it on the screen but it's it's faded back it gives a real overview that signals the quality and the brett's of the catalogue without the users getting bogged down and going navigating through a ton of pages so of course we won't stop there we'll continue to design tests and we'll learn each time we each time we do the testing a B testing is such a powerful tool for designers and I feel really lucky that I can test my ideas and then learn whether I'm right or wrong each test too close I do want you to take a moment and think about what assumptions you might have made about your product and maybe that you could look at from a different angle it's like Isaac Asimov says your assumptions are your windows on the world scrub them off every once in a while or the light won't come in challenge your assumptions and open your mind you might just be surprised we were the test may have failed five times but we got smarter five times thank you

Channel: Manifesto

Views: 13,798

Rating: 4.9295154 out of 5

Keywords: ui design, ux design, user experience, product management, product analytics, product design, user interface, mobile, mobile apps, mobile metrics, iOS, Android, growth marketing, growth tactics, growth measurement, success metrics, retention, engagement, analytics, data science, marketing, marketing analytics, marketing tactics, San Francisco, Manifesto, technology, app development, app measurement, startups, entrepreneurship, professional designer, design, Netflix

Id: -Gy8TnoXZf8

Length: 13min 11sec (791 seconds)

Published: Mon Jan 11 2016