Measure What Matters: Crafting UX Success Metrics

it's super fun to be here you know I've heard a couple people say this is such an odd situation to be in it's kind of a start-up person cuz it feels very academic and in my little tag here says professor but I'm less of a professor of professing ideas and more of a Tasker so you're gonna be doing stuff today and and so just be prepared for that so and let's go ahead and get that handouts passed out you can do it any time I work a lot on paper even though I've been in digital product for the last 20 years and and sometimes I get you know beat up for it people are like oh so many sticky notes oh my god so many pieces of paper isn't that environmentally expensive I'm like you know it's really expensive failed companies so I just you know go with the paper get things working out on paper and then we'll take some tech to it later on all right so as introduced my name is Kate I've been in a variety of places the podcast that I host is a ux oriented podcast it's not safe for work also I'm probably not safe for work so but let go with a couple you know expletives it's just kind of the way I roll but I'm curious oh and everything I say here is tweet friendly there's nothing confidential I'm not all that tweet friendly sometimes old tweet mean but um I encourage you to tweet out anything that you would like to so y'all have had lunch and I think you've been here for a couple days but that doesn't always assume that you couldn't know each other so I'd like to spend our first two minutes building trust two minutes no trust Falls required but just say hi and introduce yourselves and I'll walk around we're really gonna take like two minutes for this it's gonna be super quick but just say hello introduce yourself you know we're all kind of working on hard work here so uh so let's get to the room all right so I'll start the timer is that is the way I roll and turn around say hello looks like at the room tabbing a bit now but y'all know each other so closely and intimately and our old friends here's a few of the companies that I've been able to work with over the past you know a few years of my career many of them let me go back one many of them you might have never heard of because sadly that is kind of what we sign up for in the startup world some are pretty big others or incubators or accelerators or investor firms similar to true etc and and throughout all of this you know the one thing we always say is our work is hard and and it is but I've kind of built my career on what if it wasn't as hard what if we just ask better questions or simpler questions and kind of came up with clean and simpler ways to to start to build our products because they're gonna get complex anyway how can we start simpler and one of the ways that I've started doing that within my own practice is really with metrics and so that's what we're going to talk about here today is UX specific metrics along the way you'll be doing some other things that won't feel quite as number II and and so that's how we're going to roll so here's a couple things of what to expect so I'm not obviously an expert in your business but the good news is you are and my role here is to help you think about your company in a way that you can build your product that works for real humans and measure that so that you can actually make your product better over time that's kind of the whole deal of what I've been talking about lately many workshops if you've been in one are like take a half an hour and do this thing and it's kind of fake and you're working with an example product but I want you to work with your own product and all have some help on how you do that we're gonna be working super fast so all the activities are two to five minutes I assume you already know the information you're just gonna be answering questions that I prompt for you so sometimes workshops you know feel like this hours is gonna feel more like this and and that just is kind of the way we roll so this might not work for everybody right like you might not like the pace you might not like the prompts you might not like the perspective and that is totally okay permission not to buy right here but all I ask is you just try to put yourself in my hands for the next 90 minutes and let's see how far we can get together so how do we know our work is working and this is the most terrifying question it keeps you Xers up at night it keeps product people up at night it certainly keeps founders up at night and I've been that founder up at night and so I'm carrying us around here like who here is from a design background or whose design is your primary responsibility in your company and raise your hand okay couple what about technology or development engineering okay we're so what about like product in general like you're a product manager or your product owner etc and what about business sales marketing they hate it when I put it all together but I do it anyway yeah right then the middle I like okay well all of these things kind of roll up into this idea of what is our company doing and why so the way that we can start to do that is by measuring our UX UX is such a strange term because everybody wants it but nobody really knows what it is so I'm curious are some ways that you all are measuring UX now is anyone measuring anybody nice so you actually meld qualitative with some of the quant related to conversion or funnel or some of those types of KPIs that are fairly fairly you know concise around your product nice anybody else doing similar things yeah maybe testing ok this one's better that one's yeah Net Promoter scores yep so would you recommend yes nice so you prototype you test um we've never test our users we test our prototypes with our users and then we and then you actually have some qualitative data as well nice I love that so you have health outcomes and the interactions within your site are actually correlate or causation for those kinds of outcomes yeah I love those goals like I want to lose ten pounds it's like you can't just decide to do that you have to do these things that are tractable in a different way that's an outcome metric it's not a process or a progress metric nice so that's a pretty common handful of things um I've noticed that there's been there's such expertise around data and metrics collection especially in data instrumented products like probably what we all have but there's not some way to correlate that to the human behaviors that's really what I'm gonna talk about today so as far as our questions in order to get to what we measure if we have to know these three questions we have to know what are we trying to accomplish we have to know what that actually looks like in a fairly specific and and vivid way and then we also need to know how we measure and track that so I think those is like the roadmap on how we actually count things and calculate things I talked a lot about the three languages numbers pictures words and how they all go together so even though this is a numbers message there's pictures and words that we need a help with that so to get warmed up we're just gonna pick an idea for the day you all are here with companies I'm gonna assume you pick something with your company but you can pick either a product you're working on or an idea for a new product or feature if that's working for you everything I'm going to talk about today scales from like company to product to feature to interaction or customer problem so if you have some product features but you're actually working on really digging into another customer problem you can pick that um you're gonna need pens and sandpaper you've got your little handout with you use that for notes if you want we're gonna take one minute and just write the the topic that you want to focus on today on a sticky note and that way when your brain gets crazy you can like look at it and say no no no I'm focused on this all right so what you're gonna do now is do a quick product snapshot this is gonna be familiar to you but there's really a system in all of our products there's people the problems it solves and then the thing and anytime I go to a conference or I'm amongst the people with startups pretty much everybody comes up and says hi can you doing UX review of my thingy let me give you a demo and I'm like who put that away but problem do you solve for your customers because without that information I can't give you any meaningful insight at all there are no best practices just practices for your company so I'd like you to go ahead and use a template or a blank sheet of paper and just answer these three simple questions like who are the people who are the users and what problems do they have I'm gonna use an example throughout our session together it's tasca doodle it's kind of a mobile sharing application and so for Tasca doodle the people are working parents with kids you know it's kind of a common problem the problems that they have is they need to be able to share tasks while they're on the go they're too busy they need to get more things done and they need to know when something's completed and really everything in that footprint of the product needs to go towards solving those problems anything else is just gonna be a distraction and so then the solution is a mobile app for sharing tasks and really this is the level of simplicity you want that's why I've only got two minutes because if you have a lot more time you just add a lot more words turns out that sound helpful so we're gonna take just a couple minutes and go ahead and write down who is it what are their problems and what's the solution if you're an enterprise app and you have a buyer and then a user pick one of those if you're a marketplace app and you have a sellers and buyers pick one of those you only want to focus on one customer set for this time together all right it's only on a piece of paper you can refine it if or a CAPTCHA or keep going if you want to but I'm gonna move us along because we only have our 90 minutes I'm gonna make sure we get through the full art so in our three questions what are we trying to accomplish three questions we need to answer what are we trying to accomplish is the the first most important one like what is everything that we're trying to do otherwise you just have no chance at all at trying to capture tracking and I'd like to introduce with you a stack for what I call the UX stack and being in the UX field for a good season period of time it's hard people say like what is UX what do you do and it's a it's hard to kind of grasp actually but that purpose is and so if you do a few Google like what is UX and you look at images see all these fancy colored valdon diagrams usually have a lot of overlap with a lot of fancy terms in them I found those meaningful only if you already know what they're talking about and not really helpful for explaining what we do and why we do it and as product people we're doing all these things so this is a whiteboard a bowl really simple sketch that you can use and pretty much all of your activities as a business should be mapable onto this simple diagram so I'm going to go ahead and introduce it to you then we'll use it as a framework for our time so we have humans users people in the world and these people have needs and goals and problems turns out we can learn those by observing them watching them talking with them listening to them and those needs and goals get translated into something you can think of as uses and the phrase that really encapsulate uses is what can someone do with our product that they can't do without it okay it's a very simple phrase surprisingly tricky to answer sometimes but just a very simple phrase from those uses we distill out our features and this is a feature prioritization diagram kind of two by two and only from those users users should we get our features and from those features collections of features put together with a look and feel an aesthetic a brand a voice that actually creates this coherent it um product so the key part about this diagram is through well actually when we talk about UI there's this little pieces like it a lot of people talk about UX and UI together like yeah those aren't the same things all the UX fiddly bits kind of live way down here and so sketches and interface and pixels and mock-ups and all of that like lives way down here and throw uses runs this waterline and everything above this line is about people in the world and everything below the line is about product as an answer to those people in the world so if you're trying to make a great product you can't just focus below the line or make something below the line and try and push it up through sales and you know paid placements above above line it's got to be reflective now this doesn't really have much about your company and people start companies with a vision or a mission and so your company lives kind of up along in this area and where your company purpose and those needs and goals intersect is going to be your value proposition so you need to really have that defined and understood before you should really invest anything in the much more heavy intensive and cost on costly efforts down below that line when you start building product now we think in solutions so it's not uncommon to talk to someone that says I want to build a data analysis app or a data analytics platform well that's all very well and good and you can start here with the ideas for features but you really owe it to yourself in the whole team and your investors to explore what those people are who those people are and what their needs are otherwise you can put a lot of effort into buying into making a product nobody wants and I've been through that world that is not a pretty role that's an expensive and heart-wrenching world so don't do that so we can have these users and needs and goals to help shape and frame everything we do as far as our product and that well diagram is with you the key part that we're gonna be focusing on is uses because that's really the meat where you get to measure things that actually make a difference so let's take a talk through a word version of what some of those uses are and I'm not a huge fan of like fill in the blank templates but sometimes just as a guide point of how not to get too wordy they can be helpful so if you were gonna define use this entire stack to walk through say an existing product something like Facebook would look like this so meet Erica social engaged college student hopefully there's some kind of validated persona to stand behind that and her needs or goals or Erica needs to feel closely connected with friends near and far everyday that's probably feasible with Facebook Erica can share her latest thoughts and see what her many friends are up to again nothing about the product yet it's just about what the benefits for Erica the problems and uses Erica will have and then using status updates updates on her wall messages comments and likes so those are the features but all those features are dependent on making sure that Erica's uses are met and in a way that's universal clean consistent and fast so that's kind of the brand attributes of the company so it's this really this key uses that's what needs to be defined pretty clearly so those are it again just time with a task with tasks a doodle so you can kind of see what it looks like but it's not quite so obvious so the task of doodle are you sir marry she's a working professional two kids and she needs to share the load of having her tasks accomplished and get help completing those things and the tasks a doodle meri can create tasks share them with their spouse and kids and no one something else's gets done it sounds so basic but you know that's actually kind of an important thing for her to be able to get done and the features are going to using the task scheduler task sharer and dud notifications and then in a way that's friendly clear visual and fast right so those are more on the marketing or the brand attributes side but again those key part like what are we going to measure that helps us understand if our product is working is going to be right in that middle line so those key uses are something you want to really nail and to in order to kind of shortcut our way into that so then we have to go through the whole stack you can just ask what can our customer do with our product that they can't do without it so let's go ahead and write one of those for yourself and again depending on where you are you can use product you can use feature you can use interaction sometimes you really have to work in a very small scale for this and just answer the question with this blink the name of this thing your persona name if you don't have a fully validated persona talk to me later we can talk about those techniques what can she or he do and so that goal statement those people are good so now let's move to the next part like what does that look like and if there's any part out of your comfort zone in this session it's probably gonna be this so you know shake it off and just relax go along with me I see you and you ask well who are your users what does it look like when they're actually in the moment of need or getting the moment of benefit from your product and what are they thinking feeling and saying and in order to really communicate this words are not sufficient so we're gonna talk about pictures so here's a couple examples of those key uses from a company called Foodspotting which isn't with us any longer it was a great startup incubated out of user experience design company adaptive path Alexa and reduce ski was the the founder she took it she got funding she had a nice product it was acquired by OpenTable and now she's kind of over in the product team at Facebook but one of the things she did before putting any code or any interface around that if she wanted to understand how her idea connected with human uses and so she would draw these little picture with a little short phrase that key use phrase and show them to people get their feedback get their insights and she iterated through about 36 of these little snapshots over time trying to see if where the energy was and what people responded to and why what really what really spoke to them and she found that just talking about it was insufficient by showing people a picture that a much clearer idea of what she was trying to communicate and if you'll notice like these are super simple pictures you know these are not sophisticated but they're they give you a little bit more of a sense of context and they actually put the human in the center of that description instead of getting caught up with all the technology so to do this you can either grab a sheet of paper or an extra one I'm gonna walk you through a super simple sketching activity so that you can feel more confident drawing pictures of people which is not really common in our businesses that needs to be more common all right so be prepared there you get to see the information on the slides but then also draw it on the whiteboard as well so let's start talking about rapidly visualizing people in context so our first one is star people has anyone ever do people sketch are there people around here that actually routinely sketch pictures of people and how do you puts one of the purposes that you use to do that to think nice nice context something about you do to entertain your daughter all good party trick apparently so they're always nice anyone else sketch yep nice a visualized document and share all kind of you know being it punches one everybody else so alright let me show you a few things quick tips about this so if you're drawing star people and the first step is to draw a circle being around the size of a quarter and then directly under that draw a straight line about as why does that as that circle head this is the only time he gets complicated you're gonna go down for a point and up for a point and then down twice as far that's what the arm is for the leg and up for a point now once you've got about half the person it's pretty easy to complete it you just mirror that down and up down and up I've never completely you know completely even but that's okay and anyone anywhere can look at that and recognize that as a human being now this has a lot more merit than a stick figure because it's hard to draw a proportionate stick figures people aren't really built like that it's awkward and there's a huge amount of like juvenile associations with this that just don't seem to get the pickup but if you draw a person like this it starts looking like there's a bear there so that's why star people are so helpful now once you kind of get the basic form you can iterate on it and I'll draw a few other examples go back hang on I'll draw a few other examples if you want to draw something that's sitting you start out with the same basic form draw I'm on the side give them something to sit on like an l-shaped chair maybe a computer to work out and pretty much anyone knows that someone working in some kind of computing environment right you knew other handy things it always starts with that circle in the line and then a point in a little bit of a jiggle trust me none of this has to be super accurate and then the same formula and this is someone pointing at something if you want to give them a direction give him a hat to wear you can get a little bit fancier with some squirrely lines and a foot now you have someone waiting for a bus it's really kind of a very flexible form so start observing people and start looking and thinking about how your customers can be visualised with what they're doing honestly then and this worked most the time people say how do you drop persons sitting down flush on like from the front I'm like why don't because I just I'm trying to get the point across I'm not trying to draw a portrait the other thing you want is you want to be able to capture some some some sense of expression and expressions and emotion again are a little hard they can feel awkward in a business setting but if your customers aren't having an emotion related your product like what's the point like who cares we are humans we have emotions and for that I use this shortcut called the expressions matrix so go ahead and draw nine circles with little dots in the middle they should look like buttons once you're done with that all of the expressions have to come from two places they come from Alice they come from our eyebrows so we can use this as a system to kind of crack into the basic human expressions that we have so the first one we're gonna do adding a mouth and on that first column on the left go ahead and add a smile to each of these three folks can you see around I know in the middle one go ahead and add a yeah straight line and on the far right you know where this is going all smart people add a frown line so now you have three people that are happy three that are like and three that are unhappy the second place emotional intensity comes from as your eyebrows so now on the rows across we're gonna leave the top one house is in the middle one we're gonna give everybody our up brows and on the bottom one we're gonna give everybody down brows and now you have a range of nine discrete specific emotions that you can use this comes in incredibly handy and usability testing because this the person was confused just what your words might say but this is not the same as this right this is a much more important problem to solve than this one so when you're talking about like the level of your problem or how much joy your customers experience at that moment of release when you sit and you solve their problem like are they going from this to this or are they going from this to this like you can use those faces and emotions to really communicate that so what we going to do is we're going to take your key you statement and draw a little picture of it something that you could share and say this is kind of what I mean similar to the country the participant he was talking about having a documenting kind of shareable way of showing the people okay it is absolutely okay to be obvious and this is a judgment-free zone so as you can show your work to someone else just don't laugh at each other's work because that's a shitty way to be in the world all right so take five minutes and then we will keep going and pair up with someone and share your work back and forth they should be able to kind of get a sense of what your company does from the picture in the and the phrase alright at least you got to see a taste it's amazing how when you see someone else's work it just you know broadens your worldview alright let's move on so now you have on this stack you have this phrase and you have this picture so you've got your key use so now you're ready to start to measure something if you don't have this everything else is premature was anyone else in Amanda Richardson's data talk this morning did anyone go to that one you know how she talked about about the basic mechanics of data she did an excellent job but a lot of it was like what's your product trying to do that it was a course source foundation before you can really invest in a lot of data sophistication well you just did that so yay all right so let's move on so now how will we track and measure progress to that out that outcome in that key use Josh Picardo is a fantastic writer and he's a great UX and product thinker and his phrase really resonated with me when I read it it says your metrics are gonna be as unique as your business but I don't think that's as common at philosophy is we have really in the product world because generally what I hear when I ask what people measure or things like this over 10,000 downloads average time in sight is 22 minutes or 450 more sign-ins this week and I'll admit I said things like this so if any of you said or heard anything remotely like this how many downloads how many new adoptions yep well what's unfortunate about that is that these aren't really helpful because they don't measure the usage of your product by your person okay they're just other things that kind of go around that and so for when we talk about understanding human behavior and human usage there's a handy 2x2 that I always go back to about where metrics live in the overall broadness of understanding human behavior for our products and it basically looks like this this might not be new to you it's a fairly common way of communicating user understanding and I like it still works so you have generative concepts you have tools that help you come up with new ideas and then you have a value ativ types of techniques and tools things that help you assess or figure out the merit of something and then we also have quantitative analysis things that speak the language of numbers and qualitative which we referred to earlier on when you were talking about how you currently measure things things that speak the language of stories okay and within those two sets of categories those axes we can plot pretty much all of these or understanding methods that we have so common in the UX field we have interviews we have observational research we have field research we actually go out in context and see what people are doing understand deeply the problems they have and how they're currently trying to solve them right it's fairly common then we also have usability testing usability testing watching someone perform with your product or try and do something and then assessing and seeing how well your product is design to actually help them accomplish that and that's an evaluative method and both of these qualitative methods are really the best source of unique specific metrics that you can probably find for your company and Jared spool he's a UX thinker product guy very well seasoned he's been doing a lot of thinking about metrics and how they tap into actual product performance instead of some of the more business types of KPIs that are more common in our in our companies today so they're never on the quantitative side everybody wants to put surveys up here right like put out a survey at scale get statistical relevance for various things but ask people about stuff and you know surveys are tricky not only because they're very tricky to write effectively but also because once you've already proposed a question to someone you've already shaped their answers they tend to be more evaluative right it's kind of like when did you last feed your dog yesterday today not at all it's like well what if it's a week ago like I haven't forced into answering these things so surveys are tricky although I think there's an interesting opportunity with big data here but nobody's really figured that out so if you have a big data company like we should talk about this because I think there's some promise of how we can be better generative through data patterns but that leaves this evaluative and quantitative around metrics and analytics and that's where we're gonna be focusing our topic today okay so anyone using 2/3 of these quadrants a bunch of these quadrants has anyone seen this before like this is kind of old hat you know look at that okay good somebody this is new really handy when you're talking to your product team about the kinds of learning you need to do to be able to plot your techniques on here so make sure you're using the right method for the right kind of outcome so let's talk about metrics and analytics these are all kinds of just only a little smidgen of the types of things that tend to emerge when we start the metrics and analytics conversation and of all of these yes they're very important to our businesses but all of them really the usage stats are the ones that are related to whether or not our product is performing for our users so we're going to kind of double down on that and on those usage stats there's some common I call it the alphabet soup there's some common metric categories for this anyone using any of these we talked about Net Promoter Score before at the very early life time value CAC you've got your CPA they are PDA you you know all these things right so these are pretty common hold field of growth marketing etcetera to try and make sure that these are performing and measurable for our companies but that is a big toolkit and it's not as helpful when you're specifically trying to figure out if your product works so let's talk more only about the retention like what are people doing within your product and this is a handy diagram I like to use about retention because it kind of brings at home how big you want to grow your funnel well it turns out growing it too early can actually have damaging effects on your company so if you imagine your product gets a little product and everyone else have put a funnel on it hopefully not like as narrow as this one hopefully your funnel is pretty wide you get a good conversion through it and then you pour traffic traffic into the top of that funnel but if your product isn't working people just follow the way through and so now you've burned through a lot of those resources because it can't retain the users that are really appropriate for you so all those retention metrics that we need to optimize first need to be focused on the product and they need to be in service to the users okay so how do we do that well I think lean analytics isn't even read lean analytics at all it's a book it is your lucky day because this is a fabulous book and it really talks about how you identify those behaviors and customer needs at various different stages of your startup as it scales through different maturity so it's it's you know almost got a recipe book for that super fabulous Allister crawl and vinyasa cubits have done a really nice job of kind of optimizing metrics within a startup kind of a lean behavior type of way and they're one of their philosophies is that the good metric is the kind of metric the message measures the usage of your product by a person and they have some attributes of what makes an effective metric we're gonna walk through these together should be clear and specific it should be normalized it should be comparative actionable and then also it can change your behavior there's no purpose really in measuring anything unless you know what you and your company are gonna do about it otherwise it's just kind of data decoration it's not really helpful data action so let's walk through a couple of these so clear and specific this means you need to know what those specific interactions are that represent desired user behaviors metrics are basically user behaviors and it's the interactions that start to put those two together and you want in your product to support the specific interactions to actually link directly to those key user behaviors that you're trying to increase and you need to find the numbers that you would then use to track those interactions right so in something like Taska doodle it's a mobile task app there's going to be the ability to share a task and I want to be able to instrument or sketch out where are within that UI the chain of interactions or this chain of events are that represent that sharing a task has happened and that gets very detailed very fast which is why you kind of want to focus on a really meaningful metric at the at the start of the process something like normalized as a rate or a ratio which means whether you have peaks and valleys and your users you want to make sure that it balances out so that you know if your product performance is getting better and the best example for this is from a start-up I worked with that lexer which is they had a certain number of total users and then of those they actually shared a task ok so over 420 so that's about a fifty three point seven percent yield not bad certainly could be increased but then they were featured on the app store and so the product manager came and they're like oh my god we tripled the number of people that shared a task whoo that's a win except it wasn't a win because our user base was so much bigger that have actually dropped dramatically so you don't know if these numbers are good unless they're balanced out related to a proportion of your user base so you gotta constantly more normalize them the next is comparable so time-stamped you want to capture it at regular intervals and for this I love this because the case example is well we have a big dashboard up we know who our metrics are at any point in time like me to see them it's like that's great but that's not this purpose because what you want to be able to do is compare what it was before you change your product in some way and what it was after so you can correlate product changes to changes in those numbers okay so in order to do that you need them directly comparable over a span of time and in this example it might look something like this you have on one week you have a certain percentage of a of a metric conversion and then as you timestamp that week to week to week it will highlight things like well what happened here clearly this number is out of sync for the other numbers something happened how do we correlate that difference if you just watch it all the time you never really kind of get these comfortable timestamps and then actionable not vanity now this is a set of metrics that are very common and in the startup world we have to feel good about our work so we use them they're also influential for media they're good for team morale they're great for investors sometimes required by investors but because they don't change your meet your actual behavior they kind of make you feel like this so they're meaningful for one reason but don't expect that they'll make your product better because they're not about that they're about something different we need a way to actually make our products better so actual not the vanity and the best quote from the lean analytics is don't just ask questions ask questions to where the answers are gonna change your behavior so a good metric has various characteristics and this is the super simple like cheap tout version of it it's as clean and as simple as I could get it and it's been working and it's to look through a continuum of what kind of metrics how metrics go from kind of crappy metrics to great metrics and what attributes actually make that happen so something unhelpful and metrics is like sign ups you guys a category now something incisive you can start to measure something that would be a vanity metric might be the total number of registered users okay it's only ever gonna go up it's only gonna ever get added to so that's more of a vanity metric something that starts to sound like good would be comparative and maybe have a timestamp so like percent of new users per week starting to be more measurable something that we'd better would be something if your product is a habitual eyes usage type of product something like the percent of users who sign in three or more times a day per week like that starts to get it really frequency of behavior but so far all these could be true for a whole bunch of different products so the one that starts to be awesome is something like the percent of users who share a task three or more times a day measured weekly now you might need to count up a bunch of numbers and do the calculations to be able to get at that one specific number which is hard and so but this is the type of behavior that if they're not sharing a task they're not getting benefit out of tasks you doodle so we have to tie our metrics to the actual behavior that we want once you have a history of measuring these you can start to look at things that are more leading not lagging and this is a nice category as well and again it's gonna take some sophistication and some time in the in measurement to be able to track it but there are certain things where if a customer has some kind of behavior it just correlates more effectively hopefully even causation but have another conversation about that later correlates to ongoing usage of your product so in Nike+ probably none of you remember that but it used to be all thing you put on your shoe and it would kind of sync with your iPod but waited for iPhone days and it was indicated that by the third run a user of Nike Plus would stay a runner and a stay user that product so then a lot of the design effort could go specifically shape to how do we get people to that third run because once that happens the product kind of takes over from there and that is a powerful thing you can do that and there's some other examples then get has you know if you come back frequently within a day of buying a game and Facebook has this mythical you know plus the ten days of and friends I love the Dropbox like one one folder I'm like well it's kind of a basic can't really get any value out of Dropbox without at least that so but your product is gonna have this hidden in there and if you don't already know it this is a technique you can use to start to get at that so how do we get our awesome metric well let's take a look at the checklist a good metric measures the behavior the usage of your product by a person and a great metric when you see it says oh my god if we don't measure this then we don't know anything like but you won't really be able to get there until you actually do some practice making metrics so we're gonna do that right now and you're gonna brainstorm in three minutes you probably need less cuz you've been working pretty fast so take a couple minutes and brainstorm five ideas for metrics that you could count but directly would attribute to that key use that you sketched out earlier there have to be great metrics yet well get there but just go ahead and what are some things you can count that would indicate that key use is happening okay when you go back and do this with your teams I like to do it on sticky notes so you can put them all up and kind of pick the best ones from there but for now pick one that you want to focus on and will make it will improve it together so your first step of improving it and making it awesome is check does it begin with a number preferably a balanced number like average or percent of if it doesn't then you'll want to change that it needs to have some kind of numerical quantity to it so it should start out in the phrasing with number of or preferably average number of or percent of the second way is is there a time basis this is surprisingly easy to forget but how frequently are you going to measure that that snapshot of time and depending on your product it might be monthly it might be weekly might be daily but that's really going to depend on the kind of usage that you have usually weekly is a pretty good place to start I might have worked with a company that is seasonal so they actually only read measure a lot of things annually makes it very hard for them to experiment so they have to kind of figure out ways to hack into that there's kind of a seasonal thing so the tour short err time stamps can be good when you get down to hourly is actually an help because you're probably not releasing product hourly in that same way even if your continuous deployment so weeklies good and then lastly there's this other element you can use which is an object basis so in many of our products are some kind of object concept like a file or a thread or an email there's this interactive piece that you can use to to associate image metrics with it so the average number of messages per thread or the average number of comments per file would be those kinds of things you can kind of chain these up so that you can be very specific and very incisive about the types of behavior that you're measuring that's good we depend on your product but I want you to know it's out there is a possibility so go ahead and take a couple minutes and make sure that metric that you that you've shaped out um has these comport and elements to it at least number one and number two and then it's specific to that key use and then you'll get some help from your peers and if you already hit it like nailed it the first time out you can just glow it quietly for a couple minutes just proud of yourself it's kind of amazing how long two minutes feels why don't we go ahead and move on and pair up with someone if you're here with someone else from your company go ahead and pair up take a look at it and they should be able to look at your key use the sketch of the phrase and if this metric and have that feel like a coherent whole so go ahead and take just a couple minutes share back and forth and see if you can offer any advice to make those metrics strong there are more trackable more incisive better and read your hand if you have any questions and I'll hop over who's got a metric that they want to share as an example awesomeness yes so percent of users per week for seven days so is it the the so the two plus posts for the first seven days is their actual user behavior so how frequently are you gonna are you gonna count it cool per week and what is that what would that help you do if that number were to go up what would you do creating nice interesting and then if you're not getting that you can kind of do some things like usability testing and you could use a B test for various different things and kind of optimize where that goes what else just got one the number of passes our customers nice especially like that those scans are them related to some your actual customer so you've got users that are using it for your customers the assuming AMC is the enterprise that pays you right for this kind of sponsorship thing so if you can do that you can correlate that to potential revenue or some that business I'm gonna show rate your payment bottle is but it's close as we can get to revenue that's always better so and when they track what they're eating can you be a little more specific about like what does that look like in the air is the is the food journal an actual interactive element of it are they doing that in some analog process and then you want them to tracking food is nasty and hard and a lot of people have struggles with adoption for those things we and at Luxor we had almost a completely analog process for some of the learning tools we had and we needed to then design a way to instrument that kind of feedback so we had people take a photo of something and send it to the product as part of their like digital holding spot and that was the interaction that we had to instrument because we had no idea of what was happening in the physical world like that's its own challenge interesting one one more the number users editing our design system daily per client number of users editing your design system daily per client and what company are you but you X pin which is a platform for so when you have that number what happens if one of your design system clients like doubles our design team how do you know that what if they doubled their design team but only half of them are using it like how do you understand capacity of potential users as a relationship of actual users because let's say more and more people your design teams are growing but fewer and fewer on that design team you might not be able to have an indicator that you're losing a customer got that kinda terrifying thought but there was a I was working with a company and they were so good at their metrics if they actually had leading indicators of customer behavior that said within two months that enterprise customer would cancel and so as soon as this behavior started they'd call them like hey you don't know this yet but in a month you're gonna want to cancel our service and so let's talk about how we can avoid that from happening and make sure you're getting the value and of course the customers were freaked out they're like well that's weird cool but kind of creepy but they appreciated that personal touch because there was this indicator that wasn't a positive indicator but it was a potential risk for the business and so you can really get in front of that so think about how your customers lives are growing too and how you can tap into that so we're just gonna do so in the real world when you're working analog I know many of you have many sophisticated tools but I swear to God like a piece of paper up in your team space really amazing so you can do this as a team and you can put it on this physical dashboard low fidelity and it's sticky size so you can do this over time and you know iterate through a few different ones to see which one which metrics you really want to start to optimize also you'll see on this whole sheet that there's a goal and a timeline if when you want to reach it by because these are just containers for numbers in order to know what an appropriate goal would be you have to benchmark it and measure it once and kind of see where you're at and that might take some product adaptation and actually building in some techniques and tools within your UI so that you could even measure it and trust me that takes effort it takes attention and it takes precision so that's why having the metric first and actually change their design behaviors I gotta tell you I've been making UI for a while and the one piece that's changed how I actually construct interface is knowing what customer behaviors we're trying to instrument and design for because amazing if you can't capture it then you can't measure it so as you exit your practitioners or your UX people or your product people really it'll change how they design to be able to do this work so venema so you've got all this and now we're talking about the instrumentation depending on how mature your product is you might already have a platform you know google analytics is fine except out of the box it does nothing but make you feel safe it's filled with kind of vanity or what I call the health metrics for uptime and that type of thing to really get it something like goals or conversions you need to be very specific with how you instrument and connect up Google Analytics so having this set of a precise metric will be key for doing that but there's other ways you can do that you can gather this information you can do it technical by hand you can call people and ask them what they're doing if you're at a really early stage you can do data dumps into various just different flat files and then see if you can start to assess and analyze those and everybody's favorite tool excel or you can instrument it with some of the the event tracking software and platform tools that are that are available so there's a variety of ways to do it it can get expensive fast it gets confusing fast so start specifically with what is the behavior we're trying to increase and then measure that now when your folks are making UI sketches and hopefully they are sketching because I still believe like wireframes hopefully should be dead I know it's like they're gonna kick me out of the UX Club but I think wireframes are often a distraction from the kinds of techniques we do I'd rather go from sketch to code and prototype and I just think that's a faster way to work so if your designers aren't coding they should probably continue to learn to do that but being able to do a sketch specifically so that you can instrument and figure out where you're gonna count these things and then talking to whoever is responsible for you're doing your data capture it might be our engineers might be you like your product people it might be a data team it doesn't matter you need to be able to figure out where are these events and what kind of snapshot it doesn't need to be time snapped as well as the actual event value there's a variety of sophistication with that so write up a whole list of like what happens in the UI that indicates that that customer behavior is happening and it's very rarely like the easy button you know that some of hits once and you count it and you're done plus because you get a time snap this you have to figure out what your cadence of time is so is it 24-hour time is it weekly time what do you do is it monthly or is it every thirty days there's one well over thirty days that's gonna shift your deadline throughout the year because not every month has 30 days do you have people in global time zones how do you know where they are how do you normalize this these are sophisticated hairball problems and they'll come at you so at least knowing why you're trying to do this is super important and then how do you track progress that's all about setting the goal getting a benchmark even if you're wildly off it's some place to start and then how you can start to track from there now when you track over time whether you use a sophisticated tool or something as simple as just a list mm-hmm you will have these data values so that's really handy is to put them into some kind of visual story right so it could be a chart be a bar chart but I like the line chart and like the mountain graph and being able to annotate what the numbers were doing and then what your product changes were that's the story you really want to capture right so you're gonna release product and then from that release you might have a change in the metrics and that's the muscle you're trying to build as a team so when you get to something like being able to annotate we change the label of an interaction from one thing back to something else and you see a lift then you can correlate that product change to that actual metric change and that's why you want to start small and very specific because if your team can move a metric intentionally I can guarantee that is an ad is it like a superpower that is a superpower skill and you want to be able to do that routinely over time because there's no one metric that's gonna last forever let's get a shift as your product goes through its different maturing and as your customers change over time as well so how do we know our work is working well we've answered these three questions what are we trying to accomplish what does that look like and then how do we measure it and track progress and you've got the map and what you did is you made at least a word statement of it and then a picture so that anyone can kind of step into the shoes of what that key use is and now you've got that key metric as a blueprint for what you need to measure in a sophisticated way with your product so all that together it means that you can start to correlate the user behavior and the product releases that you do and since those are the two edges of this stack there's a lot of in between in between those and so being able to have a true and authentic and correlated story between those like that is product gold and that is what I would hope for you so that we can all go out and make our products awesome so that our customers will have their dreams come true so that is our time to gay here's some really good books that you can geek out on measuring the user experience goes through a lot more I'm not sure it hits at metrics in the way usage is that we covered today lean analytics and UX for lean startups are both excellent books and and the deck will be available so I want to thank you for your time and for working on things today and please go out and metric and I think we have a couple minutes for questions we have a few minutes for questions are we okay so I'm gonna leave that slide up because it's so deeply disturbing right this I like kind of just standing in front of it be like awesome all right so that's it what questions y'all have so the question is is we get paid for sign ups so it's okay to monetize that that is definitely something you want to track umm do you get is it then a monthly type of thing once they paid you once whoever's paying you different milestones I know got it and we measure so that it brings up a really terrific thing you know I was having a debate with Laura Klein she and I fight a lot um in our podcast about user stories and which ones are good etc and you know the classic like a user wants to log in but that is such no user wants to log in right like that is not something I think I'm gonna log in today no a user wants to protect their ship from other people looking at it like that is the youth underneath that so no user wants to sign up I'm sorry but they don't but they're gonna sign up so they can do something else and that's what you'll want to measure from your product perspective from the business metric of course you'll want to measure anything that directly correlates with getting paid yeah yeah yeah it's called what is wrong with UX and it's not safe for work that's okay there's about I can't believe there's like 50 episodes out there it's two old ladies drinking and kind of swearing about product design what else why everybody was dead it's not really dead it's just kind of limping and the UX pin folks hopefully we'll have a better platform address to this so as far as wire framing it's a it's a thinking process of putting you know boxes and arrows within some screens so that you can kind of understand how those interactions work but what I found is that because most wireframe tools are very static I mean God God forgive you if you work in Illustrator even but um even things like sketch like they're static and there we go so there's static and right now without understanding the interaction feel and the play out of something that's very hard to design an effective set it's kind of like trying to make a movie out of just individual stills but what you do when you make of movie as you make a storyboard so that then the actual cinema can fill in a lot of those gaps do you make decisions about a movie with a storyboard right and so I think that this idea of storyboarding or interaction sketches are just faster and that when you try and put together wireframes our tools are not nearly as flexible I think probably things like prototyping wireframe II stuff like gosh for some of them other than you expend that's what I've been using more of but like Flint IO or any of the mock-up kind of things those are fine because they actually help you walk through that but I come from the days where you were doing redline in very specific high fidelity mock-ups of every potential page state interation iteration and that's just it's just a waste of time if you're building that for spec so um I think the team approach for agile where you're using more of coming up with ideas and then rapidly iterating through code development to see how they behave within a design system for the aesthetics and the brand I think that's a much more healthy system for the types of products we need to build that's right go anyone love wireframes do you like gonna stand up and like you know protect wireframes from big bad hate hate sighs approach I skipped it out on a sketch and then I'd like take it and it's just as fast and neither like communicates nice when you deliver your wireframes and then you see what results like on the actual building code is there ever any change between those two is 14 it's do this actually that's that's not that's the key right so let's let's so it's a it's a good approximate not approximation I'm sure there's a lot of thoughtful thinking in that but what can happen is because the nature those tools for wire framing are so different from the technologies we're using they kind of need to change a little bit anyway and sometimes there's better ways to solve that solution based on the capabilities of the code base that you don't have access to in the same way for a wire frame so I think that that's why I'm kind of moving away from wire flowing land other ways that static Thanks anything that reduces questions especially when you're moving something into code is key so if it's working for you great but if you have that funk value of like well hello here's my 50 page PDF deck of all of the possible interactions and you expect code and interface to come out that looks like that then without any communication well you know welcome to the waterfall world of enterprise it's kind of what I would say on that yeah am I cursed but no well maybe you've been listening in yes there was a big fire at this conference I was out last week it was Enterprise UX speaking of enterprise I think it's enterprise see I said Enterprise your power went out so welcome to the startup world maybe I am cursed I don't know I'm not going to claim it but most the power came back on so that's good that's questions gonna vary dramatically when we were working let me tell you my personal story I haven't seen every company but when I've been working with teams they start with that like a customer behavior that they believe will that indicates like adoption and then you have to measure that actual customer cohort over a period of time and when people leave you kind of attribute that to whatever cohort and see if you can do any analysis or any understanding of that I think that's really for qualitative methods come in talking to people why are they staying why do they leave what are they using what's not satisfying them and having an ongoing practice for sometimes we call it usability testing but for customer discovery I think it's key for that I don't know that we'll ever really know I mean you've heard about the Evernote smile which is a visualization of Evernote usage I'm not sure if it still holds but Phil Libin discovered this which is people would adopt Evernote and they'd put a bunch of stuff in there and then they'd stop using it and then they'd forget everything that they had put in there which is the whole point of Evernote you don't forget it and then they'd be like oh wait it's in Evernote and then they'd start using it again but it didn't really have the value for recent stuff because people weren't wanting to look that up again right and so there was this kind of strange they called Evernote smile you can google it that's weird and once they figured that out they they could stop the Battle of trying to make sure that their hockey stick always went up they're like is just gonna be a flattening but then if it didn't pick up again on cohorts then they could start to do more analysis but every shape for those products is gonna be different so how do you prioritize I do don't have my number I got to like rank those it's a good question via I think start with basic and puts fairly as long as it correlates to a user behavior that's going to be directly related to what your product is designed to do I think start as simple as you can and maybe it is something is in Katy I mentioned in signups because that's the first step of that next thing but then you need to figure out like what is that what is that one thing that somebody needs to be the first time and in something like social media maybe it's making that first post or something try and get people to that stage I've noticed that metrics do generally sequence out over a life span of customer behavior so once you start with one and it feels like it's it's starting to move and you understand how to measure it then you can do move to something more sophisticated I'd say start simple even simple is gonna be more detailed than you thought but starting simple and having it be about what the product is designed and purposeful to do that's your first step and you know products don't just have one thing they do but you got to try and figure out the one thing that your customer really wants to do otherwise you probably there's a risk you have feature bloat anyway you know we start with features we think they're great ideas we release them we never measure whether or not people use them and then we release the next thing and the next thing and the next thing and now our customers are confused they don't care because they didn't want that crap anyway or you have three features that kind of all try and real realign with the customer behavior and you didn't know that because you didn't realize you were trying to move customer behavior you were just shipping features no I know it's a common situation it's really kind of tricky no no let's end on a high note like that's a good thing someone tell us a story of data or customer measurement success anyone got one of those no one's smiling yes [Music] what we now call that customer success ages one it is I'm gonna get everybody to save five by the end of the trial and like once we had those aligned and knew what each things with each bucket was we could say hey this must be said this many people and we started we are able to notice problems and the trial UX or twenty people on duty it's been a very helpful metric for our company and it also to get drill ship but they just very likely to go first so it also I didn't apply readiness there's a there's a restaurant in Marin somewhere this is an old Harvard Business Review article but it was one of the ones that it was a catalyst for even thinking about measurement of customer happiness or adoption or success it was quite a few years ago but this restaurant had a team approach to the tables and high-end and every time a server came by they would they would kind of assess what was going on at that at that table and then they would go back to the host station and like put a number on it and their goal was to get every table to a5 it was like one was kind of they were having an okay you know kind of crappy experience and five was as everybody seems really awesome and so they would do things if they went back and looked and there was like a tattoo or at a1 or something dropped him back they'd make changes so anniversary you know dinner and they used this example anniversary dinner and the very attractive server is getting maybe a little bit too much attention from the husband swap out the server right so you think about that but it was directly changing their behavior and they had this one story where this woman was dining alone and they just could not get her to a5 like she was quiet and risen reticent not really very interaction interacting and she was just sad and so they're like oh my god what are we gonna do so after the meal instead of bringing her her dessert they invited her into the kitchen and sat her down and the chef was there and they got to talking and she was kind of just part of the kitchen and it was later in the evening and so the chef as he's like why are you here today and she was there for their anniversary and she'd lost her husband the previous year so this was an emotional threshold that she was dealing with with deep deep sadness and being in the kitchen and being able to talk to someone about that being asked about that in that kind of very special private way got her to a5 but really ultimately it's not the five that matters it's that human experience and the care that we take for each other to get there so kind of similar your stages as well all right and that is our time thank you so much you have just been marvelous and the decks available

Channel: True Ventures

Views: 2,551

Keywords: User Experience, Marketing Metrics, Marketing, Startups, Metrics, Measurement

Id: x4T_Eg46L4c

Length: 68min 19sec (4099 seconds)

Published: Tue May 01 2018