Usability Testing: How to Do-It-Yourself with Steve Krug

hello little timer here so I keep more or less on track oh this was the advertised topic and and that's actually what we're gonna do I'm gonna talk about why I am such a big fan of usability testing and why I think everybody should be doing it now can you raise your hands back row if you can hear me if you can hear me raise your hands okay that's good if it gets too quiet let me know I have a knob here I can turn I'm not afraid to use it as we like to say all right yes awareness now unfortunately I know this is what my slide should be like they should just be big rocket of photos one slide one photo per slide and then I should talk as though I'm talking extemporaneously based on the photos and all that stuff but unfortunately that's not not me I am NOT the big evocative photos guy and the sad part is I'm not even sorry about it I'm unrepentant so it's bullets lots of bullets and a template straight out of office 2004 but again can you read the slide from the back of the room raise your hand if you can read the select resume yeah so okay I go to a lot of usability conferences and the most distressing thing to me about going to usability conferences is when I'm sitting in a presentation and there are slides that are like gray on gray type small gray on gray type and I'm sitting in the second row and I can't read them I find that distressing all right so me I am a husband father this is my wife looking at the crane from the balcony of our lovely room at the at home hotel is not quite right it's it's just the block away and it's the fourth floor of an office building that has like ten you know studios in it with kitchens and dryer washers and dryers it's very cool and has balcony looking out over a construction this is my son whose 25 and actually is taking up the family business he's going into UX which shocked me the raccoon in the corner is probably photoshopped because this was on a ferry and we live just outside of Boston this is our house ordinarily there's not a mermaid in the front yard but my wife was having a wedding shower for her niece on the other hand sometimes the front yard looks like this two winters ago we had 110 inches of snow all within 30 days all within so we had a record winter snowfall within 30 days the whole winter and then then there was this big contest to see on what date the final pile of snow that they had removed was finally gonna melt and it didn't melt the snow was in February it didn't melt until July the last of the snow so anyway but don't feel sorry for us it's a very nice City and it doesn't know that much every winter in I mean been a usability consultant now I am getting a usability consultant for 25 years maybe maybe even more my so-called company is advanced common sense but it's really just me and a couple of well-placed mirrors it's one person company once they call in the states of DBA doing business ass I've been doing business as advance common sense and even though we're just one person we actually have a trademarked corporate motto it's not rocket surgery which came in handy when I needed a title for my second book because it was hard to come up with a title after the first one worked really well so it was hard to come up with a title and the the that motto reflects the fact that what I do isn't that easy isn't that hard sorry isn't that hard sorry radiance Lux there in fact I think my son decided to take up the business because he's his office when he was growing up was right outside my office and I think he saw that I wasn't doing any real work so but but I do believe that what I do is really not that hard to do anybody who's in this business of ours this digital world who's creating stuff should be doing a fair amount of what I do for themselves they should be validating the usability of the stuff that they're creating themselves and that's what we're going to talk about tonight a lot of what I'm going to say may sound familiar because it really is common sense and that's the thing about common sense as soon as you hear you say oh I knew that but it's always good to be reminded of it I feel like my job a lot of time is reminding people of things that they already know and there's this cartoon from The New Yorker from a couple of years ago of the flight attendant holding up the seatbelt and the passenger saying sweet mother of god for some reason that oh I always identify really with this cartoon it's like that's my job so so first help me calibrate oh how many people have read some edition of don't make me think whoa geez okay three quarters easily great how many people have read rocket surgery made easy even wow that's pretty good that's like thirty thirty percent that's that's about the right ratio it's good ratio okay I never expected everybody who read don't make me think to read rocket surgery made easy I was thinking of it kind of like as the kid who goes to the same high school as his older brother who was on the football team you know I say it's never gonna quite live up to it but does does what he needs to do so III like rocket surgery a lot alright so you're probably wondering why I asked you here well what I won't really want to do is try and convince you that usability testing is one of the best ways to improve the quality of what the stuff you're creating okay which I believe it is I think it's one of the best and most efficient and effective ways to improve the quality of anything that you're designing that people are going to use and I'd also like to convince you that it's much easier than you think so you really can do it yourself and that you should be doing it routinely so that's my objective so how we're gonna spend this time first thing to take a little bit about why do usability testing why I'm so enthusiastic about then I'm gonna do a live demo test gonna ask for volunteer and I'm going to show you how I think you should be doing demo Billy tests which is very simple there's nothing to it and then I'm gonna talk a little bit about the six Maxim's from the rocket surgery book I tried to pull out I usually learn like three things from a book if I'm lucky so I there were like six things in rocket surgery that I thought was really important for people I would really be happy if most people who read the book ended up kind of remembering these things so I pulled them out and made them into Maxis so we're gonna talk very quickly about them and then we're gonna try and do as many it's much question-and-answer as we can so hopefully a 30 minutes maybe at least 20 all right so another show of hands your experience with usability testing so the first one of these which is true for weight no this is this is in the wrong order somehow okay you you you've observed tests you observed tests okay you actually conducted tests yourself you were the facilitator whoa what goes on in this town geez okay well that's great all right then here's the essay question like maybe two or three people to answer this and shout really loud when you do if your team doesn't do routine you use really testing routinely like on a regular schedule often many times during the course of development of a product why do you think that is one of the reasons for not doing it more often or for not not to or for not doing it off you don't do it at all somebody yes right there stand up and shout finding users finding users can be difficult okay that's very common common yes okay customers will freak out okay that's it that's interesting - yeah ignorance of the value of testing okay generally speaking yes it's gonna cost more you're gonna have to charge the client more or you're gonna have to ok fixing okay so fixing the problem so you're fixing the problem so you find is is gonna is gonna cost more okay one more yes time constraints okay and which which kind of time are we talking about are we talking about not having enough time to do it or the fact that doing it is gonna slow the process down or both probably both okay yeah there are those two different kinds of time constraints that that keep people from doing testing all right so first what is this usability stuff well for me it's it's just an attribute of good design it's it's hard to come up with more than that and I have a definition of it for myself that this this attribute something has this attribute of usability if a person of average or even below average ability and experience is ie almost anybody who has a reason to use it can figure out how to use it for its intended purpose and then the one that I like best which is without it being more trouble than it's worth because very often something may be hard to use but if you really need to use it then it may be worth the trouble so I'm sure all of us have has struggled really hard to use something that's hard to use but it was the sole source to get the information we needed or it was the only you know it was our bank and we it wasn't worth the effort of changing banks to get around it so so that constraint comes into a lot it has to be not be more trouble than it's worth so these usability problems there are a lot of usability problems out there their usability problems everywhere one of my favorites is you've probably seen one of these these coffee makers I've seen I saw a brilliant CEO and one of the one of the smartest and nicest guys that I know of spend ten minutes trying to figure out how to use his company's new coffee maker to make me a cup of coffee so we all know them sometimes they're not they you know they don't get in our way that much they just slow us down so lately I see this more and more where it comes back and I enter an address and it comes back and says is that what you really meant so here for instance I often find that I'm not sure whether it's telling me that it doesn't like the facts I didn't capitalize my address or it doesn't like the facts I spelled out the word Road but I you know I kind of have to figure out what they met just to make sure that it's not going to get delivered to somebody you know someplace else I also love these the the embossed low contrast labels on manufactured objects so like these were creamers you know cream and skim milk and and whatever and they were impossible to read they are really cool sometimes just the wording this is every time I go to go away on vacation or to our business trip I suspend home delivery of the New York Times and so I now know how to find the right page in the New York Times website which has taken some getting used to because it's not always easy to get there but now I can do that routinely but I find that every time once I get there I'm brought up a little short by these two fields so even though they're nicely done and it's gonna pop up a calendar and ask me to choose the suspend on date does that mean the first day it's not going to get delivered or or the day before when it's gonna really stop so I you know I kind of usually know really how to answer it but I'm not a hundred percent certain but but but again if the worst case is newspapers gonna be lying on my lawn or I'm not gonna get the newspaper one day so it's not it's you know it's not this we're not doing nuclear power plant controls here it's really not a big deal sometimes they just cause us anxiety this is a guy in the in the states called Lewis anyway knew who louis c.k is okay I love Louie CK Lucases fabulous this was the thing that he did where they were looking at parking signs and they were looking at this one as a parking permit after midnight enough to read you this text is it after midnight right now or is it before and it's before midnight right now yeah but it's also after and this was this was great because we you know many cities in the states we have these these trees full of parking signs that are sometimes contradictory and often just unclear and somebody actually came up with this kind of you is a proposal for a uniform method for signage on the right oh but you know it's not gonna happen because there are so many legacy signs out there but it's a great idea sometimes these usability problems are just really annoying like here I was trying to was I trying to do oh I know I was trying to do something with JetBlue and and so I I think I was using it on on my phone or something I didn't want to bother doing much typing so I just try typing one letter for my first name and it came up and said first name should be between 2 and 20 25 characters so I figured alright I'll just add I'll add a numeral after it and I did too and it said first name format is invalid so I did what we all do which is I made up a name and I don't know if any of you ever seen the Saturday Night Live's get where where that's a parody of Jeopardy where we're what's his name what's the actor's name Burt Reynolds signs himself a mr. Ferguson so it's become kind of popular alright sometimes they just scare us off I was trying to do something on Google+ I think was just trying to read something somebody written or something on Google+ and Google Plus wanted me to give it all kinds of powers it wanted to know not only know my basic and profile info but include people in circles that are not public on your profile and allow people in those circles to know that I've signed into this app with Google view my email address the most puzzling one was manage my calendar I just wanted to read something Google asked me to manage my calendar all right and then sometimes they grind us to a halt this one's sweats I file does not exist click retry to read from the same location to create a new version according to the current scheme click ignore my options are retry ignore or cancel how long would you spend puzzling over that one I was just trying to do a backup for heaven's sake and this is it actually in said hotel room at at home that one of the beauties ever we're halfway through our chip and what I liked about it when I when I chose it was it had a washer/dryer so I said well we can we don't have to take as many clothes we can do washing it with you but it turned out that the the dryer didn't want to dry it washed great but it could like pretend that it was drying clothes for an hour and they'd still be soaking wet and obviously there was no way to figure out I figure it out from any of these controls even those people who run the place had posted very nice photograph of instructions whatever so I went online and searched for the manual for this dryer and after getting through those hateful people who put up all those sites where it looks like you're gonna get the manual for your thing but instead they're asking you to download their software that will download the manual so that they can then steal your computer or your identity or whatever but I did find one and I found the section on drying and I found the thing that I said just to press the Hydra high dry button F and then I actually managed to find an illustration the F and it actually worked so you know I mean we do amazing stuff now I mean we were walking around Australia and and now New Zealand and I'm thinking you know 15 years ago I would have been carrying big thick guidebooks that I only needed a small number of pages from and I've been at folding and unfolding maps all day you know and just all this stuff that that works so wonderfully so if you're interested in usability problems in general I highly recommend it if you haven't read the design of everyday things by Dawn Norman it's a nice thin book and it's where he wrote it back in 1987 it's still relevant he talks a lot about you if people say you'll never look at door handles or shower controls or elevators elevator controls of the same way again as those of what he uses for examples and they're really fabulous and we're in I ran into this elevator recently where the round buttons the round buttons were there and they were nicely labeled and for some reason they had put labels next to them that were around and that looked exactly like the buttons very confusing okay on the brighter side though there's a lot of stuff that's done really well I always think Amazon's done amazingly well I live on Amazon Prime and read Amazon reviews all the time and now they added questions and answers so you can actually say will this work with my model blah blah blah and somebody who has one will answer it comparison charts uber obviously works fabulously well although it works obviously well I'm not sure about the how I feel about uber in general for a number of reasons but but it does it gotta admit that does work well they sort of started late so web app so mobile applications were you know kind of fully formed so they started right with a good base to start on and just did an amazing job of making something incredibly convenience that had not been convenient at all so they get really good marks on usability whether their ethics are questionable or not um my own my only disappointment here has been that you don't have open table anybody know when a open table is yeah open table I was surprised I was shocked that open table worked in Australia but it's not here yet Oh at least doesn't seem to be I was you know searching for this and it came up and said no no restaurants found in Wellington first in Queenstown oh it's fabulous you can basically just it'll show you what what the local restaurants are it'll show you what reservations are available like around the time that you want rather than you having to poke and say well give me a reservation at this time no you musician at this time no and you can get a resume you make a reservation in seconds so look forward to it showing up here so why usability testing well a long time ago when I first started writing don't make me think I'd been consulting for 15 years at that point and I realized that I had every time I butt was brought in as a consultant that part of the reason why I was brought in as a consultant was because this same discussion had been going on for a long time about how to handle the interface and so they decided we'll bring it outside expert and so I was the outside expert so I may type it in the form of a comic and don't make me think there's a usual cast of characters project manager developer designer guy from marketing the designer tries to get the ball rolling and makes a suggestion about how to handle this part of the interface let's try it try pull down menu but the developer comes back with a personal opinion disguised as fact which is very common in these conversations nobody likes that stuff and but but the design is no fool and she knows that's what it is so she comes back with one of her own the marketing person knows that in the world of marketing there is this thing called market research and then if you do well for in the market research you can actually find the answers to marketing questions like you could find out how many people will pay more for a certain feature and so he hopes that maybe this same approach can be taken to this problem the problem is that most usability and UX design issues are contextual it really depends on the thing that you're building there are very few you know absolute truths in terms of usability and UX design and in fact a lot of people with the joke is that usually people always say it depends which I I kind of like I think it's actually pretty true and then the developer comes back with the developers trump card which is we can't do that and if the person who is responsible for implementing it says we can't do that then unless there's somebody with a higher ranking in the room they're gonna win so it's effective trump card the manager poor manager so so she tries to you know she's plucky and her job is just to keep this thing moving so she tries to get it rolling again which is followed by the long hypocrisy silence which is followed two weeks later by the same discussion so this is what I sort of knew I knew that when I came in these situations this discussion had been going on over and over and that it was that is that there was it had no resolution and I thought of them as religious debates because they're based on deeply held personal beliefs and nobody ever changes their mind okay so that's why I thought that way and the deeply hurt held personal beliefs I realized eventually were that because these people have the jobs as they have because of who they are so for instance people become developers because they like complicated stuff okay it it's just true they like complicated stuff they like figuring out how things work all right they like looking at a complicated interface and figuring out how it works reverse engineering in their head figure out what they can copy from it but it's and and the thing is when they doing that there are endorphins involved you know and that's why they became developers because of those endorphins and because it's happening at the endorphin level they're never going to change their mind and it's gonna be very hard for them to imagine that anybody else doesn't get pleasure out of that all right and the same thing with designers designers you know if you've got lots of hair line rules and gray-on-gray type designers there will be endorphins released where the designer looks at it and so it's hard for them to imagine that that's not what people like most about these online experiences so that makes it very hard for people to find any common ground when they're trying to make those kind of decisions the thing is we're all users we use this stuff ourselves so we imagine that everybody uses it and thinks about the same way they do but we're not our users we are not them we're definitely not them and the other thing is we don't appreciate how diverse they are generally speaking it's so easy to imagine that other people look at and use this stuff the way I do but it's just not true once you start doing usability testing once you start spending time living among users you realize that it's incredibly the use is incredibly diverse but you know I am I like to say all uses idiosyncratic you know everybody is approaching with a different combination of prior knowledge you know exactly what their goals are what kind of situation they're in all kinds of things are very different and the other thing is that it's it's when we're building something it's almost impossible for us to remember that people don't know what we know and that's kind of the biggest problem is when you're working on something you know too much about it you know how its works or how it's supposed to work and so it's hard for you to imagine somebody approaching and trying to use it who doesn't know that that's very hard and that's why usability testing is so great because it gives you a chance to watch somebody else have that experience hey this is Lisa Reichheld who just moved back from the UK to take over the government aux basically for government sites good news is that I do think that there's a solution for this kind of conflict between team members coming at it from different directions and I have been going around for a while now trying to teach people to do it do their own usability testing so alright what's the usability test use BHS is incredibly simple it's just watching people try to use what you create so you're watching them use it it's not a focus group where you ask people questions about things or you ask about their opinions or their prior experiences you have you have if it's a usability test you're watching them try to use it even if it's a prototype you're watching they try to use it at the same time that they think out loud so that you have access to their cognitive process while they're using it you know what they're looking at what they're trying to do what they're thinking about what's frustrating them I always say you're looking for the question marks forming over their head that's where you get the insights from and that's kind of it watch them use it while I think out loud so we talked a little bit about this earlier most sites don't get tested every thinks it costs a lot of money which it used to and if you don't do it yourself it can cost a lot of money takes a lot of time the time spent doing it and the time the the time that it will add to completing this thing because you have to wait for the tests to be done and then you have to go in and fix things that were found while the test was done and if somebody said over here failure and understands value so all right so we're going to swing into demo mode and try an actual test so I'm gonna ask for volunteer but wait until I get through the slide first yeah the process is painless a nice thing about usability testing is people actually usually enjoy it and they often learn something it's brief in this case it's gonna be like 10 minutes and you get a round of applause when we're done now and then you actually use a bit I don't get a round of applause but here you will but you get something so qualifying criteria for our volunteer you have used a web browser I keep the standard very low and this was the question came of a comment about recruiting the recruiting is hard okay you can make recruiting easier by keeping the standard very low and I'll talk about that that's one of six Maxim's is keep it low so I say here's somebody's used a web browser some which by which I mean somebody who knows where the back button is and somebody who knows that things that have an underline underneath them or are all the same color if you click on them you likely get another screen full of stuff that's that's it english-speaking adult which only means is is a reminder that you shouldn't do this in translation that so much of what you're looking to learn depends on nuances of language that you actually can't have a test being conducted by somebody else and a translator for for the observers you need a facilitator who speaks the same language as the participant and he needs some observers who speak that language to not a low talker just for benefit of people in the back room and if you know the Seinfeld notion of a low talker so if ever people are always saying excuse me to you then please don't volunteer and finally as a bonus question not a necessity but a bonus somebody who hasn't used the New Zealand immigration site so I'm sure there must be somebody in the room so who would like to volunteer and we'll keep moving right the yes please come on up and now while we get going the rest of you are now in the role of observers so you're going to pretend that you're sitting in a room down the hall from us and buy a screen sharing you are watching what's what's going on on the screen and listening to our conversation and eating snacks Oh so hi I'm Steve hi I'm Tina Tina hi please sit down here all right so I'm gonna read my usability test script which is in the rocket surgery book but don't tell anybody you can download it for free from my website in fact all the documents that the checklists and forms and consent forms or whatnot later in the book are on the website and you're welcome to them just go to sensible common CNS ible calm I actually got an english-language word for my domain a long time ago and go to the part of the site that's for rocket surgery the rocket surgery book and you'll find a download link so so and I will read it verbatim because that's why I recommend you do I can tell you why during Q&A if you want to know all right uh hi Tina my name is Steve I'm gonna be walking you through this session today great before we gonna have some information for you I'm gonna read it to make sure that I cover everything okay you probably already have a good idea why we asked you here but let me go over it again briefly we would have tell Tina and when we recruited her in a phone call or email whatever basically what's gonna happen we're asking people to try to use a website that we're working on so we can see whether it works as intended a little unusual the session should take up about 15 minutes first thing I want to make clear right away is that we're testing the site not you you can't do anything wrong here in fact there's probably the one place today where you don't have to worry about making mistakes as you use the site I'm gonna ask you as much as possible I try to think out loud just say what you're looking at what you're trying to do and what you're thinking this will be a big help to us okay also please don't where you're going to hurt our we're doing this to improve the site so we need to hear your honest reaction if you have any questions as we go along just ask them I may not be able to answer them right away since we're interesting how people do when they don't have someone sitting next to them to help okay fine if you still have any questions when you're done I'll try to answer them man and if you need to take a break at any point just let me know okay you may have noticed the microphone there would be a microphone on the desk or in the laptop with your permission we're going to record what happens on the screen and our conversation the recording will only be used to help us figure out how to improve the site and it won't be seen by anyone except the people working on this project great and it helps me because I don't have to take as many notes great also there are a few people from the web design team were observing the session in another room they can't see us they can they can't see us just the screen okay that would be true we wouldn't have a webcam waiting Athena if you would I've gotta ask you to sign a simple permission form for us it just says we have your permission to record you and the recording will only be seen by the people working on the project okay it has those two sentences on it and so it's really easy and then I would lean in and start a screen recorder like Camtasia because prior to this point we didn't have to use permission to record okay you have any questions so far No all right before we look at the site I'd like to ask you just a few quick questions first what's your occupation what do you do all day I'm a business analyst i what i work for trade me oh okay oh okay it's a it's a web site it's like New Zealand's eBay okay great roughly how many hours a week altogether just a rough estimate would you say you spend using the Internet including web browsing Annie at work my every day all day every day all day yeah like like eight to ten hours a day is that I tell ya what so what kinds of sites work and or personal are you looking at when you browse the web what kinds of sites uh usually trade me I'm not allowed to say Facebook but like at work or outside I like to read lots of like technology news sites yeah like ink.com Fast Company HP uh that kind of stuff hmm all right so we're done with the questions we can start looking at things first I'm going to ask you to look at this page and you can use the mouse that's there too if you guys Oh Club it should work let me take this anyway first look at this page and tell me what you make of it what strikes you about it whose site you think it is what you think you can do here and what it's for just look around do a little narrative you can scroll if you want to but don't click on anything just yet okay cool um I know it's for a web site for New Zealand immigration it looks pretty modern I like the design looks really straightforward really easy I like to come to New Zealand to visit study work that's you know there's not much ticks on Oh which is good so what do you think you would do here from here I'll probably click on one of these buttons depending on what I was interested for oh I see there's like a search box here as well so I could and I assume I can apply four different phases or something to come to New Zealand is a Centauri surname so yeah alright so now I'm gonna ask you to try doing a specific task okay I'm gonna read it out loud and give you a printed copy I don't have a credit copy for you but I would give you a printed copy so just read out loud no Hannity I'm going to ask you to do these this task without using search you can use search within the site but don't go out and use like we'll learn more about how well the site works that way and again as much as possible it will help us if you try to think out loud as you go along here's the task I'd like you to do you are a United States citizen with a USA passport and you live and work in the United States okay what you do assume that your company is sending you to be a presenter at a conference in New Zealand and you and you think you may need to get a visa for the trip okay okay use this site to determine if you need to apply for a visa and if you do start the application process so here just in case you want to check anything yep there's a copy of it so you're gonna be a presenter yep all right so just go ahead and do whatever you would do and just think out loud what you're doing okay so I'm a United States citizen with the USA passport so I kind of want to just go into a search bar on actually do I want to go to visit now I think I'll just go in to visit I'm gonna let you click yes okay click whatever you would do okay so I'm thinking about visiting New Zealand explorer visitor visa option I'm just gonna scroll down to see if there's other options planning my visit and staying safe in New Zealand I'm just gonna go back and click on that because that sounds like what I want to do so I'm thinking about visiting New Zealand there's a things to do in New Zealand recoving trips Vista Information Center I'm thinking that's not really what I want to know that sounds like more like if you're going there for holiday or something well maybe there is me I'm not sure Explorer I know okay so there's a Explorer Vista visa options so I think that sounds like me okay so I'm gonna click on that explore Vista visa options Vista visa allows you to come to New Zealand for a holiday sightseeing adventure activity yeah that sounds like me yeah I'm a I'm travelling on a passport from okay cool so I can into United States of America that's me that's pretty straight forward I'm planning to stay listening three months he's the main purpose on my visit is business or work call so that was really easy so display options I assume that's what I'm into click Next compare options that may apply to you to decide which suits you best okay so there's a lot of options but I think it's I think it's business business stands out to me and you can come to New Zealand for business reasons for up to three months in any one year okay yeah that sounds like me so I'm going to click there okay so this is actually comparing so how can I actually find information okay okay there's view view cool so what was my task again use the site to determine if I need to apply for a visa okay so business visitor visa this business for you if you don't want to if you are if you want to come to New Zealand for business reasons yep that's me couture you must me you must show that you're a genuine business visitor okay you must have enough money okay you must also have a ticket to leave yep that's me up to three months yep that's fine what this visit you can visit New Zealand for business okay so what's considered business reasons I'm just gonna open that in a new tab because I don't want to lose this page and that just goes back to another page shows on there okay any interest time we're gonna stop there that was perfect exactly [Applause] okay now I hasten to say that I did not choose the New Zealand immigration side chose it because I had used it because I needed to answer that same question but I did did not choose it because it had problems it's in fact a very good site and once I got past the top level here it worked incredibly well and gave me exactly the information I needed my theory is that there that all immigration sites in the world was the possible exception of the UK site because the government digital service there has been working on it and they've been doing remarkable work making things easier to use I haven't looked at it lately that they all have basically the same problems and that the problems are not for lack of trying on the part of the people who work on them the problem is that immigration and and and guiding people through the process of immigration is a wicked problem and it's mostly made wicked by the fact that that it has to present laws which are inherently complicated and in some ways subtle so I I only use this because I had already used it a couple of times and and it worked out quite well other people had much more trouble than Tina oh and which I won't go into oh but if she actually was doing doing fairly well but but she was not headed in the right direction I can tell you that much and and and it's because that there's that question of do I need a business visa or or can I get by with a visitor visa is complicated and it's complicated in all countries and it's just not easy to answer but basically that was not about the site it was about the process so the process anybody want to suggest one problem that she ran into that if you were observing this that you would have thought was a usability issue that you might want to deal with too many options it did it you did to have this feeling when it asked if you wanted to compare that there were a lot of options there that you could compare and so there was a lot of reading to do and in some ways there was a lot of reading to do but those little snippets that were there weren't really enough to tell you which options you wanted to compare so it wasn't actually it didn't it didn't even work that well as a basis for choosing which ones you wanted to compare and that's partly because it's a wicked problem so that may not be quite the right interface to deal with that anybody else want one other quick one yes oh yeah sorry sorry I'm preferential to the people in here because I can't see you back yes it did circle back yes yeah yeah when she clicked on that so I'm not sure whether that's a bug or I think I did run into that myself so in a nutshell what I recommend for do itself usability testing is that for each round of testing you test three users not not five users not seven users not eight users three users and the reason for that is that it's if you only test three users obviously you don't have that much recruiting to do which is great and you will find generally speaking all the problems that you need to find you know as many problems as you can fix or more problems that you can fix by testing three users if you test more users you will find more problems but the fact is you don't need to find more problems than that you'll find you'll find very serious problems with testing just a few users so keep it as simple as possible you don't need a lab you don't need to wake to a glass to a mirror or whatever it's properly called screen sharing works incredibly well now so you just set up screen sharing in another room of the monitor so people can observe you can record it with something like Camtasia or screen recorder you don't need entrance and exit questions because you're only testing three people so there's no there's no validity to anything statistically you could you could gather so there's no point in doing it you don't need to create a big honkin report historically what would happen with usability testing is you bring in an outside consultant who would run a usability test they would then do all the observations themselves they'd go off and spend two weeks writing a big report an 80 page report that to find the fact that you were spending $10,000 to run this around to usability testing it didn't really benefit anybody and so instead of a big honkin report what I recommend is that you debrief as a team right away after the test and draw your conclusions right away after the test about what you saw and what you're going to fix and then the report consists of a two-page email maximum two-page email with some with a bunch of bullet lists on it that basically just says here's what we tested we here's what we had available to test this is what people were shown and you have a link to the prototype or the part of the paid part of the website or whatever it was that they were testing so people can go and look at it for themselves they want to hear the task we gave them the test and here the major we were the worst problems that we observed that people ran into and here's what we're going to do about them but you do in a very simple statements as a bullet list because that's what people will be willing to read okay sent down an email okay so real quickly then just so we can get the question is that the six Maxim's from the book one start earlier than you think makes sense people always want to wait to start doing usability testing and one of the reasons there are a whole bunch of reasons for this but one of them is that you always have a better version in your head than you have available to show you always know things that you're gonna fix you always know things that you're going to improve you just haven't had time to do them yet so people say well why should we show people this version that in a week is going to be much better why don't we wait until it's better and then show it to them because if we don't we're gonna waste our time and their time having them point out problems to us that we already know exists the problem is that's not what you're looking for you they're going to point out to you the problems that you don't know it exists because they're gonna look at it differently than you are and the other thing is that the earlier you start testing the sooner you discover what these problems are so that you don't build them into your product you know the longer you wait to discover these problems the more expensive it is in time and money for you to go back and fix them okay and sometimes they become unfixable if you don't discover them one too late in the process so earlier is always better this is was my flowchart for how people think about it if it's finished okay so our testing and my the way I think you should think about it are you working out yes you should start testing as soon as you start working in some ways even before all right you can tell things you can test you can test almost anything if you're redesigning something you should test the old one if you haven't even started designing one of the best things you can do is test competitors products or comparable products people who do trying to do the same kind of thing you are there's absolutely nothing ethically wrong about doing usability tests on somebody else's website I always say they they basically they like they worked out a completely flushed out fleshed-out version of a design approach to the problem that you're going to be trying to solve and they left the keys in the car and since there's no reason not to do it you can learn an enormous amount just by having a few people do the tasks on somebody else's site so that's actually a very good way to start you can test the sketch on a napkin you have a sketch on a napkin basically take it to take it to a couple people and say tell me what you think this is described that's like you did with the narrative of the homepage tell me what you think this is for and what you do here and they'll say well looks like you're selling like copies of research papers and I guess over here you could search for them and here you could order them and blah blah blah and if what they describe is what you're trying to do then get a bigger napkin and keep going but odds are when you show them a sketch on a napkin they will reveal something to you that you did not think of okay and it's an incredibly cheap and dirty way to to find out wireframes obvious people done forever prototypes much more popular now whether they're in balsamic or they're actually built in a framework prototypes page comps you don't have to have a whole thing built you can have portions like you have what they call a garden path where you just have the pages where if people click on the right things they will go to that succession of pages and if you have the garden path built then you can test with that and if somebody clicks off the garden path then it just brings up a little screen that says this part of the prototype has not been built or whatever so oh and obviously you can keep testing all the way all the way through development my proposal for how often you should test and and when is a morning a month I recommend that you pick and obviously this is not an agile version of this if it's an agile version that's a whole different story and it's whether it's once a sprint or whatever we can talk about later but but basically if you're not working in agile I recommend that you pick one morning a month there's a agency in the US government that the woman who took this up and started doing what you called third Thursday's and so the third Thursday of every month was testing day and on the morning of the third Thursday they would have recruited three people to come in and they would test basically whatever they had available to test at that point now part of the butte is a two things I really like about this one is if you say we're just gonna do it once a month and that's pretty doable almost everybody can devote one morning a months of doing user experience design usability testing to it unhinges it from their decouples it from deliverables so traditionally you would schedule a usability testing based on when you were gonna have a particular deliverable we'll have a prototype on this date okay we'll do usability testing that week or we're gonna have have alpha on this date well we'll do usability test that we never worked out because the stuff was never on time and so it's hard to plan the usability testing the deliverable would slip the usability testing would slip and never get done so if you just say okay every every on this day every month we're gonna do usability testing with whatever we have available that's a reasonable shape for us to test then it makes it much easier to keep doing it gives you one less excuse recruiting loosely and grade on a curve I talked about earlier in terms of keeping the bar really low people historically when you start saying we're gonna do usability testing everybody start saying well where are you gonna get people you're gonna get people from our target audience right you're giving you people for our target audience right you're gonna do that it depending on what your target audience is it can be hard to get them defend it so and in point of fact when particularly when you're starting to do usability testing you don't need people for your target audience you can take people who are not and feed them some of the domain knowledge hints that they need and it works perfectly well because in the beginning the problems that are built into your stuff are so gross that anybody is going to run into they don't have to have domain knowledge where it works really well okay make it a spectator sport this is probably my favorite what it means is if you want real value out of this get people to come and watch your monthly usability test which is another reason for only doing them once a month everybody knows their Thursday is usability morning ok so what you want to do is gather as many people as you can possibly get to come and watch one or all three of these sessions sit in the room and watch them together and what that does is so I'm talking everybody I'm talking from c-level people to stakeholders to developers to designers everybody who has an interest in it and the more you can get people to come and watch some of these the more they have develop a shared experience of seeing people actually try and use the thing that you're building and they learn a lot about who the audience is and how people are going to use this and whether your approach is working or not they get to share those experiences in the in the conference in the observation room and they get to take part in the debriefing right right after it I the most important thing in this illustration is the plate of doughnuts in the middle of the table I say if you want to get people to come the best thing you can do is provide the best snacks that you can so provide some snacks that no other department in your company provides get the $4 chocolate croissants or the bottles of Fiji water or whatever if you're going to spend any money spend it on snacks people want to come okay focus recency and small number of the most important problems um the problem is that as I said earlier usability testing uncovers a lot of problems in a hurry okay and it can be and it's not so much as it's overwhelming it's that you can it's very easy to find more problems then you actually have the resources to fix you probably don't can't allocate that many resources to fixing usability problems and there is the the low-hanging fruit problem supposing your usability test uncovers 20 problems that the developer realizes they can solve them in one minute each and their performance review is coming up in two weeks are they going to tackle those are they going to tackle the one problem that everybody - that's really serious but that may be kind of intractable okay and and may not pay off there's a great tendency that tad for low-hanging fruit and so you really got to focus on the most serious problems otherwise they'll just persist and you'll end up in when things finally bill it will still have these serious problems because nobody nobody wants to put the the time in to tackle them I and hit this off by giving people a form to use it in the observation room so the each observer is told that this is the form they're supposed to use to capture their feedback for the debriefing they can keep any notes they want they can write down thousands of usability things that they that they notice but at the end of each session at the end of each participant session they have to write down the three most serious problems that they observed during that session before they forget them and this page with these nine nine problems on is what they bring into the debriefing session and when they go into the briefing session they have to pick three out of those nine to give to the group and that tends to focus it on the most serious things okay and finally since we're focusing on these serious problems you know what's going to be able to fix these serious problems at least perfectly and there's a real temptation to say we got to fix it perfectly okay we always saw this problem with search well we could redesign search if we redesign search then that problem would go away along with a lot of other problems you don't have the capacity redesign search most of the time so what you want to do is tweak instead of redesigning okay so you you're you should always be thinking what we just observed a serious problem what's the smallest change we can make that might solve that observed problem okay ready going in and making it perfect what's the smallest change that we can make right now that would eliminate that for most people as a serious problem and then make that change you know and then it's done it's done right away it may not it won't be perfect but it's much better than waiting for a perfect solution so I'm gonna skip past these and this is my list of nine reasons why tweaking is better than redesigning my favorite is the last one which is a redesign mean you know it means involving a lot of people in a lot of meetings enough said all right and I'm done with my slides except for these four tips one is if you or anyone you know works on writing or editing for the web they if they don't have this book buy it for them they will be intensely grateful to you it's called letting go in the words it's by a woman named Ginny reddish and it's a fabulous book about writing or editing for the web um I also happen to like Laura Klein's book UX for lean startups it's very thin I think it's still only in hardcover but it's it's a really good book about doing UX in a hurry and remote testing if you haven't done remote testing same thing as what we just did here except we wouldn't be in the same place we just be using GoToMeeting or WebEx or some kind of screen sharing Tina would be at home and we did what we do exactly the same thing the only thing you miss out on is you're not sitting next to the person so you may miss some body language but turns out it's not that important if you have good audio so don't use a speakerphone and use voice over IP built into the screen sharing you know go to a meeting or whatever so you have high quality audio and then you'll be able to tell what they're thinking the only caveat about that is you shouldn't start off by doing remote testing you want to do a bunch of in person like maybe eight or nine in-person tests so that you really have the process under your belt because the one thing that is true about the remote testing is you have to pay really close attention you because you may not know whether they fell silent because they're thinking about something or they fell silent because their dog just came into the room and they're petting their dogs so so you have to be capable of paying really close attention and also remote unmoderated testing which is things like usually testing or comment you might know user testing outcome it's basically service you send them a URL and you send them the task and you say how many people you want to have perform that task they put it out to their pool of testers who then record with the webcam record and do the task and record a think aloud while they're doing it and then you get the video and you can play the biggest beauty of that is you can get the video back in like an hour so if you have something quick and dirty that you want to test it so it's a really good way to do it this is a box fish because many people like that anybody ever seen a box fish yeah most people there they're really cool right they're like really cool so a box fish I put decide in because most people have never seen a box fish and there is if you took a ping-pong ball and put it in a press to make it into a cube and then you glued an adorable nose to the front and adorable tail to the back they really are they're really cubic and they're they're just just wonderful to see so I don't know if any query an aquarium query I in New Zealand have box fish but I highly recommend them and I would be happy to hear from you and take questions now

Channel: Optimal Workshop

Views: 57,932

Rating: 4.8917294 out of 5

Keywords: #usabillitytesting, #stevekrug, #usertesting, usability testing, usability, steve krug, usability test, how to do usability testing, how to run usability tests, how to conduct usability tests, how to do a usability test, when to do usability testing, dont make me think steve krug, how to do a user test, how to do ux research, usability testing benefits, usability testing (literature subject), steve krug book, how to improve ux, steve krug ux book

Id: VTW1yYUqBm8

Length: 55min 46sec (3346 seconds)

Published: Wed Jun 29 2016