Open Lecture by James Bach on Software Testing

good well my name is James Bock I'm going to talk about testing today has anyone here ever studied testing at all taking the class in it and he's raise your hand if you're taking a class in testing you know you have Oliver you've taken a class in testing is you taking it now there's a class and testing right now is in school standards okay that's useless totally useless you don't want anything in there that you will ever use again to actually help society do anything and I am absolutely serious it is completely useless if I had my power if I was if I was empowered by the United Nations I would abolish classes like that but you might want to humor the professor you know don't tell him I said that Sarah feel really bad but actually it's completely useless and what the idea is that that are used in that class are 30 and 40 years old and they didn't work 30 and 40 years ago they don't work today they didn't work in 1990 when I first criticized them and was first ignored by the people who were using them and they didn't work in 1995 when I criticized them again in writing and in conferences all over the world and was still ignored by the people who used them no I'm going to talk to you about testing today I'm going to talk to you about real testing so get ready all right a little bit about my background first of all I love the sign that says dr. Bach I'm a high school dropout I'm not a doctor I am opposed to the whole idea of PhDs as matter of fact so I love giving little pictures of these signs because people assume with my I'm kind of famous as a tester they assume that I must have a great deal of academic achievement but no I'm I'm an academic revolutionary I'm the only high school dropout you know who's written a book about being a successful high school dropout okay not a lot of high school dropouts dude and I started at when a 16 is a video game programmer on the Apple 2 in Commodore 64 right after I dropped out of school well not right after I went through a little stint as a computer salesman and I was the worst computer salesman ever I never sold a single computer and I broke one so I'm I'm negative one on computers selling Apple 2 computers in 1982 and then a guy walked into the store one day and said I'm looking for programmers do you know anyone two programs and I said I program and I showed him this little program I wrote in assembly language all it is it made dots fly around the screen that's it it's all it did and he said you're hired for $1,000 a month which to me was a princely sum at that time so so I became a video game programmer I did that for four years then Apple Computer hired me I was the youngest manager at Apple Computer when I joined app a computer as a test manager having never tested before and never managed before Apple Computer hired me anyway which tells you right there how screwed up the testing industry is because you don't see that so much in hospitals that people are hired to be managers of doctors and don't know anything about doctoring but I was hired to be manager testers and so I decided I better learn testing and I read every testing textbook that I could find and about six months later I looked around the 400 people in the group the building where all the testers were at Apple Computer and I realized nobody is reading testing textbooks but me I'm exaggerating a little bit there may have been ten people in Apple Computer total who were reading testing textbooks but it was mostly me and I found out why later on they're all useless the books are useless they're written by people who don't understand how to test but it's okay because no one else understands either so people who read them don't know that they're that they are full of missing for me the people who write them don't know that they're giving misinformation and have you if you find this a little bit incredible I want to remind you that in the world of medicine for more than a thousand years gay lemak medicine ruled Europe galenic medicine you know like where you bleed to cure all kinds of ills and and where there's four humors that make up the human body you got your black bile your yellow bile your phlegm and your collar and it's the combination of humors that lead to all sicknesses right and they had certifications in this all around Europe you know you had to go to school and learn to do this medicine that doesn't work it never worked but they but for a thousand years that was the the best practices of medicine until when until some people after the Scientific Revolution got got started started saying hey wait a minute this stuff doesn't work what what really is going on in the human body and in the last 300 years the entire world of Medicine has been revolutionized because people started saying no to the textbooks that were wrong and that's what I'm doing now in the testing industry and I'm fighting a whole bunch of people especially in Europe which are they they're saying this galenic testing medicine is the best practice and you have to bleed your software and it had all software testing consists of yellow bile and black bile and I'm trying to say no it doesn't work so I stand before you to try to reframe things for you and thus create even more trouble in Europe so if you are expecting something different haha all right let's get started let's take a look at this here is a simple flow chart and I want to start very simply here and just ask you a question if you had to test something represented by this flow chart how many test cases would this take can anybody call me - why - what are the - you want to do yes and no and what what what will that accomplish by doing that all right so you will have achieved what they called you know the technical term for what you've achieved you've achieved decision coverage you also achieved predicate coverage you've also achieved all paths coverage by doing that so it's three kinds of coverage just by doing a to test anyone else have a different answer than - or do you accept - yes what are the 5 you'd like to see not 5 tests I'll repeat whatever you say you're thinking about it something is telling you 5 there's an intuition yes if a is 70 that's an interesting test for you okay less than 70 well they chose that yes so then there's three less than 70 equal to 70 and more than 70 where the other two if you have a is less than 70 a is equal to 70 and a is more than 70 are there two more no so it's three for you so you've added a new kind of coverage boundary coverage because a equals the 70s is a boundary condition and now that is redundant to the coverage that these two young gentlemen have suggested their tests are going to cover the same paths the same decision and all that and you just added one additional thing is just making sure that the behavior right exactly at the boundary is going to is going to work correctly course you could still do it with two tasks all you have to choose is a equals of 70 and a equal 71 and bam you've done hurt you've done coverage and you still done it in two tests so you could do this and it's still in two tests so what is equals you mean what if the what if the the code actually says a is less than or equal to 70 instead of less than 70 yeah yeah oh yeah yeah so actually it's a good point so I misspoke it's it's because I changed this exercise last night so I'm I have to up this is the first time I've ever done this exercise in public as is this is my beta test at this exercise so it actually has to be a is a 69 and a is equal to no a is equal to 70 and a Z is 70 and 71 because less than 70 know that 1s be 69 right 69 69 and then 70 70 s going to take the yes thing okay yeah right 69 takes yes 70 takes no got it less than 70 there greater than said okay I think that's backwards I think that's supposed to be greater than 70 come to think of it okay now this illustrates one issue in testing right away which is that your model may not represent what your what your idea you know actually is and so that's that one aspect of what testers do testers question models but let's move on from there one this today that I meant to do that a is less than 70 now you still have two tests that you can cover both of these things and one of them can be can be an equal case so then the question is is that enough is that anyone else have another test you can suggest yes call it out green guy you're not wearing green I'll ask you next yes now how is that going to cover something that's that's not already covered bigger than the long integer bigger than the longest possible integer so in this case bigger than what 18 quadrillion or whatever it is and what what how is that going to cover a new path so if it doesn't cover anything new why are you doing it what where would the error be where's the earth how could there be an error is there anything here this checking the lung it what ah your intuition is telling you something and your intuition is correct but you can't yet put it into words so you're going to get a good half point for that how about you sir what did you have an idea for another a might not be a number well so what so what's going to happen what happens if a is not a number which path does it take what do you think it doesn't take a path or do you think the system interpreted is a zero if it's not a number in Pearl the Pearl language if you give a non number of a new Alpha string to a numeric routine it just treats it as if it were equal to zero so we take the the less than path and it would just stop but you don't know so well I mean why do we have to do this test because you don't know what language is written in so you want to do this test you just want to see what happens so you're not trying to test a new path you're doing something else or or are you actually testing a new path by doing this you're not sure anymore okay you two have an intuition and it's a very important intuition and you're in my view your intuition is correct you don't yet have the ability to explain and defend your intuition that's okay that's what training testers is all about getting them to not only have these ideas but to be able to explain and defend these ideas but both of you are on to something both of you have added something that's not here both of you have brought something to this that is not explicitly in this model and that's something the testers must do anyone else have any kind of idea of what a what a what we can do to test this yes so that you think the models just flat-out incorrect it just hasn't accounted for something it's missing a third path okay or the third path is somewhere else it could be that it doesn't need a third path because the input gets filtered somewhere else and never anything other than pure clear numbers come through that are already underneath maxint because they're one byte of values from 0 to 255 for instance which I happen to know is the case so if I know already that 0 to 255 is the only thing that's ever going to come through here then what are we worried about well there doesn't need to be a third path right okay so anyone else have a comment on this on any of this there's anything you want to say about how to test this yes oh it's just a single byte integers it's just a it's just a single byte integer you think the story might go wrong okay good okay so you're questioning the storage stuff okay all right well let me tell you something that several of you have have implied but you're not saying outright and I want to say it outright for you this is not the system that you're testing this is a picture and pictures aren't the systems that you test are if they are the systems that you test they are a tiny tiny part of the system you test testers are not here to make up stories about rumors we're here to encounter reality that's what testers do they discover the ground truth somebody gives me something like this you know my first impulse is my first impulse is what is this thing can you tell me about this please what is this what's going on here so let me give you a little extra information about this this a fully-qualified context-driven tester would do they would say well I don't know how many test cases it would take I don't even know if that's a reasonable question but I'll tell you if I'm figuring out how to test it first thing I want to know is where am I did you just kidnap me put a put a bag over my head bring me to an undisclosed location whip the bag out and and now now I'm staring at this this flowchart probably not probably I know something about what's going on on the project so if you pose a testing problem to me to make it a realistic testing problem I've got to ask you have to tell me what's going on on the project here's what's going on in the project this code that have asked you to test it's part of an experimental system okay it's supposed to defend the spacecraft from space junk it's part of this this elaborate defense radar system okay the system continuously evaluates potential threats and there's these 50 processors that are running continuously evaluating threats from a threat server connected to all the radar dishes and then stores them in a threat database which is then handled by another system so this is the system that does the filtering the code I just showed you is running on each of the 50 processors simultaneously ok now that you know that you might have different ideas about how to test it you might do things like well let's saturate all the processors let's get a whole bunch of threats going here and see what happens when this code is running on all these processors at once now that we know that it's a distributed system and it is a distributed system there's there's ground-based radar systems and there's a system on the spacecraft itself and data flying all over the place well now what you should be doing is asking yourself well what's going on well first of all who calls this let's find out what the calling context is by clicking on this okay so there's this infinite loop well that's a problem right there maybe you want to exit the system once in a while take it down for maintenance but there's no way to do it other than to what cut the power if the processor is available and a data is waiting then create a new processor object run the threat tester program which is the program that you just saw and then kill the that that processor which hopefully will free up all of its memory and restore it to a state of readiness but maybe it won't so we that's another part of our testing so that's what's happening outside of this routine look what's happening inside each of these boxes you have this box called input a well look at all the things that's doing you have to establish a connection to the data server acquire the next data record handle any errors then there's an analysis process that comes up with a letter grade which is that single byte value this process takes between 25 milliseconds and 10 minutes to do for each threat translates that into an ASCII code which is the explanation for why I use 70 there because I'm looking for letter codes that are from A to F that that we want to store except this is reversed this should say this should say greater than 65 and less than 70 so I've actually messed up the program which that by the way is a real bug I actually didn't I put stuff in here that are fake bugs but that is a genuine mistake I made which just proves my point doesn't it but you should never trust any picture a developer shows you they say here's my intent and you can look at it and go that's that is that really your intent and they go oh I always reverse those greater than and less than signs and isn't it a good thing that the tester asks about that isn't it a good thing that what it means to be a professional tester is to be a professional skeptic we are not in the belief business we are in the observation and inference business you want the belief business become a marketer become a become an academic but you want to be a if you want to be a tester then when people who are in positions of authority tell you things you don't believe what they say ever I honor what they say but I don't believe it I use what they say but I don't believe it belief is a sin for testers now I hope that you're getting the idea that each one of these boxes might have a little hot link and might have some additional information like if we look at the actual code here we say that it's Oh less than or equal to 70 and look what it is this is the opposite of what the of what the the the flowchart says because this says it's going to store the threat and it doesn't just store a single bite it's during the whole threat record well it's actually storing it's sending a threat handle which then copies the whole threat record so it's not just a single bite that's being stored this whole idea of store a is an oversimplification it's like pseudocode the developer is just giving you shorthand but that's not really what's going on when you look at the code and also you see that it's less than or equal to so that's just completely wrong what does the storage mean well we have to establish the connection to the server wait for credential validation you might think here you know why is it designed for credential validation on a spacecraft you really worry about space hackers here the may be a performance issue there okay so there's a bunch of things that are happening anytime you see a box in a diagram you have to ask what's going on inside that box a box is an abstraction okay here's my point first of all two test cases what are you crazy I know why you said that you said that because you took some kind of class and in that class some professor is talking about what's called basis path testing or decision coverage testing and that's the correct answer on the is tqb thing okay and is dqp has nothing to do whatsoever with practical software testing it has to do with making money for consultants that's what the is tqb is all about making money for certain consultants now I make my money partly by attacking lianas tqb so indirectly it makes money for me too but the difference is I admit it so here's the here's bottom line here let me just bring this up if this works here the less okay they got the concept of how many test cases is generally speaking a useless and stupid question ever to ask that's like saying how many files will it take you to write your novel well I don't know I might I might save different drafts of my novel and different files let's see it might make a backup draft every couple wait why am I even answering this question what a silly question there has nothing to do with my productivity whatsoever to ask me how many files is it going to take me to write my novel that has nothing to do with writing a novel that is a irrelevant technical detail and it's in irrelevant technical detail to count test cases has nothing to do with the essence of testing if I if I happen to to think about it or break it down one way it's going to be 149 test cases and if I break it down or think about it another way it's going to be three test cases and and there will be the same testing either way it's just a matter of what I decide to put in a file basically so that's a silly question the better question would be how should you test this and if you're going to answer the question how should you test it number one first thing you have to do learn the product learn everything you can about the product you know what testing is apart from professional skepticism software testing is rapid learning if you love to learn about new things and you get bored when you know about something and you want to move on and learn about something new you're probably better going to be better testing than a development because in development you're slogging through the same thing day after day week after week month after month you're still working on the same code god make it stop when I was a developer I hated being a developer it's like it's like I was good at solving crossword puzzles and someone says you know you can make some money solving crossword puzzle I'm like whoa make money solving class word puzzles it sounds like some kind of and so I start doing it and I find that I have to solve like 25 crossword puzzles a day and after the first couple of days it's like man I'm sick of crossword puzzles and it's oftentimes almost the same crossword puzzle with slight little differences I am so bored I used to love crossword puzzles now I hate them I hate them so much but with testing it's not like that at all with testing it's a new problem all the time I love the variety of testing and a lot of people think it's there's no variety a lot of people think oh testing it's such a boring slog and it's all repetition it's kind of like sex if if it's not fun you're not doing it right you're doing it wrong if you approach testing not as the most challenging intellectual exercise you've ever done in your life then you don't understand testing because that's what it is and yes there's some repetitive stuff in it but even within the repetitive stuff there are clues subtle clues if you are smart enough and observant enough to see them that's why I love testing and I also use my development skills and testing you don't need to be a developer to be a tester I being an X developer I write my own test tools but you don't have to write your own test tools all you have to do is make friends with someone who is a developer who will write test tools for you so you either need to know how to write test tools or you need to know how to be charming and manipulative and then other people will do that work for you either way you're covered you need so in real industrial life nobody has all the skills and that's okay in real industrial life you either have the skills or you're learning the skill or you're friends with someone who has the skill and you borrow their work and that's in industrial life not called cheating that's called professional behavior in academic life it's called cheating that's what yet another reason why academic life has so little to do - with the reality that after your academic life so in industrial life it doesn't matter whether I solve the problem what matters is I get the problem solved no matter what the problem is and I'm usually working with a group so when I come into a situation I size it up the way miyamoto musashi or is it musashi miyamoto I never know which of his names is his first name but there's a guy he lived in 1645 he wrote a book called the book of five rings in the book of five rings it's all about how to win duels to the death he starts out his little book which you can read in 45 minutes I have survived 16 duels to the death here's how you can do it - and he basically says there's no one right way to win a duel you do whatever it takes to win and he makes fun of people who say they think that there's the best practice of using one sword or the best practice is using two swords or the people who are in the the Morningstar school and the mace school and the and the short sword school or the long sword school and he said ha they're all wrong here's what you should do learn every weapon and then use whatever it is that put your enemy of disadvantage including furniture he has a whole section on using furniture to defeat your opponent and that's my attitude - I walk into a situation a technical situation as a consulting software tester very highly paid I'm generally paid $300 an hour or more and most I've made in my career if you're interested in what the career of a tester might be like most I've ever made in a week is $35,000 I made 35,000 week for 12 weeks once in 2002 and it was very depressing by the way because I knew at that moment I was making more money than I would ever make again probably in my whole career and that's really depressing when you realize that because you know you're going downhill after that but it was a multibillion-dollar court case and they needed to get the tester they could find and they thought that it was me and I decided to let them believe that and try to be the best tester that I could possibly be and and it was very very exciting but what I do when I walk into a situation like that you know I'm very highly paid people are depending on me I size up everyone in the room everyone in a room is a test tool that I am going to use if you know something I don't know congratulations you're my new test tool I will use you I will find a way what is it what's your price chocolate brownies what is it money women what is it I shall bribe you with whatever it takes do I have to smile at you do I have to offer you respect I will do whatever it takes to get you to use your skills on my behalf because I am a rapid software tester I'm I took my lesson from the book of five rings I use any tools available to solve the problem so it is not true that you need to have every skill but I agree with Musashi that we should be constantly developing our skills throughout our career all kinds of different skills so I might not know how to run a tool for our time being but if I turn out to need to run that tool for any period of time I am going to study that and I'm going to get good at it and that is just an ongoing process its ongoing self education okay so a better question than how many tests is how should I test this and a tester must always seek the underlying complexity beneath the apparent simplicity that is the one of the essences I would say one of the essences of software testing is testers seek to dispel illusions so that's my little opening thing and now I want to do a little presentation for you which which talks about some of this stuff this is a different presentation this is a presentation I do usually for developers and for managers hello I'll be your tester today introducing you to the great and marvelous art and craft of intellectual software testing of using every neuron in your brain to test stuff ok so here's an example let me give you an example now I want you to listen to the words that I use to how I'm using words as I describe what's going on here because this is a key to one of the aspects of professional intellectual testing years ago I was in the lobby of the del Coronado hotel in San Diego California and I was about to go on stage to speak in front of several hundred people at an at a conference for into it it was an internal Intuit Developers Conference and they wanted me to speak on the subject of good enough software quality so I thought I would gather some fresh information while I was there so I'm in the lobby of the hotel is wandering around and I see this dental discover Club kiosk and it's this thing with a little keyboard and a mouse and I thought I know this is this is a tester workout station and I'm going to do a little testing workout right now so I plunge in it's called a pic cycle decided to a pic cycle a plunge in and quit cycle on this and so I just approach the system and I begin to interact with it I click on it and I see this immediately when I see the fields I'm thinking input constraint attack it's a natural thing whenever you see fields so I decided to do some input constraint attacks as part of that I'm going to use a little trick and then I'm a type of a few characters I'm going to go go control a control C control V control V control a control C control V 12 V so if I just hold the control key down and just go AC V V AC V V AC V V every time I do that it doubles the input pretty soon I've got several thousand characters in each of those things and I say submit and and it seems to except it doesn't seem to be any performance issues so I shrug and I decide to D focus and I do a little click frenzy I do a left click frenzy right click fine see when I do the right click frenzy Oh click frenzy that's where you move your wrist like so and move your finger like this on the mouse clicker button and just click all over the place just go crazy with the clicking ok it's one of the many quick test heuristics that I use well when I did the right click frenzy look what I found now when I see this I immediately think rumble strip heuristic am i right the rumble strip heuristic now the rumble strip heuristic that means that just as on the sides of highways they put the corrugated strips of you know bumps so that when you when you are driving and you drift off the highway because you're falling asleep your car starts vibrating you know you've ever seen that before they have that here or do you guys just fly off highways and and well in in civilized countries anyway what happens is we value life and it starts vibrating and when it vibrate your car vibrates like that you don't your first thought isn't I can live with this it's just a little vibration now your first thought is I'm flying off the road wake up so so a rumble strip heuristic in testing means that often when we see small problems we get excited because it means there's a bigger problem about to happen and boy I want to see that happen so I want to pull the string wheel even farther and see if I can get this thing to fly off a cliff alright so I see this it's not a big problem but it's an empty it's an empty error message and you know that means that the developer maybe was falling asleep at the wheel there so maybe this developer hasn't fully thought through the code that's involved here so that leaves something called error message hangover testing it's like a hangover you know the error message is a big party but then the now your place is a mess and there are people sleeping under tables and there's tables outside in the lawn and broken glass and the all's you deal with your hangover okay so I think that it's probable that whatever the something after this error message might be messed up so so I'm going to try to do stress testing on this because that will exacerbate whatever the problem is so I'll use a variation of a shoe test shoe tests of course is any test that is consistent with pounding on the keyboard with the shoe or sometimes gently placing a shoe on the keyboard in this case I'm going to click to make the message come up and then I'm going to make the message go away and I'm gonna do that as fast as I can because I'm thinking maybe some bad thing will build up and then somehow cause a crash well as I'm doing that I notice the faster I go I get little flickers in the corner of the screen well I'm distracted by those flickers but that's okay because that's called branching and backtracking and we're encouraged to branch and when we do when we're doing professional exploratory testing which if I haven't mentioned it already this is professional exploratory testing so I'm going to do what Kem Kanter calls follow-up testing which is you let yourself be distracted by some bright shiny object you follow up on that you try to maximize the power that that also is part of the rumble strip heuristic so I decide I come up with a theory of this that's called conjecture and refutation conjecture that perhaps this flicker is related to timing so I begin varying the timing of my clicks and enters and lo and behold I find out that if I right click in on that screen within one fifth of a second of pressing enter on that error message I get that menu which allows me to break into their system which I break into there it is I'm in there pictures a little blurry there because I had to snap the picture quickly the storm troopers have been alerted sirens were going off and so the total elapsed time to gain full control of this computer from walking up to it is seven or eight minutes this is the kind of thing that hackers do this is the kind of thing that computer gamers like to do it's the same kind of thinking there might be some people here who've played computer games before or I don't know I see some people who might have that that kind of dazed look of computer gamers I discovered recently a few weeks ago when I took my son to an emergency software testing situation this is never well it's happened to me twice now in my total career where I get a call from a client and I feel like James Bond I just get this call like we need you in Los Angeles to test something no you don't have time to go home we've made arrangements go to the airport from where you are right now pick up your first-class ticket but my son's with me I was taking my son home we live on an island and take me several hours to get him home and they said I'll just bring them along so I two first-class tickets are waiting for us we fly to LA and I have to do testing on a medical device for one day these guys just begged me to come down if one day is all you can give us and that's what we'll take so I thought what would happen is that my cat-like son who occasionally will let me pet him but mostly sits on top of a cabinet looking down at me balefully my son would I thought he would stay in the hotel room or he would sit next to me playing a World of Warcraft or something while I did the testing but he decided he was interested in the testing and pretty soon it turned out he's really good at investigating bugs he's a natural at it maybe 13 years of figuring out video games is actually a good education because he was looking at it as like beating a boss on level 39 in or in some and some dungeon in Azeroth you know he was he was looking at this as a real challenge he would he would find a buggy say Oh dad look look and I'd look at it and go okay can you do it in fewer steps he go let me try and he come back ten minutes later and say I got it down to three steps and I'd say but it shows an error message yeah you can get around the error message but can you make it happen so that the error message doesn't even show up because that could kill people and that would be great if we could find one of those he eventually found an example of a particular bug I'm thinking of that caused a multi-thousand dollar piece of medical equipment to melt and melted it failed it melted and we confirmed from the technician that oh you've destroyed this piece of equipment the good news is no one will be killed by that method now because they'll fix it right so we are high-fiving each other you know that's like when the ultimate good things about testing medical equipment is if you can find something that could have killed a patient then that's a very very good news for the for the tester bad news for everyone else well sort of bad news you know they'd also don't want to go to jail and they also don't want to kill people but it's it feels like bad news to them because they they go oh there's a problem in the product but testers are high-fiving and going oh you know why testers high-five each other when they find a bug a really bad bug you know it's not because we don't like developers it's not you know that's what developers think but that's not true it's because on that day at that moment we go yay we're useful because most the time a tester doesn't feel very useful to you I mean you you have to really work at it to feel useful as a tester it's kind of like being a guidance counselor in school I mean most the time you know that your life is meaningless and none of the kids are going to listen to you and all your aptitude tests are wrong and and but once in a while you know you actually give a kid advice and then they contact you 20 years later and go that was great advice to change my life you know and and you go wow I was useful something happened and I did it right and it was these so testers are like that too we we we often have trouble feeling useful but when we find a huge bug we go yes today they won't fire me for non-performance so that's terrific and maybe tomorrow I'll go back to not being able to find another bug like a gold miner who's just wandering around with the donkey and the wilderness hoping to make a big strike so what I've just shown you is I've shown you a little cycle of exploratory testing using skills that video gamers have but there's something that I showed you that video gamers don't do that my son can't do yet and that is I explained what I did in specific technical language I can tell a story of software testing that sounds effective and compelling and professional and I wrote up a account of what my son and I did at that medical company and I sent to my son I don't know if he's read it I hope he has because I wanted him to see look what you thought was just playing around with the system look I've turned this into this professional-looking report that makes it look like we meant to do all this stuff okay and that's part of professional software testing is to tell a story about the testing a compelling story and that means that you must examine the patterns by which you work you must put words to them most of the terminology that I just used in describing this I made it up I invented that terminology it's not part of any testing textbook one thing that I borrowed well I borrowed follow-up testing from campaigner I borrowed input constraint testing from James Whittaker but the rest of it is all my own terminology and I've spent years and years noticing what testers do putting words to it but there's a lot more of that left to do and there's no official vocabulary of software testing despite what the is tqb wants to tell you there's no what they are doing is they're borrowing terminology from 30-year old textbooks and saying oh it's standard it's not standard it's tremendously controversial but they ignore the controversy so it turns out if you ignore enough people everyone agrees so you know Qaddafi found that out too if you ignore enough people they're all in agreement what uh yeah that's right he's still alive so you know his life philosophy works great so um here's another little I'm gonna skip this example I'll just skip this I was just hacking this system and found out the their secret phone number and and but it's going to take too long to talk about it so there we go all right let me get on to this I'm gonna challenge you now you've already been tricked once at least these two guys haven't or the rest of you weren't but these two bold gentlemen at this time I want you to demonstrate for everyone thinking like a tester okay okay mr. two tests okay so you're going to do it this time because you're now you're on your guard now you know now you know what what I'm going to do okay you are carrying a calculator you drop it perhaps it's damaged what might you do to test it this isn't this is one of the opening exercises I give to anyone who seeks to have me coach them in software testing it's a diagnostic exercise very simple testing exercise you're carrying a calculator you drop it perhaps damage what might you do to test it this very example was given to me years ago and I had to answer this and now you have to answer it I'm focusing on you two gentlemen right here use it enough okay is that is that your final answer find what buy a new one okay okay is anyone else have a do that's their answer they've locked in their answer anyone else have a different answer to this question okay so here we have a tester in the audience now how I can tell how can I tell the cheese a tester cuz she asked a question before giving an answer like a professional tester would do you don't know what the situation is you know you don't know what the situation is so you got to find out what the situation is before you decide you know what to do oh come on guys was it on or off oh let me just say this I note your question I will not answer it but I note that you have asked it and I will gently set that aside for the moment do you have any other questions that you also want me to answer for you that's it that's the only one at the moment yes okay we have people calling in with more answers anyone else have anything else they want to say about this ah okay that's pretty important isn't it because if it was an abacus we will do different things then if it's an electronic calculator or if it's one of those I used to have an adding machine before I ever had a computer I had an old adding machine from 1965 that my mom got at a rummage sale at a bank and these things didn't work they never worked but we just like typing on them and pretending we were computers so that was exciting if it was a calculator like that it weighs like 40 pounds drop that thing you might do something quite different of course in that case we already knew it was broken when we dropped it if it's broken when he dropped it and if the reason why you were carrying it was you were carrying it to the dumpster to throw it away you're probably not going to turn it on you we not going to pick it up and use it you're probably going to just kick it towards the dumpster and then when you get to the dumpster maybe you'll lift it up and put it and put it in and knowing what state it was in at the time you dropped it might affect things what kind of calculator it is sure anything else yes what how was that relevant where was it drop how is it relevant where you're dropping it ah very good very good now notice what I did she asked a question I asked what is the relevance of that question so she can't just guess questions she has to support and defend her question in this way I'm finding out how she really thinks and so if you drop it into the water say the Indian Ocean and it floats down 10,000 feet down into the debt I'm probably not going to worry about toasting it after that or maybe it's a calculator designed to sink down into the water calculating currents or something and I go I drop it perhaps it's damaged well perhaps it is damaged it's always a possibility that the thing we designed to drop into the water might be damaged when you drop it into the water what might we do to test it well why don't we check the telemetry because it's beaming continuous telemetry back up to us we'll check the tournament tree and we'll see if we're getting a message from it and if not Ram will shrug and drop another one of the these things that cost 5 euros that we've mass-produced by the thousand and we'll drop another one in and no big deal we don't test that the testing happened months ago this is when we're using it well I was asked this question years ago and here's what I did I spent two hours and I wrote a report which I submitted back to the person who asked me the question and this is one section of the report one of the three sections of the report okay well I want to understand the assignment and I'm aware of the possibility I've misunderstood it so let me interpret it back to you and I have a bunch of questions what does it mean to drop it when did I drop it did I drop it on a surface with a hard surface with a force that makes me suspect something to drop it into a disrupt chemical environment a biological environment radiological environment was it connected or anything else at the time that I detect anything wall was dropping it leaves you suspect a problem is it ruggedized is it mine or someone else's maybe it's someone else's and I don't want them to know I broke it what am i using the calculator for maybe every calculation doesn't have to be correct anyway maybe it's my job to test calculators and my answer is I follow the test plan of course I dropped it as part of my drop test now let me get you a couple of exhibits watch this this is a calculator watch let's see if it works oh it works yeah I'm not really worried when I drop something like this question depends how you drop it look edge on weight edge on there see kind of absorbs the impact huh and I did this in a store once with one of these and it actually wouldn't work after that but I've never seen one break since then here's another one here's a let's calculate children's calculator inside a protective packaging oh no I'm really not concerned about that you know what I would do it would just tip is go and hang it back up again and a no problem at all no problem no but if I were to test it I would be constrained by the fact that I can't see all of the the buttons so I would have to restrict myself to these so there's everything has a situation everything has a context testers who are good testers they check the context before they give a technical answer to any question there are abdi million ways to test things there's no official right way to test anything instead we have situations that we read we interpret and we solve the problem in those situations here is a requirement this requirement is a real requirement on the medical device that I am testing as a matter of facts it's the one I went to LA to test and this is one of the requirements for that the system shall operate on an input voltage range of nominal 100 to 250 V AC and I like to ask people how would you test this how would you test against this requirement if you if you will sought now I'm not going to actually engage you and normally in my class I'm I would engage people in a question about this and take good ten minutes on this what I'm going to do here now is I'm just going to tell you that here's a really bad answer try it with an input voltage in the range of two 100 to 250 and here's why this is a bad answer this answer requires absolutely no thinking this is a mechanical answer it's generated mechanically it's just taking it's basically taking the requirement text and putting the word try it in front of it printing the words try it that's essentially all that was done here it does not actually engage the issue at all good testers don't just take requirements and add the word verify in front of them and then try naive things instead good testers break down requirements they model them in their heads I look at something like that I go what's the system I need to understand that system I got to learn the system what does it mean to operate the system I need to learn that input voltage range of okay looks like I need some kind of special equipment to control the voltage coming into it what's that equipment called turns out it's called a programmable power supply and we have one in our shop it's called a very act yeah so yes we have a very equitable power supply sure did you need one I didn't know that so I've got to identify the need for special equipment then I have to decide what I'm going to do and notice it says 100 to 250 well some testers like to say well let's test it 90 and I go why why would you test it 90 it's outside the requirement are you just making up requirements does anybody I feel that we should test at 90 volts given that this is the given that this is a requirement does anyone here feel like okay Oliver Vil s'en I was a very accomplished tester runs testing company thinks we should test it 90 anyone else think we should you get you all right I'm challenging you tell me why why should we test it 90 when that is clearly outside the requirement that's like someone saying hey this equipment is not designed to be smashed with a mallet well I'm smashing it with a mallet then yes but it's a cupcake it's oh now it's all over the place now what have you shown yes my cupcakes are smashable by mallets are you satisfied now tester big tester smashing cupcakes does that tell you anything about the quality of my product I knew was Mashable and and and now you've just shown what we already knew that it was smashable the thing doesn't work at 90 volts we know it doesn't work at 90 volts how doesn't testing at 90 volts help us do anything so what we know it breaks at 90 volts so what of course I'm but I know that it breaks but I already know so what you haven't taught me anything I told you that it doesn't operate below 100 volts we mean does it break I don't understand it mean doesn't operate it does not operate it does not work under 100 volts I knew that I designed it I know that it doesn't operate under that's like saying does this boat that you've designed work on land too but no it doesn't well I want to drag it up on land just to see no don't do that what does someone said something over here well yeah but so what I told you it only works when it's within this range why do we care to test something when we already know the answer what so what then it's like free extra functionality I guess but it doesn't have to why do we have to know everything why why do we have that we already I already say it probably doesn't work if it accidentally works beyond the engineering tolerance that is specified then you know that's gravy it's like an airplane might be designed to go only as 150 knots so the small airplane and moki maybe it won't centigrade until you get to 165 knots but don't fly it over 150 knots and and you put that as a do not exceed speed and then the pilot knows not to go faster than that then that speed so what's the point of testing beyond situations where you know already that it won't work finally you say something that's brilliant thank you I knew if I gave you three chances the third one would work he's got it you see it's not just that the system it it fails outside the require mean it's going to fail outside the requirement if it doesn't fail then we would go oh well it doesn't fail but we don't expect it to work but that's not the issue the issue is is that there's another requirement an unwritten requirement it's usually not written and it goes like this thou shalt not kill it's one of the commandments I guess and so this is a medical device and okay I know it's not going to work but the question is is does it fail to work in a way that is dangerous to the patient or does it gracefully fail and stop itself from from hurting people in other words I know that if I fall asleep at the wheel of my car I'm in a dangerous situation what I want to check is if I'm falling if the driver is very tired does the driver detect that the driver is very tired and does the driver pull over and take a nap or does the driver just fall asleep at the wheel you know I want to know what happens when we take it down to that level you're not inventing requirements notice that I'm not testing outside the requirement I never test outside requirements but I do test unwritten requirements they're real requirements and if it explodes that would violate a requirement that may not be written but it darn well is important so what I'm saying here is that every time I look at requirements a tester I don't just lovely copy them into something called a test case it's not simple like that instead I'm questioning it I'm modeling it in my mind I'm imagining different ways that might interact with other things and in so doing I'm expanding the world beyond what is written down explicitly for me that's part that's a big part of a testers job to make the world clear so you have something which is the world that is described that's the world that's in this pack right and you have something that is the actual world and the testers job is to compare the actual world to the world that's described but this isn't the whole story is it there's more to it than that because we also are interested in the imagined world people have ideas in their head some of those ideas get explicitly written down and other ones never get explicitly written down the tester must be aware that the spec they're looking at is but a slice of the world that they must cope with so I'm testing the actual against the imagined I'm in other words the actual world against the intended world and as well as testing it against the described world and any of these areas of the Venn diagram are interesting for testers there might be a situation for instance where the product actually works the way it's described to work but that's not how the programmer intended it and here's an example the programmer says oh this this webpage is going to be compatible with firefox browser and then they create something that happens to be compatible with firefox browser even though they have no idea what it actually means to be compatible with the firefox browser now how is that possible the developer could mentally create something that fulfills a requirement that they don't even understand who happens all the time when people use libraries they may use a nap Lakai ssin framework that is already designed to be compatible now all that that's fine you know you can have a developer accidentally a cheaper requirement they don't understand the problem happens when they as at some later point change something and then it stops being compatible but they don't think they changed anything that made it incompatible testers are there to catch situations like that testers catch all kinds of interesting situations and one of the things that testers do is to help educate the developer about how their own technology works you could say that testing is just the process of try it and see if it works but this is very complicated there's lots to this idea of try it and see if it works and it's kind of an infinite process so I prefer to use this phrase I say try the product to learn how it might not work that gives an edge to testing I'm constantly focused on how the product might fail by doing that I'm axel find useful information for my clients now what I'm illustrate here with these different examples that I've been reading off are the following things okay we've got you've seen audaciousness testers must be audacious testers must say I know you think it works well I think it might not work I think I can find a way to make it not work I might fail but I will audaciously try I use my imagination I question I'm rapidly learning about anything I encounter and making sense of what happened happens and creating stories out of that I'm telling those stories you've also seen focusing and D focusing which are funding mental dynamics of testing sometimes I must focus which means I must look at a very narrow thing very carefully and other times I D focus which means I stop looking at whatever I've looked at and I'm now going to look at something different I use heuristics as well heuristics are ways of solving problems that may not work in other words they are fallible ways of solving problems I don't do any best practices there's no such thing as the best practice actually if you ever hear anyone talking about best practice they are not engineers they are marketers engineers know that each solution that you might suggest is relative to the context of the problem that's why doctors don't have the best medicines they have medicines that they prescribe for specific illnesses and specific situations and they ask you all these questions make sure that this medicine probably will help you they don't say oh I'm so bored hearing your symptoms look take aspirin it's the best practice medicine there's no best practices what they are though is there are their skills there are heuristics there are context variables and testers when they're crafting their tests and they're crafting their test strategy they must be aware of these things constantly feeling out the weak spots of a product constantly reanalyzing risk I love it it's like detective work it's the most ultimate detective work in multi-dimensions wonderful and it's not just one murder there are hundreds of murders to solve for this detective this complicated looking diagram is is what I do whenever I sit down and even play with software for a few minutes I'm going through all these loops I'm referring to my knowledge I'm analyzing what I know I come up with experiments I ask questions of the product I get answers I form my testing story which then guides my testing I check with other people and not their ideas guide me lots of complicated things are going on even if I never write down a test even if I never make a mark on a piece of paper even if I just stand here and test off the top of my head all these complicated things are going on testing is a highly interestingly structured activity even if you do it entirely improvisationally entirely ad-hoc it's still a highly structured activity sometimes people use the word structure and when they're trying to criticize the things that I do they go well I use a structured approach what they mean by that is they use approach an approach where they understand the structure of it and they don't understand the structure of my approach so they go oh there's no structure there it's kind of like a giant cockroach standing next to you going I have a skeleton unlike you and you say well I have a skeleton and they go you look pretty squishy to me and you know like no no no I see the confusion giant cockroach you have what's called an exoskeleton your structure is clearly visible to the outside I I have what's called an endoskeleton my structure is within and they go oh you're just you're just using rhetorical tactics to try to fool me you really you really have no structure like a jellyfish aren't you and then you and then you you punch the cockroach and spray rate at it and kill it then you don't worry about that anymore also giant cockroaches can't live very long because they can't breathe well when they're that big so don't worry about it now there's three to four loops his third loop there's a self-management loop here there's a testing loop looping between analysis and experiment there's a learning loop looping between knowledge and analysis there's a collaboration loop between analysis and other people these loops are all part of professional exploratory testing and most of testing is exploratory there are lots of people although you'll run across people like this if you if you read any books about testing or if you take testing classes at institutions like this you're going to hear a lot about test cases you're going to hear a lot about writing down tests and most of testing is not written down most of testing remains tacit only a small part of it ever gets written down and to write down a test makes that test vastly more expensive than so it's actually it's a dangerous thing to write down tests economically it's very dangerous I mean imagine that that you could only eat something if you if you log a description of it before you eat it it's not just that you say I have mashed potatoes it's you have to draw a picture of the mashed potatoes and then you're allowed to eat it okay eating would become quite expensive it would become this big big production and you would probably do a lot of secret eating kind of off-the-books or else you'd starve to death I don't want to make testing so expensive that people won't do it but if I insist on making my testers write down their tests before they run them I am making testing expensive and that means I'm limiting the kinds of tests that they will ever do and they'll do a lot less tests so they'll do a lot less variety of testing if I insist on trying to document all this stuff so the art of test documentation is actually an interesting and challenging art how can we minimize the cost of documentation how can we minimize documentation entirely help we maximize the value of the skills that we have if anyone here drives a car you already know that you do this because you were trained to drive a car instead of being given a set of instructions that you have to carefully follow so that they wouldn't have to train you and then you follow the instructions you go ten feet you stop then you read the instructions again go ten feet and stop I mean you're driving would be incredibly slow and inefficient if instead of training you to drive the car somebody tried to control you with a script so I recommend a book if you want to know all about this if there's a brilliant book about this several of them actually there's one called the social life of information which is based on a study of how people learn to repair copy machines at Xerox and there's another one I just read called tacit and explicit knowledge which is a brilliant analysis of the difference between what can be explained what can be put into words its knowledge and what cannot be put into word where it says knowledge you're holding up a blank piece of paper what does that blank piece of paper mean means fifteen minutes okay I got to go I can little faster here okay I talked about I talked about rapid learning I want to give an example of rapid learning that I do as a tester a friend of mine sent me a link to something about space-filling curves and specifically sierpinski space-filling curves that are being used to solve the Traveling Salesman problem and I thought interesting maybe I can use it to solve other testing problems if you can solve a coverage problem with a space-filling curve maybe I can solve a test coverage problem with a sierpinski space-filling curve so I worked on that for about an hour and couldn't think of a way to do it but while I was doing that I discovered about L systems and Penrose tiles and Penrose tiles are interesting because they're non repeating patterns and I thought non repeating patterns I can use those in testing maybe and then I couldn't think of a way to use them so I went over to the travelling salesman problem and I decided but there's got to be apps that solve the Traveling Salesman problem I can learn more about it so I went to this place that had a bunch of physics apps and one of them was an app for diffusion limited aggregation which is totally fascinating and relates to learning but has nothing to a testing right now so forget about that and then went over to simulated annealing which is right next to the thing I was looking for simulated annealing is a method of solving the Traveling Salesman problem and when I found out about that I realized that I had learned a technique of software testing that I'm now waiting to use in fact I think I have used a technique much like this simulated annealing is a process of using random testing to to solve a problem it's it's a formalization of exploratory software testing here's an example of the thing working this is the this is the little algorithm trying to solve the Traveling Salesman problem is trying to calculate the shortest path through this set of random points and the way simulated annealing works is it first it's very random and then it slowly becomes less random it sort of turns down the temperature is what the the what they're trying to do meaning that they perturb the path by less and less and less and in this way you avoid a local minimum a purely local minimum and you get closer to the global the global minimum and it's pretty cool but this is an analogy that also has a lot to do with testing what I'm trying to say is as a tester what I'm constantly doing is I'm exploring ideas from physics from mathematics from all walks of life and I'm trying to use that to invent new solutions to testing problems that I might face in the future what I love about testing is I am constantly learning about cool deep new things this whole process of discovering why notes you know no two snowflakes could ever be alike I found out why that's actually true I always thought it was kind of a myth but it turns out mathematically that it's it's rather true for the length of the entire of a universe will be the time that the universe will be in existence there will never be any two exactly identical snowflakes that are visible to the naked eye and I found out why after I was learning about diffusion limited so well this whole process took me four hours to go through and then this was happened three years ago and this is just one of many many many little learning events that I go through as a tester testing is full of self-education oh I told you not to count test cases yeah that says that you do you don't test count test cases and here why here's why it's just like counting unicorns if I ask you how many unicorns would fit in your bedroom and I asked is some of someone in Sweden once and she had this faraway look in her eye and then she said - I think it's like she was really working on this problem you know she's trying to figure out now if I put this unicorn in this way and then tuck it up against the wall I can lead the other one in I guess she wasn't thinking about the unicorns of teleport the thing is is that if you calculate it I it all depends on on what you mean by a unicorn and it's the same thing with test cases you see their test cases are tiny and test cases are huge or test cases they're copies of other test cases and if you think of a unicorn as the letters U Nico RN put into a text file copied 100,000 times put in a directory with ten other files like that shrunk with 7-zip packed onto a 32 gig thumb drive and then packed into one cubic meter area my calculations say you can fit 15 peda corns in a 1 cubic meter area or 15 quadrillion unicorns each independently addressable example and of course anyone who has studied data compression knows that there's a much better way to fit more unicorns and that is just to write on a piece of paper of a 1 x 10 followed by some number like 10 to the the one billionth power all right over a minute along you want and then just say instances of unicorn and there you've got data compressed right there that's just that's a piece of paper that has as many unicorns on it as you want but my solution is independently addressable instances of the word unicorn so depend where you can't test cases you're not counting anything you're counting fluff you're counting ideas because test cases are not independent they're not interchangeable now this brings me to test automation because some of you might be thinking why don't we just make the computers test themselves and that way we can watch them fight each other like gladiators and we can be the humans in the stands going yea or nay to them and that's the computer to testing each other isn't that marvelous well it's not marvelous here's why imagine that we know my wife she likes it when I say things like I love you and all that stuff right so been married 20 years you got to know about this stuff I could create a cron task that would just send her an email every hour saying this she doesn't like that so that's what I get I don't get the credit okay for making it extremely efficient to reminder of this fact that she seems to find very suspiciously easy to forget it's the same thing with testing my boss might be comforted by me running tests over and over and over again so I might I might write a lot of nation thing and then it's just running this action over and over and over again but here's the problem problem is that when you do actions over and over and over again that's not a test that's just an action the test was the test or thinking at the time of doing the test and that cannot be automated now the action might have some value might definitely have some value in and of itself you might say I'm going to run a task just as is the website still up is the website still up is the website still up it's a website still up and maybe it'll go down and maybe I'll find out that it's gone down and maybe that's a good thing that I found out that it's going down but it's not testing I think what we should call it is checking that's checking any fact that you can gather with a computer should not be called testing it should be called checking because it will help us not get confused because what a tester does when a tester checks that fact is not just checking that fact when you personally look to see if the website is up as opposed to having your script do it you not only will check if the website is up you'll also start noticing things like the websites getting slower and slower load that's odd I brought the website up and it comes up but part of its changed I think we've been hacked or the hundredth time you bring it up you might say I never noticed this before I just realized it's been happening all along but I think this is misaligned to think this paragraphs misaligned with that or this isn't being rendered correctly I just realized it that computer the program that you wrote to do that checking it will never spontaneously change itself to a program that will start noticing things it's never noticed before but humans will I a human tester who is trying their best to act like a robot will probably fail and probably some humanity will get through but if you're trying not to act like a robot even more likely you will learn something new the 10th time you run what you think is the same test but your program will never learn your program is not doing testing your program is just performing a behavior which can help testing I use test automation don't get me wrong it's just that test automation is not automating testing its automation in support of testing I mean here's here's the thing though here's why it's so hard to explain this to people because managers think testing is this they go hey you look at these clean white plates and just tell me if there's a bug on it okay I mean surely it's easy just glance at it you'll see if there's a bug on it you know what's also hard testers constantly complain about this it's so easy look I can look at the plate there's no bug on it and I go yes my esteemed manager but that's not the testing problem this is the testing problem I'm testing the professional kitchen for bugs this is a very complex space to cover it's a very dynamic space this is a very simple space very narrow space when people don't understand testing they think testing is like this just look at it what's so hard but people understand testing they realize it's more like this so here's the thing developers developers they have already figured out that they should not call things programming if computer does it you never hear the programmer saying I'm going to have the auto programmer take my source file and prepare a program from it no they call it compiling but what is a compiler it's automated programming so what programmers have done for years and years and years is as soon as they make a tool to automate programming they don't call it programming programming gets redefined to whatever the programmer does that's what programming is I'm doing the same thing we should call testing whatever it is that testers do and anything else that supports that give it a different name I prefer to use the term checking to remind myself every time I say checking I remember the automated checks can't use judgment and they never learn or evolve they might be very helpful but they're not humans and therefore we should not get too confident about what automated tests can show us recently the on the performance tester there's one performance tester for Facebook okay one person who does that her name is Veronica get off and she told me that her bosses told her that she had to automate looking at the graphs the performance graphs and she said no no that's bad idea and they said well it's okay we'll have this other guy do it then so they had this other guy automate looking at the graphs so that they wouldn't even have to look at the performance graphs shortly thereafter Facebook suffered a major performance problem that was not caught because the person who automated looking at the graphs didn't put any alert in in case the performance got better but only if the performance got worse so one day the performance tests are running are one evening they're running and suddenly performance is improved by something like three times it's suddenly three times faster than it was yesterday and a human tester would go that can't be right something has to be broken in the tests in fact it wasn't running some of the tests but the automatic automated graph reader just said cool no problem it's even farther away from the red zone so they never got clued in until people started saying wait a minute there's something wrong with Facebook I don't know what happened after that but I am talked to Caracas since then but I think they they pulled off out those automation checks now and they have a human look at it that's what as humans that's what we're good at okay I'm not going to say too much about testers being like tamagochi this is something for developers and there's a little bug in the tamagochi site anyway a tester must develop their reputation in order to be effective on a project it's very very important developing a repetitious comes from the book the prints you know the most the prints must be this gives striking proofs of his capacity and I'm a wrote a book about Buccaneer but being a Buccaneer scholar which applies the kind of the concept of the pirate world to learn throughout your life so I applied this to buccaneering here's a little segment from the book Buccaneers of America and I like this because it kind of relates to me it's about this this this famous Buccaneer and notice what it says it says ah though in his private affairs he governed himself very well he would often appear brutal and foolish when in drink running up and down the streets beating or wounding those he met no person daring to make any resistance so I as a tester kind of I'm like that though in my private affairs I cover myself very well I I work hard to develop a reputation as a tester that he could test anything he walks into the room and the bugs just surrendered like the first Gulf War they just surrender they just they'll surrender to his boots I want people to think of me that way as a tester and that's what you need to do if you're going to develop yourself as a career as a tester you need to practice testing you need to be like miyamoto musashi you need to become a famous testing duelist and there are ways to do this there are communities that will help you do this I can introduce you to those communities and if you do this you can easily get at this point in the world of testing you can easily achieve with with some work an international reputation as a tester right now for instance as far as I know there's one tester in Bangladesh now that's probably a lot more testers in Bangladesh but there's one who has an international reputation and I know his name Sajid dual Hakim and that's just because he blogs and the rest of them don't and he makes up interesting new ideas and then the rest of them don't and there's one tester as far as I know in Germany and I'm sure there are more testers in Germany but there's only one that has an international reputation in my community there more could but they choose not to in Estonia there are several this Christian oboe who's here in the audience who's a internationally known tester and Oliver Wilson who's internationally known and those are the two so far although there are some people working for Oliver who deserve to have international reputations I don't see their blogs so I don't know and oh well there's also Ave although I never I'm always afraid to say your name because I can't pronounce it so it's something like I have a Oh today holding holding kind of like Oreo with the letters mixed up um uh-huh there was someone in Israel a services secretary in Israel I had to deal with when I was teaching there and her name was uh Ken and I would say that I'd say Ken and they go what who is that and then like 10 10 10 and they would say oh you mean him I'm like yes that's what I said like no oh you didn't say that you said something different you said like pinball or Sun we don't know what that is but you win huh clone there some kind of ultrasonic sound I can't make oh it drives me nuts one last thing here's my last thing whenever I show up as a tester and I want to establish my reputation with developers I do several things to do but striking proofs of my capacity so that they know they're dealing with someone who's an intellectual who's a rapid learner who is no lightweight and I've worked many years not to be a lightweight one of the things I do to disarm the developer and charm them into submission is I give them a piece of paper and it has 13 commitments on it I say these are my promises to you and they've never ever received this from a tester before no one I've given this to as ever ever encounter tester who has ever made any promises or commitments to them but that's what I do I provide a service you're an important client of that service I'm not satisfied unless you were satisfied if you're unhappy with my work I'm unhappy with my work there I said it I'm not the race car driver I'm the pit crew guy I know that I don't have to deal with spinning off the track and crashing you're the guy who puts the bugs in the product you know testers like to say they break the product I know I don't break the product I know you spent all night breaking that product and I'm just noticing the result of your work I didn't break it so I know the pressures not on me pressures not on me I know that so I support you all the pressures on you developer and I'm going to help you with that pressure I'm going to support you I'm going to be the guy you can count on to try to see the best in your software and also to help you notice how there might be some things that other people won't like in it not me I mean I love you but then you don't love you like I love you so let's keep it in the family show me the product developer I'll complain about it but you know I love you it's okay you can hear complaints from me look when I'm about to leave the house and my wife says and there's a little bit of mustard on your chin I don't go oh how embarrassing I go thanks honey that's your job I expect you to do that thank you very much she wants me to look my best I didn't buy these clothes I don't know about clothes buying she figured all that stuff out so I just wear it and I hope for the basket if fights not the best swerve all she's support crew for me in that respect I can't do what I do unless she does what she does and I support developers I work with so I make a list of all these things that I promise to do I'm going to learn on my own as much as possible I'm going to try and find important problems quickly I'm going to write clear respectful problem reports I'm going to if you have special requests developer I'm going to try to fulfill those special requests I'm will never fake my work will always do work that I think is real and useful and if I'm asked to fake my work which I sometimes am I refuse to do it you will be asked to face fake your work to everybody is asked to in their career when I say fake your work someone will say something like create a bunch of test scripts written test scripts and you'll go but I really can't do that because the products really not designed yet but our process says you have to create the the script so just to do something my response is no I will not do that I will not make documentation that is useless under any circumstances I will never do that because you pay me too much but if you only paid me let's say two euros an hour I'd be happy to do any fake work you want but you pay me so much money that I must only do my best work and not waste your time or anyone elses or pretend to do things that can't be done this is stuff that's hard to say to your employer when you're new and young but the older you get the easier it gets I find and the less patience you have for bad work okay well let me just wrap this up what I've tried to show you is some of the thought processes that go into being an excellent tester you need to think laterally you need to think vertically you need to learn things quickly you need to become excellent at asking questions you need to model products in your mind and respond to those models I haven't had time to demonstrate some of the finer points of test design and that sort of thing but I have demonstrated some examples of testing and specifically talking about testing because your ability to talk about your testing is critical to how well you are thought of as a tester Google me you'll find out that I am considered a world expert in software testing you'll find out if you count the hits not that they matter but I do get the most I get more hits than any one with respect to software testing type in anyone's name it could be like Mahatma Gandhi I blow him away my great man not in software testing okay like how about about Julius Caesar no no reputation at all as a tester okay compared to me so I have a reputation as a tester people think I'm a great tester it's not cuz I am a great tester because they don't know they've never seen me test all these people citing my work they've never seen me tastic maybe a few of them have here and there once in a while they see glimpses why do they think I'm a great tester because I talk really well about testing I'm aware that that's not the same thing as being a great tester so I have a secret program that I am working on secretly which is to actually be a great tester and I work on that really hard because I don't want to be exposed as a fraud but the thing is is there a lot of people who are really great testers and they are unknown in the industry unknown because they don't know how to talk and explain themselves so I encourage you first of all come into the world of testing I think you'll enjoy it if you are a strongly intellectual person with a short attention span who loves to learn new things all the time and solve puzzles then you'll really enjoy being in testing and if you have some development skills all the better and a lot less pressures on you then will be on you if you try to build things if you are going to be a developer try to insist on getting testers who work for you or work with you who are intellectual rapid learners who like to solve puzzles we need to hold testers to a higher standard and as you encounter the is dqb I hope you will notice as I have that it is almost content free and says nothing whatsoever about skills and pretends as if testing is all a bunch of words that you have to memorize and most of the words will be rather useless I predict but maybe some of them will be useful I just think if you never heard of the is cqb you'll be better off and you'll be a better tester and you'll be more educated then if you study the is dqb syllabus which is 40 years out of date all right that's what I have to say that's my challenge to you from across the great great Atlantic sea good luck and I hope that you that you could get out there and get involved in the testing community thank you very much

Channel: Eesti Infotehnoloogia Kolledž

Views: 378,519

Keywords: estonian, information, technology, college, james, bach, software, testing

Id: ILkT_HV9DVU

Length: 98min 35sec (5915 seconds)

Published: Wed Sep 07 2011