The Case Against TDD - Eric Smith

you can clap yeah I forgot let me put this on it'll help pad out the runtime of this talk is can you hear me now do you feel better okay uh it's good to see like Emmanuel SEDs and familiar faces the familiar faces do me a favor tell the unfamiliar faces when they're supposed to laugh because sometimes they don't know I said my name is Eric Smith I've been with a plight for a little over ten years now and the reason I actually join a plight was because I was at a company that thought that quality was not a big deal they had a defect list that had over 7,000 open defects a product they shipped every year and they had a system whereby if a defect was a medium or low in new code it didn't have to be fixed to ship if it was a high medium or low in existing code it didn't have to be fixed to ship because once your customer is used to getting home they're fine with it right so I came here because I wanted to do higher quality work and I've gotten to do that right now this is a software craftsmanship Meetup which means all of you know TDD is already one my talk title is bullcrap and we can just get done eating pizza which is good because I didn't finish my dinner right so TDD is one that debates over you guys have probably all heard this guy right you know the software craftsmanship movement in many ways got kicked off at agile about nine eight years ago where Bob stood on stage and yelled craftsmanship over crap all right and he has gone on record usually like well he starts pacing back and forth if any was in the talk and just glares everyone kind of like that and says you are unprofessional do you write a single line of code without a failing unit tests know if you've met Bob you know his favorite language to deal with to write code in these days is closure closure is a language created by rich Hickey we're going to see some code enclosure script today actually rich Hickey has also given talks about TDD where he is compared to guardrails and called it stupid but you know what maybe rich doesn't know what he's talking about maybe rich isn't a professional maybe he works for free he also Bob does works on a Mac exclusively well max if any of you remember when the Mac was the NDA was released and we all get to write iOS apps an occasional they give these big presentations I was at WWDC that's Sharon occasionally someone would go how do you test your code and they'd say well that's easy you plug in your iPad a phone and you run all the code on your phone there was no way to do unit testing it literally could not be done I mean while all that is based on an operating system called Unix which doesn't have you which doesn't practice TDD largely doesn't so in order to be a professional you can use a unprofessional operating system to run on profession on an unprofessional laptop with unprofessional languages as long as you write a test somehow we can build this casts a lot of sand and say well each level is completely unprofessional crap but my bowling game is perfect okay I did not put this on my face right as entertaining as it might be for you I am NOT going to let this strangle me in the middle of my talk TDD has three claims it makes and today's talk is called the case against TDD I'm going to make a case for you today that all of its claims are bubkis the first is that code with tests doesn't have bugs has minimal bugs but maybe not none at all not a single bug in the code base its perfection Yeah right because there's no bugs in the code base we can refactor without fear I mean just change big old swaths of code pull stuff out put things in right and we'll work fast by working well and that is actually the second claim TDD even tends to make whoa somebody wants to written unit tests for this piece of software this one it's faster we're going to be faster the reason we're going to be faster is because we're going to get fast feedback on whether stuff works and we're going to be faster because we're always working on good code on clean code and it's all going to be working so we'll be quicker and finally the third big claim is that the design is better TDD promotes simple design it promotes the minimal amount to do in order to make sure that you've got just enough to make your tests pass it follows those laws of simple design it's it's wonderful and everything's good and we can all go home so let's dive into the first claim because it's the biggest the reduction of defects okay I have here's some closure script just a little bit don't worry we're going to use this little piece of code to demonstrate how and where a TDD can fail so does anybody out in the audience care to take a guess what this code does updates the game Thanks perhaps there's another detail in there do you think the code could oh sorry go ahead right okay it's updating the game state I can give you a little more detail what this actually does is takes the game state which is a hash runs it through a series of transformations and because this is closure everything is a value so it calls a function and returns a new hash that gets the new state and then associate it back right to the state that it started with and returns it pretty simple could the code be improved something in there really bad or seems pretty good right nothing particularly wrong about it maybe state with a game and it's a little redundant how would TDD improve it if we want to ask ourselves that question we have to look at the tests this is a set of the tests a subsection the reason it's only a subsection is because if I wanted to show you all the tests for those six lines of code Sigma we can even count them one two three M is seven or eight lines of code it's over a hundred lines or so love tests of test code and the reason it is is because this test code needs to make sure that you're getting the time correctly which required a function wrapper it needs to make sure the invasion gets updated but when you needed to write a test to make sure the laser got updated well you had to make sure you set up a good invasion update when you set up a good laser you had to make sure the bullet was updated nicely each test built on the others this thing and I are really getting along well each test built on the previous one and they got a little longer and a little longer they just keep going now let's assume for a moment that by writing all these tests I reduced my blood count by 75% in the production code I'm going to show you a very scientific graph now completely provable that compares the amount of defects found in test code for people that do right tenth unit tests and for the people that don't write unit tests the people that do TDD find this many bugs in their test code people that don't find this many because we often say there are less bugs because we just don't count all that extra code we wrote I didn't write a little bit of extra code for that little update game I wrote a ton of extra code and all of it has the potential for defects if you don't write test code you don't have bugs and tests pretty obvious when you think about it but the fact is the people that are not writing test codes don't spend time debugging those tests they don't have bugs in their tests now you might have looked at that code and said there is something I can do to change it because when it comes down to these tests you can sort it you can see that each one is making sure of the state of each one of those updates you can kind of see it don't you know try to do it too much because you're not gonna be able to parse it all together there but you can see that each test is they're not mocking their non Macha's tests all right they're actually making sure to call those functions because all those functions are immutable so they can be called safely no side-effects so we could begin by mocking this we could change and inject the updaters all that little functions would greatly reduce the amount of tests I wrote by not testing anything all my tests would do is validate that I can calculate the time between each an and even that is a lie because as you well know if you've ever maintained a test suite with more than about four tests in it or if in fact you have internationalized code and know that we have two different daylight savings times on this planet you never ever actually want to query the time in a test which means this is a wrapper hence the T slash epoch and it's not just using a true test thing it returns a known time so if I inject mocks here literally this line is tested everything else is just fakes it's me shuffling a bunch of things on the table and making it look like I cleaned up but really I just made a different mess and in fact that messes in my tests which are again going to be loaded with potential bugs recently you may have seen the news is on hacker news programming reddit somewhere or maybe was on slack that there was a scientific study done measuring the effectiveness of TDD and the study said in a paper called an external replication on the effects of test-driven development using a multi-state blind and animal and analysis approach I almost said it right TDD does not affect testing effort software external quality and protectors developers productivity when they tried to test this it made absolutely no difference the people that wrote tests after took the exact same amount of time have the exact same amount of defects as the people that wrote them before now fine you might say okay so you can write a master and catch no more new bugs but I'm faster right I'm faster I have living documentation I have clean code I can refactor making me fast drugs I can make it simpler for next time let's start with the argument that we're making living documentation what is this test do what is it testing you can read the string part good for you sir that's okay I wrote this test and that's the only part I can understand good it does set a static time value to two there's some mocks in here there's some keys let's get a show of hands how many of you do test-driven development most of the time all the time come on now I know there's apprentices in here you better put your hands over here in trouble right one if you find that your most complicated code is in the unit tests yeah happens all the time especially in code that's really procedural or really just do thing do think do thing do thing the code the tests actually get more complex in the code because as hard as we try to make our production code clean our test code gets complicated and we all know that if you try really really really hard and make your test code really really really clean you'll have no idea what it does later when you come back to it because test code needs to be self explanatory and the more abstract you make it an effort to make it clean the harder it is to actually follow the next time you need to do it which is why tests tend to be built by copying the previous tests pasting it and starting to tweak the numbers and the bits we all do this writing unit tests all on all my code for 10 years and I still do it and I tell my apprentices not to so fortunately Honda's not here right now once again this code can be riddled with defense guys got continuous built and gals you got the continuous build servers okay when the build breaks how off you can just shout out a number what percentage of the time is in an actual defect so code you say zero almost never right in fact the better and more diligent you get at it the rarer real defects found in the continuous build system are which means we spend an enormous quantity of time debugging the build system has anybody ever spent a day debugging their build anybody ever spent a week because I've done it I was on a team once that had a team to debug to build because the suite had grown so large that any attempt to change any order they had it sprawl over multiple machines and what would occasionally happen is someone would inject a new test which would make the tests actually jump to another machine so another five tests would go from machine a to machine B and it turned out there was a dependency on a test that was only running on machine a now and those tests would fail tests are hard to maintain we say we're faster because we ignore the maintenance cost right we do like funny accounting we go and we say we're gonna show you when we teach other people this is how you should write tests will show how we do a bowling game or prime factors or any of the different sort of this is how we teach TDD examples that are out there and we stop the end of it and we say look see how well that worked but we ignore the fact that I know have to check that in and those tests are there forever and they are now everyone's maintained maintenance problem we sometimes say code is a liability not an asset well tat unit-tests ad easily twice as much code I'm being kind if you're been doing TDD all the time having a ratio like I showed you or it's a hundred lines to seven is not uncommon at all doing test maintenance making them in the living documentation making them reliable takes time time we don't factor it doesn't really make you run faster and does it help with refactoring I put this up here it's the same as before it's got the updater and the updaters in the do seek here alright for those of you don't know closure you can think of this like a for loop that just says loop through every updater and call it if I want to refactor that section does it help do the tests help they're not gonna because I'm going to written unit tests that gave fake updaters I can break this prompt like this all the time and that's another thing that unit testing can add which is an entirely new class of defects we didn't have before say what you will about people that write thousand line methods or functions they don't start injecting dependencies which then break because in all the wiring didn't get worked out right we do this and we introduce defects that way and we own and I would have only done this for testing same idea imagine I had done this and actually injected updated what if I switch to actually inject the updaters test well if I'm doing true isolation testing there's a good chance I'm knocked out update game everywhere I call it no test they'll tell me I screwed it up and it'll be broken everywhere and finally though we always talk about feedback right the tests are running all the time we get feedback we get feedback we get feedback okay well let me do a quick demo hey not so quick demo okay this is my current pet project space invaders enclosure script 139 unit tests right now take a second or two to run the tests not bad but let's go over here I've got the actual game over here do the test over there actually tell me if the game works I'm sunk to some extent yeah but in other ways they don't I don't have a test that tells me that once the bullet gets to the top of the screen I'm stuck in the upper left-hand corner did it take me any longer to look at the actual app no not really the tests on the app were pretty much the exact same speed and of course I have a continuous build because I am really big on self-immolation I gave myself a continuous build despite the fact I don't actually deploy this app anywhere and I'm not working with anyone else on it because that's what you do dammit it's the right thing the continuous build fails does it matter probably not yet I have it there what are the odds I'm gonna finish this if I got things like that causing me problems I think I'm gonna finish anything when I'm working on my little toy ha me on the train up builds broken forget it I'll just do something else alright but finally right we have better design the designs better it's better how do we define better design oh that's easy we defined it by saying it's more testable so you see TDD produces better design because it's more testable because it produces better design and that is what you call the circle the fact of the matter is we don't even have and except a definition of what better design is we've got a lot of principles we don't know if any of them are real or full of it is it better than my coats immutable versus mutating I don't know there are days when I'm really happy it's an immutable hey everything's just calling a function I can tell what comes out the other days I'm like you know look every damn functions got a worry about returning in this and I've got to keep track of where stuff is all over the place because nothing just holds the goddamn state and in the case of design where we start injecting mocks isolation testing right now we're saying heavily mocked better code is better because we skip the testing right we mocked out the object by decoupling it so that we don't have to have it affect our tests and so we're better I can't actually even follow the logic at this point anymore like was it 2 years ago and ehh did the whole TDD is dead thing and we made fun of him because he has goofy hair and says silly things sometimes but he talked about test-driven design pain this exists it's not a thing he made up sometimes people change the code to make it more complex solely for testability Bob Martin the angry face at the beginning for those of you that didn't recognize them often says the second guy pays for abstraction so what you're teaching the single responsibility principle people can use that to just tease everything out and say well no you can never actually do anything because every method has to only do one thing and once it's two lines it's doing two things oh no we can never get anything shipped so say lacks the second person pays for abstraction don't make the abstraction until the actual product forces you into it but when we write these tests we frequently have to force abstractions that aren't necessary because the client is as we often brag about in TDD those tests are the second guy and now we're making the first client pay for it every time you have flexibility for the sake of testing you're making code more complex on some level so compare my two different versions of this function this function is simpler because it's a loop with one call it's more complex because it has dependencies that can break it's more complex because there's more you have to follow it's more complex because when you read you have to understand what an update err is the one on the top is not what we would call OCP I have to change that function every single time I update you may have noticed the bullet does not run into any of the little space invaders because I haven't added that function yet but the one on the top I know exactly what it does now there's the rules of the game there you go there's space invaders I'd say don't copy me but I'm copying someone else I think you're okay so I think I've proven beyond a shadow of doubt that all your unit tests are a waste of time we shouldn't have bothered the last 10 years of my life were complete and unadulterated waste what do we do oh I showed you all this stuff enclosure script which is going to make the next thing I show you be a little ironic but one of the things that happened with TDD is that it came out of the Smalltalk community which is dynamically typed language got popularized in Java but it was really made popular in languages like Ruby and JavaScript where Lord knows you don't have any freakin types so you need something right we need static typing I abandoned it years ago and I'm looking back thinking I regret it right static types are free tests and we throw them away cuz we didn't want to type them like literally typed them and they happen before it even runs for all our unit tests anybody who here works in dynamic language is at some point had code that blew up because the types want what they thought the closure code I showed you before every function takes a hash and returns a hash right you sure because I know I'm not okay you can't forget to write the type checker you can forget to write a unit test and make sure all your parts of your contract are there funny I showed you closure right in the closure world they don't do TDD they think that's guardrails rich tick you told them it was silly they do everything in the repple they even call it repla driven development and apparently they have rebels with menorahs and common lisp which is kind of cool what if all that maintenance pain I talked about was just gone and we toss it away we get the feedback from the tests we want but we don't make everyone else maintain our half-finished thoughts that tests often are other alternatives fun fact designed by contract is older than test-driven development and a lot of your object oriental schemata the same people who invented it not going to do an entire talk on design by contract now not the least of which because I am not even remotely close to an expert at it or even all that good edit but the basic gist is every method function class what knotheads invariance preconditions and postconditions closure has this built in at least in the form of pre and post can tell you a story about a project I was working on a couple years ago tested the crap out of it really really good unit tests and I had two gnarly functions I despite all my best efforts could not break down the way I want it and continually caused bugs come up again and again and I would write tests when I'm sure this test reproduces it not that one passes to dam till one day I put the contract in and I let the app run twenty minutes later the bug was fixed now I kept all those other tests to because I'm a big fat weenie pains but fact is the design by contract at that moment gave me a lot better feedback next logical step ends up being things like quick check property based testing Rick wind freeze here he can tell you all about Haskell and property based testing he he's shaking his head no now but he is the biggest expert in the front row I'm almost certain in fairness I haven't met you for before tonight preconditions and postconditions only give you some more they can give you some more complicated feedback what this does enclosure we're not going to go again in super details is it starts putting types and requirements on the methods so enclosure this is a multi method you can think of it like polymorphism or switching on type that says when the game state is named playing call this version of the function there's another function elsewhere and you can start writing these specifications that say exactly what your contracts are it's kind of it's the next logical step from design by contract well you say this is what the state of the system needs to be when I call in to the function and when something comes out of it and you can even put them in the middle those don't do anything at run time but they have this beautiful thing down here where you can just call it's called checker that will generate hundreds of valid scenarios maybe thousands depends on the setup I suppose maybe millions but I don't have the patience for that anything within the range of valid inputs and outputs and makes sure that when for all or as many invalid inputs as it can think of you'll get valid outputs I set this up on my test and looky here I have a failure even though it passes all my unit tests and there are many so should we abandon TDD right can't wait to a craftsmanship meet up but all you here instead of having a kata battle or showing you the latest way I write unit tests I basically told you that your whole lives are crap so there's a different code checks the collision between two bounding boxes very simple stuff I have no idea how to write quick check tests for this it only returns true or false Wow there's a great contract I need at that level I need unit tests I need to check that if I send it to boxes like this they don't collide if I send it boxes like like this I camera they collide and I need to do all the sides and I need to do the corners and I only know how to do that with TDD fact is if you wander around online long enough you'll find somebody saying something along the lines of TDD is great but it's a tool in the toolbox and I recommend a pragmatic approach then we'll never tell you what the pragmatic approach is because what they're really saying is I don't want to do it and my boss told me I had to until such time as I know how to do a completely better way and when to turn on and turn off TDD I'm still going to write unit tests for all my code and I fibbed a little at the beginning of this talk in a different paper called on the effectiveness of tests first approach to programming the science said main result is that it worked those who write road tests first wrote better code less defects it's all much better now what does this tell us we have one study says it doesn't work another says it works great tells us that we're crap studying our field we're terrible at it I believe if you add the tune the number of participants in these two studies together you get to almost 40 people I'm pretty sure given 40 people I can prove almost anything all right but it also says your software craftsman or people software craftspeople I'm going to get better at that we don't need to be TDD is not equal to craftsmanship it is the bomb and he helped popularize the term but truth matter is there are a lot of ways to write great products and you can TDD away to a big honkin pile of garbage people do it all the time testable is not the same as good and on top of that we need to define good still we need to if you'll pardon the pun prove our assertions about what we say is good and what we say is bad I was in a training class recently a few weeks ago and I said shorter methods are good because that's not debatable right at which point one of my students said well actually I write them more like a lot longer because then I can tell what everything does right in front of me I didn't really have an argument for that other than you and you is not actually in arguments we need to kind of not be jerks when we get into other people's code bases maybe I see this more than some of you do because I'm a consultant I go places every six months every year right but I see people check out code all the time not me I would never do this but other people would check out Co and be like oh my god what idiot wrote this there's not a damn test in it well there's lots of great code with no tests so maybe that person is an idiot they might be but you don't know that yet and maybe you could try to find out why there aren't tests written and finally we need to remember that everything is a trade-off as of today November whatever today is November 16th 2016 a mere eight days since the Cubs won the World Series kind of Mets plan to clap I still think TDD is the best way to write code but I don't think it is in all scenarios anymore and I don't think it's universally a good thing and I think I could be wrong which is not something that I necessarily felt five years ago and I think that when we start telling other organizations other developers you have to do this we have to keep in mind there are downsides 10,000 tests in a build is a downside on many levels another levels it's not ultimately TDD is actually a fad now there are multiple kinds of fads right pet rock was a fad and he made a million dollars right it's always jogging weight lifting exercise is a fad eating healthy that can be a fad fat doesn't mean it's bad and a fad that stays around for a long time ends up being a good thing but sometimes it gets overdone CDs a fad right now it's born of good intentions it has some primarily anecdotal evidence behind it and it's time will probably pass when we move on to something much much better some of the things I mentioned today might be the better things some of them might not it may be something we haven't even talked about yet maybe formal proofs will come around to being possible in a reasonable amount of time you know or maybe someone there in the back is going to invent something that or has invented something I never heard of but it's a fad and we should remember that so that we don't judge other people who don't do it and that is my tongue should I ask those questions I don't know I'm a little frightened asked that question what the hell is wrong with you is not a valid question before you ask life that's why so um yeah so the question was our company doesn't do TDD but there's quite a movement to do automated testing there's the same kind of thing applied automated testing depending on the kind of test you're talking about it actually argue some of its worse um so my wife is in QA and whenever I mentioned to her automated testing her eyes get all big and she starts to get pale and she's like not that again so if you're talking about automated testing of like on the small on the medium level that has a lot of benefits the larger tests get the more maintenance goes in so if you're going to try to test your entire application through the GUI and every single test through like the GUI which is what most of the time teams take a big jump in automated testing they want to do the cost of that is extraordinary even if you managed to write all the automated tests for the entire system and you won't but even if you do it's going to be an enormous amount to maintain and it's okay which is why you know companies usually try to solve it by sending it to another country where it's cheaper which also has its own class of problems there are other questions so yeah so mr. shingles asked about the benefit of design during TDD so the benefit of design is there I don't think it's completely thrown out and the way I brought it up in the beginning but I do think that we have to consider the trade-offs and when we're doing TDD frequently we decouple code way too soon we make too much stuff public we break encapsulation we like to say we don't we get defensive when people accuse us of this like no no I don't do that and you're like how big are your classes they're all six lines damn you might have broken some encapsulation right so when we use TDD as an excuse to turn our brain brain off then it can absolutely harm our design we need to balance making things more testable with actual thought you know the the XP folks the ones who popularize TDD in the first place also popularize crc cards and most you probably don't know what I mean by CRC cards because nobody uses those anymore but we probably should a white board is still ok just don't laminate the white board costs a fortune you can never get it through the machines yeah so you're asking okay so the question is if I understand it right would I invest time in TDD if my boss just wants a proof of concept no I wouldn't it's not worth it because I'm not going to keep it in fact I would intentionally not invite tests for it because the closer you get it to production ready the more likely it is you'll use in production so I want that to be as far from production ready as possible because I know it's throwaway code so I don't want it to look like it's not yeah yeah so I think everyone heard it but probably not the recording so the question was would we benefit from essentially raising our level of unit testing to be a little higher to be more integration level rather than being more isolated on it one unit at a time the answer is sometimes right just like anything right it is a trade-off so tests that are really localized are really fast and they're really specific you don't miss stuff tested at a higher level probably catch more defects catch more mistakes but they can be slow I just about a few months ago rolled off a number J s app and if you've used Ember you know that the default testing suite that comes with it kind of discourages unit testing and pushes things into what they call integration and then acceptance level and that could be so frustrating to work with because when you're trying to write a test just to make sure a dang button gets checked I really don't need to start the entire app and login so in each case I think there is a trade-off um and in some cases I think that trade-off might be so in the example I gave today I mean like that's real code that's my real hobby project and I might really throw out the tests for that one section because I don't want to inject the dependency to make it truly isolated I don't think that gives me any benefit I think it makes more complicated and I also don't want to maintain these tests are really hard to follow and I can spend a lot of time refactoring them that might make them easier to follow and a little less funky but I also want to actually make the invaders blow up I only get an hour to on the train I think you had a question okay um so the question was in a language like C sharp like a no language how do you write unit tests around internal do you and how do you write unit tests around internal methods so we could actually extend this talk out for a long the short answer is no I write the test through the public interface and make sure I the internal methods get exercised through the public interface you know if it doesn't get called through the public interface why does it even exist most of the time my private methods get extracted now to your point or maybe this wasn't your point but um those can get overtime confusing especially as your classes turn into more of an iceberg right small public interface large limitation and that's where we as TDD people tend to break off tests and then we make our buttock off classes we make new classes and we test those sometimes that's a really good design choice sometimes it's not and we just did it for our own benefit and I kind of apologize I realize on every single question I'm kind of saying both sides but that was sort of the point anything else oh no I did not expected few questions yeah so the question is that you know I have I seen a difference in chat between shops that started without unit testing and then with new code they start adding unit tests to their code so broadly speaking the new the new code will be much better however the developers are also better at their job because they've been doing this longer and have pretty have learned from the stuff they left behind so some of the improvement may be to do two tests because I'm really I've knocked tests a lot but the truth is they do have a lot of benefits that out of craftsmanship meet up you guys really don't need a numerated but the fact is a lot of the benefit also comes from the fact that you're a better developer huh and when a system grows you usually know more about it all the hardest stuffs at the beginning when you're sort of like you know blank sheet of paper yeah and then usually they ask how the heck are we going to test this old stuff hmm prayers heavily involved yeah and I actually saw that I mentioned that you know in many cases you don't even know if you broke stuff if you have to change the old things yeah and I do think that is a case where higher level integration tests actually really shine is when you have to identify a section of the code especially if like there is a part of the system that is just broken testin or not tested or I like I've written suctions of sections of code actually suctions of code is probably a good description um that I tested the heck out of it the unit level it just sucked and they didn't work and then you try again and it sucks and it doesn't work and then someone else cries and it sucks and it doesn't work it happens right we're all human and or it sucks and doesn't work and it's not unit tested at that point high level tests can really help I call this the in case of emergency break glass approach to integration test right like bust amount you just don't want to keep all of them because once you've put a big scaffolding around it you've cleaned it up and it's been green and good for a while bunch of those tests are going to have a high maintenance cost start selectively chucking them they're not actually providing benefit anymore because you fix the root problem if you ever get to the point where you fix the root problem which does not always happen anything else all right well thanks guys just you to paying attention and asking all the questions

Channel: Chicago Software Craftsmanship

Views: 5,242

Rating: 3.48 out of 5

Keywords: software, craftsmanship, tdd

Id: nRdn5k5jKyY

Length: 47min 33sec (2853 seconds)

Published: Thu Nov 24 2016