Ned Batchelder: Getting Started Testing - PyCon 2014

all right good afternoon everybody thank you for coming I'd like to introduce Ned Batchelder who's going to talk about getting started with testing hello everyone so this talk is starts at 1:40 but the printed program says 1:55 so there's going to be some people coming in late and you can all make friends by piling up with them later and explaining what they missed in the first 15 minutes I'm going to have a lot of code on the screen it might be difficult to read your two options are to move forward or to use the bitly URL at the bottom which will take you to a page on my website which has all the slides in the code and even the text if you want to follow along so my goal here today is to show you a way to test your code I'm assuming that you have not written tests before and you are test curious let's say and that you want to figure out how to do it so I'm going to show you a way to test code there's lots of different ways to do it and if you talk to testing experts they're probably going to tell you that this is all the wrong way to do it maybe but this is a way to get you started I want to remove the mystery from writing tests first a little bit of philosophy about this why test testing automated testing is the best way we know to figure out if code works what I want to hope when I'm trying to show you in this talk is to look at tests as a solution to a very important problem that you have you have written some code and the problem you have is that you don't know if it works and tests are the solution to that problem some people approach tests as a checkoff on some sort of form that they need to complete before they can go home at the end of the day it's just sort of paperwork to get done and that's the wrong way to look at it you'll be very unhappy if you write tests that way testing is a very serious problem too it's a serious solution to a very serious problem and if you approach it as such as real engineering you'll get a lot of benefit from it it will save you time because you won't have to go back and fix problems later and it will give you better code because the tests will actually help you write better more modular code to begin with most importantly for me it removes fear from the development process if you've ever approached a large piece of code and had to change it and thought that you were certain things up and make the code not work anymore testing is a way to make sure that you will not do that anymore it turns fear into boredom as a way that I've heard it put and the other way to think of it is that debugging is very hard but testing is easy so testing is a really good way to write really good code this is most developers first impression of testing they know that they should be doing it but they're not doing it and so they feel bad and and just like our poor developer here they are paralyzed by this feeling of inadequacy and uncertainty and doubt about what they should be doing and and and they feel bad and I don't want you to feel bad testing is hard it's a lot of work people and by people I mean you will not want to actually do it but it pays off so if you approach it as an engineering task to solve a problem that you really have and that you want to solve it will work out for you the reality is that the universe is a chaotic place so here's another poor developer she recognizes that the universe is full of chaos that is trying to attack her code okay you know how it is you write a program it seems to work great and a week later who knows what's happened to it some gremlin has gotten to it and it no longer works tests are the defense against this our developer here has a spear and a shield those are the tests that I'm going to teach you how to write them and use them alright our goal for this talk is to get these two developers happy and confident at the end so the road map there are three main parts that we're going to talk about today the first part was we're going to grow some tests organically we'll take some real code and we'll just start testing it's sort of ad hoc making up as we go along and talking about what we like and don't like about the way those tests are coming out and then we'll the main piece of the talk is talking about the unit test library in the standard mount in the standard library that is the way that the right way to write the tests and I'll show you the some bits and pieces of the unit test module and show you how to use them to build your tests and then I want to get into a kind of an advanced topic mocks which is something that you don't often see in a novice talk but it's a really really powerful tool for making your tests much more powerful and flex so let's start from first principles and grow some tests so here is our stock portfolio class and again if you can't read the code there is a bitly link at the bottom bitly slash PI test zero where you can see the code so this is a stock portfolio class it's very simple we define a class it keeps a list of tuples named shares price you can buy some shares for a certain amount which will append that stock lot to the list and then later you can ask what is the cost of the portfolio the original cost which is simply the sum of the shares times the price for all the stocks in our portfolio very simple code how do we test it well the first test is excuse me you just launch the interactive interpreter and you start filling around with it you make an empty portfolio and you can see that its cost is zero and then you can buy a hundred shares of IBM stock and you see that the cost is gone up and then you buy a hundred shares of HP stock and the couple the cost has gone up some more so this is really good so the good thing here is we're testing our code let's be honest there are developers who have not gotten this far right so right off the bat let's just congratulate ourselves that we are actually testing our code but the bad thing is that this is not a repeatable process if we change the implementation we're going to have to go and do this again and we won't remember what it is we test it and we'll probably miss a case and that's where the bug will be it's also labor intensive you have to type this out every time and you don't know if it's right you're going to get out your calculator and see if the arithmetic works out right so let's fix some of those problems the second test will write a Python file and we'll import our portfolio class will basically do the exact same things we just did in the interactive interpreter we'll print out what the cost is at the empty portfolio and then after we buy a hundred shares of IBM and that's good and we can see that you know this price has gone up for the IBM stock and then after we buy the HP stock it goes up so that's good we're testing the code it's now repeatable we can run that Python file anytime we like it's very low effort it doesn't take much to run that file but we still don't know if it's right we have to look at that output and you know is it right that it comes out two to one two six three you have to get out the calculator again all right let's add some more to our tests so now here what we've got is we've got a portfolio cost where we print out what it should be and then we can print out what it should be and we can print out what it should be and here we can see that it's printing those I apologize for this code going off the side of the screen now it's even better we've got repeatable tests with low effort we've got it's better it's better because we got explicit expected results we still have to check the results ourselves here the line of code if it were wrong we just have to look and see if the two numbers were the same or if they were different right the computer isn't telling us whether the number was right it's just showing us the actual result and the expected expected result so again let's add a little bit more to our tests keep in mind these are not the way you're supposed to write tests okay this is you exploring the problem of writing good tests I'm going to show you with unit tests how to do this the right way but here now what we've done is we've taken our cost and in addition to printing a message about what it should be we use an assert statement to assert that the cost should be zero if you haven't used the assert statement before it takes a condition and if the condition is true it simply carries on executing the next statement but if the condition is false it raises an exception so here we can make an assertion that the cost must be zero because our tests are passing the assertion never happens so now we have repeatable tests with low effort we have explicit expected results and the results are checked automatically good the problem is that if the test fails it will actually throw an assertion error and the tests will stop this blue code is actually incorrect there should be an assertion error there so if one of those assertion fails the entire program will stop and you won't run the rest of the tests so if you would this were a real program that we're doing many tests you'd have a hundred or a thousand or even 10,000 checks to make if the second one failed you'd never find out about the rest of them what we'd really like is for each check to be run completely independently so as you can see this is getting complicated right if we wanted to approach that problem of how do we make the tests run independently of each other that would be yet more engineering to put into these tests that's going to grow it's going to become a real program with real engineering typically engineers when they find that they're encountering problems that other people have encountered they go and find a library to help them solve those problems that library is unit test and we'll get that to that in a second but first let's talk about the characteristics we want from tests we want our tests to be automated so that there will be low effort and they'll be repeatable because if they're too hard to run or if they're too confusing to run you're not going to run them we want them to be fast for the same reason we want them to be reliable remember the point of a test is you want to run it and have it tell you something you don't know if you doubt the test then you haven't really gained anything now you've simply moved the question from do mic just my code work to do my tests work and you haven't really solved anything so you need your tests to be reliable you need to you need to believe them when they say that something doesn't work and when they tell you it doesn't work you'd like them to be as informative as possible because if you think about the workflow the workflow is you run your tests and if any one of them fails you're going to immediately try and figure out what what is broken in your code and so the more clues you can get from your tests the better off you're going to be during that debugging exercise and lastly they should be focused your tests should exercise as little code as possible which is a little counterintuitive usually we want any line of code we write to do as much as possible but with tests we want each test to be as focused as possible because if it fails and you dive into your code to debug the problem the less code it runs the Morpha narrowly focused your debugging task is so this is what we want out of our tests and now we can talk about how unit tests actually gives us these properties so unit test is a lot module in the Python standard library it is the infrastructure that you use for well-structured tests and it is the lingua franca for tests it is based on the same patterns as J unit and n unit and CPP unit and a lot of testing libraries across many different languages for that reason unit test has kind of a java flavor to it which a lot of people pythonista is don't quite like which is why there are alternative ways of writing tests but we're going to talk about unit tests here because it is the it is the way everyone starts out writing tests and all the test runners and test tools support these tests so this is an actual unit test to write a unit test and by unit test by the way I just mean an automated test you import the unit test module you import your tests your code that's under test and then you're going to define a class and your class is derived from unit test test case so all of your tests will be in a class that derives from test case and the tests themselves will be methods will be methods in this class let's start with the word test so here we have a method called test buy one stock and it's going to do the test that we did before where we make an empty portfolio we buy a hundred shares of IBM and then we can make an assertion that the cost is what it should be the way you run tests with unit tests is that you use Python - M unit test if you haven't seen the - M flag it means instead of running Python code that you find in a file I'm going to name you're going to run Python code in a module I'm going to name and so by running you the unit test module as your main program it knows how to actually take the next argument test port 1 import that and find the tests in it so it will automatically find the test class on the test methods and run them this single dot means that it ran one test and then it tells you that it ran one test and everything went ok so that ran your test and the test passed behind the scenes what actually happens is unit tests instantiates your portfolio test class to get a test case object and then it runs the test method inside a try except block if it got an assertion error if your assertion failed then it records a failed test but otherwise it records a success test so conceptually it's very simple and this is perhaps how you might have approached the problem if you were building on the earlier examples and trying to solve this problem yourself wrap each assert in a try accept and catch the assertion error and print something out if it failed but unit tests will do all this for you alright let's add some more tests so here of course there's more conditions that we have to test so we'll also test the empty case the portfolio should be 0 here's the buy 1 stock that we had before and but we'll also test buying two stocks so we buy two stocks and assert the cost is that now you see we have a dot dot dot dot dot is not me leaving things out of this code example that is literally three dots that unit test prints ones for each test you get a dot for every past test some people who are really into tests talk about being dot addicted that just want to see lots and lots and lots of dots because every dot means the tests the past and then it tells you that ran three dots and it tells you that it went okay under the covers what happens again is it makes a test case object and then it runs a test method and then it makes a whole new test case object to run the second method and then it makes a third test case object to run the third method this is a very different way of using classes usually with classes you make one object to represent a thing and then you call many methods on it here we're going to make a class that has many methods each method gets called once on a separate object from each of every other method but you can see this gives us test isolation so not only do we have a try except block or out in every test method so they run independently but because each one of them is running in a brand new test case object it's very difficult to do anything in the first test that's going to affect the outcome of the second test test isolation is all about making sure that every test is independent of every other test if you do something in test one and it passes and you then but it's side effect is visible in test two how do you know that test two is really testing when it's supposed to test so the idea is to keep the tests independent of each other and to make sure that failure doesn't stop the next tests from happening so we keep them all completely isolated in theory you should be able to have a 10,000 test test suite and be able to run any single test out of it all by itself and have that test passed just as if it were run along with all of its 999,000 however many it is brothers and sisters it's hard to do arithmetic on stage now what is failure look like so suppose we had either broken our test code or broken the test the product code itself when you run the tests you'll get instead of a dot you'll get an F the F means failure okay so here what we've done is we had an assertion that was wrong we had the wrong price in there or something and it raises an assertion error so what happens is all the tests run and you get an F for every failed test in a dot for every passing test and then at the end every failed test prints a stack trace so get a full stack trace it's not printed when it fails because that would interrupt the whole field of dots thing so you get all of the dots and and then you get all of the failure to trace backs and those trace backs are very useful because that's part of the debugging clues that helps you go and find out what product code has failed so this is better because the failed tests didn't stop the other tasks but it's bad because we don't actually know exactly what failed here right we can see that the the cost was not 17 648 but we don't know what the cost actually was right it could have been zero it could have been 17 649 we don't have that information so instead of using an assert statement the unit's tests test case objects give you methods called assert equal for instance so here you can see that we've got our test by one stock method and instead of using an assert statement we've called a method on the test case object itself and instead of using an equality operator we've passed the two values into assert equal so this is kind of stylized compared to what you're used to but the advantage is that when the assertion fails it can actually print out the value it got and the value it expected so we can see the assertion failed because 17600 is not equal to 17 648 and that's a clue 17 6 17 600 is the thing that went wrong we can now we have a piece of information that can help us diagnose and debug the problem so the unit tests module gives you a lot of cert helpers a cert equal and not equal assert that this is true this is false that that is in that that this is almost equal to that for floating points etc etc etc regex matching all sorts of interesting assertion helpers for you to write your tests in a way that the test Runner can give you lots of information one pro tip that large projects often get to but people rarely do at the very beginning make your own base class to drive all your test cases from so here what I've done is I've created a class called portfolio test case derived from test case it doesn't have any test methods and in itself a test method remember starts with test underscore but it has this helper assert cost equal and here instead of passing in two numbers to assert equal I can pass in a portfolio and a cost and then this method will assert that P dot cost is equal to the number that was passed in and so now I've raised the level of discourse up into my own domain instead of tests that assert to numbers are equal I can write tests that assert that the cost of a portfolio is equal to something so here can I can say a cert cost equal of P is equal to zero and I still get the nice assertion because I'm using assert equal under the covers and I get the assert cost equal method because I've derived from portfolio test case these base classes tend to grow over time and especially if you're focused on that kind of base class then you have an opportunity to write really good domain-specific helpers for instance it's very common in web testing to say I got a response back I want to assert both that it is a 200 response it's successful and that it has this word in it well why not just have a method that says assert word in response and do the 200 check in there as well and you can have one line in each of your tests rather than two because your helper method has taken on some of that domain-specific knowledge that you're going to use over and over again this is one of the things I notice about tests is people approach them as wrote boilerplate rather than code to engineer as if it were really code you cared about you're allowed to refactor your tests and move common pieces into helpers and functions and classes to help you build those tests by the way there's a third possible outcome besides dot and F and that is an E and E happens if any exception other than assertion error comes up so here for example you see that for some reason in our test we call this by xx method which doesn't exist it doesn't have that attribute so it raised an attribute error that test ended with an E and the reason that's distinguished is a good test should either successfully pass or it should fail in an assertion if it fails in some other way that really means that the test dropped the ball right the test should have a thing in its hands and say it failed or it succeeded but if another exception happened then something is truly wrong with your test and you have to go and fix the test by the way this brings up one tricky thing for beginners which is negative testing let's say you want to call a function method a method and you want to test that it raises the exception you expected you can't just call the function here we are trying to call the by method with the wrong number of arguments so it should raise an exception if we wanted to test that we can't simply run this test method this way because that will actually raise the exception will actually get an error result because the type error will be raised by the by type error is exactly what we want to happen but we haven't managed to assert the exception happens we've simply managed to make the exception can happen unit test has a way to do this there's a self dot assert raises method that you can use as a context manager in a with statement so that you can put the P dot byline into the with statement and you can assert that it raises the type error and that with statement we'll catch that exception for you and it will check that the exception was actually the type error that you expected and the test will pass let's test the loud another method to our portfolio class we're going to add a cell method so the cell method is going to take a name in some shares and it's going to find that holding and it's going to subtract that many shares from the portfolio to test cell we're going to make a portfolio we're going to put some things in it to it we're going to try to sell 50 shares of Microsoft this cost should have gone down if we try to sell Microsoft but we trying to sell more than we had it's going to get a value error if we're trying to sell IBM and we haven't even bought IBM we're going to get a value error but notice there's a lot of repetition here and again just like any code that you write if you see repetitions if you can squint at code and you can see a repeating pattern that means that there's stuff that you can refactor and the same is true of your tests than of anything else just because you feel like your tests are boilerplate that doesn't mean they should be boilerplate you should make them as expressive as possible so we want to refactor all that code out luckily the unit test gives us a way to do this a class can define a setup method and the setup method is invoked before each test method so they hear the setup method is defining a self chief which is a portfolio object and we're going to buy the three stocks we want to buy and then we need within each of our test methods we can simply access the self dot P that was created by setup so here we can have much shorter tests that are much more descriptive of what they want to do what's happening under the covers is that for each time that we create a test case object we're going to call the setup method by the way it could fail and give you an e output but if it succeeds we'll call the test method will record the failure in the and success as always and by the way there's also a teardown method that will get invoked at the end of all of your tests to clean up whatever the setup did we didn't need one for the previous example because it was just creating in memory objects but if you were creating files on disk you could clean them up in the teardown method so setup and teardown also give us a great way to create isolated tests they establish context they are placed for common pre and post work to live and they give you isolation even with failures and as a pointer off to something else to look at if you find yourself creating lots and lots of big common data you want to look into a thing called fixtures which is a term of art and testing but it's also the name of a Python library again tests are real code you're going to have helper functions and classes they can become significant and by the way you're going to write tests for your test helpers because you want to know if your test helpers are working so that you can trust that your tests are working and you should not resent that that is a good sign that your project has grown to a level of maturity and a level of testing that you are a really good software engineer so pat yourself on the back all right let's talk for a bit about mocks as I mentioned before it's really important for your tests to test small amounts of code the less code each test tests the more narrowly focused your debugging task will be when that test fails but we always build systems so that components depend on each other right we're all used to building these towers of abstraction so that at the top level you've got all these things you're depending on right how can you test just one component in your code dependencies in your code are good because you're building on the work of others but dependencies and tests are bad because it puts more suspect code in each test in addition some of those components might be slow and remember we want our tests to be fast right the problem with testing missile-launching sites is that if your test really launches a missile that takes a long time you want your tests to go fast not only that but your tests might be unpredictable right if you have a flaky component that isn't really the focus of your test you want to get it out of the way so that you can test the code that you do want to test this is the wake systems are often built with a code under test as depending on all these modules which you want to do is you want to replace those little compose big components with little pieces so that you can reduce the amount of code in each test let's add a little bit more to our portfolio stock portfolios are only interesting because stock prices go up and down so let's add some code into our portfolio that's going to actually go out and talk to financial comm to get the actual price right now of our stocks so we can find out what the current value of our stocks is so this value method is very cool we can actually buy some stock and we can actually get the current prices for IBM and HP stock and write our current value is now think for a minute about how we're going to test this code we cannot write an assertion that says I assert that IBM is trading at $100 right now because we don't know what IBM is trading at that's the whole point of this code is to get information that we don't have how do we write a test the tests that this is working properly right it's live data it's unpredictable that's its entire point it's slow right it's actually pretty fast to go to Finance Yahoo comm and get a quote but it's probably slower than you'd want it to be maybe it's unavailable maybe Yahoo goes down right there trying to fix the heartbleed bug you can't run your test that day that's no good so the question should be for your code assuming Yahoo is working does my code work and the challenge now is how do we get Yahoo out of the picture right how do we remove that dependency on that server so that we can just be talking about the piece of code we actually care about we don't want to test all of Yahoo comm we want to test our code one way to do is to have a fake implementation of current prices so here is a test that we derive from test cases always and I'm going to have a method in here called fake current prices now if you remember from our product code current prices simply returns a dictionary with the keys or the names of stocks and the values of the prices and this is going to be a fake implementation of it that just returns some cans data and in my setup I'm going to make a portfolio I'm going to buy some stocks and then I'm going to actually monkey patch the portfolio to change its current prices method into my function and this works so this is a little odd and if you haven't dug really deep into Python internals you may be surprised to see that you can just replace methods like this on objects but it works and now when we run our test and we look at self dot P dot value it will actually call self dot P dot current prices which is our method fake current prices which will return that fixed dictionary and we haven't made any Network calls and we can assert that with the value is what we know it's going to be because we know what we what lies we told old about the prep current price of IBM and HP stock so good the test results are predictable the bad thing is that there's some code that isn't tested so here I've used the coverage tool which is a way of measuring what lines of code get executed when you run your tests it's a way of testing your tests the theory of your tests is that they're testing all the code you have how do you know you run coverage it tells you which lines of code you haven't actually run under your tests and here we can see the lines 53 53 through 56 have not been executed and 53 through 56 are exactly the lines of code we tried to snip out of the system which were the lines of code that talked to yahoo.com but this is code we wrote we need to test this code somehow right so we've got a good system here but it doesn't test all our code so we can try faking URL Lib you are L open instead so here what we do is we make a new object that's going to be a stand-in for your lib for us and it's got a method on it called URL open and in our setup we're going to save away we're going to we're going to jump right into the portfolio 3 module which is our product code and it has a reference to URL Lib we're going to save that away in our setup we're going to make an instance of our fake URL Lib and monkey patch it into our product module and now in our product module URL Lib refers to our class and therefore URL Lib that URL open refers to our method and when it gets called it's going to return the data that we want up here this URL Lib is returning a file like object which returns exactly the CSV data that Yahoo would have returned I know the details here don't worry about the details about the CSV and all that stuff there's no way to keep up with me right now just keep in mind the idea here is to make an object that works exactly like the component you don't want in your system any more and put it in place so that the product code will actually call your code instead of that code and by the time we get down here and we call self dot P dot value value calls our current prices module function our current prices function calls URL Lib dot URL open which it thinks is a standard library module it talks over the network but is in fact our fake object that we put in place it gets back that file object it parses the CSV it makes the dictionary it computes the value and then we get what we want and by the way at the end the teardown method is going to restore the real URL Lib it's going to unmark a patch things now all of our code is executed right we've got 100% test coverage all of our code is executed the standard lib is stubbed out and we've made no web access during our tests so this is really good the thing that's not so great is we've had to do a lot of sort of handcrafting of engineering there there's an even better way to do this which is a thing called mock objects mock objects are these amazing things that are I call them automatic chameleons they will act like anything you want them to act like here's an example where I from mock I import the mock class and they make a thing called funk which is a mock and I do all I say about it is the return value should be the string hello and then I call it as if it were a function and it returns hello right it did it just said I'll return that sure you told me to return that that's fine and by the way the 17 into something it just stored them away and later I can look at that and say funk what were you called with and it says oh I got called with 17 in something so this is great this is perfect for putting in place as a mock as a fake object and it's really easy to do because the mock library also has a patch decorate a patch context manager so here what we can do is we can say to the mock library I want you to mock out you are a live that URL open and give me the mock object that you made and then down here what we say is I'm going to make URL open have a return value of this fake Yahoo data and now when I call self dot P dot value it goes through all of our code it gets the URL open which hits the mock which gets the return value that we gave it and it gets that back that data and it comes back up and gives me the value on to now by the way down here I can say to you are all open I assert that you were called with these values because by the way in our previous example we didn't actually look at what our fake object got called with maybe the URL got broken in some refactoring and the actual code wouldn't would have failed where our mock would have succeeded here we can actually assert that we're getting called with the URL we thought we were being called with so now we've got mocking with no explicit set up we didn't have to write an implementation of something we simply had to make one out of this magic mock module and tell it what we wanted it to do the more the more comp of the the overarching term for all these things is test doubles which is kind of like stunt doubles in a movie if you take the star out and you put in some guy that you don't care about as much and let him do the hard work so this is very powerful it isolates our code right we've removed dependencies and focused in more and on smaller code it focuses our tests and it removes speed bumps and randomness a lot of times you'll take this do this approach not because it's not your code and you want to get it out but just because that code is slow the problem is it makes your tests fragile if you think about what we did the only reason our mocking worked is because we knew that the product code was calling URL Lib dot URL open if someone reef actors that product code to use the requests library instead these tests will all fail and by the way the product code will now be running out to yahoo.com to take a long time before it fails so we'll have lost all of the benefits so the only reason we could mock this stuff is because we knew enough about the implementation that we knew exactly how it worked inside and that's a dangerous situation to be in also by the way there's another way to do this called dependency injection which you should look into so there are too many things we can't talk up at them all testing is a big topic but I wanted to give you pointers and plant some words in your head about some other things add cleanup is a method on unit tests test case objects which is a nicer way to do things than teardown doc test is another thing you'll hear about in the standard library only use it for testing code that's already in your Docs if I hear about any one of you using doc tests or anything else I'm going to be mad because we've spent a lot of time here together and I expect you to listen to me knows and pee tests are better test runners that unit tests they have a lot of cool features like plugins DDT is a library lets you do data-driven tests so if you have one test method but you want to have 10 test cases out of it because of these different data it will do that for you coverage is a really awesome thing and you should use it selenium is the tool you'd use for in browser testing Jenkins and Travis are continuous integration servers which are way of having a place that you can push your code and it just runs your tests every time you make a change other topics TDD is writing tests before code which is very interesting BDD is a way of writing tests in a way that your business people can perhaps write your tests and focus on external behavior integration tests get at larger chunks of code which is a way of testing more realistic scenarios in your product load testing test about there's how much whether you have enough I see for your traffic and lots of other topics I'm sure so summing up how do we do we've learned that testing is complicated it's important but it's worthy and it's rewarding remember the two developers we had at the beginning of the talks we wanted them to be happy and confident here they are if you think testing is hard these drawings the two original drawings were drawn by my 16 year old son if you think testing is hard try getting a 16 year old to draw you a third picture any questions how much time do we have questions about five minutes thank you for your talk what happens if finance EIU returns JSON instead of a CSV if Yahoo returns a gzip J'son instead of CSV oh well then that API has changed and your code is going to have to change okay so our days are going to pass but the applications are going to fail right exactly so that's one of the failures of the mock right is that we're assuming that we understand the interface to that mocked component well enough that we can replace it if that maths component changes in this case our product would also change so we're going to have to fix it either way but yeah there could be scenarios where it changes in a way that your product could would have been okay with but your test is now mimicking the old behavior and you've lost track of that that's one of the ways that mocks are really dangerous yeah thanks for the talk one of the things that I've run into as a barrier to get in is just into testing is getting all the pieces together and in the right place so a little bit blatant self-promotion I've made a github repo that has a Python unit test skeleton with dashes between them you can search for it that has a bunch of kind of boilerplate code you can take two to get up to speed much more quickly great so I find that super helpful to make the burden of getting into testing as small as possible fabulous Thanks Python unit test skeleton great all right so when I'm aiming for 100% coverage I often have trouble with the main method you have any advice on testing that make your main well first of all in the if dunder name equals dunder main yeah you should have one line of code in there which should be sistah exit of main of sistar V and then your main function is now a pure function that takes a list of arguments and returns a status code and you can test that right that just takes arguments and it takes a return value and then all of your connection to the external world happens in that one line and your dunder named under main and it's hard to mess up that line so you can just leave that alone that's my advice oh thanks two questions I've heard it said that dry doesn't apply to testing but when I see my co-worker just cut and paste one test to the next and I go back and something changes with the test it feels like it should be dry that's silly yeah well I have no idea why people would say that DRI shouldn't apply to testing i testing is one of those mysterious worlds where you will find experts and they will they will say things to you not me I can be trusted but there'll be other ones who will say these mysterious things and you will believe them because testing is mysterious but maybe they're wrong well I mean in one case where it makes sense to me is that once you start making your tests dry your tests become more complex and then don't you have to test your tests yeah you do right but testing all the way down yeah so I know why some temp of why they say they should be dry because some people like tests as sort of almost documentation of how things should work and they like a test case to be completely self-contained but if that means that you have the same ten lines setting up the same fixture that you're going to use in sixty tests that just seems silly to me okay and in the second part I still have problems with my reusable test data so I put some of it in the setup I put some of it in the individual tests and a little bit in fixtures which I see a lot of bad talk about fixtures and I experience that sometimes so any advice on where to put your reusable test data it's really going to depend I mean like I said testing is real engineering right if if I were talking up here talking about web performance people wouldn't say to me so what should I do to make my website fast right everyone knows that making websites fast is like serious work where you have to really consider all these complicated factors testing is exactly the same I don't know what your test data is or where you know I can't answer that question we're gonna have to sit down and really think about what problems you're facing and what the best solutions are in a well tested system it's not unusual to have more lines of tests than you have of product and why wouldn't that code be rich and complex and have thorny problems that you're not sure you solve the right way and you're going to go back and refactor etc etc it's engineering okay so JSON fixtures that's always gotta be there yeah it sounds great hi I have a pretty essential question you said that it's a good idea to reduce some of the modules that you're testing like if you've got eight that are all feeding into one test do you just mean to just stow the code somewhere else briefly and throw a pass in there so it'll just no go through a turn okay and the reason why is because that makes your tests hard to run right you don't want to have to edit your product code before you can run your tests you need to have an automated test that you can run it with the push of a button some people have it happen automatically when they save a file in their editor every time they there control s twitch comes in like the test is all run like what like Jenkins or Travis let you even know even more than that every time literally the file hits the disc their tests run so you want your tests to go as quickly as possible and as automatically as possible so when I say removing code I mean engineer the tests so that it reconfigures the system and finds just the code that it wants to test thank you but the test the session runner went away so we're going to be here all afternoon this was a very good talk thank you very much umm I saw a recording of a talk you gave a few years ago also on testing yes brought in the issue of testing the missile silo yeah what have you learned in the last few years about testing what have I learned I have learned if anything I've learned that it is even more complicated than I thought it was before that as your system gets complex your tests the tests seem to get complex faster than the system gets complex and that the more work you can put into attacking that problem early the better off you're going to be and I've learned about the downsides of mocking thank you sure okay I'm going to thanks like a great talk and thank you for the permission to refactor test to test tests yeah gotta try that as soon as possible yes I'm so uh my question is I've been starting to do unit testing and stuff and I've been and suddenly I've been forced into dependency injection land mm-hmm and so I wonder if you had any thoughts on mix-ins since we're in Python land here and we can have multiple inheritance versus dependency injection and what if you have any you know global thoughts on that whole concept well so dependency injection for for people who have an encounter the term means that the things you depend on like URL Lib that URL open that we used in the product code instead of it being an explicit reference to a module that we would keep patch by name it would be a thing passed in explicitly into the portfolio class on construction so the portfolio class when it was constructed would know when I need to talk to the network I have been given this thing and it will it's the thing I will use to talk to the network and that means that when you do your tests instead of doing weird monkey patching you can just pass in a different thing and you default it to URL URL open so that your real callers don't have to think about it the downside of dependency injection is that you're cluttering up the signatures of your methods with all sorts of weird stuff that starts with an underscore and the doc says don't we ever use this and weird stuff like that that's unfortunate and by the way it's hard to even get all of them and in a big system you don't want a hundred dependency injections makes sense I'm not sure how you'd use mix-ins for that because mix-ins are things that give you give you flexibility at the time you define a class but that means you'd have to define a different product class to get a mixin that has different dependencies so I guess we'd have to talk about how a mixin could solve that and maybe you can teach me something about it because I haven't encountered that as a way to solve the the dependency problem for during tests well for instance you you know you have your your hierarchy of the thing you're doing and then you just mix in URL live with it I see but and so but how do you get it out of it that's the thing that thing that you've derived from both your product code and URL open your a Lib now is the thing that you need to test or are you deriving a different thing that mixes in something else in which case you're creating new you're grading another right right so that's another way to do it right you could define a new class in your test which is your product code mixed in with the dependencies yes that sounds cool see you learn something that you do Mike on tox there's no one over there go alright it's mean it oh hi it's the runner yeah so we're out of time but I'd like to thank you thank you for a great talk and thank you to all our questions you

Channel: PyCon 2014

Views: 51,303

Rating: 4.9681592 out of 5

Keywords: PyCon2014

Id: FxSsnHeWQBY

Length: 42min 44sec (2564 seconds)

Published: Wed Apr 23 2014