Automated testing with pytest and fixtures

so I'm Steve hello we're gonna talk about automated testing with PI tests and it should be a lot of fun so why do we test well obviously to find bugs but also for some more positive reasons so when we're adding new features to our software we want to have confidence that our new features are working as intended so we can write tests for them and we especially want to have confidence that our existing features have not gotten broken so hopefully we have a suite of regression tests that will assure us of that and also if we want to do some refactoring meaning take the same take code that does something and write it in a better way to do the same thing if we have good tests for that code we will hopefully know that our refactoring didn't mess anything up now a lot of people do manual testing there's definitely a place for that but why would you want to automate your tests well big reason is it's pretty boring to run the same manual tests over and over I'd rather let the Machine do that but there's some more other important reasons one of those would be repeatability so I want to know I'm running the same tests in the same way on the same platforms time after time human beings just aren't good at that kind of consistency and if you're doing any kind of continuous deployment or continuous delivery you will almost certainly want automated testing to be one of the steps that sits somewhere in between making a change to the code and deploying that code to your production servers there are lots of ways you can automate your tests there are all kinds of tests automation Suites out there why PI tests i really like pi tests first of all because i'm using python it's optimized for use with python it's also really easy to get started with and i'm going to show you guys real-world examples in a minute of the very few things that we need to install in order to get started with pi test and if you have a whole bunch of tests it makes it easy just to run which tests you want to and if you don't want it to be easy if you're the kind of person who enjoys complicated features a lot of flexibility it does have a lot of capabilities very flexible so louder yeah okay so yeah I'm a little closer to the mic thank you for that and as far as what you need to install pipe tests it's really pretty basic we should be able to take this command so pip install - cap you PI test that's going to install PI test if we don't already have it and if we do have it it's going to upgrade us to the latest version I'm in luck I already have the latest version very possibly if you have Python installed you already have PI test installed it may have come with it by the way the examples I'm giving today if you see me type anything at the command line that is for Ubuntu and PI test is available for other flavors of Linux it's available for Windows some of the syntax may vary just a little bit so a really cool thing the PI test does for you is test discovery and this means if you name your test files and test functions properly PI test will run them will find them and run them for you and I'm going to show a couple of real-world examples but the concept to keep in mind when you're looking at those is if you name your module test underscore something where something underscore test dot pi then PI tests will find it and in those modules if you named your functions test underscore something PI test will find them and if you want to put your test functions inside classes you can as long as those begin the class names are going with a capital T test and that's really it well it's not really yet it's very configurable you can make test discovery work lots of different ways but if you want to just follow those two basic rules it will get you very far in your testing life so some real-world examples let's open up this repo so this is our cat spin car this is our main repo it's a suite of different applications it's got lots of tests in it and we have elected to have a whole directory just full of tests and we've got lots of subject to reason there and in each of those sub directories we've got some test files and as promised their names all start with tests let me see if I can scroll this for you and inside those test files we have functions whose names all begin with test underscore so just occurring to the naming convention I mentioned and that just works I can run PI tests and all those tests will just happen and here's maybe a little if you couldn't see that if it was too small here's a kind of zoomed in view of what I was just showing so module named test manager won some functions name test homepage test vehicle and so on that's all there is to that so test discovery is really nice it will look inside subdirectories for you if I want to run my tests on just one module I can do that and I have here some example tests that maybe are going to be a little easier to work with and the full-fledged repo so we have here PI Gotham tests and this repo is on github I've got a link to it in the slides I will share the slides with anybody who wants if you want to just drop me a business card with your email I'm happy to send the slides out to anybody and so first of all if we're going to run tests we need something to test and I have a really simple program here that just has a couple of functions that test that determine whether a number is prime and returned its prime factors if it is so I've written some tests for these functions and for example some of those are in this module test primes and we can see what a test looks like sorry test Prime's the open that forever that one okay so we can see we have some test functions and they're not doing very much they're printing something so you can see that they're running and they're doing some assert statements but those are real tests and we could run these and if I simply say PI test and the name of one of my test files the tests run those five dots each represent one passed test if I had a failed test of it a print of the letter F four fail and it would have printed a little stack trace that showed me why it failed so you can also run a subset of all your tests by name so it will find your test functions and these are based on regular expressions so if I do something like if I use the - K parameter that says the next thing that follows - K is a regular expression and we should run any tests whose names match that so I could do something like that and it found all the tests matching that and it ran them and hey I'm gonna lock all three passed again so so far we've been running one test at a time but you can actually run tests in parallel why would you want to do that because it's faster and to run tests in parallel we need to install one additional thing which is called X dist and we're just going to pip install - cap you it's called PI test - X test and again this might have come with your installation of Python but if not that's all you need to do to install now that that's installed I can use PI test with the - n command line parameter and I can set n to some number like three or five it will run that many tests in parallel or I can set - n auto and PI test will automatically determine how many processors my machine has and run that many tests so we can try this I just go PI test - an auto we can see it's spun up for processes and it's running my tests it's running them really fast and it looks like a bunch of them passed and two of them failed that's pretty much all there is to get our tests to run in parallel but there are some things you have to be careful of when you do it first of all how many tests is too many to run in parallel well there's no one-size-fits-all answer to that you probably won't get much benefit from running more tests at a time than you have processors and even that might be too many like if you're launching browsers if you're doing browser-based testing with something like selenium I wouldn't try to open 12 copies of Chrome probably crippled most machines I found like three or four is a pretty good number for that but just experiment see what works for you couple funny things about or one funny thing about X dist is you don't get output to standard out and I actually have to take a little digression backward to show you what I mean here so I think I have a test yeah so this test prints the statement this test this is test crime so let's run test prime spy okay didn't print that why not because I didn't give pie test the - s-parameter that's what tells it I want to see output - standard out and there I don't know if you could see on the screen that says this is test Prime now let me do that again it's got a couple tests in it I want to run them in parallel will do - and - what happened to my output it's gone that's a quirk of X dist it does not print to standard out if you want to print when you're using X dist you can and the trick I'm gonna try to do this in real time for you guys whoops not going very well so far we're gonna print to standard error so we'll just print to sis dot standard error try it again and did it work did I get my output I can't spell it on the screen but I hope it worked anyway if it didn't good so yeah that's that's just a next dist quirk to be aware of more important thing to be aware of when you're running your tests in parallel is design your tests so they can run in parallel it's really easy to do dumb things that make your tests interfere with each other and I can show you an example of that so this is just pseudocode but imagine you have two tests and each test creates a test user does something with the test user and then cleans up by deleting all test users woops sorry about that cleans up by deleting all test users you don't really know what order these are going to these statements are going to execute in when you run tests in parallel so maybe sorry guys maybe this test runs its first statement and creates a test user maybe then all three statements of the second test run create a test user do something with it delete all test users what happens when we get to the second statement and our first test our test user is gone the test unexpectedly fails trivial example in the real world it's hard to design tests to run in parallel you can do smart things with fixtures and mocks and so on to try to make that better just be aware of it speaking of fixtures so fixtures are maybe the coolest thing about PI test show of hands who knows what a fixture is well a lot of people good so maybe like half the audience or more so maybe you guys even know more about this than me I'm kind of new to them but I found them to be really powerful fixtures a weird name I don't think it communicates exactly what they are fixtures are functions and they're functions that PI test calls for you and they can do useful work and/or return useful objects so what's an example of useful work maybe before any of my tests run I need to make sure that my database is cleared out of leftover test records where maybe I need to make sure a particular directory exists on the hard disk what's an example of a useful object maybe I need to make a test user and make that available to my tests so a fixture is a function that does anything along those lines the cool thing about fixtures is PI test runs them for me and they have three different scopes I can take advantage of by default a fixture has test scope and that means that anytime any of my tests refers to a fixture function the function gets called so once for each test that runs but a fixture can also have module scope so a module is just a dot py file I could have 50 tests in there that all refer to the same fixture that fixture function will run once so going back to my example of creating a test user if I need a fresh new test user for every test I'm going to let my user fixture have test scope if I can get away with just one user to be shared across all my tests so I want one user shared across all my tests I'm going to give a test scope I can even give my fixture session scope which means I could have lots of different Python files that is lots of different modules and my fixture function will only execute once for the first test encountered across all those modules it needs it so let's look at a little bit of code and then maybe I'll come back and talk more about this slide so this decorator at the top at PI test fixture is something we import from PI tests and that's what tells us our function is a fixture I made a fixture called user and it's job is just to create some test user and this particular test user has user name Python and password rocks and then look at my test functions they take user as an argument so that's PI tests naming convention if I have a function decorated as a fixture then I take that function name and pass it as an argument to my test function when PI test executes my test function it will first execute the fixture function according to those rules of scope I mentioned before it tests the scope or module scope or session scope so here my fixture has the default test scope when these two tests run we should see this user function called twice and this test user is going to be available to these two test functions and maybe you can see it's actually using that user and a call to his prime so let's see if we can actually make this happen which module is this I think it's this one let's try okay this worked it printed creating user twice and printed that because I remembered to say dash S printed it twice because the fixture has test scope but if I go into that code and modify the scope of the fixture so we just say this is an argument now to the decorator function scope equals module that goes in quotes save that run the same thing again that printed creating user once so now those two tests are sharing one test user whereas before they each got their own fresh user think we covered all of that a one one little note so I showed you an example of a test function that refers to a fixture but a fixture function can refer to a fixture so if you have some fixture that needs some other fixture if my test user needs a test database to be fired up before I create the testers I could have a fixture function that itself takes a fixture as an argument odd infinitum it can get really confusing but it can be really powerful kind of related to fixtures a semi related subject is parameterizing tests and the first question is why would you want to parameterize tests well what this means is you can get the same test to run multiple times by passing at parameters and a classic example of this if you're doing browser-based testing you might want to run your test on a lot of different browsers well simpler example that I might try to show if we have time if you're testing a function that sees if numbers are prime you might want to pass it a bunch of different prime numbers to test little note about spelling I'm pretty sure that's how you spell parameterizing with those two e's but the PI test imports spell at parametrize TR ize so if it says you can't find your import that might be why or maybe I'm spelling it wrong so that's why you would parameterize how do you parameterize tests one really cool way to do it is with a parameterised fixture function so here in the decorator you can pass a parameter called params and that gets set to a list and that list can be anything you want Mayon has two elements it could have a hundred elements or three or whatever number you want the tests that use this fixture will run that many times as many times as you have elements in your list those list elements can be anything might happen to be tuples and my tuple happens to be a password and then whether that's a valid password or not so rocks it's true that's a valid password bogus it's false that's an invalid password when I return this user object to a test like this tests off this test is going to run twice let me see if I can make that happen for you so this is in we're just going to try to run yeah so if we do - Kay test off it should run that test function and you see two dots I ran one test but I got to passing tests because it ran the test for me twice as did I forget to say - yes and I remembered so why didn't it print more stuff oh it did printed creating user and that's all that these tests all right if we have time at the end I can talk more about parameterization there's some more cool stuff and in case we run out of time first I want to do configuration files so configuration files are totally optional PI tests will work if you don't tell it anything but you can have a PI test any file and you can put directives in there that mostly do the same things that you can do on the command line so PI test takes a lot of command-line arguments if you get tired of typing those in you can have a PI test any file the other thing that PI testing is involved in is determining the location of what is called your tests route Darer and that sounds simple and it's kind of simple it's basically PI test starts looking from here down to find your tests and test discovery it has some other implications it's actually a little bit complicated and I put a link to the documentation and again I'll share these slides with anybody who wants them but that's the basic concept and just know that if you're like deep in a subdirectory and some of the tests you're thinking are supposed to run aren't running or a fixture function you think PI test should discover didn't get found it might be because PI test in he is confusing PI test as to where your test route is the other kind of config file you can have is called contest PI and you can have 0 or more of these and they're really powerful because you can put them in the root directory of your tests and you can have different ones in subdirectories and PI test finds these files when it finds a contest PI file it adds the directory where that file is to sis dot path so contest is on the path and the tests in that folder are on the path and it loads that and runs the code in it for you so is a wonderful place to define fixture functions or any other code that some of your tests need like the ones in a particular directory or all of your tests need put that up in the top-level directory of your tests so cough desk spy files are really cool for that and I think I have a minute to prove to you that PI test really imports this for you and you don't have to do it explicitly so here's a contest oh I just closed it by mistake sorry here's a contest PI file that defines a fixture called func in contest and here is a test that uses that fixture I hope it's in this file it's not in this file I'm sorry it's in I'm not gonna waste your guys time finding the file I have a file that has that test in it and does not import contest pi and if we run it like this it printed this function doesn't contest pi and you'll have to take my word for it if we have time at the end I'll find the code I didn't import that pi tested it for me okay I understand we have about probably two minutes left so at this point I want to take questions and if we run out of questions I've got other stuff I can talk about but I definitely want to ask your questions so anybody nobody I explained everything about testing okay cool let me let me cover some extra topics so I talked about parameterizing tests using a parameterised fixture function but you can also parameterize a test function directly and there it is spelled without the e at PI test mark parameterize so here I have a parameter I got a subdue named N and I have a list of three parameters to and five and what this is going to do is run my is prime function three times and you really have your choice you can parameterize your test functions you can parameterize your fixtures you can do both they both work Oh another really nice thing about PI test is PI test raises so let's say you want to test whether your code raises the expected exception when it's supposed to and an example I have here is some of my test fun some of the functions I'm testing are supposed to authenticate the user so we might want to pass it a user with an invalid password and verify that in this case my code is supposed to raise a runtime error so you can do that in very few lines of code with PI tests using something called PI test raises and as PI test raises the argument it takes is the name of an exception class so if you don't care what exception you could just say exception but best practice is to check for exactly the exception you care about and this code is equivalent to the more verbose code below so it's like calling your function in a try block catching the exception you hope it will raise passing if you catch that exception I don't mean the test passes I mean the Python keyword pass and if you get passed call on your function and an exception wasn't raised then you have to raise an exception so much more readable syntax using PI test raises okay now does anybody have any questions yes and back i Sark you say louder okay so yeah question is how am i running parallel browser tests and the parallelism is coming just from PI tests and X dist and my favorite tool for browser testing is selenium I use that really extensively and it marries very well with PI tests Yeah right in front yeah the question is higher yes okay so the question is our fixtures passed into your test as an Arg and the answer is yes that's exactly right you're not obligated to pass you're not obligated to use that argument but the presence of the fixture named as the argument to the test function is what says this test uses this fixture and that's what tells PI tests to execute the fixture and the second part of your question is what if you mix parametrized tests and parameterised fixture functions and the answer I think I've never tried it is you'll get all the parameterizations so let's say you've got three parameter i3 parameters on your tests and to an ax fixture function and all together they have 20 values your tests are gonna run like 2 to the 20th times so watch out for that could be really cool it could be really bad anybody else yeah question is what are good resources for learning this stuff and there is PI test documentation I believe they have a tutorial I don't remember if I put a link in here I did doc stop PI test org and also I'm not going to say it's a great resource but all the code that I showed add more is in this github github repo on the second link yeah question is what is it using under the hood to do the multiprocessing and the answer is I don't know what X dist is doing under the hood I'll have to read the documentation on that anyone else more questions do we have any time for one more question I think we're good thank you everybody so much for coming [Applause]

Channel: PyGotham 2017

Views: 23,315

Keywords:

Id: 8mp_1Jt-xHQ

Length: 27min 16sec (1636 seconds)

Published: Sat Oct 21 2017