Testing with Pytest for Data Science - Ravin Kumar

if you never seen pie test or unit testing you'll find this the idea for this talk is to get you over the hump where you don't know what it's about and you don't know how to run it to a point where you feel comfortable enough with it to move on with more advanced tutorials that are going on so I will and so I don't want you at any one time if you're looking for a super advanced PI test tutorial this is this is not it this is getting getting started and getting familiar with unit testing all right so let's do the traditional about the speaker and all that other stuff so about me I work at SpaceX I taught a bootcamp here for six months so these two awesome girls were in my class philately little Linda lots if you think I'm a terrible speaker you go complain to them they had to deal with this for six months so they'll they'll be commiserate is with you I contribute a lot to RVs that's gonna become relevant later with testing and we're happy that y'all came to PI data so I'm one of the committee members from PI data so thank you all for coming to this conference the first year that I did is this it it's fantastic that you found it and you took the time on a Sunday to come out and I super encourage everyone to come back on Monday and Tuesday as well when we have our keynote speakers and the rest of the talks we have a social event tomorrow we have lightning talk so if you didn't get a chance to submit something and you still want to come and talk with that that's available as well I want to give a shout out Nathan Danielson is in the back he is he's the primary chair one of the three and he spent a lot of we weeks and weekends getting this going as well so go shake his hand he's a great guy we just get hold of me you have my my LinkedIn and all that and then disclaimer everything is my own opinion I don't represent anyone else but myself in this talk so we'll get that out of the way the other thing I want to say is you guys will be doing the majority of this tutorial I well I'm gonna pause for like 30 or 40 minutes a cumulative over the the time this talk we're year's gonna be coding so you know otherwise we'll just be staring at each other awkwardly for 10 minutes if you don't clone this repo and install the dependency so super encourage everyone to to get a copy of this repo PI test for data science there's actual activities we're going to go through and so prevent it from being boring let's do that I'm so the I would clone this and I would install the dependencies some of you probably have them installed they're pandas high-test and requests pretty small ones but do a pip - are so let me let me jump in the next section tutorial format so why am i doing it this way why do I sort of believe in this method so git clone install requirements how this will go I'm gonna just conceptually talk through PI test a little bit so it'll be a little abstract I'll wave my hands and then we're gonna just dive right into concrete stuff I'll show you an example and then you guys are gonna try it out this is super similar to the bootcamp format which I found super effective I want everyone to find a partner in this and I see people sitting alone and I'm not going to have that because this is a conference and we want you guys to network and meet other people as diversity coach here I want people to meet other people of different backgrounds but there's also there's also real reason to this well this is not just the like let's be buddies type of thing it turns out when you're when you pair up with somebody yeah I'll give you say yeah go ahead and review cig Risa so the reason I'm making you awkwardly introduce yourself is well one you're at a conference so I make you know you signed up for this a little bit but it actually makes you learn if there's loma linda has a has a effective ways of learning and and we're gonna go through all of them we're gonna learn multiple ways I'm going to I'm going to show you I want to talk about it I'm going to show you in code and you're gonna do it yourself we're gonna build examples from each other and the biggest one why I want you guys to sit next to each other is if you teach somebody else so I'm gonna be teaching you but you're gonna learn more if you then work with your partner and you guys both teach each other I think the knowledge and the retention will be there so my goal is not to just give you a bunch of information my goal is to have you guys actually be able to use PI tests by the end of it so you're not gazing I feel like a lot of tutorial sometimes you're dazed and confused at the end I don't want that to happen I want you to feel secure in your knowledge about what's going on so let's do Y testing and we'll talk about that so why do we bother with testing things at all so I do data I see type stuff one of the data science things that I do is I go to dinner with my girlfriend or whatever and then I want to split the bill up it's a data science problem I've been in foot I'm also kind of cheap and don't wanna keep paying for hair so I want to cut it in half and I wrote a function to do that because you know we have computers why do it do I do it myself so the function is really simple and I'm gonna use VI a lot for because vim is just a text editor that's happens to be the console so this is my function I have a Python function split bill it takes input cost take the cost I divided by two and then I'm just going to print the cost so I know how much to you how much this bill is and then that's about it so it should be pretty simple I take this one Python bill split two point five dollars mean surprises he was amazing nope crazy nothing crazy will do this anything over here two dollars it's the same function does anyone want to guess why what happened exactly the same huh yes yeah but I'm running Python right yeah so it's the same command Python dual split yeah weird huh a little different yes somebody got it python 2 versus python 3 so when I flip side flip the screens python this ones 2.7 and this one is 3 and I know this like where it means like 2019 everyone should get on board Python 3 what did someone crazy but it's there's differences on it with code right you write code on your computer and it works and you're like cool I wrote this awesome function let me just give it to my coworker or let me we're done focus all but open source let me open source this thing suddenly somebody else is running your code and nothing works I guess does anyone agree that division is the thing we do in data science seems like common yeah so you would want your division to be correct and if I wrote my library in Python 3 and I my bill split function works correctly if I give it to somebody else there that's everything's gonna be off like you're known that so can we officer division they're your classifier is gonna be off your mat plot Lib bars are gonna be like everything is just wrong so the point of testing is that you're supposed to write down what you expect and let's let's go through our first test so oh by the way questions who who answered you answering yeah okay you get your chance your Kit Kat Reese's Hershey's Hershey's they got almond and you got normal only yeah well who got 2.73 you got something all right you guys are you guys also gonna find out that I'm a nerd and don't play sports so I hope look hit someone in the face yeah the facts that I'll hit you with a candy bar but I were welcome to add questions they're super encouraged I will gladly throw sugar your way yes where did they live okay good point so me I mean particularly anaconda to split my environments up so I I used my condo package manager and I have in this in this computer I have six versions of the Python from anaconda then there's two from from the Mac by the way and if you go to user bin Python this is a little bit off the Taco Bell go with it this is the this is the version of Python that's built into your Mac and there's actually two there's Python 3 and there's Python that's Python 2 that's in Mac OS this is I guess I don't have a Python 3 in this one so I have at least seven on this computer this is a little bit of site will go with it it's with Python it's important to split your environments up so there's there's virtually envy is one way of doing this another way of doing this is Conda that's my favorite so I used anaconda versioning and then you have docker containers are really popular so there's there's actually doctor talk happening right now doctor provides a layer of isolation and then the Hat most heavyweight one is VM but that's how I step how I split my Python environment at work I have like 26 of these because I have 26 different projects that I work on I hope that helps so this one was activated this one should have been the this is 3/5 yeah this one it's a 2 7 you can see the difference here 3 5 and then 2 7 and this one's the PI data environment and this is just my normal route environment it's cool all right you get something to you that's good question ok all right you wanted to me a favor you wanna help me pass like I tell people okay a great question yeah so it is a that's but the point is people have different versions of Python and the computer is like there's there's a lot of difference of Python but you say Python that's there's at least two seven and then within two seven there's minor and major Reds then you have 3 1 3 2 3 4 3 5 3 6 and minor versions between you have to be it you have to be sure that your code runs on on the ones that you support right so that's why we write tests so I'm going to show you that let's talk about why testing how much time do I have I should probably keep a lot of yeah okay alright so notes I put some notes in here too so you guys I have to memorize what I'm saying I might talk a little bit faster to get so python 2 versus python 3 let's talk about what is a bug wow if my computer wants to cooperate with me alright so what is a bug it's an error flaw or failure in a computer program one of my programmer had a bug in it it's not on python t7 you get the wrong answer so that's a bug in my program it is not a bug in Python because python was designed with that explicitly so python 2 does not have a bug that's the way python 2 works if you divide 5 by 2 you get you don't get 2.5 you get to language design choice it's documented the bug is in my program where I'm using it to do a thing and it's not doing the thing that I wanted to do so testing helps catch bugs so the reasons the motivation yeah testing helps catch bugs we're gonna see that in a second I'll show you the exactly the example we just saw but the other two benefits of testing is that they help users understand the program and they help with code development refactoring so these two will become a little bit more apparent as I show you some examples so let's jump into basic testing examples all right so let's do let's go back to my single bash pain or see you go for okay so in my test I have test bill split and what I'm doing here is this is this is the first structure for a PI test program the structure is you bring in the code that you want to test so in this case I want to test my bill split module you then write a function and the function as a just has to say test somewhere in it so in this case I have test underscore in the front and I'm testing Bill splitting it doesn't have to be the same I could make this test bill splitting whatever but the important part is that test is right here in the front I have a I have a doc string because you know doctrines are good and then what I'm doing is I'm I'm writing my function and then I'm writing an assert statement down here that says the split is equal to 2.5 that's what I want out of my program yes sir PI test does not require imports so this is a really good we'll go into we'll go into the different testing libraries in a second this is a PI test style test so they are deep I thought as a number of different testing style sometimes this is where Python fails that the one obvious way of doing things but there's other ones we'll talk about that in a second this is just this is just five tests so let's run let's run the test so the way I run the test as I run PI test test bill split WI and run it and I get my air and I get two equals 2.5 this is good because we're on Python 2 so if I run this in Python 3 high test test they'll split yeah well o source activate whoops source activate root this works this is good because one test when I run it in one environment I get a nice hair message that tells me I have a failure this test failed to is not equal to 2.5 my search fail in Python 3 you can see it right here actually it even prints it for you this is Python 2 this is Python 3 it runs the dot means that it passed and it's a little unintuitive and I'll make this a little more verbose if you like if you like to read and see that it's really passing you it's an English language you can put a minus V flag run - me and it'll it'll tell you explicitly that this passed this is the test this is the test module this is the test function and this passed so where this is a let me pull up the reading for this thing I'll show you how this works in practice so in practice you run a PI test by running it like this in open source this is actually how it runs we when I want to contribute to open source and either PI mc3 your rv's or whatever if you look at this graphic we're testing across two seven four two three one two three four five six seven pi 2.7 both in 32-bit and 64-bit versions and 3.6 and we have both 64 a 3.6 we're only testing in 64 so not only testing python version we're also testing the bit of the iPhone so 32-bit 64-bit versions of Python this is to make sure that that it works across every single supported version of Python PI test will run in the default so this is but this is a good argument to make sure don't have super messy environments isolate your environment so then there's only one version in each one so this is this is where I super messy environment but this is a good reason this is where super messy environments will hurt you the code will work on your computer and it will only work on your computer because there's some magical combination of dependencies that somehow conforms into being the thing that makes your code running and will never run on anyone elses computer and you can't it'll be hard to make around anyone elses computer so it is what this is a good maybe quick CI thing and let me actually pull up an RV seat I'll use Arby's because it's my it's the library that I contribute to I'll show you how this sort of works github.com go to Arby's so I'll be doing a lightning talk on Arby's later it's a library for doing Bayesian in Bayesian model criticism so if you had a Bayes theorem there's there's libraries out there to make Bayesian models it's a type of machine learning and then you have to evaluate the models and so these is a library that has a lot of plots and whatnot to help you evaluate the models we want to make sure that this runs over a lot of people's computers so what we do and I'm gonna go all the way into you actually won back all the way into my CI engine and in rvs in Travis CI every time we make a commit to master or commit in the and the make a push to the repo travesty I sees that and it runs tests across three five and thirty six RVs only supports three five and thirty six and we're actually running across two versions of stamp in here when you open this up and you look at the job log it's going through and it's running a hundred and no two hundred twenty unit tests and this is actually a small project if you look if you try to test pandas it's got thousands of tests so this is why we test we want to make sure RVs works not on our computers on all the people who want to use our visas computers that's the motivation behind testing so this is the end of the theoretical section does anyone have any questions or feel like testing is still a waste of time don't need it any quick any in this yeah okay so you're running that's a good point if you're running if using a docker image your your environment is the same right because docker isolates the environment for you but does doctor guarantee that your code is correct so that's why you still want to test so this is when people say about test-driven development well one one one point of testing is to you know that it runs across all environments in this case we don't need it with docker but the other point is you write code and your code can have a bug in it you don't know about it testing helps you catch those bugs there's another class by the way people this is a little more theoretical there's a whole notion that in software the moment where some people think that if you write tests you end up writing better code so if you write tests first and then you write your code your code is better than if you write code first and then test later this that's the arguments back and forth about test-driven development but that's probably the farthest reason I'd say the test the first two are makes you make sure it runs across all environments the second is that you know your code works for sure actually in the third that I think that is nice is that if you look at the if you look at the code we'll go back to and I'm flipping all over the place if you look at the code it actually makes it easy to tell what's supposed to happen so my test split bill code actually when you read this now if somebody can sort of tell what it's supposed to do because it's it's one example of it in action so testing helps new people understand your code base - all right so this is where I stop talking and I let you guys run your first test so I'm gonna admit all the solutions are in this solve folder you can do your you can cheat and look at it first but I love it you'll you'll probably learn less so my advice to you is try doing it yourself and banging your head against the wall for just a little bit and then if you're really stuck we'll go through this solution so what I want you guys to do is I want you to just to be consistent I'd want you guys to navigate to this directory oh whoops went back too far it is labeled activity so everything labeled activity is things you're going to do I want you to write a test to make sure this outer function works correctly I which it's a it's a five in here what I want you guys to do is write write a PI test file yourselves that verifies that this add function actually does addition addition correctly so this add function there's any I guess does anyone want me to demonstrate the add function it's easy enough okay if anyone has questions about about how to use the outer function let me know but I want you to write a I'm not gonna show you the answer but uh this is this is the test bill split function I want you to write one that's like test add and make sure that adding works works well and I'll give you guys like ten minutes to do it ah the font okay while you guys are doing the activity I will also change my terminal font size because it takes a little bit of effort but otherwise the code is all online if you get it on github it's everything is there so everything you see me doing is on the github repo alright I get her repo is um let me pull it up is this readable how about that that there we go the first the fun part is github calm and then this is the actual like part that matters I mean am I using PyCharm I don't know what I'm using oh now I'm using the regular camera I think I have to go and change it I will do that alright well well look at that perfect all right so ten minutes I'll give you two about two o'clock so good discussing trying to figure it out you got this Tiffany it's normal right you just way back like I forgot to do myself all right who asked questions are good guess on candy candy candy no you're not stupid all right so okay so oh you want to run the test okay no no okay so if you want to if you want to use outer okay that's fair I should have put an example you know that you have to do I just know that's okay VI work so I'll make one for you real quick I should have this was an oversight on my part in the next one ones will have it I don't know how to use okay VI has a really high learning curve so don't worry about it if name equals no no that's a good point you should know what it does to test it like that is the exact mentality you could have almost done print I would stick a Python no you what you have is like I shouldn't use that it was right there oh boy okay so now this will this will actually it's a two plus five is it yeah so what's happening is when you run this function it you get seven yeah and so it's we're like cool that's correct now just what you do is try and write there's a new file is they write a test that does it okay that's all yeah so fair point copying and pasting from the previous one is not cheating you can just copy the test from the other example and then just modify it to work that is totally fine writing code from scratch is a waste of time it's great to just copy code that exists and make it work for you yeah which is you've cannon I mean I okay so just try Python hmm oh well I mean I can't oh you need pie test oh did you saw it yeah you need pie oh so you guys will definitely have to install pie test it's not a built-in it's a it's a something you need to install so you need to you guys definitely need to pip install pie test for pie test to work and then for this lecture I go into pandas and requests as well which also need to be pip or content salted whatever it might be and I have about two more minutes running from the console I would run it from the console it's just running oh I see what's mine okay I'll explain what's going on here second all right okay yes okay see it's not Python actually that's a good point I see what's going on okay so we had the same question a bunch of times which I'm happy because you guys are trying but we're getting the same problem so for the purpose of this tutorial let's go through it and I'll show you I'll show you the common mistake that you guys are making and then we'll I'll show you how to work this so in my case I'll run the bill split one first I shall just run the ad I run the Beltsville try thing yeah I did this one okay so the bill split when you guys type in Python test bill split you get nothing right nothing excessive this is the first this is probably the first mistake where I got frustrated with PI test if you try running this this Python program under Python it doesn't do anything because Python doesn't see anything to do you define a function and nothing happened all right the trick is you have to run it through PI tests so you don't type in Python because then Python runs the file and Python does sees that there's nothing to do you have to run it through PI test which is actually the program that reads it in and does a bunch of stuff that's a really good question it's the first hurdle I have with PI test that and it confused me forever actually keeps me like a good three or four hours before like I finally stumbled upon it yes yeah that's not how high test runs that's an interest that the good point so tests don't take arguments because they're isolated so each test is self-contained and PI test has its own test discovery so this is good this is why this is why I want you guys to try things for us because we when you make mistakes and you feel really embarrassed and it like it's really painful and you remember high school and you don't want to live that experience over again now you're gonna remember how PI test works so there was a really good question when you run this like why despite the how this practice and know how to do anything all right and where are the arguments well one if you're really unit if you're quote unquote truly unit testing each each function is completely independent from every other function so we're gonna let me show you are B's or actually I'll show you I'll show this near the end but each usually test don't take arguments tests test a specific amount of code and then assertion and then you're out you don't actually provide anything there's in real code you do have to provide something there's ways to fake it out we're gonna get into ways to fake out functions later but you're not testing your whole library in one shot you're testing tiny sections of code and you're you're building up all it's like like testing a car or whatever you don't test it on the road you build these fixtures that hold it and then you test one component the way PI test works is PI test looks for every function or a class if you if you're familiar with the action unit style testing that has tested in then either the front of the back and if it has tests in the front of the back PI test recognizes it as a test and runs it so even though I don't directly call this function the way PI test is written it it it does what's called test discovery and gets the test I think I wrote so I tried putting a helpful reading ease in everywhere I could so let me go through this one the documentation for how this works is right here oh shoot it's not okay I will add that as a readme but really not sad didn't okay yeah so your first test run it with PI tests and I will I will add in the the link to PI test auto discovery but if you type in PI test test right here the first link will be how the documentation on the full documentation on how high test works you'll see that this documentation is really long this is why I made the tutorial otherwise we'll spend like a week trying to go through the documentation and pick out the relevant parts I just want I'm gonna get you to the point that works and you're gonna have this whole world the documentation to help you out yes ma'am so the question justice in the Mike is that when there is a hundred things that could go wrong with your code how do you know which ones detest so it's not fair I the answer is and this is this is where people get in a lot of arguments online the the answer I want to give you is what's reasonable like a reasonable amount of test for your code and I'll preface in two ways there's some people I think you have to test everything so there's this thing called hundred percent test coverage and then if anyone's ever heard this concept but there's some companies have this thing where every every line of your coat every code every line of your code has to be tested so and it's got to be at a hundred percent so I'll go back to RV's when we're testing code we're running we're running a test over portion of our code so I'm gonna go through Python because it'll be a little easier to show so when we're testing this outer function for instance in this case we're testing a hundred percent of our code but I could have another function in here that's like test def subtract and then I say num1 and then um - and then I don't test it I'll just say something I don't test it I just test one function and your question was I'm if I'm testing this some way but I'm not testing that another way you you should write enough tests to what you think is reasonable in in this case I provide a number input so it works if I provided string inputs it wouldn't work so then I ask myself two questions if this is an internally facing function two people that are from like familiar with Python and in the middle of the library I will I'm assuming they there I'm gonna say they're more familiar with how Python works and I can assume they'll use it correctly and I'll write just the test of functionality so maybe one test but if I'm writing test for something at the user interface level there's a lot of users online they don't know how my library works they can be at varying different skill levels so then I'll write a lot of tests to test everything they could possibly throw at it so like things that are public like it's your car right when you when you go to your car your car's an interface right then you got a nice steering wheel and everything like that and everything is good as soon as you dive one level in you take your dash up or you bring your hood it's suddenly a lot more confusing because the designers of your car where it's where it's uh it's meant for everybody it's nice you've got cup holders you got steering wheel you got your stereo that's all good as soon as you get one level of abstraction lower into your engine it's a lot more confusing and at that point they're assuming that you're a mechanic or a car expert and so I write a lot of tests for at the user interface level where I expect a lot of people to work and I write less tests as I go down into the code where I expect a more people that are more familiar with the code to be at that's officially into question I feel like good okay yes yeah that's fair but what I do with my bills for example what do I put the assertion there you can't because it's the it's the Python code itself that's running it poorly so there's not I can't write in a search for the for the value that I want to get so that's the reason for testing that's another reason that's a good question the other reason you don't have you don't have put assertions maybe in your code is it clutters up your cut so testing if you put off a billion assertions your code to test for everything in your code you'll have like 20 certs to make sure it's a string it's an integer it's above 1 if it's square root all this and then you have like one line to actually do this the addition if you write your test you can leave your test you can leave your function has to be simple and all the extra testing stuff's on the outside so it's easier to maintain over time as well I think that's another it's another good reason okay so in the interest of time I will barrel on so let's let's do this basic guiding let me show you the the answer for the adding example um so I'm gonna just copy okay so I've added my test function right next to this so I'll actually show it to you in PyCharm because it's gonna be a little easier to see here's my adder here's my test dad and I'm we're actually running two cases I'm adding yeah I'm sorry about this I'm adding two and three and I should get five and then I'm adding two and negative three and I'll get negative one to your point I'm not testing a string input if somebody use this library and they put a string and put it and it failed that'd be on me because I didn't adequately write enough tests so the way you run this and I'll do it the wrong way first if you run Python test add py nothing's nothing's gonna happen I guess Python will look at this there's nothing to run it's just function declarations just to exit doubt I want to run it as a PI test file so I do test dad and I'll get I messed up my code as I was talking so this is a good example one of my tests passed and then I screwed up my code while I was talking to you guys and now it doesn't work so when I stuck my second my second test failed so I'm gonna fix it and then we're gonna run it again I think that saved a I run it now two dots cool if you want the extra verbosity you put minus V and you get T test testing cases I would think about is it's does the code work at all did I did I actually implement my ad incorrectly because you can implement your adding incorrectly like what if I have my outer function and I accidentally add I add one because I don't know I'm tired and it's late I want to go home and sleep suddenly my codes not working it's not adding things together correctly so that's one version one reason to test another reason to test is it's for Python versions it's to make sure it's the biggest things that make sure your code works the second one it's cross compatibility the third one is it helps people understand what your library is doing so when somebody sees this they can the lis see some examples of how your code is working so for developers it makes it makes it easier to see what's the expected and put the extra goblin actually in the last one I forgot this the biggest reason to test so I should need the jump is all the way back on the front for refactoring your code so your code will work at one point in time but then you'll keep editing it and you'll accidentally introduce a bug testing make sure it's what's called there's no regressions in your code so regression testing which is a subset of what real testing is to make sure your code keeps doing the right thing even though you keep adding more and more features to it that's actually sorry that should just leave Lud with that that's the primary reason for testing so it's how pandas keeps working even though there's a crap ton of developers because you have tests to make sure that the code still functions the way you want it to function if and the opposite is for people that are usually new to Python they do Python they do a bunch of print statements and then they wash it all and then they delete the print statements well if you go any modify your code you don't know if those it still works because you're not looking at all the print statements again make sense all right so let's ramp up another another level so I'm gonna show you this is everyone actually before I move on does anyone feel confused about testing at least how to run a Python okay because we're gonna go up one more level of difficulty it's gonna match it up I'll use a this is this readable no ELISA directory structure okay so in in in most production projects let's use pandas for example I'll use ivories first and then I use pandas you don't actually put your tests right next to your code so in this example our tests were right next to our code so in this one here's my test here's my code this is not how most projects look what they do is they usually put the test in a different directory for for keeping your projects clean so I'm gonna show you this in pandas and RVs to show you the real thing and in pandas I have my top-level repository you can see here's all the code that goes into pandas the actual module the actual module is right here and right here I have all my tests away from all the other libraries so here's there's a directory called test and in here you have tests for arrays computation everything else pandas has a whole bunch of tests and arvy is the same thing our code our topple over repository is where this package is so this is the RV's package in here there's a directory called test in here we have all of our tests that test out our our functionality so tests are usually separated from code and uh normally on all production projects if you look at x-ray or any numpy the same deal why I'm explaining this to you is because in in the next example this is what we're this is what we were doing I'm splitting out the tests from the actual code to declare that they're actually our tests so if I go to the fixtures example I go to my library my tests are in here are in here they're in a different folder then then my module so my module is the actual module it's the pandas of the RV's or the numpy or the matplotlib of the world and then my tests are in the directory called test this is very great typical of testing libraries so bring this over Oh why this thing is always gonna fill me alright we're gonna use pycharm to show this thing to you so again there's a readme uh should bring it up on github but yeah bring it up on github okay sorry I know I'm all over the place well get their fixtures example ok just read me the concepts we're gonna test module in directory structure that's one I'm gonna show you fixtures and then we already did PI test flags in here again there's the entire documentation for that coverage even more of what I'm gonna cover so let's run this test same deal so in this case my library is the directory right here in here there's my module is this piece of code right here all I'm doing is loading data so here's where the data science component comes in sometimes when you load data in for things it takes a really long time so large data sets can take like minutes or hours to load and you don't want to it's annoying to keep doing it over and over again so this is my data loading function it's it's sleeps for three seconds because I have a a ton of data and then it returns five I want to test this thing to make sure that it works so what I can do whoops boom tests and I slow and I'll explain why it's gonna be slow in a second I'm gonna test my function right so I'm gonna import PI tests and I'll explain why in a second but we'll leave that there the next thing I'm going to do is I'm going to import my code so this is a Python relative import so from two directories up I'm going to import the my module the code Python has absolutely imports and relative imports usually tests that are contained inside of a test directory are relative because you're importing the code that's two directories up this you can ignore me if this doesn't make any sense I'll explain it for a little more advanced instructions in Python the important machinery if I did import pandas or import RVs here it would not import the one it would import the first one it finds in the suspec import discovery which might be in a different environment than the one that you're in so you might be testing a different version of pandas then one that you're trying to test the reason why this is important I I have three versions of arteries on my computer at any point in time I have the production version and then I have two different divisions and I want to make sure I'm testing the correct version so if I do if I do absolutely imports if I say input RV's I don't know which one of the three it's importing I mean I do because I know how Python works but you could easily screw it up so sometimes I don't and then I think I'm doing my cousin great then I upload it and then my fellow contributors are like what the hell you screwed up the codebase so I do relative import to make sure I'm getting the right thing there's three tests in here so the question before the prefix buy test so I've test test not three test object is an integer and test justice five so I felt like in this case we should test three things because you know I don't know like three things with the thing to do I want to test that it's not it's not three so shouldn't be three because it's five I want to test that it's an integer type so in data science you're loading in lots and lots of information if you guys can use like scikit-learn or whatever you throw an integer or you throw a string and an integer column and the regression just throws up and you've wasted like an hour that happens all the time so let's test data types and then I just want to make sure it's five so I'm gonna run this so directory LS so I'm in this case gonna run PI test my library and then test so when you when you do your example in like the five minutes this is the function you're gonna run because you're gonna run the tests that are located in in the my library function I'm gonna run this and we're gonna let this go yeah let's go oh shoot I know what I did wrong whoops right example so I'm gonna run actually just I just went on run of them and this is a little contrived but I'll explain why in a second so I'm just going to run the test slow function sorry the test slow module I'm gonna run it and it takes in this case it took 9.0 four seconds so it's super slow so I'll have someone do maths for me anyone want to volunteer just free candy if you volunteer why do you think it's took nine seconds yes yes I have three test cases and because I'm reloading the data every time I'll just bring it up over here because I'm reloading the data every time it takes three seconds to run each time so in this case time sleep is three I ran three tests you have nine seconds and then you have point oh four seconds for the overhead of high test right that's annoying when you run a lot of tests it's a pain to keep doing the same thing over and over again I'm literally just wasting time so if I was testing data loads if I loved the data in every single time I want to run every single test the tests will take hours and then nobody runs the test and then there's no point to testing in the first place so this is the motivation behind fixtures let me show you what a fixture looks like so in tests here's test slow there's no fixtures in here here's another version of height test and this one's gonna be a little confusing so let's talk through it and then I'll show you the difference or did someone want to see the result the sum want to see the results first and then I'll show you the difference all right results first this test took nine seconds I'm gonna run a PI test oops and then I'm going to test test save time with fixture and I know we're getting a little safe okay test save time with fixture I run this and it's gonna run in probably 3.0 four seconds 3.0 three seconds here's here's a test with fixture here's test slow so test slow has three functions one two three test safe time with fixture also has three one two three they're the exact same test I test that five isn't three that result fixtures is not an integer and this equals five the way I'm saving time is I'm only loading the data once through a fixture so fixtures syntax and PI test is a little confusing so what it what you do is you you write a function and you call it you can call you don't have to call it a fixture but you call it something you have it do the really expensive slow operation so this is the the really slow thing and then you have it return the result and then with all the other functions you give it you give it an argument that's called result fixture and what this does is it passes this result over here and then and then does your test you have to import PI test here so this is where you have to import PI test to get access to a PI test so there's a lot of the if I go through all the advanced I'm gonna say all the advanced Python words that you can immediately ignore but you're gonna relative imports are good this is called the decorator this is the fixture function and then I'm passing fixture as an argument to these other fixtures and then this stuff and in plain speak just add this thing at the top of your code that says high test out fixture it means this will run in this case once and I'll explain in a second why and then it'll it'll save this result and give it to these three functions so it loads the data once for three seconds and then runs three tests yes you could and I'll explain the scoping in a second so for now I'm just gonna leave you to you guys to just know to just take this as magic when I say scope equals session and then I'll explain after you've done the activity with what the scoping means I don't know I know I've been talking for like 15 minutes and this is where people start zoning out so with that I would I want you guys to do is I want you to try the next example and here's what's gonna happen in the next example I want you to test this this function or this object so you can ignore all this code but there's one trick here if you if you try and make this object twice the second load is gonna take a minute so the trick here is you need to write a fixture so you can write two tests for this object but only run at one time that's the point of fixtures any questions on why this is this contrived but I wrote it in a way that forces you to do it this way otherwise I know what would happen you just write it twice and you miss the point does anyone have a question why I read this way or what it's any you will get is you will get really fast you'll get one second if you try if you try running this object twice though you're gonna end up getting one minute because the second time it loads it's gonna take a minute to load so if it's already was created it's gonna sleep for a minute before it finishes this time if you want to see what it does you can run it so I'll here I'll give you an example I'll run the code to should give you an example what's gonna happen so CD CD o7 so if I run Python my library so I'm going to actually run this I'm not gonna run a pie test Python my library by module it used the object only once it this was nearly instantaneous if I tried doing this again though and I say print finished again I'm gonna make this a little bit less because I don't want you to stare at me for a minute but I'll make this five seconds the first one will be incident and the second one is going to take five seconds before loads so this is where you don't want to do a thing twice what I want you guys to do is use a PI test fixture to test this and the test should take less than a minute and I want you to test that adding still works that I can I can give this thing a number and then I can give it a number in another number and then it does the adding so see ya I'll give you 10 minutes to give a give that a try so we'll get will end it t 30 or 231 yeah again copying and pasting code from the previous example not cheating totally valid yeah the fixture isn't method me oh you can have anything yeah your fish give me anything it doesn't have to be called results picture it can be called whatever you want to be and I'll bring up the just see okay since you guys have laptops this is the this is the example so you can use this as a guide admittedly by the way it helps have two screens when you're testing because you get one screen is your code the other screen is your testing so you can look back and forth it's a little hard doing it sideways getting it on one one small laptop screen definitely doesn't give questioning I'm sorry I answered it in a kind of vague way but did it make sense for your code is that people that are outside of the company that are working around or inside yeah that's a good one okay so yes that's a really good use I forgot operating system but yeah same thing so and as much do you think is reasonable because if you had an infinite time then you test everything well like you can read like 50 you could write a 50 test to test like a square root function like the ad you can test everything right eventually you could test really big energy it'll cause memory overflows like I tested for memory overflows but like who's gonna do that you know I mean like you're gonna sit at work for like a zillion hours and write a thousand tests yeah but I will say the other way I do it if I'm being lazy is I write like a minimal amount of tests and then if when somebody messes something up and like it causes a problem then I write a test like cover that one like a another to keep writing band-aid tests and then over time I'm like okay I've got all the things that like normally get caught well probably will have me yeah I don't know it's like it's like seatbelts and cars after in cars you know the first cars are really crappy and is just ran into things and over time they learned like oh we should put an airbag here and we should put in there back there and we'll have it everybody got here and now we're we're like safe but it took a lot of car crashes for that to happen it's like it's sort of a similar thing with code got it make me proud [Laughter] [Music] yeah yes for the outer class yes so so I'm making so I'm making one of your objects [Music] add more does that change its number in just returns number it's very close you're really you're really close you're like one line up you get that you get the gist to it see you're good oh yeah exactly so these fixtures have a question I got is what a fixture is used for fixtures are used for the biggest using reason to use fixtures is to save computation so if you're running out if you're running I'll give you an example from rvs it's a Bayesian computer computation library doing Bayesian running a Bayesian model so it could take like five minutes or more but we want to test the model I don't want to run one model every single time we got to run a test so I run the model once and then we just reuse that model over and over and over again and PI test ensures that it's just there and available there's there's more there's more advanced uses for fixtures I'm not gonna get I won't go into in this basic talk but there's the link will tell you barbata so we good there all right I'm gonna show you the I've run the example and I'll show you how it works so alright so vim test outer solve so in this case I instantiate the fixture one time and then I and then I run this twice so here's the fixture it instantiate s' the object and then I reuse the object here by saying outer fixture this doesn't even have to be fixture by the way this could be outer anything I'm gonna break my code by doing this for instance but you don't the name it fixture in this case and then I have my code so when I run this Python I put the wrong place so CD my library CD yeah so I'm just out here CP dot slash solve okay just gonna use a GUI okay um hi test my library test this should run oh no all right Tracy yeah there you go it runs twice if I would do minus V you'll see both tests come up this this one runs really fast in two seconds so what I'm gonna do is so the code looks like it looks like this right now I'm using a fixture so here's me using a fixture it takes point two oh two seconds to run if I took the fixture out and I can die instantiated this twice like like this so what I didn't want you to do this is why I'm just trying to show you you could write the test like this and then say adder and then I they say adder here and I get rid of it so now here's my code does anyone want to take a guess how long it would take to run yeah it'd take it over a minute because it's going to instantiate once and it'll be fast but this one's gonna wait a minute before it tests can run this we can you can see that the first ones gonna run and then we're gonna be standing here for a minute while this thing still goes that's the point of fixtures this is a little bit more of a contrived example this one ran in five seconds because I modified my module here to be 5 but in real life example is you guys work with data dating loading data takes a long time for really big data sets do not do it more than once besides wasting time when you're in it actually puts a huge load on the so my coworker did this he he ran he was running tests but he was just hitting the database every time and then the DJ's got really angry with him because he was hitting a like he was just slamming the database with queries every like second to get a crap ton of data out to do his tests and he's like Oh everything's fine it's running my computer but like this the server was like on fire so you know you don't don't do it besides wasting your own time you're gonna piss off somebody that has to run like the spark cluster or whatever that you're running on so don't they just use a fixture load it once and then you're being like sort of responsible with usage that's my sage advice to you that's what the data portion comes in we're going to go through I'll do sit I'm gonna skip to the skip the end and skip back a little bit we're gonna definitely run out of time before I get to how monkey patch works but here's I will be at this conference all three days so if you guys want to go through it you can just come find me and I will be happy to walk through any of this code with you any time so we can go through there you can you have my contact information - you can send me a message and I'll try and help you out so yeah don't feel if you feel confused don't leave this conference feeling that way come find me we'll talk through it like the toilet oh I'm not like gonna be a really mean person after this lecture ends you can we can still still chat so let's do mock I'm gonna explain how mock works so let's go back to mark example since I have github up let's just used by that example oops computer doesn't want to cooperate all right well great so are we load things locally okay so Mach lets you create fakes fake stuff in Python fake objects the reason being is some of an object everything in Python is an object you just might have heard that means that like with data frames there's all these a little dot methods I'll let you do cool things like rolling sums or whatever they might be but it's really complicated sometimes that you create these objects like creating a data frame sometimes isn't easy you got a little whole bunch of stuff but you want to just test one piece of this entire thing out what mock lets you do is basically fake out the majority the object and just add in the little things that you that you need and let's go through I'll show you an example oh I'll give you three world real world examples to motivate this when you fit a model if you have to fit up a model and then you want to call dot like dot predict and scikit-learn dot predict takes to function it takes probably an x and a y and it gives you an output if you have to fit an entire model just to test that your dot predict works you're gonna be waiting like you're wasting all that time fitting that model you just want to make sure this like one tiny thing works why do you have to do all these steps beforehand that's an example where you use mock you don't want to test that the whole model fits you just want to test that your dot predict does something it doesn't totally fail out and maybe in this case you're got predict fails when you input a string you don't have to fit a whole model just check that your dot predict doesn't take a string another example is calling the API alright you don't always have a bazillion dollars to pay Google to call the API a thousand times so I have two students here that wear my bootcamp they part of my class they were hitting an API and then all of them hit a rate limit within like the first thirty minutes and they couldn't use the API anymore so testing against a real API is expensive usually someone's gonna charge you money for it so what you can do is you can mock the API response which is what the next example is so you can test against something that's not the real API so when you get to relay fail you're pretty sure it works and then my example is calling a database that I call databases a lot of work if I'm testing against a real database somebody's gonna be really angry at me then I'm opening connection zillion times to test a tiny thing out so let's go through an example of mock so in my library I have I have a module um and in this case I test that a data frame is over if it's if the data frame is over a million rows looks like it's it's big otherwise if it's less than a million Rosita return false so I want to make sure that I might is is dataframe big object it works now here's a trick if I want to test this out and I have to make a data frame that's a million like Y and we can actually just make this even more ridiculous if I have to do this this is gonna be super expensive like I'd have to generate a data frame that's like this case looks like multi billions shape Y just to make sure this function works that's what that's really annoying so what I can do is I can create I can import mock and we'll go through the import in a second but I can create this mock thing and then I can just give it an attribute that's huge and then I can see what the function is so this is not a data frame this is a mock and then I can add something to a mom so I'm gonna show you outside of the context that is test real quick just to make it a little more clear so ipython so from this one you get it from from pipe pythons inbuilt unit tests a unit test mock import mock and I can say equals mock and the AAE equals mock so I'm instantiating a mock object here and if I can say a dot name it's nothing but I can say it nine equals like test and now I can do a dot name well that's a test I can do a dot shape equals two and 1000 and now when I do a dot shape I get that back it's basically just like this a more first thing that we can just shove attributes on to see what happens you can make another mouth cops gotta do this all the time you can do mock return value equals hello we return value now when I do this it acts like a function that returns something so now I don't have to run in the entire function I can just fake out a function by having this mock objects so mock is a specially designed object and it's in pythons inbuilt unit test that that is complete built to be this like I don't know you guys my Pokemon did oh you did oh did it can be any Pokemon this is like the the ditto of Python objects where it can be any object so you can just slide it in and that way you don't have to deal with something expensive in our particular case we I do not want to make it any frame that's like 20 billion integers long just the test that my big function works because that's a huge waste of time what I'm just going to do is that I'm just going to say that the shape is big and then I'm just going to test that this is true and then this is false any any have a question why I'm doing this oh yeah actually good point that would have failed good catch that would've been fun so mark is just this thing that you can just make it act like anything else you can tell it what to do it's the most nice of Python objects that will just that'll just take any attribute or return any value and just sort of sub in and act anything we want yeah you think yes so I should say is Marcus an object that will act any way you want it to act in this case I want when I when I ask it what its shape is its gonna tell me this number and this is gonna to this function the way that this is gets a little bit more in the way Python works but this for all this function is doing is it's calling it's seeing what the shape object is and then seeing if the if the rowcount weight yeah row count is is bigger than two million I don't need a full data frame to dismiss function I just need something that looks like a data frame because python is what it's called duck types this is a whole statically type is ductile Python is duck type which means that it's Python doesn't check that all the attributes exist before you test and so it makes it easy to make either sort of testing where I give it an object that looks like the data frame and it's just happy and it like life goes on it's the the CS like 300 way of saying that does it make sense why I'm not creating a data frame yeah okay so I'm gonna run this test yes so that's a good point you better really know what your thing is supposed to act like because if you don't and you create the wrong Mach that you're not actually testing what's gonna happen so this requires this requires you to know how your code works that's a really yeah I've done that before I've had mocks and then I'm like wow my codes great there's somebody use it like your code sucks and then I'm like oh nobody tested it and like your tests are wrong like shoot no that's embarrassing so yes you have to you have to actually know how your code is gonna work so I'll run this for you real quick and then we're gonna go through the mock example and then I'm gonna do questions so Python my library tests work no up because I made my mistake not Python should be PI test and then of course I run into my own problems maybe this is what happens when I do live coding did I have it right before it well oh I know why sad okay this example definitely is broken so let me see I'm gonna see if I can fix it in a minute otherwise I'm just gonna say we're gonna do the example yeah I think actually is what it is is because I call I call a magic so this is a good example I screwed up on my mark and now the tests fail and now I look like an idiot in my own Python to toriel so this is a really good example the dangers of testing I know I know what the areas is gonna take me more than a minute that's all for now all you guys except that mock does the thing that it does I'll let you try doing the example and I'll show you the solution which I do believe works so I'll give you I'll give you a like three four or five minutes to just try out the mock object so see if you want to try out number nine and then we'll run the solution and I will talk through it there I apologize on your computer work oh yeah maybe I hear I'll just try that okay get checkout dot yeah this is sometimes it's nice to have slides because then you really can't mess up slides but this is a good example for testing right I actually screwed up my code in the middle of my talk and then my test told me that I screw up my code so maybe this is a it's like I don't know it's like those those videos you seen YouTube of people on bikes crashing into things and you're like I should wear helmets like oh yeah okay so yeah I did screw up my code when I modified it so there was an accident a little explanation of testing I thought I was doing something that if I would work and then it clearly didn't work and then PI tests told me that it didn't work so the the example on github works the one that you see in the video is the one that doesn't work as we saw with the PI test so there is a good example testing for the win I'll give you till 250 and then I'm gonna cut it off and I'll take questions okay mm-hm so we're trying to fail you for mock is so I'm unit tests for mark and you do mock our return value equals forty here oh I have this I have to put this in yeah so you can say see series equals mock and then you can say series dots dots sum equals mock return value equals forty so you're using two so in this case there's two objects there's a series of check and then there's this some object that does something so you can this can be multiple layers deep alright so in the interest of time I'm going to talk to the solution that I hopefully works sure does but that's going to find out so you can do this two ways I showed one way and I'll run I'll see it let's see if we can like for the other without making a mistake so let me try it real quick I test my library tests so there's two ways you can do this this is getting a question you can you can say a series as a mock object so I'm marking it out and then series dot some dot return values 2000 mock does this this thing where if you call a dot and you don't tell what it is it just creates another mock object and then it gives you the Train value the other way you can do this you can say series dot sum equals mock and then the return value in this case I'm going to call this return value equal one so these two end up doing the equivalent thing I can either I can either this case I guess I didn't explain it well but I know that I know the way mock works and if you just call dot and then this it returns another mock object and then I can set the return value manually myself this way or I can create a mock object I can assign this object to be another mock object that returns the return value I know this is getting like Inception like but I mean this this is uh this is pretty reasonable and test this happens all the time okay I'd you mark that shape or you can do that so you need whatever mocks will take on whatever attribute you give oh oh yeah so I will give another disclaimer this maybe I screwed up this example I believe I I did make an error on so what I'm gonna do is don't use it don't use this as the actual example I will right after this talk I will fix this and I'll pushes the github so it is 100% correct it works but there's there's a reason why it works but it shouldn't work that way would so for now example eight that you've seen on this on these slides please just ignore there's an error on it I will push right out for this tutorial a completely correct version this this solve version actually is 100% the other one was assigning an attribute to a modular level namespace which wasn't which is actually using a mock object it's the mock module so that's why I made the yes the mistake so this is a really good example of me writing a tutorial and then screwing it up and it works but they're still bugging my code this is I don't know I can't explain it I would I didn't do that on purpose that's actually just what happened so this is why you should test because we make mistakes and testing helps you I catch mistakes most of the time usually all the time but and this is this is also why we still do code reviews by the way so there's testing and this code reviews it's for both aspects mm-hmm yeah you can assign an object return value so python is a crazy language and that like everything is an object and objects can return other objects and functions or objects and like the whole world is objects which is not true of every language if you work with other languages like C or even are not everything is an object in our like our has integers and they're just integers but in Python integers are objects I know this is like this is far off computer science but when if you start writing a lot of Python code these things like start matter because that's how the language itself functions all right with that I'm just gonna leave it open to just any general questions that you guys have at all for 9 minutes I don't want to take up the whole time talking you usually shouldn't write tests that depend on tests because then if one test fails you don't know which one is the failure the up the upstream test the downstream test so while you could is usually frowned upon because it's just like it then it becomes really hard to tell what's going on so one of the values of doing unit tests if you're reading small is that only one test will fail and then you know where the errors in your code if you write tests that were found on tests I'll depend on tests suddenly your test is running through like 10,000 lines of code and the all of them will blow out and you can't tell what's on fire there doesn't make sense so yeah sometimes you want to ideally yeah it's it's fair so this is just this is just how you decide to code there's some Pierce that's a unit test should be to be unit eyes there's tiny as possible you test one function and they're completely unlike right a bazillion unit tests and mock everything but that means you write a lot of test code and write a lot of mocks and the nice thing is then you get really isolated test to test one time specific thing some people are like why don't I just test my whole function at once so why don't I just call my model fit and just that's the entire model and then see if I get the correct prediction and then in that case yeah it's easier cuz you wrote one test and you're testing all your code in one shot but if you get an error it's hard to tell where it goes I'm like somewhere in the middle and RV's where like right in between we test our entire plot function of like 100 lines of code and we think that's close enough so it just depends on where you're where you're at yeah that's a great-looking example how do you split up test requirements and EV requirements let's just go look at our B's because it has it so I'm just gonna keep using RVs because I know it very well we have we do we have regular requirements so requirements of txt so if you want to use RVs this is your dependencies and if you want to develop on RVs then you also install the development which is all these other things so we split up we split alone yeah we split apart and our B's we split up our our dev requirements from our regular requirement so the to go one level deeper there's people that just use a library they don't need they don't need PI tests because you're just running the library but then there's people that are developing and arteries and then you do need PI tests because we make that a requirement of development so I don't want everyone who uses our reason to stop high test it's kind of a waste of time but I do want everyone that develops of what I receive by test I should say we if you look at our pandas and things they should be gracefully this is a hard question because I'm just like so much less familiar with testing in the libraries because they don't strongly develop another libraries I'd say the bigot oh I know actually here's a big one so python requires more tests in my opinion then actually a for a fact then like c-sharp and whatnot because in c-sharp because they're statically compiled you get interface checks by by the compliation themselves so the code doesn't compile that itself is a test that the code works with python and even javascript because their duck type you don't know what an object is until you run it so you end up with more I end up with more tests at chests that interface exists and that the object types are the types that I want versus in c-sharp you know I don't test for that because it's like a or Scala I can only be that one thing because it's just it's in the definition of the code so make sense so just test to make sure my code runs or my code does the thing I wanted to do in the other in all languages but in Python I definitely check interfaces and types so in data science a lot I have to make sure like integers are integers they're not floats that strings are actually strings and then there's Python to Python 3 so I have to double check that in Python 2 it's it's the correct string type because there's there's differences between Z languages all that well Python 2 and Python 3 or 2 different languages so I have to make sure that the string type is different three Xena codes Python twos blanking at the top my head but it's the string handling is not the same yes it's really cool if you use Python three six which a lot of them don't so then we can so we read a lot of testing RVs because we have support three five and PI EMC still supports two seven which doesn't have like type in thing at all so then you can't but to your point like if everyone was on three six are higher than instead of having the right as many tests to test the type of objects you can just have the the linter like my pike try and verify that it's not type but it's important another they called type hints because we're not guarantee they're just hints so that's still not a sit still just because they have hints doesn't make it a strongly type of language it's a good question yeah no it's a good question so there's a your the question was do you always have to do tests that are that is this thing a boolean or not no there's other types of tests and I'm gonna see if I can find one really quick yeah see if otherwise okay in this one we're not testing and we're in this particular function we're testing that a value error is raised so where the question was I'm in Mic functions I'm just checking that like in our outer function we're checking that two equals two then we also had a question should I check string types when an outer function I wouldn't want to check that an exception is raised if you pass in a string you can write a test that is if if you input a string do you I get a value type error or here or whatever the case might be so you can test for exceptions too it's not just conditional testing things you can also test logs I can test this logs I can't that files are written so you can write tested to make sure that your your program is writing things on the director on the file this a file system so if they do sense a lot I dumped excel files I don't actually write a thousand Excel files I just want to make sure that it would write a thousand Excel files so I can write a test that checks that a thousand Excel files would have been written which is not necessarily like a boolean but I checked that like a another thing would have happened also a good question yes most open-source projects will say run will want you to write tests nearly all of them will this is the only way open-source works because if people just checked in code like every library would have we broken all the time just for a fact like you guys have written code yourself like you start writing and there's gonna be bugs and if you check that in then everyone's installation of pandas are broken and worldly on fire so every library usually wants you to write tests because they want to make sure that you're validating the functionality road and you're not your library still runs and isn't gonna break everyone's pip install pandas so yes that's another really good reason to test all major I'm gonna sit down and say all major and stable open-source libraries are gonna require testing and most companies require testing internally to most companies don't like the critical package is breaking like right before the end of the financial quarter so I can imagine that a couple a couple of people here have companies that are asking them to write tests you don't really you guys any more questions they're all really good completely out of time [Applause]

Channel: PyData

Views: 7,588

Keywords:

Id: dY1nNtDTruE

Length: 88min 8sec (5288 seconds)

Published: Thu Nov 29 2018