Property-based testing with jqwik and Kotlin

hello it's Duncan we developed our temping bowling simulator with a combination of type and test driven development we started by designing types that we F Express the domain and then used test to drive the implementation of the operations on those types the resulting test coverage is good but only consists of examples that we wrote by hand given the complexity of bowling scoring I for one have a nagging doubt that there are edge cases that we haven't covered property based tests are excellent at finding such edge cases we specify the property of the system and the framework generates thousands of inputs to try and break it so stay tuned for the battle between jqu and our bowling code we currently have a lot of what I would call example tests we found an example a game with Fred and Barney and a frame count of zero and we SE something about it and that's the patent all the way through here except there are some places where we start with an example and then we move that example through time in some sort of way but we have to divise these examples by hand so here we're only testing that we can play a game with Fred and Barney and a frame counter two we don't know that our code would work with Wilmer and Betty property bestas testing would generate examples for us so let's see how far we can get at the moment we're using junit and coton test assertions and also strict let's ask AI assistant to generate code for add jqu which is the property based testing library that I want to try cotlin support as a test dependency okay that looks quite plausible so we'll accept all although I do note from the jqu release notes that we're up to 1.8.3 by now so chat GPT is a little out of date anyway let's go off and next door to bowling test we'll create bowling property tests add that and now for jqu tests instead of annotating with at test we annotate with at property can we see that yes we can and we then write a function with some sort of name a property holds and that will take some parameters that we'll come back to but should return Boolean for whether it's whether the property does hold and providing return true I think we might manage to run this test and have a look okay well the test passed I noticed we've gained a jqu database here let's have a look at what the tests say it says warning a property holds in container without a foral parameter for, tries so basically jqu called our a property holder a thousand times and every time it turned true because had it returned false for example had we said random next double up until one if we were to compare that we say greater than 0.99 we might expect that to happen in a th thousand runs let's have a look and well a bit happen straight away that can't be true can it tries one I don't know let's find out what that was also BR run that oh I'm an idiot I should be starting from a small number so I should be saying let's only fail if it's not greater than 0.1 that should run quite a few examples there we go lots of those but it still failed at some point there we go so far though all we're doing is using jqu to repeat a test what we'd like it to do would be to give us the numbers so in here we could say for all a double and now we'll use that in our test so instead of taking random of our own we'll let jqu give us something that's probably random so we'll say that this is a double is greater than that now then I suppose a random doubled about half of them are greater than Z or .01 so let's change this test and we'll say that it should be that every double is greater than we say Min -1 e uh call it a billion let's see how we do and so the name of this property is that every double is greater than minus a billion now if we run this it's j Quick's job to prove that that isn't true by throwing random doubles at it let's have a look okay so let's have a look at what we got we got 6.3 3.9 n so so far all of these it is true that they are more than minus a billion zero well we might expect that have happen sooner or later we've got a range of Little Numbers here they're getting smaller and smaller and smaller and smaller and sooner or later you can see we had a failure actually there we only had a failure after seven tries why is that well we go back up here you'll see that after seven I guess here which was less than minus a billion at that point J quicker started looking for the smallest number or maybe actually I suppose the largest number that this is not true for so it started throwing these numbers at it and you can see as we get down here it's iterating in on -1 by 10 9 until in the end it's thrown a lot of other examples at us in order to come up with the answer that ah actually it failed with the sample of 1x 10 9 that was the simplest thing it could fail with okay well so far so theoretical how can we use that for our bowling to warm up let's start with the first test we've got in our example tests first one here is a game with no players isn't playable and that's saying that if we call game with no players it's a completed game but the fact is that game has a Constructor here that allows a frame count of defaulting to 10 but any number here we've shown that a game with no players and 10 frames isn't playable but the question is is a game with no players than nine frames or 15 frames or one frame and so on not playable you can see the issue here is that we've a test that explains a property but only tests it for one particular example of that property so let's take this over to here and now instead of a double we'll have the number of frames so this is a number of frames and it's going to be an INT now we can create a game with the number number of frames we're calling it frame count is a number of frames and we want it to be true that that's a completed game that's what we did here but instead of using expectations we're going to just return a Boolean so we can say return game is completed game let's try running that and that passes good and if we look at the output you can see that we tried with a thousand different numbers of frames and all of those result Ed in a completed game because we didn't have any players okay that's good next okay then a game with no frames isn't playable let's take that move it over here and make it a property and it wants to return a Boolean now in this top test we held the names of the players the first parameter here constant and Vary the number of frames this time we're going to vary the number players and keep the number of frames at zero so it's going to look like this and in here we're going to put the player names there is just one problem and that's varag here I think jqu can create us at for all a list of player names which is a list of string we're passing it into here as vog would be a bit complicated I think in any case it would be hand to have a list version of this Constructor now last episode I made the Constructor here an extension on the companion object and the reason for doing that was that my experience was that if you started a function with a capital letter like this that intellig would complain that you shouldn't start a function with a capital letter turns out in fact that it only complains if you start a function with a capital letter where that doesn't match the name of a type or an interface I haven't really worked out which yet i' we call that Fame it would say well a it's unused H let's work our way there if we rename this to be Fame then that would change the names of these but you note here there is a warning that it should start with a lowercase letter but if we call it game and that matches the name of this type we don't get that warning so that would allow us in fact be a bit simpler we could get rid of this odd companion object and just have a fun game note that are the call sites all the places that call this here for example The Source hasn't had to change although in fact the thing that it's called has changed and I think the code would have needed to be recompiled anyway back to game we want a version of this that takes player names as a list we've also got playable game here with player names I think maybe the easiest thing to do is going to be to take this one make it not vogs but but rather take a list of string this one can then call player names to list it might be somewhere where we're calling playable game directly yes there you are so this is now going to have to say empty list that's fine and the same here this is list of Fred don't mind not having the varag with playable game because we have the game function that's more usually used okay and now back to games we want a version of this that takes a list so I think if we pull this into a variable and then make this a block body move this up to there and now extract this thing it's also going to be called game and I don't understand why we have player names twice let's just go and fix that up by hand so this is player names which it wants to be our list of string get rid of that one this is player names and that we can sort out like bit of a that's an expression this goes back to being a single expression and there's everything still run it does good now why was I doing that ah it was to pass a list of player names into here let's try that okay so far so good it called this one a thousand times let's just see what sort of things it called it with so we say also printland run again and now if we look you'll see that it turns out the most random strings are Unicode and quite pictogram but you can see here there's a special case of an empty string and so on in fact lots of empty strings maybe these are blank perhaps don't know okay moving on I think now we've got this Constructor I will pass empty list in to this one one so we can compare them easily and it's not clear to me we're running all the tests now we've got two sets so let's set up a test run of all the tests good moving on then this test here can't create an invalid playable game you can see does actually have three examples but where we're using a frame counter zero here for example we're only using one list here rather than several this is a bit of a strange test because it's talking to a playable game Constructor which isn't really part of our public interface but there is this case here I'm interested in where we pass a frame count of minus one in and this should apply to our game Constructor as well so I think there's a property of cannot create a game with negative frame count so here to be exhaustive we want to test for different sets of lists of player names and different sets of negative numbers so we want to keep that and we want to add a parameter of the number of frames and now what does it mean that we cannot create a game well it's that it will throw an illegal argument exception so we'll say return try construct a game if we manage to do that then this has failed but if we get X which is an illegal argument exception then this has succeeded and and I need to pass a number of frames into there now you can probably see what's going to happen half of the in are pass into here are going to be positive and this will succeed so if I run it is that true uh yes it is it turns out that if we passed the second parameter as zero in this case then we managed to actually create a game so we need to find a way to constrain this number of frames one way to do it is when an assume and we can say assume that some condition so now we can say assume that a number of frames is less than zero and that will have bought any of these runs where the number of frames is not less than zero should pass oh that's interesting let's have a look ah well in this case what it's saying is the first parameter is an empty list and then we're passing in minus one frames this feels like an actual bug let's have a look how we got there if we go down into game then we pass the frame count down the frame count won't be zero so we pass a playable game playable game it's Constructor here says player names map playable line it and frame count playable line here this is the place where if you have a negative frame count we're expected to fail because you can't create a list with a negative number of frames but block for this map won't be invoked when there are no player names so we'll never hit the constraint in playable nine that cause the illegal argument detection to be thrown so jqu is making us consider which of two behaviors we want do we want to say well there are no players so it doesn't matter that frame count is negative or do we want to say well there are no players is but we're going to fail because you pass an invalid input for the number of frames I think probably our software would be less confusing if it always failed for negative number of frames so where can we put that constraint that's probably we should do it in here so that it also applies to playable game in order to do that we need to find a way of making this expression into a multi-line thing and I think what worked with before was to say unit. run and then we can put the require in here frame count is greater than or equal to zero otherwise cannot create a game with a frame count of frame count does that get us running oh no well I think it shows why we need tests let's have a look at the failure it failed for an empty list and a negative frame count how did it manage to fail let's see whether we can work it out let's look down for our game Constructor ah here we actually specifically say player names is not empty if the play names is empty we go straight to a completed game okay I think we're in a good old fashion compromise territory here I can't think of any good way of funneling the frame count into playable game in order that this require is hit but I don't really want the require in both places I think I'm just going to say that we should make this a block body we should move this require up into there and say that if you call playable game directly then you don't see this Behavior so we can take that out return to there is that okay still no oh dear but now you see we failed a different test we're now failing a game with no players isn't playable why is that well here we are game is now checking that number of frames is not negative but jqu is passing a neant number of frames in here well I suppose that's all right we can fix that with an assum so we put that in there and now things are still not right oh I'm an idiot in this case we need the number of frames to be greater than or equal to zero can we finally get get there yes we can having said that of course jick is throwing random sets at us so it is just possible that it will fail another time but I'm quite confident about that now a problem with what we have now is that this assume will be throwing away half of the number of frames that's passed into us I suppose given this greater than or equal to zero technically almost half so instead of assume we can use a provider let's see whether chat GPT can generate one for us so we say generate code and I want write a jqu provider for integers greater than or equal to zero okay let's see what we've got it's changed a number of frames to this for all with a name non- negative integers there it is and down here oh it's actually done negative integers for us as well well that's nice thank you and it's added non- negative integers they are an arbitrary int and they're defined to be integers between zero and max value and here we've got integers less than or equal to minus one you know that will seems really plausible I'm going to accept it all and run well there you go so we can use these providers to affect what values jqu is injecting that seems like a good place to check in to me let's see what changes we've made we've got this unversioned jqu database that's interesting I suspect that we should add to G ignore bowling tests have changed let's have a look why ah because we changed the interface to playable game build has changed to reference Jade quick game has changed and that's because we've added this play names that takes a list of string and that's just more convenient for software to call rather than the varag which is more convenient when humans type things in okay I think that's a good change so we'll commit that with introduce jqu tests for creating games and there are a couple of issues let's just have a look oh that's interesting intellig doesn't know that provide means that this function will be called I think there a way to add annotations to the list that make something used let me see if I can work out how to do that ah it's sort of hidden in plain sight here you can say suppressed unused warning if annotated by J quick API provide so now if we build that warning has gone away good okay then before we get on to actually playing a game we have one more test here a game with players is playable again this has two examples but they're just a single player and two players and they're not testing the frame count anyway let's take that copied over into here and we'll say that this is a property and in the same way here we want a list of player names and in this case we do want it to be playable so we want positive integers so let's write ourselves one of those which is positive integers and that's all integers that are greater than or equal to one so now we want to throw positive integers for our number of frames and we should be able to say that return game a list of play names and a number of frames is playable game and fix this up to return a Boolean let's run that good I must say that I'm suspicious because I haven't specified that this list can't be empty and I would have thought a game with an empty list of player names no matter how many frames should not be a playable game in fact I thought we had test here that showed just that for an empty list let's just print out whether this list of player names is empty and run just this test aha well that's interesting because as soon as it was we failed look there but why didn't we fail before let's try running all the tests that's very suspicious isn't it what did we actually run oh am I running the wrong thing I am do and yet there we are I do not know what's happening now look at this we don't appear to be running all the tests in Bowling property test why not well I want whether in the edit I'll see how I mess that up but we do seem to be in a state now where we're running everything I expect to run and getting the results I think we should be getting so let's take this out run good and now given that most lists of string will not be empty I think we can just solve this problem with assume that a list of player names is not empty good Ah that's interesting rerunning the tests has just run some of those tests aha there we go look Java Heap space finally got to the bottom of it and in fact it was in the back of my mind that we could be passing 100,000 frames into here whereas the real bowling game is limited to 10 I'm glad we got to the bottom of that and we can solve it by saying in fact I positive integers here what we really want here is positive integers less than 11 say so we can say here between one and 10 and we'll just check the definition of that that's a Min and Max so it's not actually between it's including this end point good so this I think we'll say is valid frame count we can use valid frame count in there and in I was I was going to say and there but I think in fact we still want any non- negative integer here because we're passing an A List and we always want to be a completed game so I think that's fair enough just check everything's fine so valid frame count seems to be a good thing let's run that good and we'll check they're running all of our tests Splendid finally one little cotlin tidy jqu provides int extension functions for int. any and then we can put in a Range so we can say int any and this will be zero to int max value if we import that that would tighten this up quite nicely this I think then is int. any from int. Min value to minus one and valid frame count is int. any between one and 10 all good all good except that if we'd messed this up we might now be injecting fewer values into our tests as a cotland developer there's something a little bit irritating about these annotations and the fact that this thing has to line up with this thing I'm sure a pure cotlin property based Library would be based more around first class functions but I don't hate it and it is quite expressive okay then given where we are I think probably that's only an amend commit we've gained a misc.xml h that's where we Mark The annotation okay let's just men commit that one analysis point which is the intell J is still lying to us about current player well testing our game creation has been a good way to warm up on jqu and I think the formalism has certainly led us to improve our interface a little bit of course the really interesting properties though are those that apply to games as we're playing them if you've experience in property based testing and can think of good properties of in progress bowling game then please let me know in the comments below if you want to see the sort of thing we come up with then please subscribe to the channel so that YouTube will tell you when the next episode's available I'm sure there's a like button somewhere that you can press if you want to give me a little dopamine hit or even better you could buy the book that I wrote in that price called jav toot refactoring guide book details of which are on the show notes below thanks for watching

Channel: Refactoring to Kotlin

Views: 573

Keywords: property based testing, jqwik, kotlin, bowling kata

Id: xgDPvlgUSLE

Length: 27min 22sec (1642 seconds)

Published: Fri Mar 08 2024