John Hughes - Keynote: How to specify it! (...) - Lambda Days 2020

so those of you who know me will know that I love testing and in particular I love property based testing but I found out fairly recently that many people actually find it quite tricky to come up with good properties to test their code with so I thought today I would try and demystify that a little bit now I am going to talk about testing pure functions so if you are sitting there thinking when is he going to start talking about testing code with side effects the answer is not today but don't worry it's going to take me a full hour to talk about testing pure functions so baby steps you know and I'm going to be giving today's talk using Haskell for the examples and using Haskell quick check as the property based testing tool but the ideas are going to be generalizable so whatever property based testing tool you like to use you're going to be able to take these ideas and I hope you'll find them helpful to write properties of your own so let's start off for those of you who maybe haven't come across property based testing I'll just explain what it is let's take a really simple example the reverse function on lists so suppose we wanted to test that well of course it will be easy to write a unit test right here's one and this is just an example of the form that unit tests so often take we called a function that we want to test we make up some sample argument for it we figure out what the result should be and then we check that that is actually the result that's returned and this triple equal sign that's just an equality test but it's part of Haskell quick check it'll generate an error message and the test will fail if those two values are not equal okay so so that's straightforward do I like it no I'm only testing reverse on one input what about all the other so what I like to do instead is to write a test for reverse using quick check what does that look like well I just define a function for testing reverse function I call a property and so I'm calling it prop reverse it's going to take an argument list X's and tests reverse on that list and the great thing about this is the quick check is going to be able to generate arguments for this function so I'm going to be able to test reverse on thousands of different inputs millions if I want excellent you'll notice that even though reverse can reverse lists of any type I've written a type signature here list of integers and the reason I've done that is I need to tell quick check what kind of test data to generate so you'll see some time signatures they're there just to choose test data and I'm going to use lists of integers because why not integers make quite good test data okay so now I'm going to be testing Reverse on a random argument that's great but of course I hadn't finished the property yet I need to write something in place of the question marks what should I write well in the case the unit tests I knew what the argument was I could figure out what the result of reverse could be and I could just compare against that but in this case the expected result of reverse is going to depend on axis isn't it well of course that's no problem I can just write a function to compute it from the argument yeah but that isn't actually any easier than writing the reverse function is it because it is the reverse function now I'm going to write the reverse function twice once of my code and once in my test and do you know what's really stupid about that if I've made a mistake in the reverse function the first time I wrote it how almost certainly make the same mistake the second time I write it so that means not only am i doing the same job twice but there's little value I'm going to be making the same mistakes both times I've gone to all this trouble of writing a property and it's like back in high school when you did lots of algebra and you proved of zero equals zero it's not actually going to be a good test and this is something that people encounter all too often when they try and write property based tests they replicate part of the code in the tests that's expensive because you do the same work twice and it's low value because you probably won't find your bugs expensive and low value is a bad combination in a testing technique and some people when they discover this they just say property based testing isn't for me the wrong conclusion of course but it's understandable so what can we do instead well if I can't predict the result of reverse in my test I have to do something else what I can do is I can test some property of the result so for example in this test I test that if you take the result of reverse and reverse it again you should end up with a list you started off with right that should be true and now this property is it's quite easy to write and I'm going to be able to run as many tests as I like by default a hundred okay so that that in an instant will test reverse in a hundred random cases and I can go on to test many many more there's just one problem with us what if I had written reverse in this particularly stupid way now you might think surely nobody is that stupid but you know sometimes I don't want to write the code of the function straight away maybe sometimes I just want to put something there to keep the type checker happy and go on and now come back to that later what if I forgot to do that this will keep the Haskell type checker happy but that definition will pass my test because it's also true so that's sad but what's particularly irritating is that if I make this mistake and I run that unit test I showed you in the first place of course it will detect the bug so what have I done I've gone to all this trouble to write a property-based test and perhaps I've got worse testing though when I started that's not good well now of course I can write a property that will distinguish this wrong definition of reverse from the correct one here's one example this is the wrong property it tests to see have I written the bad definition so it just checks that reversed X's really is the same as X's and if I test this one with quick check because my reverse is actually correct I can write reverse of course this property will fail so this gives me a chance to show you what happens when a property fails quick check will complain and it will produce a counter example a failing test case in this case the list 0 1 and why does it fail because when you reverse it you get 1 0 and that is not equal to the original list X's so it fails the wrong property there the interesting thing here is that quick check almost always will choose 0 1 as the counter example occasionally 1 0 never anything else now I told you that quick check is generating random cases what's the chance of hitting zero one as a random list it's very very small the reason that we see this particularly simple test case is that quick check after generating the random test and finally one that fails has simplified it as far as it possibly can we call that process shrinking and in this case shrinking will discard unnecessary list elements will you need to to falsify this property and it replays it into integers by smaller ones if it can and so that's why we get one zero why don't we get zero zero is the list because when you reverse zero zero you do get the same this back again so that's not a counter example so the smallest counter example is a list with two elements 1 and 0 and they might appear either way around okay so that's a quick introduction to property based testing and we've seen two of the great things about it first of all that we can randomly generate lots and lots of test cases as many as we like and so get much better testing hopefully secondly it's really important that when a test fails we don't see random ball we see a really tiny example that is usually very easy to debug but we've also seen the trap that it's often very tempting to replicate your code in your tests and if you do that you're really wasting your time it's expensive and low value so what I'm going to talk about today is I'm gonna try and show you five if there's time systematic ways of thinking about how you can formulate properties for property based testing to do that I'm going to need a simple example but not as simple as the reverse function and so I'm going to use binary search trees containing keys and values so this is just a representation for finite maps and I'm sure everybody has studied binary search trees sometime in the distant past maybe but we're going to test a Haskell implementation of them on this slide we see the definition of the Haskell data type that a binary search tree with keys of K and values of type V is either a leaf representing an entity map or a branch and a branch node contains a left subtree a key a value and a right subtree so that's the data structure that I'm going to test of course I need to test operations on the data structure so let me show you the operations I'm kind of I'm going to work with first of all one that can find a key you give it a key and a tree and if the key is there maybe it'll return you a value I've got four ways of building trees no is going to build an empty tree insert can insert a key in value into a tree delete deletes a key from a tree and just to make things a bit more interesting I'm also going to use Union the given two trees gives you a tree that contains the contents of both so these are the five functions that I'm going to concentrate on testing you will also see a couple of other functions here and there in the talk but they're not interesting to test that I just they just help me formulate properties to lists is going to take a binary search tree and just return the list a list of pairs of the keys and values in the tree and keys is even simpler you give it a tree you get a list of the keys of the tree so you'll see those cropping up okay so now I want to write some properties of these operations but before I can do that I have to tell quick check how to generate random trees now I could give an entire talk on writing generators but I'm not going to do that I'm just going to briefly show you the generator that I'm using it looks like this so the code is a bit complex but what it says is choose a random list of key value pairs and then insert them into the empty tree okay so we're going to be able to get random trees by randomly and thirteen keys and values I hope that sounds reasonable I'm also I also need to tell quickcheck how to shrink this type of data but that's easier there is a generic method the quick check provides and I'm just going to use it so that's fine okay so what properties might I write of these operations so I think the first thing that one should always ask oneself in this kind of situation is is there an invariant and if there's one thing you should remember about binary search trees yes there isn't invariant that the trees have to satisfy and I can easily write a function to check that that invariant holds I've called it valid so what does it say it says well a leaf is a valid binary search tree and when is a branch valid it's valid if the left subtree is valid and the right subtree is valid and all the keys in the left subtree are less than the one at the root and all the keys in the right subtree are greater than the one at the root ringabel what what if you have the value twice well this says that's not valid okay so that this is why it's helpful to write this function down I'm defining what's valid here and I'm saying you must not have the same key twice so some of you may be looking at this and thinking yes that's obviously correct but it's not very efficient is it no I could be clever couldn't I because since I know for example that the left subtree is valid that I only need look at the rightmost key in the left subtree and compare that with one of the root right is that obviously correct well it's plausible I'm not going to write the code that way it would be more efficient this is quadratic in the depth of the tree but who cares generated trees generated tests are small you can afford to run a quadratic function what you can't afford to do is make a mistake in this code so i optimize my test code for obvious correctness so that when a test fails it's not because I screwed the test up it's because there's really something wrong in my implementation I think that's one useful lesson about writing property based tests optimized for correctness not performance so once I've defined this function then I can write a bunch of properties for every way of constructing a tree I can write a property that says that should give a valid result okay so first of all the antah tree should be valid secondly if I insert a key in value into a tree I should get a valid result thirdly if I delete the result should be valid and fourthly if I take the union of two trees the result should be valid you'll notice some greyed out type information on this slide remember I said I have to specify it fact I have specify a bit more I'm also going to say that the keys I'm gonna use my tests will be integers the values will be integers and so on it has to be there it's boring I'm going to elide it on all the rest of my slides but if you when you try this at home you do actually need to have it okay so now I've got four properties testing the four ways of building a tree if I plans a bug these property should tell me where it is right so let's see what happens I planted a bug in insert that constructs a tree that is not valid so prop in search should fail the other should pass right here's what happens what they all failed so insert valid the first one fails I expected that but delete and Union can also return invalid results even though I did not put a bug in them what's going on well if we look at the test cases that quick check reports these are the trees that I passed in to delete an Union look at them closely there are the keys in the first tree wait a minute there's a zero key in the right subtree and a one at the root that's not valid likewise the one that I passed the Union has the zero key twice I said you can't do that so quick check is complaining that if you give delete an Union invalid inputs they'll produce invalid outputs but duh I didn't expect them if I give them a garbage to fix it up so that becomes a valid tree what I expected was that if I give them valid trees they'll produce valid results so what's going on why did this happen do you remember how I generated random trees I used insert right insert is buggy so now quick check is generating invalid trees and passing them to delete an Union so of course the property's fail one so what this shows us is that when you do this kind of thing there's another property that you must always write this one this looks weird right it looks as though it's testing that every tree is valid what is it really testing it's testing the generator it's testing that every tree we generate is valid and if this property fails there is no point looking at the results of any others so it's a critical property we have to test our test infrastructure so that we can have confidence in the results that it produces actually it's important not only to test that I generate trees valid trees but also that when I shrink trees they shrink to valid trees so the shrink function that you have to define it returns a list of smaller trees than its argument and the property I want to write says that all of those should also be valid but you know what even if I fix the bargain insert so that that first property passes the second one still fails look quick check tells me that the first line there shrinks to the second line okay and if you look at the second line it contains the key 0 twice okay that's not good so in what case does shrink my shrink function do that well I can see the argument I gave it but what that is also invalid so quick check is showing me that an invalid tree contribute invalid tree again yeah so what why is it showing me that so I'm not I'm not generating invalid trees anymore but this still happens why because before quick check shows me this result it shrinks the test case using a buggy shrinker that's why I get a useless result so you can stop this from happening and the way to do it is to add what we call a precondition to this property which says it changes it to say if the tree is valid then everything it shrinks to must be valid and the effect of that is to tell quick check don't even run tests where T is not valid and in particular don't shrink to tests where T is not valid even if the shrinking function says suggests you might and if we do that then quick check reports something sensible it shows that that first line there containing key 0 and 1 which is valid containing key zero and zero why is that well quick check shrunk the one key to a zero key why because I was lazy because I said just use the built-in shrinking function the built-in freaking function knows nothing about binary search trees it just tries to shrink everything so of course it will shrink that one to a zero I need to fix this to get any useful results how can I fix it actually it's dead easy to do so I can just say use the generic function but only keep the valid shrinking trees that's the easiest way to fix it okay so that's good we talked a lot about validity testing testing an invariant and I want to emphasize this is important because while you know just producing valid trees isn't everything you want to test if your functions are producing invalid data then it's you want to fix that kind of bug first so this is a good starting point what can we do next well it's always useful to ask yourself what is the post condition of each function now I realize this is the vaguest advice in this talk because it's like saying what is the property of your function but nevertheless it is useful to ask yourself after I've called for example insert what should be true and here's my suggestion after calling insert we should be able to find the key that we just inserted and we should also be able to find any other keys that were there beforehand and I can write this down as Haskell property I call it the insert post condition and it just says that after I have inserted a key in value into a tree if I try and find any key at all if it's the one I just inserted then I should find the value I just inserted otherwise I should find whatever was there before so that more or less directly translates the sentence into code this is a good property certainly worth testing you might wonder if you look at K and K Prime those are two randomly chosen integer keys how often will they be the same and the answer is not very so when we test this property we're going to spend most of our test effort in the else branch in other words testing that insert did not corrupt other keys in the tree that's important to test but whether we should spend you know 90% of our time testing that is another question okay so there's one slight problem with this property but I have a lot kind of focus on that it is a good property I'm going to show you a more serious property with post-conditions and we'll consider another function to see that what's the post condition of find okay well let's think about it after calling find there's two cases right if the key was present in the tree then we should get just and the value if the keys not present the result is nothing how am I going to write this property well I'm gonna need some auxiliary function that can tell me whether the key is present in the tree or not and if so what the value associated with it is but that's the find function I just took my foot and I'd put it in the trap how can I avoid it well there is a nice trick that we can play at this point we can say rather than trying to write a test that determines whether or not a key is in the tree let's write tests that guarantee but the key is in the tree or that the key is not in the tree and how can I do that well I can write one test that tests find on a tree where I guarantee the key is present because I've just inserted it at another the tests find where I guarantee that the key is absent because I've just deleted it okay so those two properties together capture the same intent as the as the sentence that I wrote previously so we ensure what we want that construction there is still a reason to be suspicious of that first property for example and that is it's supposed to test find in every case where the key is present in the tree but how do I know that every tree containing K can be constructed by calling insert with K may be insert only puts keys in particular places in the tree so maybe I'll miss some important tests by writing the post condition this way well if so yeah that's a problem but we're not gonna solve it here let me go on to another idea a very very useful idea for testing that I call metamorphic properties and the idea here is that you ask yourself you say well maybe the actual result of the function like insert is very hard to predict but if I change the input of insert a little bit maybe it's easier to predict how the two results should be related all right so you see what we're going to do we're going to take a tree we want a test inserting a key and value into it but it's hard to predict what result we should get so instead we're going to modify the tree in some way but maybe insert a different key and then do the the first insertion now what can we say about the to question mark trees at the bottom well what we can say is we don't know what either tree will be but the one on the right should contain K Prime and V Prime as well as what the one on the Left does in other words I should be able to construct the one on the right just by repeating the insertion of K Prime and V Prime so this is an example of the commuting diagram and I'm going to write a property just tests the both paths around the square produce the same result so I want to think about how many property ideas I can get with this approach if I've got n operations to test and if every operation can be used to modify an input then I got a quadratic number of test ideas so that's great this is a really fertile source of ideas for writing properties when I write the property down it's always gonna relate to calls of insert so I called it insert insert because we're testing insert with an insertion into its argument so here is the first call the original call I wanted to test here is a modified call where I've given it a slightly modified argument and then there should be some relationship between the two and that relationship it could be I could write a property that tests something as simple as the upper tree is bigger than the lower one that would still be useful but here I'm gonna say that actually I can get the upper tree by repeating the insertion of the other key in value into the other call okay so here we are here's my first metamorphic property now I can see some of you are looking a little worried you may be wondering is this really true and when I write this kind of thing down I see many people started thinking is this really true or not so here's the more important advice about property base testing do not think let quick check do your thinking for you is it true we'll just find out by testing it and then one of two things will happen either the test will pass least that case is almost certainly true or quick check will tell us precisely what case we missed and in this case it's not true the test fails and if we look at that the test case of quick trick reports we can see that we're inserting the saying key twice the key zero is used twice the property says you can swap any two insertions over but of course you can't swap insertions of the same key right if you insert the same key twice the last insertion wins so what do I have to throw the property away no of course not I could just refine it a little bit okay so here's the property that I actually want to write where I'll make that modified insertion and then I can just say if the two keys were equal well I can also predict what I should get the last insertion wins right so it should be the same as just only doing the last insertion otherwise the original thing I wrote and now I've got a refined property that should be correct only it's not oh darn what happened here well now I've got quickly chosen show me a test case where I insert different keys but I in different orders and the order makes a difference you can see there at the bottom are two trees that are clearly different but if you look at the keys and values in them the two trees contain the same keys and values they just have different shapes do I care about the shape of the tree I'm gonna say no so I'm gonna say that morally this property is true of course now I have to teach morals to quick check and we very often need to do this when we do is kind of testing so I'm going to define when two trees are morally the same or equivalent and I'm going to say they're the same if when we convert the trees to the list of keys and values they contain the same content and then I just rewrite the property to say that the trees that I get by these two roots are always equivalent this property passes and it's a really good property for testing and now I could go on and I could write you know insert with argument modified by delete delete with argument modified by insert delete will argument modified by delete union with argue it's modified by Oh lots of lots and lots of properties so I get great testing and this idea of running the code under test twice on related arguments and checking that you get related results that's not limited to property based testing that's called metamorphic testing in general and it's an extremely useful technique it's also a research area in its own right there is an international workshop that runs every year of metamorphic testing with examples of applying it in all kinds of different contexts to all kinds of different software and that runs together with the International Conference on software engineering so it's a really really useful idea ok have a little time to show you the fourth idea a fourth idea is what I call inductive properties and this is something that Kuhn Classen came up with Kuhn was my co inventor of quick check in the first place 20 years ago so look at these two properties for Union the first one says if you take the union of an empty tree and any treaty you should get the same tree back again it seems reasonable the second one says if you take the union of at rebuilt with insert that's the same as taking the union of the tree you started off with and then inserted the key in value into the result also things reasonable and I'm using the equivalence comparison there because they might have different shapes but if you look at these two he almost looks like a definition of Union doesn't matter because look at that second one I'm saying that the union of at rebuilt with insert can be constructed by recursively taking the union of a smaller tree and then in doing the insertion at the end now this is not a suitable definition of Union it doesn't work as a definition but we can argue by induction on the size of the tree that if Union passes these two properties it must be correct only one function should do that and it should be the right one okay so that's nice and inductive properties are properties that you write down and you can make an argument by induction but the only function that can pass these tests is the correct one in other words you have a complete test of your code so that's really nice but there's a hidden assumption here and the assumption is that every tree can be built just using insert every shape of tree can be built only using insert is that true well you could think about it but what did I say don't think can I test it turns out I can to do so I need to write a function that given a tree returns a list of insertions that will reconstruct it and that's easy to do in this case because I'm not it would be harder if I were using some kind of balanced trees but I'm not so how do i reconstruct a tree well first of all I better insert the key and value at the root and then because insert never moves keys and values they'll stay there so I can then take insertions that will construct the left subtree and insertions will that will construct the right subtree and re running these insertions should reconstruct the tree I start off with how do I convince myself of that I write a property obviously here it is so once again this is Phi the intricate code but what it says is any tree can be reconstructed by making this list of insertions and just using insert to rebuild a tree and here I have to use the structural equality so I'm the best is why I need both these comparison operations okay so this property says that every tree can be built only using insert does it anybody the least bit suspicious about this property as a test well how did I generate random trees oh I only used insert wait a minute this property is saying that every tree that I generated only using insert can be reconstructed only using insert zero equals zero what about trees that are returned by delete or Union okay so there is a problem but it's easily fixed I just have to write a couple of additional properties one that says that yes this property really holds of trees returned by delete and another that says yes it really holds of trees returned by Union and with these two additional properties then I really am going to be able to test that every tree could be built just using insert there's one more kind of property I want to show you they're extremely useful and they're what I call model-based properties so once again suppose we want we have a tree we want to test inserting into it and we want to verify that somehow the result is correct how can we do that well one thing we can do is take the trees these complex data structures and convert them into something simpler let me just extract the list of key value pairs in each tree now then it's hard to say whether the tree on the bottom left is right but what can I say about the two lists that I've got on the right hand side of the slide now clearly I should be able to get the bottom one just by using list insertion to insert the key and value pair into the list right so this is a very general idea what we do is we define a mapping from our complex probably efficient implementation into something much simpler data type we call that mapping an abstraction function we abstract the implementation to a model and then we define we implement our operation on the model list insertion is the model of tree insertion and then I just write a property that says for this commuting diagram you get the same result both ways round there it is these model-based properties are really simple to write all you do is you call your implementation insert you convert it into the model using two lists in this case and then you compare that to taking the same input mapping it directly into the model and using the model of the opera operation the abstract version of the operation so we're using the simple list functions as a kind of reference implementation to test the complex tree functions so this is a great property there's only one thing wrong with it it's false why is it false well quick check shows me right it's false because in this case I'm inserting the key one into a tree that already contains it and quick check complains that these two lists are different the rightmost one is the one produced by the model by list insertion and what you know list insertion does not object to duplicate elements in the list whereas the tree insertion will never insert a duplicate key all that means is that list insertion was not the right model for tree insertion but it's easy to create the right model all I have to do is delete the key before I reinsert it and then I won't get duplicates on the list side either and now this property will indeed pass and it's a great test of the insert function so one interesting thing about this idea is that it comes from a paper by Tony Hoare in 1972 he wrote a paper in apt acta informatica called proof of correctness of data representations and all you have to do is take the word proof and replace it by property based test and you get the same thing I really like being able to go back to old theoretical papers and turn them into effective tests I love that so now I've shown you five different ways of thinking about your properties how effective are they well I'm going to compare four of them the inductive properties are real interesting but I'm going to focus on the other four and first of all this table shows how many properties I was able to come up with by applying those ideas systematically to the API I wanted to test so there are four ways of building trees until there are four invariant properties there are four five operations under test no four operations that I had to write to post conditions for find so I have five post condition properties look at the metamorphic properties I could think of sixteen different ways of using that idea to come up with a property that is useful for testing model-based tests only five why is that because with model-based properties you only need one property per operation each model-based property is a complete test of the function you're testing ok so you can see what I told you about the quadratic number of ideas it really plays out in practice you can also see that model-based testing means I have to write less code how effective are the properties at finding bugs to find out I planted eight artificial bugs into my code three into insert two into delete three into Union and these vary from totally blatant bugs to quite subtle so now we can ask which bugs can these properties detect and it turns out here that the invariant properties miss five of the eight bucks that's not so surprising they only catch bugs that break the invariant of course there are lots of ways to do the wrong thing without breaking the invariant nevertheless invariant properties are useful because if you have broken the invariant you want to know that you don't want to discover it because of more complex property fails otherwise the other kinds of properties together could find all of the bugs and I planted so that's good but I want to look in a little more detail at how good each property was at finding bugs and to do that I'm going to measure what I call the effectiveness of a property let me just explain what I mean by that if we look at this property this is one of the post conditions I wrote for find it clearly calls find it insert so potentially if there's a bug in find or an insert this property might find it but it clearly won't find any bugs and delete or Union because it doesn't even call them so if we think of the eight bugs are inserted five of them obviously cannot be found by this property of the other three how many can this property find and the answer is two so I'm going to say that this property has an effectiveness of two out of three it finds two of the bugs two of the three bugs but you might think it could find by just looking at which functions it calls and then I added up the effectiveness over all my properties and averaged it out by property type and here are the results so the invariant properties they were 38% affected well I already explained that they miss a lot of bugs but they're still useful because when they fail they tell you a lot of the others the post conditions were only about 80 percent effective the metamorphic properties were rather better than that there were 90 percent effective but the model-based properties were a hundred percent effective I said earlier the model-based properties are complete tests and that's what we see here the model-based property for concert' for example can find every possible bug okay so we can see that the metamorphic properties well we can see that a ranking order and that suggests the order in which we might try to think of properties of course all this says is which bugs can be found by a property you might also care about how fast can a property find bugs and I measured that too so here's a slide that shows the results when I planted a particular bug in union one of those three bugs and at the top we can see that the Union post condition finds it the yellow part in the middle those are metamorphic properties and they all find it - and the green property at the bottom that's the model-based property which also finds the bug but what I've also shown here is the mean time to failure of each property so that post condition at the top on average that needs 50 tests to find this bug I planted the model waste well at the bottom requires 8.3 tests so it finds it much much quicker even though if you think about them as logical statements those two properties are logically equivalent but one of them finds bugs much faster than the other how come it's actually obvious when you look at the code of the properties you only have to read the first line of each property to understand why this planted bug is in union to find the bug you have to construct two trees T and T prime that will provoke the bug and then you have to observe but the bug really happened the bottom property is only quantified over T and T prime is only two arguments it's a model based property it checks that every key has the correct value the post condition at the top takes a randomly generated key as an argument to find the bug with that post condition you not only have to choose two trees that will provoke the bug you have to be lucky and choose a key that will reveal that the bug really happened now you might be thinking that the lower property contains more code it's probably slower to test yes it is but it's not 8 times slower to test which is what you would need to get the same performance in practice the model-based property is going to be much faster finding this kind of bug okay so that's one example I looked at all the properties that I found I measured their mean times two failures and then I took the average mean times of failure of all the properties that could find each of seven bugs y7 not eight because I didn't include the bug that screwed up in the invariant and insert since as I told you if you have that bug the other results are not interesting but look at this now and look at the last column in this table we can see that on average the post conditions needed 68 tests to find the bug the metamorphic properties needed about 60 and the model-based ones needed only around six tests there are an order of magnitude more efficient at finding bugs so what conclusion are we going to draw well it sounds from all of this surely as though model-based tests are really winning here right they're easier to think of some post conditions because we've got a systematic way of going about it you think of your abstraction function and you think of the abstract implementation you have to write fewer properties than the metamorphic approach you write less property code that's good you only need one property per operation they're the most effective at finding bugs and they find the bugs fastest and this has to do with the fact that model-based properties form a complete specification of the code so yay for model-based properties why did I talk about any others because model-based properties can have a big drawback the model can look too like the implementation and when this happens then you're putting your foot in the trap again and in that case metamorphic properties are the best alternative the great thing about them is that they do not require a model they're the easiest properties to write you just think if I change the input to this function how will the output change and they are the second most effective kind of property in this experiment so my recommendation is first of all think about model-based properties if you find that you feel you're writing too much code your model is too like the implementation switch to metamorphic oh but don't forget the invariant once again because if you've broken the invariant you just want to know that so there's a little bit more on this topic if you'd like to know more I've written the paper about it and the paper is coming out in trends and functional programming not this year's but last year's actually this is a mock-up of the proceedings because it hasn't actually been published yet but sometime before too long it should come out and if you don't want to wait for that I tweeted a link a little while back to a draft version of the paper so if you want to know more you can just search for me on Twitter and you will find this tweet a tweet is it a while ago but I hardly ever tweet so you'll find it easily and just in case you can't read it there is my Twitter handle okay you you

Channel: Code Sync

Views: 2,453

Rating: 4.9344263 out of 5

Keywords: Lambda Days, John Hughes

Id: G0NUOst-53U

Length: 52min 7sec (3127 seconds)

Published: Thu Feb 20 2020