Code Checking Automation - Computerphile

We're going to look at an example of property based testing. [I'm] going to show quick check Which is the tools [I] work with and the idea here is your test software And that's very important to get software right? But thinking out all of the cases that you want to test is not something that is very popular with software developers actually And so [are] it's often not really done as thoroughly as you would like And what I'm going to show you is a tool that we can use for generating tests so that instead of running? Maybe you know a few dozen tests. You make up by hand [we] can run hundreds of thousands or millions and get much more confidence [for] [the] software really behaves the way that we wanted to Temple which is going to test that text message encoding, so this is actually a part of the 2g [11th] Standard, so it's got old [but] then [I] [think] we'll introduce text messaging that they were just making you for some spare bits that within the packages And they wanted to fit as many characters they could into a text message And there's only so much space so they realized that they only needed seven bits per character [from] so that [we] decided to use But when you put those seven, [but] characters into the messages you have to fit into 8-bit byte And that means that you can fit eight characters in 27 bites So of course the standard says you must do that, and that means the way you have a message [you] [want] to send you have to encode it by packing [a] [bit] closer together? And when you receive it you have to decode it and here I've got back some code that came from the company in France does that ankle Candy coating so let me just start by Showing the code that we got so here it is. This is that line code? and This is the code for packing characters. You can see there's plenty of it This is it comes from unpacking them again, and there's plenty of that And we're not going to look in detail at this The only thing I want to point out is that a lot of code and it's complex So it's quite easy to imagine that they might be an error in there So course we need test that And down here we have some test code and what it does is it says well. Let's just take a sample message Let's try [backing] it And then unpacking it and what it will call that tea and at the end of the day what we get after unpacking must be the same as the message to be started off with and then there's a little test suite here that says just try that for a number of different strings and with Somebody thought about this little bit and they tried all left that message from 0 to 10, so yeah Hopefully it works for these [strings] that will work in general what we can do now is just don't compile a program [just] done and now I can run that test code [I] do so Then we get [them] [out] [put] all of the [task] [past]. That's good it seems to work But of course what always demonstrated is it [works] for these particular ten strings, so you might wonder? if we give it another message, is there a risk that the code won't actually work as we expect, so That gives us a chance to use a test generation tool and when we use quick check I go back to code here actually have what we call it quick check property that is the way that we Write tests with property based testing we want something that's supposed to be true in general of our code here [I'm] saying for all messages or msGS not Just pretend that test which our list of [7th] values So they're made of the 7th character Then if I pack that message and one packet [I] should get the best Off with that says the same thing [as] the test [I] had before but it says it in general And that means that now I can use quick check to generate many tests [are] like and make sure that [coastal] works [let] me go back to my test execution window here, and I will just use quick check Call it and I'll give it that [property] that. I just showed an argument and we'll see what happened well After only a little more than a hundred test quick found an example in which the [prophet] doesn't hold and what you see here is first of all a randomly generated message It's a sequence of bytes which appears numbers here You can think of it as a message, and that message was not packed and unpacked correctly So after finding a random example like this quick goes on to simplify the [same] case as much as we can And we end up with this message. Which is just eight zeros So why does [quickcheck] say the problem occurs with eight zeros and not for example 70? well because if you have 70 still works So this is the smallest example that doesn't work and if I think which again You see that it starts off finding a different random example, but when it simplifies the test case we get the same 80 [this] is one of the key things about this kind of property based testing you always get The simplest case because you know it's the simplest that tells you a lot about what the problem might be [okay], so it seems to be something to do with being a characters long. So maybe it works for all other [lengths] What I can do is I can investigate when the coattails now by changing my property So if I go back to the property out here at [the] what it says for every list of seven that bike Packing unpacking [gives] [us] [a] result now. I'm going to add identification. [I'm] going to say for every list of seven bit byte Provided the length of the message is not [ate] That should work, [so] my hypothesis is that the length of eight characters problem? Let's just test now messages of other lands [I] do that. [I] just recompile test code and rerun could check Now [two] [thousand] [tests] passed. So maybe I was right. I'll just run a few more tests. Which I can do five [growing] test let's say 10 seconds [There] Are [0] It still doesn't work So now we find another example. [we're] packing and unpacking Failed this is actually consists of 16 zeros So maybe it's not just being of length [8] that matters may be any multiple of 8 is also the case that I fail But let's investigate that [I] can go back to my property and I can change it to say that provided the length Is not a multiple of 8 that's what that means then packing and unpacking the message should give me the same result back and if I We can buy that and run the test again Now quick check is Discarding some tests [with] the crosses mean so running lost every [dollar] [is] a passing tests or by now for [Dr.] 10,000 passing [test] the cross is a test that Where a multiple of 8 quick check is not found failing case after a hundred ninety-five thousand tests so that suggests that we're right But as long as the lens isn't a multiple of 8 then Code works But I can also investigate the problem more closely [I] know When the test is when the messages of multiple date then the constant of sales but doesn't always [sale] let me go back to my property and I'll just Take that out for truth and said we got nothing Instead of generating any listed as a message [I'll] just generate list of 98 and I was focusing testing on the case on suspicious of [its] every message of Mentaiko drawing. Let's find out [I'll] run these tests [that] everybody's hands and fate And what you know some of them passed [38s] past obviously many things are correctly encoded But this example is not found, or [is] this example now? I want to look at the random example that failed And [that's] that's [a] few more of them one of the nice things about Generating tests as you can see a lot of different examples of the same thing so is there anything in common between that first example This other one can see that full of different values of numbers those are all different wait a minute. [look] at the last number there [and] the last one here left one here. They all end in 0 [that's] very unlikely to happen by chance so maybe the code works unless the messages of multiple of 8 in length and the Last character 0 [let's] test that hypothesis. [I'll] just change this still [to] runtastic [Wednesday] put to say provided the last character of the message Is not 0 then the coach should work [ok]? So am I right in my postures let's find out Whoa [now] all of the test statistic? And after 10 seconds, we'll see we run to [continue] two thousand tests without a 0 in the last position [they] all passed So that strong evidence that the hypothesis right [okay]? So now without even looking at complicated code. We understand the problem very well We know it happens when the messages of multiple date. We know that happens when the last character [0] And if we think about it, we can actually understand. Why this is happening When we pack a number of characters in two bites? They're always going to be some bits left over What happens if we pack 7 characters in 27 bites? [what] happens is that there are seven bits left over? 70 bits that looks very like Eight characters where the last 10 [and] that's the problem it's a problem in the standard [car] distinguish between A message that contains eight characters with zero at the end as a message to [just] [contains] the first seven And it's a known problem. So there's nothing found it except I suggest not putting their characters in your messages We realized that we were all working with very much the same kind of programming language But because we each had our own combiner. We couldn't share any [cooked] We couldn't share our results And that just like a waste of effort so that led to the proposal to just Take the common core of what we're all doing

Channel: Computerphile

Views: 118,048

Rating: 4.9572401 out of 5

Keywords: computers, computerphile, computer, science, computer science, University of Nottingham, John Hughes, QuickCheck, Quick Check, Haskell, Professor John Hughes, robust software

Id: AfaNEebCDos

Length: 10min 38sec (638 seconds)

Published: Wed Jan 18 2017

Reddit Comments

[u/[deleted]]

He used Erlang to demonstrate QuickCheck.