Software Testing Theory + A Few Less Obvious Testing Techniques

today i'm going to cover some of the basics  of software testing now one type of test you   may know already quite well is unit test but  there are actually a couple of other types of   tests that you can run that are maybe less obvious  to you so i'll talk about them today it's a pretty   generic video so i won't touch much upon python  specific libraries i'm going to cover those topics   in more detail in the future though so stay  tuned before we start i have something for you   it's a guide to help you design better software  it's available at arjancodes.com/designguide   it consists of seven steps it's pretty short  and to the point so you can apply it immediately   to the things that you're working on and  hopefully avoid some of the mistakes that   i made in the past so you can get it  for free at arjancodes.com/designguide   now let's talk about software testing software  testing is the process of verifying that a   software application is working as expected if you  set up your tests properly it gives you insight   into the quality of your software it also provides  an objective view on your software that developers   can rely on whenever they need to make decisions  about how to improve the software there are   different types of tests different levels of tests  different views on testing and i'll try to cover   a few of those things in this video but before i  start let's first talk about what tests are not   tests are never a guarantee that your program is  correct Edsger Dijkstra dutch computer scientists   we're not related he once said program testing can  be used to show the presence of bugs but never to   show their absence let's take a look at an example  that illustrates this here's a very simple example   of a function with a few tests note i'm not using  any testing framework here because in this video   i'm covering testing more in general i'm going  to talk later on in future videos about a few   different testing frameworks such as the built-in  unit test one by Python but also for example   by test and compare the two to give you an  idea of what you can do with them now the   function that we're testing here is add  three which is a really basic function   it's basic for reason because i'm going to use  it to explain a couple of ideas in testing add   three does one simple thing it takes an input  integer it adds three and then it returns that   integer and then here i'm writing a few assertions  to test that the function is working correctly so   when i run this obviously i get all tests pass and  if i would change this to something else then of   course these asserts they're going to fail so  let me just change this back so that this is   correct again there's another way i could  write this function let's make an alternative so this also takes an int and it returns  an int but i'm doing it like this like so and now let me change the names here there we go and now when i run this of course also  the tests pass but obviously this is not how you   would implement an add free function because it  doesn't work for anything else that we're testing   it with if i were to add another test here with  number four then of course it was going to fail   but that's also where the crux lies with tests not  being able to prove that the program is correct   because obviously we wouldn't consider this  function correct because it's not but the only   way if i had just to prove that is to just add  more and more tests but for every test i could in   principle add an if statement here to also cover  that case and you could say that well okay if you   have an infinite number of if statements and an  infinite number of tests that test all the values   then you might actually be able to prove it's  correct but that's still not true because then   you're not taking into account side effects for  example what i could do is break the test system   by adding a random number generator that gives me  a different results only on a very specific date   and time for example on June 8 which happens to  be my birthday so then most of the times the tests   are going to pass except on my birthday and that  in short means that test basically can never be   a guarantee that a program is correct because  you're testing a finite number of cases and you   don't know anything about side effects but what  if you want to actually prove that a program is   correct can you do that well actually yes you can  it's possible using Hoare logic and when i say   Hoare logic i don't mean "all prostitutes like  to watch Netflix X is a prostitute therefore X   likes to watch Netflix" no i mean Hoare logic in  a sense that this is the logic that's invented by   British computer scientist Tony Hoare who in  order to avoid a lot of manual subtitle fixes   i'm going to call Tony the central part of Hoare  logic i mean Tony logic is the Tony triple this   triple describes how execution of a piece of code  changes the state the form of a tony triple is p c   q where p is the precondition that's basically  the situation before you ran the code c is the   computation that you're performing and q is the  post condition which is basically the state after   you've run that computation in short this means  if the precondition is met executing the command   establishes the post condition in a sense you  can view it as a very generic representation of   a unit test what you normally do in a unit test is  you provide some kind of input value you run your   function and then you verify that the output value  is what you expect it to be so that's basically   one case and with Hoare logic you can define  let's say a range of cases using mathematical   expressions for example you could prove that  a program is correct for a certain input x   when x is between 0 and 100 or something like that  the interesting thing about Tony logic is that   they have rules for all of the commands that are  available in structured programming so assignment   if statements loops etc so in principle if you  take let's say any imperative programming language   you could use Hoare logic to prove that  the code is correct at the time when Hoare   logic was invented that was at the end of the 60s  actually quite a few computer scientists believed   that this was the way to go in order to make  sure that you got correct high quality programs   of course nowadays we don't use logic to prove  that our program is correct because that's just   way too much work especially if code grows complex  you rely on lots of different libraries it becomes   really impractical to prove correctness of your  program all the time so what do we do well we use   unit tests and other testing approaches that i'm  going to talk about today to help us get a grasp   on at least part of how our code works so often  when we talk about testing people immediately   think about unit tests because that's like the  most straightforward way to start testing your   code unit tests and any other testing approach  where you run your code with a collection of   predefined use cases test case is called dynamic  testing another approach is static testing that   doesn't require writing test cases but involves  things like code reviews and even syntax and type   checking by your IDE and then you have passive  testing which doesn't even look at the code but   looks more at the side effects of the code so log  files that are being generated databases that are   being filled with data by the code you can look at  those things as well over time to see if there are   any anomalies or strange things that you need  to investigate for example suppose you develop   an application for schools that has students  and education programs then what you could do   as a kind of passive test is to check from time  to time that students are always associated with   at least one education program another way to look  at testing is to differentiate between white box   and black box testing white box testing assumes  knowledge about the inner workings of the code   and black box testing doesn't and unit tests for  example are a good example of white box testing   because you you know what the function looks like  you know what the kind of inputs it expects and   you can test those things if you've done  anything with testing probably you started with   writing those unit tests to cover those edge cases  but there are other things you can do as well and   maybe a not so common one that you haven't used  before is mutation testing and this is also quite   interesting what mutation testing does is that it  slightly modifies the source code of your program   basically introducing mutants and see whether  your tests pick these minor changes up   you could also see it as a kind of test for your  unit tests it works pretty well for common off by   one mistakes for example if you have a function  that uses a for loop that's accesses a list at   index i minus 1 instead of index i or you let a  loop run until i is less than n instead of i is   less than equal to n those kind of things and  what mutation tests often do is replace these   operators less than greater than by a slight  variation of them and then verify that the   tests actually pick that up there are libraries  that can help you do these kind of mutation tests   mod mod is one of them i will put a link to it in  the description below but let's do this manually   and see what happens so again i'm going back  to this add three function here and now when   i run the test up because obviously pass but  let's say i would change this plus to a minus like so and then when i run it again obviously  the tests are going to fail so in this case the   mutant that we created which was the program  but with the plus replaced with a minus   resulted in the test failing which is a good thing  but let's look at another example i'll just change   this back to plus again let's say i'm adding  another function that's called multiply by 2. like so and that gets an  end and also returns an end   and that returns x times 2 multiplied by  2. and now let me also add a test for this so i'm expecting that when i multiply 2 by 2 this  equals 4. so when i run this obviously tests are   going to pass because it's a pretty simple program  but now let's say i'm going to reduce a mutant   here where i'm replacing the multiplication by a  plus sign like so and now you see that the tests   still pass because x plus two happens to equal  also x times two so here we found a mutant   that apparently the test did not catch  and indeed here this is just one test if   i i would need to add more tests in order to  make sure that any errors are going to appear   when i test this program so let's say i add  another test here for the multiply by two   and then we're going to do this for three and  that equals obviously six so when i run this   now we're going to get an assertion error and  let's change this back to multiplication and then   the tests are going to pass again as opposed to  white box testing black box testing doesn't need   to know anything about the internal working of  the code it looks at what the software does not   how it does it an example of this is a testing  technique called snapshot testing that's actually   used a lot in web development nowadays and that's  technique where you take basically a snapshot of   the state of the system before and after a command  is executed and then you can compare the two   snapshot and see if there is any difference  or anything that you need to investigate   if you build web applications you can use  snapshot testing to compare the html and css   that your app produces and to make sure that  things work as you expect them to this is for   example what React's Jest library does while  i was explaining the example i talked about   mutation testing if you think of that in terms  of Hoare logic basically what you're doing is   changing the c the computation slightly and then  verify that given some precondition p that q still   is established there's another important concept  in determining program correctness and that's the   invariant an invariant is a logical assertion  that is always held to be true during a certain   phase of computation for example you could define  a loop invariant where you specify that the index   is never less than zero or something like this  and you can also apply the invariant idea to   software testing actually what you could do for  example is define the kind of property that you   expect to be true and then test for the truthiness  of that property in your test and if you know that   that property is true for a wide range of inputs  you can actually just generate a bunch of these   inputs and verify that the property is true this  is a testing approach that's called property based   testing there are libraries available in python  that help you do these property based tests one   of them is hypothesis i'll put a link to that in  the description below i'll talk about hypothesis   in a separate video in more detail but let's first  look at a simple example of how property-based   testing works so now let's see how property-based  testing in principle works in this same very   simple example so we have the add3 function i'm  going to add another function here that's called   remove three obviously that gets an int and also  returns an int and that simply does x minus three   still very basic obviously what you can do now is  start testing properties one example of property   is that we know that if we have a number we call  add three and then we call remove three we need to   end up with that same number that's a property of  the combined calling of add three and remove three   and you could do something like this in a simple  unit test so that would look like this let's say   add three remove three four equals four right  and then let's run this and this is going to   pass but now we're just testing one value we're  testing this property with one value and actually   when you look at this particular test the actual  value we provided we don't really care about it   we could pass it for a hundred or a thousand or  a million or minus 23 or whatever we don't care   we just want to verify that the property holds  so with unit tests you provide a particular input   and you measure what the output is and you verify  that it is according to what you expect with   property-based testing you're measuring whether  a property holds and you don't really care that   much about value and that's a difference between  the two so that also means that in property based   testing you can actually run those tests with many  values for example i could create here a for loop so now i'm running the same test  but i'm running it 100 times   and still of course all the tests pass but this  allows me to do way more than just test a single   input and output combination it allows me to test  specific properties and that's why property-based   testing is quite powerful another thing that's  really nice about property-based testing   is that with unit test as a developer you're  kind of responsible for determining what the   inputs and the outputs are going to be that  you're testing because each case is basically a   case that you test and that you have to define as  a developer property based testing allows you to   do things with random numbers so here i'm  using a fixed x in the range of 0 to 100   but of course you could also generate  random numbers and then do it with that   let's add that simple example here so i'm going  to import random and when i go back to my for loop   let's not use x anymore but then i'm going to say  x equals random and then i want a random integer   let's say between minus thousand and a thousand so now i'm also testing with 100 numbers  but they're randomly generated and this is   nice because it means that potentially you might  generate a case that you didn't think about as a   developer and that's why using randomized testing  in these very specific cases is interesting it's   a nice addition to your regular unit test i think  you still need also unit tests to cover the cases   that you are sure that you want to be tested but  these things help in kind of make the code and   make your testing a bit more robust this example  of property-based testing where you're applying   a transformation on a number and then applying the  reverse and verifying that you get the original   input is also called Bilbo testing or   there and back again but there are other things  you can also do with property based testing   for example you can verify that something won't  change let's say you're sorting a list you don't   expect a list to change its length so you could  create a property based test that just generates   a bunch of lists sorts them and then verifies at  the end of each sorting operation that the length   of the list is going to be the same another  type of property-based tests that are also   nice is things that are hard to prove but easy  to verify for example let's say you write a   data processing function and that gives you a  dictionary with fields and you just want to make   sure that none of those fields are empty after  you've called this function and what you could   do in this case using property-based testing is  that well the property you're testing is that   the dictionary won't contain any empty fields so  you just run the processing function on a bunch   of random data and then at the end verify that the  property holds they also see that properties are   not exactly the same as invariants and variants  describe things that don't change over the course   of a certain piece of code properties can also  only say something about the output such as in   the case that i just explained so i've mainly  talked about pretty low level white box testing   techniques in this video in the coming months i'm  going to explore libraries and python that can   help you set these things up and also look at all  these different techniques in a bit more detail   next to these lower level testing tools there  are also higher level testing processes such as   integration testing exception testing and those  also require a particular infrastructure such as   a DTAP street this stands for development testing  acceptance production i can talk about that stuff   as well on the channel let me know if you want  to know more about this also i'm curious are you   using unit test at the moment to test your code  and what about notation testing or property based   testing let me know in the comments below  i hope this gives you an overview of the   different types of testing systems there  are and how that feeds back into the more   generic idea of program correctness if you enjoyed  this video give the like consider subscribing to   my channel so you don't miss anything thanks  for watching take care and see you next time

Channel: ArjanCodes

Views: 39,223

Keywords: software testing, black box testing techniques, software testing training, what is software testing, software testing tutorial, black box testing, software testing methodologies, software testing tutorials, testing tutorial, software testing tools, software testing training online, python software testing, software testing python, white box testing, mutation testing, mutation testing python, mutation testing example, property based testing

Id: K47pr6lPxsA

Length: 20min 33sec (1233 seconds)

Published: Fri Sep 24 2021