Simulink and Stateflow Automatic Test Case Generation

hello and welcome everybody today we will talk about detecting design errors in Simulink and stage flow and also how to generate test cases automatically I did mute the microphones and we will have a Q&A session at the end of the meeting otherwise it's hard to organize such webinars so please no Daniel questions and I'm more than happy to answer them at the end of the webinar I will start with a couple of slides and then we will move to that lab just for those who might not know gamax is we are the local distributor for the networks so the network's is located in Natick near basta and they have offices in many locations all over the world however in some locations they prefer to have distributor who covers pretty much the same services as they do so we have presentations like this we also offer licensing trainings consulting pretty much the same services as the metals does if there are some really specific areas which we cannot cover then we can ask the network's and they always help us out in those topics so we've pretty much worked with them together seamlessly and the main topic that we will talk about is verification validation today and we based on customer feedback we realized that a lot of our customers are not aware of some of the key capabilities of the verification validation workflow so that's it that is why we organized this webinar but first of all why do we care about testing just a couple of examples Nyssa needed to go back a few million cars because a faulty airbag or maybe you still remember the Samsung Fiasco just a couple of years ago we want to avoid these kind of issues the next question is when the companies find these kind of errors and the answer is that most of the errors are introduced during the specification however they are detected only during the testing phase so our goal is to match the first graph as closely to the second one as we can so bring four earlier the error detection so we want to detect the errors as early as possible so how does design work to look like we start out from some requirements then we do the design specification and then we create our model code generation and then we deploy so we have two part we have the model verification and we have the code verification and we realize that on the model verification side usually people do all the testing however something is missing so static analysis on the model side usually is missing from the work flow of our customers what we will talk about today we will start out with automatic test case generation how can we extend the coverage beyond functional testing what we had also we will use now also we will do some design error detection and I will talk a bit about property proving why is it useful how does these tools work so simulation based testing we only test in a couple of dots but you can see here in the the entire design space compared to that form of verification in an ideal case could cover the entire design space the issue is that there is no ideal case so in reality we can cover some of these areas with formal methods and then we can include some of these dots in the design phase and the design space and this is how we can achieve a much better testing so this part is what is missing in those cases in most workflows now just a couple of examples why is this interesting for example in the case of area fine they ported some code from Aryan for however because Aryan 5 is a stronger rocket one of the in 16 variables overflow and the rocket self destructed or another example when the USS yorktown stopped and cannot restart because the speed became 0 and the engine controller faith failed because a division by zero error these are the types of errors that we can detect with formal verification methods what is similar in these two so the similarities are that both of these software's will exhaustively tested however still some failure occurred because they could not detect some potential error in case of formal verification we can use formal verification on the model level and we can also use formal verification code level and some of our customers actually are using quite heavily formal verification on the code level which will look like something like this so it finds the potential errors in the C code the resulting C code however most of our customers do not use the formal verification to us on the map model level and the reasoning behind it is that why should we use formal verification methods in the model level if we will do that on the code level anyway and answer to that goes back to my first slides we want to detect the errors as early as possible if we can detect such an error using formal methods during the design phase much earlier it's much faster and easier to fix there compared to if we find that there or only in the production code so that that is our goal for today so how does this work we already wrote let's save some functional tests and we get some concrete results but as you can see this coverage results are far from 100% so the goal is we already did the functional tests however we want to extend the test cases to achieve 100% coverage that is our goal so we start out from our logic we add the existing coverage to that and the two can automatically generate test cases to achieve 100% coverage now I will open up MATLAB and do this example real fast okay what do we have here so we have a state machine instyler model reference and we have a test harness for it so this is this is our state machine that we want to test in this case we are using signal builder block as a test harness and we have our functional tests here to get coverage we go to the configuration parameters coverage and here we need to enable coverage analysis also we want to make sure that in the results we select this enable cumulative data collection the main point here is that we will run similar several simulations and we want to collect coverage data from all of them okay twist with this wrong-o button it will run all 14 or 15 simulations and it will generate a coverage analysis report at the end you can see here in the background that still is compiling and running with different input data so as I mentioned here we will get the initial coverage analysis of after the functional testing so functional testing is always necessary based on requirement we will need to write these kind of steps by hand there is no way around it however the rest of the tests that are required to achieve 100% coverage that we do not need to write by hand and usually it's a huge effort to write this kind of last couple of test cases that cover the entire design space so yeah that does the hard part and this is where this tool becomes really handy now we are almost done with the simulations and we will get a nice report and we can have that in a PDF format or in an HTML format as well okay so we are done with the simulation and now we can look into the results the coverage analysis well here we have for each round separately the coverage analysis it's renamed it all of them around 14 but this is different run and here we can see the cumulative coverage now the next step we already have these analyses we want to expand so we already created some tests and we don't want to start from 0 to cover all the possibilities we just want to add some some tests to this so what the next step I will do is to to save this data so I save another coverage data and I will use this data for in the next step for the automatic code generation so the next steps are to set up the automatic code generation I open up the model the state flow model because this is the model that I want to generate test cases for and I go to into analysis design verifier options so here in the designer vibe verifier options I need to set up the test generation so I will select that I want to have mc/dc 100% coverage and I also set up that I want to ignore the data that is necessary so I want to avoid creating test cases for what which I already have that's why I exclude coverage from the previous run another thing that I need to set up is that I need to save the data because later on maybe I want to reuse it or I want to include in my test harness or something like that so I will save the data and I would generate separate usamos ok so now that everything is set up I go to analysis design verifier generate test cases model so now it will extend my existing test cases to reach 100% coverage that's the point you as you can see here the design verifier detected 15 objectives there it needs to generate test cases for these 15 objectives that's the the whole point and now this will take like a minute or so and we will get additional test cases as you can see this does take a while however compared to writing these test cases manually waiting a minute or two or even an hour probably it is worth the wait so it was able to generate test cases to satisfy all fifteen objectives and now we have a new harness model with the additional necessary test cases so it generated a harness model which has a signal builder block and inside the signal builder block we have eight additional test cases that are necessary to achieve 100% coverage that's the point the next step is that we actually want to combine these two together because if I will run only these eight test cases I will not get 100% coverage because this is only additional to the already existing test cases so what we want to do is combine the two together so for this I will use a command so similar in Korean we merge harness and you mentioned the name of the new harness Moodle and the name of the two models that need to be merged there's the syntax okay we get new model and if I open this up you can see that here I have all the test cases I actually opened this example of earlier so it's merged it again but you can see that here I have all my test cases and here I have the eight new ones and if I run this then I will get 100% coverage now I will not do this because it will take like another two or three minutes but you get the idea so you run this and you get the 100% coverage and you get a nice PDF document from it so our next topic is error detection so one of the possibilities was so we can use formal methods to generate test cases another thing that we can do with formal methods is to detect as I mentioned division by zero or overflow or that code let's see how can we do that just a moment I will close down some of these examples because there are too many okay and I will open up the trunk model hmm so the goal here is to detect hard to find errors some errors that are really easy to overlook even if we do a lot of testing or we are really confident that our model should be working properly so this is the model that we working on also state machine and we want to detect that code for now so analysis design verifier options and in this case in this case we will use we are interested in that logic so designer or detection that logic analysis design verifier and in these cases we will do detect designer or model so here the idea is that we do not need any test cases this error detection is not based on test cases it just looks our model and tries to find potential debt code in it so in this case we have 70 objectives this is quite frequent to have a lot of objectives in case of state machines they are really hard to to test and really hard to generate to write by hand test cases so we get 100% coverage that's why I'm using the state flow examples here okay is done with all the 70 objectives and we want to find them into the model the two objectives were falsified we have two places in our model where there's that code I want to highlight it in my model I open up the state flow model and here in red I see that there is an error however I do not really understand why is this this should work so I want to debug it just to be able to stop it I will add here a break point and run them along so it stopped here the first time when it entered the state machine and if I come here I can see that after five hold rate and in deck is one so this should work so why is this dead code and if we go to MATLAB or even better let's stick with simile and open it up in the model Explorer so we have alright and we can see it that the data type is inherited and if we go to the base workspace we see that old rate and increase decrease there are unsigned int so because of this if I go to MATLAB so holder it is one increase/decrease who quadratus five increase/decrease is one so if it's unsigned int and I've right increase/decrease / hundred then this is zero so here the issue is that I divide one by five and then I multiply it with ten however because of the execution order here so one divided by five is zero multiplied by 10 is still zero if I would move this 10 here to multiply increase decrease with them and after that divided by five it would go so this is a really hard to detect error the design was perfect then it moved from floating point to fixed point it went to ensign it and actually it is introduced an error so this is a really good example how can we detect these tools for error detection now similarly to this we have many other options we can detect as I mentioned division by zero as well not just that code so here we have digit vision by zero and we have integral workflow as well and if we well don't want to start it from here we also have the possibility to open okay so still running I will stop the simulation we can find the same thing in the model advisor as well we know that many of our customers are already using model advisor however they do not know of this feature because they don't have access to the design verifier so in this case in this case we will have some additional checks for the model advisor if if we have the Similan design we require okay so we have the model advisor here and if you select by product you can see that here is the design verifier and if I select the designer verifier I will have a couple of checks here so here is that logic here is division by zero integral over flow we can select these and run selected checks from here and generate a PDF or HTML report I will not do this step because it will take another couple of minutes but if you have similar in design verifier you can use it in this way as well for verifying your model and the last topic which I will only touch on is the property proving I will just close down the models real fast and I have one more example so here the idea is that I have my design and I want to prove something this is not proven with test cases this is proven with formal methods so if it says that this will be true no matter what the input and I can be sure that this is the case or if it is not true it will find an example and it will automatically generate the test case which shows an example okay no property proving and the spire of an example real fast well this will be a cruise controller and the issue with it is that they detected some error that when they decrease the speed they pressed speed decrease the speed increases instead of decreasing and they want to analyze this issue they want to prove that this is impossible but in this case it actually there is an error in the control algorithm of the cruise controller okay now here I have my model and I also have some assumptions so a means assumption so in this case the assumptions are that this is true and this is true and this is false and if these assumptions are true then I want to prove that this will be true as well so this is property proving and it works the same way analysis design verifier and proof property model if possible it proves that this is correct if not possible then generates an example a test case automatically so these are the three main topics that I wanted to cover today so we are already over time a bit sorry for that if there are any questions I am more than happy to answer them so if you have any questions please unmute your microphone and we are happy to answer your questions if there are no questions then thank you for tuning in if you would like to learn more check out our other videos and webinars as well or join us on our training events thank you and if you still have any questions then contact us on our web page or via email thank you bye

Channel: Gamax Laboratory Solutions

Views: 4,078

Rating: 4.9148936 out of 5

Keywords: matlab, gamax, visualizing, simulations, MATLAB webinar, wathworks, engineering, matlab for beginners, LiveLink, Simulink

Id: M4GZBvSqlRU

Length: 36min 42sec (2202 seconds)

Published: Mon Jun 11 2018