You Can't Unit Test C, Right?

and our first presenter is uniquely qualified to give this talk for two reasons the first one is that he has in fact written a unit testing framework and the second is that he has written a whole bunch of C in a context where unit testing is quite difficult so the fact that he has volunteered to give this talk indicates to me that he has probably thought at least a small amount about this which means he's qualified to talk about it for 25 minutes benno rice lives in Seattle he works on FreeBSD stuff and is going to talk today about unit testing and C please make him welcome hi so yes testing is a wonderful thing I could do the hands up who likes testing thing I could also do the hands up who like see things you like see I'll come on yes that's it C is the most fun you can have without safety equipment so I'm just going to go a little bit into sort of why testing and and just sort of because things like unit testing and stuff like that sometimes have bit of fuzzy definitions so we're gonna sort of start basic testing is awesome testing is awesome because it lets you know that your codes doing what it's meant to it's really hard to make forward progress on a code base unless you know that things down broken or that you haven't broken anything new and this goes especially for legacy code bases and let's face it if you're working in C is a high likelihood that you've got a code base that's got some years on it it's also hard to know that you fix bugs unless you test that you fix them and testing comes in a whole bunch of different flavors testing can be done manually testing can be done in an automated way which is even better because it's a lot less boring that way automated testing means you can feed your changes into some kind of test system it runs and it tells you yeah you didn't break it or no you broke it and preferably here's how you broke it and go fix it specifically automated unit testing is awesome so unit testing in the way that I'm going to describe it is aimed at testing what basic units of your code hence the name unit is generally defined as the smallest thing that it's practical for you to test that can mean different things in different environments it can end up being a function or a method but it's not really sort of hard fast well it could be a an expression within a functional method but the main rule of thumb is to keep the coverage area of that test as small as you can which has some side benefits too and that it kind of nudges you towards having small bits of code that are easily testable so what else defines a unit test the other main thing I keep in mind is I only want to be testing my code I don't want to be testing somebody else's code which generally means that if you're calling out in the library functions you don't necessarily want to be testing those you want to assume that they're going to behave in certain ways you can do this by using things like dependency injection where you hand in a thing to your code that does things for you and then during the testing time you can hand in a fake one that works the same way but is really just faked out or you can stub things out or mock tamaak them in various ways but we'll get to that later the key thing to keep in mind is you're trying to restrict the test coverage to the bit of code that's right in front of you as soon as you're moving into larger bits of code that's what things like integration tests are for and the last thing to say here is that it's fine to have more than one test for a unit if you're trying to test multiple paths through that unit so for instance if you have a happy path test you can say okay all of the if statements succeed everything's great but you could also have a second tested tested various failure paths and third and fourth so on and so forth so lastly let's talk goals with testing some people aim for 100% test coverage and that's fine but firstly coverage can mean different things in different contexts one notion of 100% coverage is that every statement in your program was run sometime during your testing but there's also a more stricter notion which is branch coverage which says that not only did every statement get executed but every way that you could have come out of a given if statement gets tested so for example if you've got multiple clauses that are odd together you need to be able to test that each one of those fired correctly at some point and that's kind of hard to do getting to a hundred percent coverage can be good but I don't feel personally that it's necessary what you need to work out is what kind of coverage feels right to you in a given circumstance that said if you can get coverage maps which you can get out of a whole bunch of coverage tools they're a really good way to find out it's where you're not testing your code properly or at all yet so generally just be pragmatic to do what feels right to you so having nailed down what unit testing is let's talk a bit more about how it's done generally you use a framework for example this is a a Python example testing a fairly generic implementation of fizzbuzz which is the world's best interview you can see that we're testing one category of thing in each test we're using pythons native unit test framework and so when we run it we get you know our tests passed but hang on this is about C so let's talk about C C has you know doesn't have the best reputation these days in terms of writing code that's safe some of this is deserved I mean it was designed back in the 60s and it was designed to be as close to bare metal as you could get without actually being assembler and because of that it's got some fun gotchas like you know the memory model being incredibly less a fair and very basic the other thing to think about is that while testing of software did exist back when C's showed up the methods that we used to do automated testing and unit testing today didn't which leads to things like you know C is statically compiled you take the source code it gets turned into some machine code and at that point it's kind of hard to manipulate it unless you really know what you're doing more many modern languages because they use things like virtual machines or much more sophisticated runtimes can do things like swap methods out from underneath you redirect things change things a much more easy way so Python and Ruby can easily change these even objective-c has mechanisms for the swapping method definitions out from classes at runtime and the reason that this of course the problems foresee is that it limits your ability to test within a compilation unit unless you split your functions out into two lots and lots of tiny small fires which might not be the worst idea in the world but it does cause they can cause some headaches the other thing that it does is restrict your ability use mocks and stubs so as I said earlier we only want to test the code that's right in front of us to do that you may need to replace or redirect some calls to other functions and see makes doing that within a compilation unit really difficult doing it to calls in another compilation unit but within the same linkage unit is doable that needs careful planning is the source layout there are some games that we can play with that and I'll get to those in a bit lastly coverage analysis getting coverage analysis out of C takes work again because again we can't sort of dynamically instrument things so generally you have to recompile your C code with certain flags that emit coverage data that you can then analyze after the fact it's not impossible it's just a bit more involved than then some other alternatives so let's actually show you how you can do this again we're going with contrived versions of fizzbuzz fizzbuzz just for people who have not had an interview at a place that these buzz interviews is an algorithm where you take a number and you return the string form of that number unless it's a multiple of three in which case you return fizz or if it's or if it's a multiple of five in which case you return buzz and if it's a multiple of 15 of both three and five you return fizzbuzz easy you've now passed all of your tech interviews so how could we test this unit testing as I said generally involves a framework unless you're don't like frameworks I don't know generally just to reduce boilerplate just how much boilerplate you can reduce and see frameworks is a bit of a good question but there's two frameworks I'm going through here first one is called check this is what a a single test looks like in check you can see we've got some macro boilerplate at the beginning and end that sort of mark up the test in other languages in other environments you could infer this stuff based on class inheritance or even just method naming but that's a bit hard to do in C so you need to tell the framework where things are so that that's not too bad but that's not all the boilerplate you need here's some the rest of the tests that we're going to be running I apologize for the syntax highlighting in this my app got a bit confused with all the macros going on and so you can see that each test is just focusing on one particular thing so here we get to the next bit of boilerplate check needs test cases to be grouped into test Suites and you have to do this on your own so there's a function that does that for you so you can see what they're adding our test cases to our suite that we've created and then returning it and then lastly you need a runner so this is just very simple main function runner which I took from the examples that they've got there you can get a bit more creative at a previous job we were using this this library in our development process and we had a run a binary that would look into shared objects and find test Suites that were in there and run those you can view a bunch of ways but assuming you've got all that done you build it you run it and off you go pass checks excellent so second framework that I was going to bring up is called ATF and QA these started out as a google Summer of Code projects in BSD originally was the ATF framework which is the imaginatively named automated test framework and the ATF tools package which had the runner system these eventually got split out into separate projects and ATF tools got renamed QR which is meant to be pronounced QA but isn't and Kure can all now run ATF tests tap tests and simple test programs its return success failure using exit codes so here's a simple test in ATF you'll notice that there's a bit more boilerplate around it the advantage of the boilerplate is that the head part lets you set things like descriptions which you couldn't do in check you don't have to do that there's an ATF TC without head I think macro that lets you not do that and that's yes I was right should look at my next slides so that's without the header part so filling in the rest of the tests again we're doing the same kind of test as before that font is very small I apologize just the boilerplate is somewhat different and that's the last bit of boilerplate that ATF needs because under ATF you don't have to construct a suite the test program itself is the suite on the other hand we do need to write a cure file which tells us where the test Suites are and what they the test programs are and then once you've done that you can run all that and you get you can list your tests you can get descriptions if you added them and then you can run your tests now you'll also notice that it stores results you can use that to do j-unit and HTML reports and things like that so next part you'll notice that in my fears buzz implementation I am calling some standard library functions like stood up and Malak so we need to work out how we deal with those because malloc is a wonderful place to discover problems in your code so this is where some of the real fun starts see binaries on most systems are dynamically linked statically linked binaries as opposed to them dynamically linked binaries everything that the binary needs is in the image itself everything's in there go binaries are a good example of this go links everything into one large blob and then you run that blob and it doesn't need to call out to anything else dynamic linking which is how most C programs work means that only the applications code is in the binary everything else is in a shared library somewhere else when the binary is executed a thing called the dynamic linker looks at the binary it works out what libraries it needs Maps them into memory and and make sure that all the calls that the application tries to make into those libraries point of the right routines this is an important because it gives us a way around C's lack of direct dynamism like this so let's explain what's going on here we've defined our own function called malloc this will override the standard libraries definition of the func malloc function in this malloc function we are going and using a function called BL sin to find the real definition of malloc and we're storing that in a pointer because then what we do is we test to see if our failure code variable is zero or not and if it's not we return that failure otherwise we call the real malloc that last part is really important because if you override malloc without actually allocating memory and return it everything will break so this is an ugly hack but it works and it allows you to get around the fact that in a lot of cases you can't test you can't stub our standard library functions or provoke the right kind of areas that you need without this if you wanted to test low memory conditions you would actually have to run your process out of memory so once we've done this we can add a test to make sure that we handled malloc failure appropriately by saying by setting our malloc failure code and checking that it actually did the right thing and we can run it and something goes wrong because as I said malloc is one of the fun functions to break and I wasn't checking that it actually returned something so the right thing to do would have been to actually check for null so as I said this is an ugly ugly hack what you should absolutely do if you're going to use this is that you make sure that anything remotely complicated in your stub functions is put into libraries and stuff so that it's easy to find easy to see and you're not just copying and pasting it everywhere so you can do something like this where you've still got some boilerplate around it but you can link against the library of test stubs you could have different test stubs a different behavior for different functions if you need to and you don't have to re-implement things all over the place in ways that could be go horribly wrong so coverage when your unit testing it's always good to have an idea of what your test coverage is and luckily for us this is another area where si makes a tricky in virtual machine-based languages like most things people use these days except for go and rust and even in languages whether the weather run time isn't just here's your stack have fun you've got hooks to emit coverage data in C you have to build it in how you do that varies but it generally looks something a bit like this I've added the dash dash coverage flag which tells it to build in coverage support and I've also added - a zero because when you optimize it often confuses the hell out of your coverage analysis after we compile you'll see some of these G CNO files they contain metadata about the metadata about the metadata that's about to be spat out so we can then run our tests and we get the G CDA files it actually contains the coverage data it's not exactly human readable at this point though but there's a package out there called L Cove which will post process that data and sort of collect it together and give you this human readable output but it also contains another tool called Gen HTML which when fed that will actually give you something you can use so as you can see we got full full coverage here so so we're done this is not branch coverage though but it's it's the best thing you're going to get that you're not going to have to pay lots of money for and the last area that I wanted to cover is testing main main is problematic because it's really hard to override you could play ugly macro games like saying that when you're testing is not actually mean another way you could do it is with things like a TF or you can write tests in shell so you can actually just execute your function so we can write a main function that has some allows us to specify various things and the output so we can specify a start point and a stride and a various things and we can test this by wrapping some shell script around it you can see that the structure of this looks similar to the C structure in a TF and you can sort of see how we'd expand this out into other tests for example testing the behavior of the start flag but actually what I find the best thing when you're when you're trying to deal with testing issues and main is just keep main as small as possible the more stuff you push out of main the more stuff that you've got that's more easily testable you could even have a goal of trying to make it so the mains only job is to you know basically call get opt-in powers command-line options and then hand the rest off to something else to do so finishing up C can be unit tested mostly just like with the other languages you should use some kind of framework just like your application code don't cut and paste stuff especially when you're doing ugly hacks like looking up dynamically linked symbols use testing as a way to push yourself towards good design this especially comes if you're in a position where you're testing legacy code you've got the opportunity to refactor if you can start refactoring towards something that's more testable then that's great and finally be pragmatic and do I get the job's done and with that I'd like to thank Chris for running this excellent mini conf and I'd also like to thank the organizers of Seigle where I presented this last year thank you [Applause] so we do have a few moments of questions is unware here at the moment the next speaker yeah if you can come and plug your laptop in that'll give us more time for questions so we can change over the laptops while we take questions so first one is here and they're all coming at you it seems like in a package you could provide an alternate version of gilepsy that is built especially for these stubs to pick out the different Lib C functions if you could yeah you could and that would actually be a good thing to do that one of the issues that you might run into there is that there are occasionally platform issues I mean as Chris said most of my stuff is more freebsd than and we don't use GMC we have our own so there are there are quirks in there there might be important but in general I think that's a really good idea and that's what you also find in things like valle grande well it uses LD preload tricks to override things like malloc so that you're going through its own thing and there are a whole bunch of debug Malik's out there that you can use as well oh we've got another question over here hi so check the vs. atf pros/cons should I bother to switch if I were starting my own thing I would probably go with ATF but there are multiple reasons to that one of which is that there's a whole bunch of ATF infrastructure within the FreeBSD project and a lot of the C code that I write has to do with that as I said in my previous job we did use check and it was perfectly fine so if you're already using one I would keep using it if you're not using one yet then I would say try them both and give it a go and I think we've got one over here hey so I generally merits are fragments of the converse a lot of different more always trying to be more portable I suppose you gave me from FreeBSD but I always like to make sure that I test on like geodyssey muscle even other environments without illiteracy or etc in which case there's always sort of LD preload hacks and snarky interceptions is you describe them all become ineffective so the pattern I've sort of ended up doing is I always you know use some macros in my file and I always just put them in a single C file and never reuse code because otherwise you end up having to include things or port things that usually aren't well portable between different Lipsey's and platforms or even compilers so I suppose the question is but if you are in a environment where you can't use LD preload hacks what do you do well LD preload gets you what is one way of getting around the the dynamic linking stuff the DL sim stuff that I did should be work in most cases that are using real-time linker I think I'm not sure if DL same as a POSIX standard or not but I think it is if you're in an embedded environment where you are statically linking everything then really you're just going to have to come up with a way to statically link in test harnesses to do that kind of stuff and again I would say turn that into libraries that you can you can reuse rather than cut and paste I forget how much exercise I get with when I run mini comes I'm over here now hi you mentioned compiling with the optimization of for car engine what about cases where the optimizer does something that is completely legitimate but a behavior you would assuming how do you test those cases in production and see more how does that compare with the other two that you I have not used it in their port could not comment okay it seemed off is the one the same the same standardized okay well I could turn that around and ask you how you have found it but we can do that out in the hallway if you like there'll be a talk in the hallway with Andrew Andrew Bala would hold forth in the hallway anyone else I'm gonna take that as a no more questions thank you again everybody please think better [Applause] you

Channel: LinuxConfAu 2018 - Sydney, Australia

Views: 27,454

Rating: 4.9506173 out of 5

Keywords: lca, lca2018, #linux.conf.au#linux#foss#opensource, BennoRice

Id: z-uWt5wVVkU

Length: 23min 8sec (1388 seconds)

Published: Tue Jan 23 2018

Reddit Comments

[u/fnoyanisi]

There is a nice framework called Unity

http://www.throwtheswitch.org/unity

Works for me