CppCon 2016: Greg Law “GDB - A Lot More Than You Knew"

okay I guess we'll we'll get going so we don't run out of time so I'm going to kick off just with a bit of context because just because it's kind of fun so this guy hope you can see him washed out there but chap called Maurice Wilkes and I think has as good a claim as anybody to having been the world's first programmer because he was the first person to write code on a general-purpose machine to do something other than just prove the experimental machine they were building worked right in first person to write what we think of as a program to do real work and in his memoirs he says he remembers this moment which I think everybody in the room will probably be familiar with which is that that realization that is going to spend a lot of his the rest of his life finding and fixing bugs and his own programs because the term bugs and debugging hadn't been invented then and I mean I see a few sort of smiles around the room and I certainly remember that myself and a little while ago my daughter started programming in scratch and she said you know it's a lot of fun but I have to do an awful lot of times to get into work which was kind of cool because now like she could understand a little bit about what this company with a funny name that dad started actually does so could you know that was then back in the 40s now nothing has changed you know if anything the problems have just got much greater right so you know we yes the tools and everything of advanced and the you know we write in high-level languages rather than the ones and zeros that I'm sure Morris Wilkes was doing back then but the order of the problem is just like exploding right and the modern machine is running billions of instructions every second it's almost certainly talking to lots of other machines and lots of other processes and if you kind of think about it and those in those sensors I mean it really is the ultimate needle in a haystack problem right and it's kind of surprising that we can ever fix anything right all that anything ever works that people get frustrated with how sort of broken things are but actually I think we do remarkably well considering the degree of the complexity but the way that we one of the ways that we do that and we handle that obviously prevention is always better than cure and lots of things we can do to prevent bugs in the first place and that's like you know just to prevent that obvious criticism that's always the preference but whatever you do happens right and you will need some cure so what do we do when things don't work in I mean it as Morris found and there's my daughter found and as we all know it never works first time right and and and more often than not it doesn't work when you ship it so what do we do to find that needle in a haystack we have various tools and techniques that we use from printf to you know defensive programming so it's stuff like Coverity and all these things I'm only going to talk about gdb today because we only have an hour and it's like it's a you know it's a huge con you could do a whole conference on just this stuff and in fact I'm only going to talk about sort of gdv itself right so I'm not going to do this like a whole bunch of front ends and and things that are cool you can see the stuff that we make downstairs we're here you come and get a demo of our stuff I'm not going to do much of it here today you can also get very funky t-shirt like this if you want to see lime people are here JetBrains they have a nice front end onto gdb that works with our stuff as well I can also show you that so there's a whole bunch different front ends Eclipse and people use it within Emacs people I'm not going to talk about any of that stuff beyond what I've just said we're just going to talk about gdb itself right and just look at the features that it has in itself rather than funky ways to drive it this is basically a kind of mishmash of stuff that I've thought I've collected over the years right so I'll undo you know there's a whole bunch of interface is actually on to undo gdb is just one but it's it's what probably setting the most commonly using the default one and over the years I've picked up like with customers and you know are just on my own different kind of tips and tricks and things that work I'm not I'm no gdb guru right I'm sort of some way I suppose between newbie and guru there almost certainly be things that people in the audience know that I don't know I'm happy for this to be so slightly interactive so you know just like shout if you you've got something to add I've missed a bit or misled or or you've got a question no guarantees I can answer any questions but I'm happy to take them gdb has come a long way in the last few years I think probably I mean 10 years ago certainly it was pretty basic and pretty flaky like it might work for you if you were lucky long as you weren't doing anything super funky like when those threads or trying to attach to a program or whatever but it's come a long way that's changes both in 50 on Linux so in the Linux kernel lots of changes have been made that made things better and then there's been lots of contributions to gdb itself it's come a long way in terms of stability I think and robustness and just kind of working properly it's also come a long way in terms of features a lot of stuff has been added over the last decade or so and that's kind of what this talk is about one other thing that where it's come West progressed quite well over the years is the documentation is pretty good it's pretty complete fairly well written and if you just type info gdb or you just Google for gdb info you'll get kind of fairly complete list of all the stuff right but as always with these kinds of things it's about knowing what's there right so the purpose of this talk I'm going to do a whole bunch of demos and we'll see things in practice that's always more fun than slight but I don't really expect realistically that people are going to remember all of the weird keystrokes and machine notions and things that I've typed but hopefully you'll remember that these things exist so that when you need this in like a month a year whatever's time you can go yeah I think I remember that and go and look at the man page and figure out how it is that it works right so it's just knowing knowing this stuff exists so I did give this talk here last year but it wasn't a proper talk it was just a kind of lunchtime session hence we're doing again if you did see it last time I've got some new stuff so hopefully there'll be a bit of a refresher and some new stuff as well and some of it will be familiar so I think just following on from that point on that on the documentation gdb s is very powerful actually but it's certainly not intuitive right and a lot of the kind of interface is built up over time especially if you're used to a kind of gooey kind of visual studio kind of world can be pretty painful and I think it's it's it full I first heard this phrase said about VI actually is one of those things I'm talking about UNIX actually one of those things that's not ease of use is a misnomer gdb is easy to use but it's hard to learn right ease of use is not the same as ease of learning okay now this is probably where life gets a bit tricky because I spent half an hour of panic earlier on because whenever I try to mirror my screen between here and there my computer hacks which is less than ideal so there's going to be a certain amount of non-slick moving between between slides and and demos but just bear with me on that so the thing that I think like gets the both of the scores highest on both fronts together of least well-known but most useful is this thing called tui text user interface very useful very badly named because gdb is always a text user interface right I mean that's like one of the main reasons that it sucks but let's just just just have a look what we mean by that so um I'm just assuming by the way I should have checked lots of check before I started is anybody not used gdb at all we're all kind of okay cool at least no one's no one's admitting to that usually people don't admit to using it but so let me try and see if I can get my shell up all right so I've had to do some magic the mirroring of the shell but we should be good all right so here is a program that is just a tiny bit more than hello world so we compile it in the normal way right you compile it - gee - put the debug info in and then we have a hello world program we're going to run that in gdb in the normal way so gdb a dot out my program starts I can type start just saves a tiny bit of typing rather than doing break main run you can do start which kind of puts a temporary breakpoint on main and runs it so there we go and like this is not the good old gdb interface that we all know right and and like the main problem I think with this is that you just lack context we have a single line where we are but you know especially if you're navigating code that you're not super familiar with which often when we break out the debugger by the way that's type one of its main jobs race is to help us understand code and it might be to help us understand code that we wrote that's not dude not behaving as we expected it to but more often than not it's actually code that somebody else has written and we're trying to just understand what the whole thing does and if you can only see one line at a time then you know it's kind of lame yeah we can type next to go to the next line and so I kind of see what line six and seven does but it's still kind of lame you can type lists list gives you bit of context yeah but you know I still need to look um at line seven and I need to look at where line seven is on there and it's just it's it's not so good it's not so good so here we are very kind of you know certainly a text user interface but this is called the CLI writing the command line interface not the text user interface not Tooey Tooey drags gdb forward screaming from the 1970s to the 1980s and I'm going to do and I can switch between the two you can do can do it at startup or you can do it interactively so if I go control X a I can to the cursors interface right now I can see that context and now pretty it's not it's also not very reliable just like completely contradicts the earlier point I made - II just I don't know I mean maybe it's just me but it does break a lot sometimes you can recover sometimes you can toggle so you can toggle between the two right so control XA again and I toggle back right and sometimes you can toggle in and out to e and it fixes itself sometimes you can do control L to get it just to repaint the screen and things get better sometimes it's just like it's just broken and you have to stop gdb and start again I've not put it that way but I still like it despite all those caveats because it just gives you that context and yes you know you can use better front ends right like Eclipse or sea lion or you can use it from Emacs or whatever and get that context but it's not always so convenient I you I wouldn't have to kind of a whole bunch of stuff around you to do that whereas you know you could be SSH din somewhere to some server somewhere that's like a minimal server and you can still do this stuff right so it's still useful and and so when I type in my normal kind of command so when I type next it goes next now I said that can't be a bit flaky and unfortunately one of those things that's unreliably unreliable so I haven't found a demo yet that causes reliably causes to e mode to mess up but you can see beginning of kind of what's happening here right isn't really messing up this is just fair enough my programmers printed hello world and and hello world has appeared kind of slightly messing up I might my tui right and now okay that's only hello world we can live with that wasn't pretty in the first place it kind of parable but obviously if your programs doing more stuff but printing more stuff at the screen it can get pretty trashed and so often not necessarily always but often I can just type ctrl L and it repaints the screen right and that can be helped and what else can we do we can actually it's good you can actually split it into two windows so if I go ctrl X 2 now I can't because I tell you well I think this is probably because I'm I've got this weird split screen thing no okay alright so ctrl X 2 gives me two windows okay so I've got the sauce top and I've got the disassembly there if I do ctrl X 2 again and cycle through them now this is quite a useful view if you're ever unfortunate enough to be debugging something that you don't have the source for right so now I can see the registers and and the disassembly at the same time and I can kind of cycle through them I go control X to back control X 1 get back to screen one thing I find in in too emo just took me a little while to figure out is I think we all use the up and down arrows to go through the command history right because there's typing and we wants to type the same thing again but when you do that in to e I hit up arrow just moves my source around which is like really tedious but I then discovered that you can do ctrl P for previous I can get my previous command so ctrl P and ctrl n which kind of works anywhere normal gdb CLI but I never didn't it that I was just doing up and down arrow if you're in tui mode you need control P ctrl n to get your previous commands and that's handy so try unlit ly to get back to my slides alright so I'm going to just do this messily I think this isn't going to be pretty but so think we've covered everything there so control X a is probably the main one go to and from you can do gdb - GUI when you start gdb in it will just jump into UI mode - e mode but you can always do that control 8xa switch between them and I went to the other things there all right next up and this is few years old now but very useful very powerful actually gdb has a Python interpreter in it and it's it's pretty complete so if I come back here so and this is since like version 7 I think and she's a few years old now so I can do price I can just type a line of Python here so I could type and I print hello world print lol right there's quite complete as I say you can do most things so I can actually type a small Python script and write I'm going to come out to you I made because it makes it clear what's going on hang on minute that was my photos finger problem anyway so so Python I can type a few lines of Python here and you can like import all the standard libraries and I can go and they I am running on appeared that yeah and the reason I'm showing you this is two things one getting the pit of jeebies not often that you saw can be handy but just to kind of demonstrate that this is look this is not you could there's also the shell command from from from from gdb right so I can do shell and then I can run anything from from my shell right and and so if I say run shell PS what that does is Forks sub-process runs this runs fashion it runs PS does the output now returns that you can be prompt when I'm doing Python it's not like forking a Python process and going running that stuff it's running it from within the gdb process itself and that's quite useful for reasonable seen in a bit so you can see that here right so if you see a gun shell PS I've lips I've got my gdb process there six two four three if I look at the output I am running on pit six two four three just within this within the process the second reason I wanted to show that import OS thing other than just to show that you can do it is in my experience this is the most common way I've seen screwed up gdb installations out in the wild because it's it's like that worst that worst thing of mostly working right so so when when you compile gdb from source it's fairly straightforward to compile as you'd it needs a bunch of packages that you need to install but once you've done that you know configure make does the usual thing and it makes a gdb executable that's mostly pretty self-contained so you can copy that gdb executable from your workstation onto your server or your target or wherever it is and just use it right and mostly works and that's good except the Python modules stuff will almost certainly all be broken either subtly or catastrophic ly if you're unlucky Suckley and and so because what happens is it you when I go import OS is importing the OS python module from my computer right and if that doesn't version match with the GD p-- with the python that's built into your gdb will get weird problems and it doesn't like happily tell you are no version mismatch it just kind of like depending on how different the instant that the distro you're running on is from where you compiled it it'll work somewhere between you know just a few things being broken to nothing working at all so that I think is the most common example I've seen of broken gdb installs out there so if you start to use the Python stuff be aware if someone's compiled duty be on a different machine it probably won't work for them now there is a way you can jump through some hoops to build a version of gdb that's like self-contained and has a directory that has all the Python stuff in it and that's actually what we ended up being forced to do it undo to get around this problem ok what's next so sorry about this alright so uh where are we so yes oh that's right sorry yes right so so it has a the GD p-- iceland built into gdb is more than just a Python because it has this really useful module that you can import called gdb which gives you pretty tight binding between the Python and the gdb now this is something that it was released in version 7 of gdb but that was kind of like version 1 if you like of the Python integration of gdb and it got better over the subsequent versions I think by about version 7.6 gdb I think the kind of pace has slowed quite a lot because most of it's done right so early versions of we've got G to be 7.0 it'll have Python some of the integration with gdb is quite limited you know although there's no or not all the features are there if you have seven point six seven point seven it's pretty complete anyway let me show you what I mean by that so so I'm going to go back into to you I made use to make this little bit clearer so if I go python so now if you just type some python at it right gdb executes and what that does that takes a string and that string is just a gdb command to run so when I hit enter here Dada goes next right pretty cool so you can now start to imagine how you can script pretty powerfully you know debug whether it's been something inside of test suite and whether you've got some intermittent failure that happens only very rarely I'll talk more about that in a minute and you can script it quite quite nicely um but it's more than just having that that like CLI interface onto onto gdb you can actually interact with gdb through the Python right so for example if I go - VP equals so I'm going to create an instance of the breakpoint you'd be breakpoint class and you give it the noodle kind of line spec so if I file name : line number let's put it on line nine so if I do that I'm going to create an instance of the gdb breakpoint class it's going to be bounded of the value the identified VP and if you watch on the on the top here is up and I hit enter pointless appeared all right so now her breakpoint I can now control that breakpoint right so I can go now another thing to notice is by the way when you create a variable and the pison it's kind of long lived this of global so that's that BP variable there it is still there so I can do another Python script and I can see it and I don't know we can go enable equals false all right if you watch that breakpoint is hit enter now disabled you can pretty complete you can like attach commands to it you can do all the kind of pretty much I think everything I think you would normally do conditions all the rest of it on the breakpoint there and I can I can look at the breakpoints with just I think I go I remember this all right and I can we go back out to you I may just forget the clearer so print yeah so I've got VP's is actually a list of all the breakpoints and you have one there so there's a list of one what's tupple anyway one thing but but then I can so I go a Tyson print vp0 dot location okay so I can interact my breakpoints and I can also interact and you know obviously watch points and all the other kind of cool things in gdb I can also interact with the program that I'm debugging and it's state by what gdb calls the inferior and so I do that using this parson eval thing so let's go back to um as you can see where we are so alright so I have a local variable here I which I can tell from code inspection is going to have a zero value zero but let's pretend it's not so obvious I can do the usual gdb thing print I there it is and in the Python I can just write some Python now and I can say I don't know I call it something different just to make it clear what's going on but I equals gdb pars and a vowel right to positive a lie now give it a string which is a bunch C code well actually I think technically whatever language my program is written in and gdb will pass it and evaluate that expression so I'm just going to do that right so the expression is very simple just going to get the value of I that's going to bind it to that Python variable so now if I go Python print is that okay and and clearly I could you know do the bucket follow pointers and operator overloading and all the usual stuff there if I am do next a couple of times now an eye net when I hit next again now of course I is going to increment fine if I control P to get that line again it's going to bar eyes not bound like dynamically to that variable I right I did I called positive owl some time ago when I was zero so if I look at bar eye it's still it's still zero it doesn't change when my debug least thing changes so just be aware of that but you know obviously I can just parse any vowel it's more and that was up later there's lots to that read demand pages with the info pages rather and and you can see there's lots of there's lots of Python is one other thing I'm going to show in a minute but we'll come back to that okay so oh yeah no the other thing the inbuilt help is quite useful so I find it useful anyway so I often do Python let's come out to you I Motors it can get a bit messy Python you know you can do that online built in health with heightened right and it will give you the doc documentation for the objects so if I go place and help gdb it gives me all of the documentation for the Python modules that you can see everything's there you see it's actually quite a lot of stuff right and then there's all these classes so and obviously we can do the usual thing like I couldn't just want to find y out but break point then there's all the stuff in the break podcast so I find that quite useful it is all in the info pages as well but it's always nicer just to get it from where you are okay okay so the other thing I'm going to show on Python is the pretty printers which is definitely falls into that not easy to learn I'm not sure they're that easy to use but they're quite useful so so you know we often have data structures that are complicated right and and we have ways to visualize them that are kind of make life easier and often what people do is you might have a pretty printer function that you call from the gdb command line right and that kind of works so let's let's let's show that so what I mean so I'm going to a new program here now I'm going to do this one so I have a pretty simple program oh I should have given this apology at the beginning on a terrible confession I'm not really a you know a huge user of C++ I tend to do more kind of kernel e lower level stuff so most of my examples are in plain old C but anyway whatever so um I've got the structure right it's not very complicated but you know it's complicated enough to check the demo so I'm going to compile that I'm going to run it in gdb let's go forward a few lines nice the now st has been initialized so if I print ster student there's my structure all right everything's there this is not so complicated that you can kind of deal with it but obviously you know the more complicated your classes and your script and data structures get of the more that the less readable that becomes I quite like and I usually do this with my GTP init file I quite like a set print pretty on I just find that slightly easier to read but you know it's still not like super easy to read right I need to figure out what that time spec is telling me now my program has in it a do have a pretty printer thing dump students which is printed in a better way so I can just call that right so I'm gonna minute very so I can call dump student right that's okay good I can I think I've just realized as a problem or pretty printer but anyway you that's fine we'll overlook that right so that's that's fine okay but the problem is I'm calling a function inside the inferior which you may or may not want to do right if you're debugging something where your data structures are all screwed up calling functions in the inferior that walk over your data structures is very probably a bad idea you know you did your debugging so probably all bets are off so calling functions in the inferior you know it's kind of it's a bit dodgy probably don't to be doing it so I can do with these Python pretty printers I can do a bit better so I've made a price and pretty printer for my student structure now it is a bit fiddly there's quite a lot of boilerplate code to write but you know it's pretty useful right and it's flexible it's powerful you can do what you need to do so we've got a bunch Python here you kind of create the class to bind that class to a type given a regular expression so this is my this is my my my pretty printer class of course I've just got a two string function usual kind of Python stuff and it walks over this this this value that I've given it in a constructor and it's like a dictionary and I can look up all of the the members of the structure in the dictionary like that and and then I need to do this to boil up late stuff is pretty painful but easy enough to copy-paste and and this is the bit here that binds it to a regular expressions I've just at any type that is called exactly student I'm going to bind it to that but also you could be a bit more flexible particularly with you know we've got C++ classes you know class hierarchies and things that could be quite useful and and so now I can source that right so if you remember when I go print stuff I've said print pretty on so it's kind of bit clearer but still not great if my source pretty high now I print it right in pictionary so that's kind of useful oh one other thing on them here's so and what they've done with the with the STL and other libraries there's a bunch of pretty printers like installed by default so if you print iterators and stuff like that then you can get kind of nice output and it's kind of the way you know rather than hacking into gdb itself all of this kind of knowledge of libraries and everything else they've just got they're pretty printers in the distro yeah I just yeah okay a minute let me get that back so be nice to show this again and sorry yeah question is all right that's that's what Python pretty printers do right so the Python pretty printer is kind of that's so sorry I should repeat the question so how come the Python stuff gets invoked when I just type print at the gdb command line right and that's that's that's kind of that's what it is that's what the inbuilt gdb Python pretty printers do so they kind of override the Python print command it's a way to hook that Python print sorry let me say that again cuz I said the wrong word which is doubly confusing it hooks the gdb print command so when you type print in gdb the command is hooked because of that because I've added that pretty printer so now gdb instead of doing what it would normally do to print something it goes and calls my Python so I have defined and it returns a string in that nice way that I define the printer that the Python to do yes yeah I'll be honest you can probably tell I just copy pasted this until it worked so it may be may be possible to do it more simply but yeah so this is just so that the those those so essentially what you're doing is you have to create one of these pretty printer objects which was so I first had to find a class and then I've created that an object of the pretty printer and bound it to that class and then I've bound it to every type that matches that regular expression a student so yeah I'll be honest my copy pasted it from the interwebs and it works does it work with templates I don't know yes yeah I would have thought so so so question is could we could we like invoke other logic and things and and and in order to kind of yeah I mean I think it's pretty just-just-just Python right so you can do what you do in the Python and you can bind back in to the gdb you know Python binding stuff so I mean I've not done exactly what you asked I've not caused it to do an X when unto a print I know and I'd advise against it I'd advise against making it so that print causes side effects but you know I think you have got enough rope to shoot yourself in the foot with there if that's what you want to do yeah you you get no you I think I think you could do it like I say I would I think I'll come to actually you can you say the other thing you can do it you can it's not just print actually you can hook you can create your own commands in Python and that you can then just hyper gdb and it will just go off and execute your Tyson you can either hook existing commands or you can create your own and I would probably recommend doing that rather than if you're going to do more than just print values out if it's going to actually move around and change the state in the debugger I would advise you create a new command to do that because if it doesn't confuse you or probably confuse somebody else later on if you're doing that but I mean you know I sir that's Who am I to say right and okay all right okay as we're going to do some different stuff now which is reversible debugging which is kind of a subject that's close close to my heart and but I'm just going to show him bill gdb reversible debugging because it's pretty good what it does and really what it's just you know sure why why it's useful so if you as I mentioned before debugging is really all about thinking how did that happen Brian Kernighan and Rob pike in their book practice of programming I talked about debugging great totally in terms of backwards reasoning right and thinking back from the failed state to kind of figure out how you got there and most debuggers just let you step forwards which is like you know if it would be really useful to step backwards is stepping forward to the opposite of useful probably bit harsh but but it you know it's not necessarily what you want so we want to be able to go back and see what happened rather than like guess what happened which is what we normally do when we're debugging and this is particularly useful with intermittent bugs so let me show another little demo here I could just do that right so here is my and my program will start as this one yeah right and so it's a bubble sort right does what you'd what you'd expect so you know it's pretty pretty minimalists just got an implementation of a bubble saw and I get some random data when I sort it and I've precompiled it in here actually let me just make sure I got the right one there so to do now I'm going to see my house so if I run it it doesn't actually do anything but I happened all it's doing is sorting the random date getting random data and sorting it but I happen to know that there is a an intermittent bug in this and if I run it a bunch of times from the shell Joe Giusti was just a little bit shell I could remember yeah so long as this is the one I think it is oh all right so we run a bunch times and every so often that happens we've all been here right so it's dumped a core file so that's something that's good let's have a look at the core file I've got a bunch of Ulthar let's get them this one so it's this one here two six five two nine so I can go gdb - see this is like one of the most common things I think tw2 stores to look at core files if you so you can see the back-trace guys just behind us here who have better ways to do this kind of thing but we're just using regular gdb here so so so I'd load up the core file just with - see in older versions gdb you need you to give the executable name but the ex-people name is stored inside the core file so it will just find that for me and and all right I've got a seg v and there doesn't seem to be any code there so well let's have a look where where are we so print I could print a program counter like that you can see on that address we're actually just hoping that a bit of wait time at the X command in gdb lets me examine memory by dig dollar one that's going to refer back to this value here right sorry you can't see like I do dollar one is going to refer back to the value there so so X dollar one is going to examine memory at that address Oh x5 a four blah blah only that address does not exist so my program has branched into hyperspace let's see what else the most common thing you do in when loading a core file is get a batteries right because back trace is either closest really that most you buggers get to directly answering that question of how did I get here how did that happen and and you know it's just based on what's in registers and what's in memory and it can kind of walk the stack and give you a bit of a guess basically of how you got to that guess is usually either correct or obviously wrong and in this case it's obviously wrong so there's garbage on my stack I've smashed my stack it looks like and so like how do I get right I've got I just got nothing to go on and in in real world you know to cut me this is a simple small program so you can just come pretty stare at it for long enough and you'll and you'll see the bug but you know you're working on real-world code base of millions of lines of code and you get this occasional problem what what do you get you've got you've got nothing right you might have some log information somewhere but that's probably a long time ago and very hard to debug so what I'm going to do is use a little bit of funky scripting and-and-and gdb is reversible debugging to do a bit better so I'm going to load it into into GDP and so G to be reverse debugging so since version 7 it's had this feature reverse debugging if I start to program I can enter this command record and now when I continue everything that the inferior does is that this isn't recording my interaction with gdb this is recording what the inferior does right when I continue everything period does is going to be recorded right programs run and it stopped that's not a surprise it we know it works most of the time and this was those occasions when it worked so that's not really helped me so what I need to do is script this so that I run it a bunch of times with the recording on until it fails and then I can see what happened right so the way I'm going to do that I'm going to add I'm going to put a breakpoint at Main I'm going to put a breakpoint at underscore exit which is that where it goes through when everything finishes normally I'm going to hook some commands off of that so you can hook a series sequence of commands to execute every time a breakpoint is hit breakpoints all have these these low typically small numbers that you can reference them by so I created a breakpoint at Main which is great point too and I created a breakpoint to underscore exit which is breakpoint number three so if I go command two that gives me a list of commands that will automatically executes when breakpoint two is hit so the first command I'm going to tell it to do is to enable the recording and the second command is just to continue and now I'm going to add commands to break point three which is the exit point which is actually just to rerun the program right so so it's going to ping pong between these breakpoints get spray point three which is normal exit run it again get to start enable recording continue I get to the clean exit repeat I've also hacks this to turn off pagination and confirmations and things so makes it a bit easier same point just run here we go I didn't take radon that was lucky open you have to wait quite a long time for this it ran out but a few times and now it's stopped it didn't get to the end point so back trace looks much like before my stack is messed up so I have limited information on how I got here but now I can do something a bit better right now I can go reverse step I to step high in gdb will step you forwards one machine instruction so reverse step I will step you backwards one machine instruction which if I do that now that I've gone back into sensible land right I'm back into my into my code here let's go into tewi okay so I'm at the return statement from Maine so you know all signs point to spat stack smashing now I still need to figure out how did that happen right how did my stack get smashed well let's have a look I mean first of all let's just just make sure that I'm I'm correct and so just to sum dis a sort of disassembly gives me disassembly the current function I can see in that little arrow the bottom tells me the a machine instruction I'm on and sure enough I'm I'm on a return instruction right so I've gone reverse one instruction it's gone back to a route machine return instruction now one x86 does when it returns is it fetches that the stack pointer points at the actual return address right so I can look at the stack pointer like that and then I should be able to look at what's there so examine that and actually that's so and I think I need to get it time I actually know that it would wanna do all right I'm gonna do this T I'm about to do that and it just cast it to a long star star and and we'll look at there look at it here so is that valid memory no right so the cop of the different address since last time because it's kind of non-deterministic bug at the top of the stack contains a pointer to invalid memory hence when I execute that return instruction it's going to blow up right but once again how did that happen alright how did I get here how did my top of my stack contain get to contain bad data well you know I could do a few things like maybe do binary search and time jump back and forth try and find what's gone wrong but we can do a bit better we can use a watch point right so watch points typically used when you go forwards and it hooks into the hardware usually to run forward until a particular variable changes value but I can do those in Reverse as well which is super cool so to do like this so so I want to watch the top of the stack top of the stack is that dollar one so if I go watch for the simplest way to do it is that all right and that's going to watch that location in memory actually what I could have done is watch - L and probably done that and that probably would have done so - knock - P - L the location that were done the same thing so just beware with watch points you're watching an expression not a value of memory which if you're debugging memory corruption bug stop it often you're just looking at that value of memory because like it's some aliased pointer or something else that's the buffer overrun or something so the watch - L actually just watches the address if you watch the expression it kind of tries to be clever and when you go out of scope ill disable the expression and if you if you're watching something that might change like a register if your expression is based on a register we'll have to do a software watch when don't have time to get into that but it's in the slides anyway I've put a watch point on the top of my snack so now when I reverse continue if I didn't have stack smashing what I would expect to happen is I would go back to the call site or the very beginning of main because that's when the return address should get pushed right just before just by main is called but as I think I've got stack smashing here it's probably gonna happen sooner than that so reverse continue and sure enough what are the chances it's in my array assignment so I'm assigning to the array and I seem to be overwriting the return address of my stack print I or is 33 I can go what is array and it tells me that an array array is an array of 32 Long's and I've just indexed the thirty-third entry why have I done that or I can see here I've got percent signs of of course here's the problem right so size of array is the size in bytes sizing elements so you know schoolboy error but you know the point is clear enough right so by by by using that little bit of scripting and the gdb inbuilt recording we were able to go back and see what happened rather than have to try and guess and particularly with non-deterministic intermittent failures like that that's that's a really powerful thing to be able to do yes couple of questions here could I look at the core phone and go back no because the core file has like the information when the reason this is hard when the program is running the computer is running it's destroying information all the time every time it writes to a memory location or to a register what was there before it's gone so in the core file that information physically has been destroyed it doesn't exist anywhere in the universe so so yeah now you can there are things you can do you could use our live recorder product which gives you kind of a cool file but with that historical information right so that then enables you to do what you want to do and and another plug for the back-trace guys who are just behind who who do stuff this kind of core file debugging but extra stuff in it basically um but yeah you can't do reverse continue off the core file yeah yeah whoa you say that let's just do that again and enable record continue this is a very very small program right and you're but watch I've got my watch point set let me remove the watch point continue I took like half a second to run right because what it's doing underneath is it's single stepping instruction by instruction so it runs I mean it depends on the test case but typical kind of measurements for that are the slow down when you've got holding compared to native it's about 50,000 times slower so you kind of need problem with inbuilt do to be reverse debugging is you kind of need to be able to put a breakpoint quite close to where it's going wrong and then enable the recording if you know how to do that you're kind of you know 90% of the way there already but that's said no it does get used right and do know people who use it particularly when trying to understand someone else's very clever code you know someone's been very smart and tried to optimize something it's quite good it's also very actually it does make debugging machine code like almost doable because you kind of get trouble debugging machine codes you get very lost very easily and you can go back to where you where and and and so it you know it definitely has has use cases but there are other alternatives and other alternatives no just are alternatives on it anyway um at least it would be there we go so show you some of these these so there is there's a number of things right and this list probably isn't it but certainly is an exhaustive list this this is evolving all the time there's quite a lot of these technologies coming around now to these inbuilt reverse debugging which we just saw inbuilt 2gd build in built to gdb it's on if you're running on a newer Intel CPU you can go record was it record record be trace yeah and that uses there's hot there silicon in the newer Intel CPUs that will like cause it to store the targets of all the branches as it's running in memory and gdb knows how to drive that stuff and so that's kind of cool it runs much faster than inbuilt gdb record still surprisingly slow measurements that I've done it's about 100 times slow down which is surprising I think the newer CPUs are much better and and but it you know it and it's come back more usable in that sense however it only gives you a history or by the program counter so you can do reverse next but you can't look at the program state so I mean you know that's useful but it's not as useful as being absolutely program state and you certainly can't do things like watch points and go back and stuff right so it's not really reversible debugging in a sense but it is a kind of log of where you've been you do need a new enough Intel CPU although goes back a few years now and a new enough colonel there's another thing called RR which stands for record and replay which is much faster that's the fastest option actually that you have depends on the benchmark but typically around like less than 2x slow down and that gives you full data visibility all of the things you can do in gdb reverse debugging and so is quite is pretty powerful it is a little bit limited though in terms of both platform support and and also kind of features so so it's kind of separate record replay step only works on newer Intel CPUs doesn't work on AMD doesn't work on arm new cut a new kernel as well but you know if you've got all of those things and it works it works very well doesn't support programs that do share memories probably the biggest limitation and but yeah if it works it works well we've got undo TB for us which is perfect all of the above works with very good performance that's cross the platform you don't need special kernels you could work with arm does all those things so our new DB is wonderful but it is expensive but you know get what you pay for and how are we doing for time I think we're getting towards the end now which is a shame I could go on for ages but just just want maybe time for a couple more quick things GDP in it files this is my GDP in it I find this quite useful one word of caution don't be too clever in your GDP in it the second most common problem we find with customers GDP installations is somebody somewhere put like a run command in their GDP init file because they were too lazy to type our enter each time and that just caught it like you know just fuses people at some point it will confuse you or somebody else to keep the gbpn it's simple that's that's all I do in mine and yeah I run out of time we could definitely do a part two to this but you know I think we've seen I've kind of put this in order but let me just better if I didn't have to do this might debugging multi-process yeah if you can debug multiple processes same time that's very cool non stop mode you can debug a lot of threads run although that's the most reliable way to make gdb Ceglie in my experience breakpoints and watch points do more than you might think so you can watch like thread specific breakpoints and watch points you can put conditions on them which is kind of useful you can have read watch point if the hardware supports it one last word of warning or and then I run out of time and gdb sometimes sets this is the third most common problem that I've seen in practice gdb sometimes sets software watch points and doesn't tell you that it's what does tell you that it's done it but it's not obvious so if I start that program again you can watch like more than you might imagine I can watch all of array like that but watch point on array it's created the watch point right however there's a subtle difference it says watch point to array what it normally says like if I watch I oh no open done that that's surprising if I go watch a really zero Hardware watch plays right so I've created two software watch points and one Hardware watch point it's a harder watch point is almost certainly what you want because that uses the silicon and line silicon feature to run at full speed and stop out subtle ah when it when it needs to that amount of time or is that like a fire alarm anyway so I'm going to do one more command so continue I've got some software watchpoints and look how slow this is that's really really slow so you almost just beware of those if GDP being clever and putting watch put software watch points in there not what you want out of time

Channel: CppCon

Views: 45,969

Rating: 4.9248557 out of 5

Keywords: Greg Law, CppCon 2016, Computer Science (Field), + C (Programming Language), Bash Films, conference video recording services, conference recording services, nationwide conference recording services, conference videography services, conference video recording, conference filming services, conference services, conference recording, conference live streaming, event videographers, capture presentation slides, record presentation slides, event video recording, video services

Id: -n9Fkq1e6sg

Length: 59min 8sec (3548 seconds)

Published: Sun Oct 02 2016

Reddit Comments

[u/Kassandry]

Slides from the presentation