Advanced Debugging with GDB

hello world my name is victor engelmann and in this video i'm going to show some advanced topics of debugging with gdb the gnu debugger and yeah in the last video i've already gone through some simple um topics of gdb like uh setting breakpoints stepping through your program examining the stack and local variables and stuff like that and yeah in this video i want to go deeper into the topic and i want to start with something that i actually already showed last time but i think it's still a bit advanced so i think it also fits here so i'm gonna do it here too okay so this is a directory that we had last time with this very simple make file and this primitive program that's just an example doesn't do anything spectacular i run it it just asks for a number and then it generates this ladder of of numbers um yeah nothing spectacular but yeah so the first thing i want to talk about is attaching to a separate process because let's say you've you've started your program normally and so you didn't start it from a debugger so there's no debugger attached to it right now but let's say your programming just misbehaves in some way and then you might want to debug it but yeah just closing the program and re-running it again from the debugger yeah it might not exhibit this wrong behavior you know it could be based on some timing issues or whatever so i can now for this i need to know the process id of the process that i want to attach to so i can run ps-a and now i can see okay gdb test has this process id and so now i can take my gdb but attaching to an already running process requires administrator rights so i run this with sudo and now i can say attach give it this process id and yeah now i'm attached to that process the process is now paused um so that i can do some usual stuff like setting breakpoints and so on and yeah i will just set a breakpoint the same way i did in the last video i will put a breakpoint here on line 10. break gdb test dot cpp colon 10 and now continue okay so now it's waiting for some input again i can give it a number but uh yeah this time it doesn't do anything because i've hit the break point here and uh yeah when you hit a breakpoint it pauses but okay so um i can now detach from the program and uh okay when you detach from your program it uh then run keeps running as if there hasn't been a debugger attached to it in the first place okay the next thing i want to show you are conditional breakpoints that's also a very useful feature yeah maybe you only want a breakpoint to be hit under certain circumstances you know like maybe you do you want your break point in this loop here for example but you don't want the breakpoint to hit each and every time that you iterate through that loop so what you can do is you can set a breakpoint in gdb test line where this time i want to put it into this loop so in line eight and now you can just add the condition by saying let's say i equals five okay now yeah now we can start the program so it's now starting put in some number and yeah now now this time the first five iterations um just went through because in these iterations you know w was just less than five so i never reached five so so in this sixth iteration is of with w equals six it's the first time that you actually encounter this uh breakpoint and if i say print i now you see indeed i is 5 so the first iterations of that loop with w equals 6 have also been not broken at so if i say continue or just c then i continue now with w equals 7 i hit that break point again when i is five continue and yeah that's all that happened so the program is now done and it's exited and yeah i just think that conditional breakpoints are a really useful feature something that i've already said in the last video but maybe you haven't seen the last video i want to repeat that here if you go to gnu.org and there to software slash gdb then you get to this official page of the gdp debugger and here you have this link to documentation and there you have a link to gdb user manual and yeah skip this stuff up here go down to the table of contents and this is a really really good manual so if you have any problems with gdb go here you will find an answer you know the the the stuff is really written well the chapters and sections have good titles so that you can easily find what you are looking for here so yeah i can only really recommend using this official manual it's really good so if you have any problems with gdb this is where you will probably find your answers sometimes you want to turn a breakpoint off temporarily you know but you don't want to delete it because if you delete it then you would have to create it new when you want it to be turned on again so so yeah you don't have to delete your breakpoints you can just disable them so i will put a breakpoint here in line 10 again and let's also put one in line eight okay and notice the one in line eight has the id two so um so now if i debug this just input something like eight so now at this point i would see the breakpoint number one a lot you know because uh um yeah every time for for each iteration of this one you see this breakpoint again and again and again for as often as you have the value of w here so so maybe you're debugging something and you're thinking maybe i will need that breakpoint later so i don't want to delete it because yeah then i would have to set it again so instead you can just say this enable breakpoints two right two is the one that i'm hitting again and again and again okay and now if i continue i only see breakpoint one and yeah then after debugging like this for a while i might enable the breakpoint too again so fine now yeah okay now it's the program has finished but yeah if i start the program again now then now i'm hitting breakpoint 2 again okay so let's do that again disable breakpoint 2 continue now i'm only seeing breakpoint one enable it again continue and now i see breakpoint two again okay the next thing i want to show you is um you can actually save your uh break points you know because maybe you've spent a lot of time setting all your break points and maybe you have a lot of breakpoints and then yeah when you then quit gdb these break points are gone and that might not be a good idea because maybe you want to to debug again under very cert under very similar circumstances so so let's say break test cpp 8 and 10 again and yeah now uh let's say i'm done with my debug session but at some later point i want to debug again with the same break points then uh yeah if i just quit now then the breakpoints are gone and yeah then i would have to set them all again when i um when i start gdp the next time so what you would do instead relatively obviously is you can say save breakpoints and then just give it a file name let's say breakpoints txt okay let's look into that file and yeah this is quite obviously basically what you told gdb you know the stuff that you that you typed for these break points so then you can start gdb new and in here you can just say source breakpoints txt and there you have your breakpoints again now this might not be the best idea actually because yeah if you if you compile your program debug it set breakpoints save your breakpoints now if you change your program and recompile it start a new debug session with your new program and then you load your old break points um the other lines might have changed so these uh so these um lines here might have become invalid in the meantime you know so i think a better idea to do this would be to have some text editor and i'm certain that vim supports that in some way and i would also guess that kate supports that in some way that you can set breakpoints and export them into this format you know so that a text editor can keep track of uh your changes you know if you if you have a bright point here and you insert a line down here then a text editor can say okay then i will just move uh the breakpoint down a line and yeah i'm certain that text editors can do that and i think that's just a better idea to to keep your break points valid the next thing i want to talk about is so-called post-mortem debugging postmortem is latin and means after death so what postmortem debugging means is when your program has crashed so it's already dead so you cannot attach to it anymore of course because it's that it's just not there anymore but what you can do is when programs crash they usually generate a so-called core dump and this car dump is something very useful in debugging and yeah i will i will now include see a search and a search i less than 10 for example okay so now if i pass a 12 or so down here then um this assert will be violated it will fail and yeah when an assertion fails then the program crashes so so i've recompiled it and now i say make run now it's running if i input an 8 everything's fine but if i input i think 12 then we get this problem here the assertion has failed um it then says that the program was cancelled and this here is german it says spikehouse sugarstelt which is german and means that a core dump has been created now where is that core dump well if you look at a directory listing there is no core dump so what does this message tell us well in systemd environments the core dumps are i don't even know where it stores them but you can tell system d that you want to have that car dump using car dump control dash o card so this is a file name under which you want to save it then you give it the command dump and the program that you had run so gdb test okay so yeah cordon control gives you a lot of informations down here and in particular it now has generated this file core dump and now that we have this core dump we can now start gdb i would give it a gdp test to reference for the symbols and yeah now now we can load the core dump using target car and now we give it the file name the file name was quarter okay so now we have loaded the core dump and yeah a core dump is like the last state that the program was in when it crashed so you can for example look at the back trace and you can now see okay there was main there was b and then there was a and this is where an assertion had failed at line ten so line ten we see okay this assertion has failed okay and uh yeah now you can do the typical stuff like uh print w uh of course we are now at the top frame of the stack and w is isn't x doesn't exist in this race function apparently but we can now say again frame four so we want to look into this stack frame the call of a and now here we should be able to print w yeah w was 11 and uh print i i was 10 and yeah when i is 10 then i less than 10 fails that's that's a given so yeah this is how you can really work with quad dumps and yeah the card arms are really useful tool for for finding problems after a program has crashed now as i said core dumps are generated when your program crashes and they are generated when your program gets a signal sick aboard sick bus sick fpe sick segfault and sixes um and also as i've shown you when an assertion fails and also when the abort function is called so these are ways the abort function is in standard lip age if you include that you can just say abort and that also generates a core dump um but you can also generate core dumps manually so your program doesn't have to crash to generate a core dump so let me quit wait i've changed the program i need to recompile it okay now gdb test line 10 again okay okay now the program is running it has paused and yeah i i don't have to wait for the program to crash to get a call down i can just say g cor or generate a car dump gcorse just a shorthand and uh i don't know how to call it you just have to give it a file name my car dump okay it's complaining a bit but it said that that the core dump has still been created let's see it as my car dump okay and yeah you can still look into the stack trace at the point when i generated that car dump i think this is probably useful let's say you are looking at some problem and i don't know it's uh it's getting late you want to uh go home and sleep or something then you can generate a car dump like this and um and then you can continue analyzing problems in your program the next day if you want let me try can i continue now no okay that would have been fun but okay if you want to generate a car dump from within your program um that's a bit difficult um i've been looking for a way to do that but the best answer that i've gotten is here on stack overflow it seems that there's some a core dumper library from google and i don't know how that works but i guess it works like the answer down here where it basically says it forks as a process so it makes an identical copy of your process and then this identical copy is then given the signal sick abort so the identical copy then crashes and this crash will then generate the core dump yeah i don't know that seems a bit overkill to me but i really haven't found a better answer to that so uh so yeah if you uh if you need to do it uh i think this is uh an a decent way to do it but yeah i wish there was a a cleaner way but as i said this is the best that i've found online yeah that's how you can work with car dumps and um yeah let's set a breakpoint again line 10 run input 8. now it's hit the breakpoint another thing that you can do and this is i find this extremely awesome you can say checkpoint and um this now generates a so-called checkpoint it tells you down here that there's the checkpoint that it generated has number one now okay so if you now say info checkpoints you can now see okay process zero and this is your checkpoint now continue continue continue now if we are already at w equals three okay so we've run through this a few times but yeah the really cool thing now is you can say restart give it a checkpoint id like here i have the checkpoint id one so now i'm back at the point when i called checkpoint and i think that's extremely awesome this is extremely useful because you can now run through your program and try different values for things without starting over and starting over and starting over because getting to a point that you want to analyze can take a while you know you might have to click through some gui and that can take minutes and minutes and minutes until you are at the point that you want to debug and yeah here you can here you can debug a point in your program multiple times without starting the program over and over and over you know i think that's really a really awesome feature and i don't think i've ever seen that in any other in any other debugger so yeah i really think this is a feature that makes gdb stand out and makes it so useful so yeah and by the way i won't quit info checkpoints as i said um you uh have these checkpoints here um the star is telling you which of the checkpoints you're currently running let's go back to checkpoint zero so this was when we were at w equals three and now you can say delete checkpoint one and for checkpoints now there's no checkpoints left so this is how you can deal with checkpoints and uh yeah as i said an absolutely incredible feature extremely useful uh use this really to your advantage yeah something that i want to mention is um you can look at a list of threads info threads now we at this point we only have one thread but you could if you had more threads then you could say i don't know thread two three four based on this id here in the front you could switch the thread because in a gdp session there's only one active thread i mean the other threads can run but gdb is only ever examining one thread at a time and with this command thread too you could switch the thread that you are analyzing at this point okay so but yeah this program isn't multi-threaded so so i cannot show you this but i still wanted to mention that um yeah switching threads in gtb is no magic either okay now now let's say you let's say you're a software company and you write a proprietary software and so you're selling your software and it's not open source okay then you of course you wouldn't want to give your your customers your source code and well if you compile your program like this with g so you include your debug symbols then these debug symbols are a pretty good i'm not sure if they contain the entire listing but they are at least a very good hint for reconstructing your original source code from the program and yeah if you are a proprietary software company that might not be what you want to do so so you wouldn't want to include the debug symbols in your in your program what you can do instead is you can put the debug symbols in a separate file and you just keep that file to yourself okay so you just don't deliver that file to your customers and then you can use this file for debugging but your customer doesn't get it so he cannot reconstruct your source code from that file okay so how do we create such a file now i've looked through that a bit but unfortunately i haven't really found a way to do this directly so so you still have to compile with with your debug symbols and then afterwards you can strip them out into a separate file okay so yeah that's a bit uh i find that a bit annoying i think it would be more elegant if we could uh if we could tell gcc to store the debug symbols in a separate file directly but as i said i didn't find a way to do that so so instead we just built them into the into the program and then um pull them out there afterwards only keep debug from our original program dot dpg whatever so this is a file name for my debug symbols okay now at this point so object copy takes these debug symbols from our program and stores them in gdbtest.dbg okay but yeah with this we've only copied them from the program into that file but they are still in the program so we still need to get them out there we can do that there are multiple ways to do that but one way to do it is object copy strip debug from our main file okay and it says it's hasn't been updated i will force it with dash b okay now um you can now see i've compiled it extracted this gdb test db3 and then removed the debug symbols from gdb test okay so if i would now load gdb and give it gdb test you can now see it's trying to read symbols from gdb test but now it's complaining no debug symbols found there because we've stripped them out so so now at this point what we can do is we can either tell it symbol file is gdb test dot dbg okay reading symbols from gdb test dbg and now we now we can debug like break gdb test line 10 run 8 breakpoint hit continue and so on and now you you see the function name here we can look at the back trace we have function names that's something that you wouldn't have if you didn't have debug symbols then you would only have these addresses here okay so so this is one way you can pull in debug symbols from a separate file another way you can do it is when you launch gdb you can say dash e is your main program file and dash s is your symbol file so like this you can do it directly and now you can see here it's reading directly from gdbtest.dbg it's not even trying to read symbols from the gdb test program and that's actually something that software companies actually do you know they strip their debug symbols out of their programs and store them on some common file server for all the employees to access so if an employee debugs something on a client machine he can pull these debug symbols from his own file server to yeah to have these debug symbols now for the last topic i want to talk about imagine you're a software company again you've written a software you've sold it and your client has some problem with your software then you want to check out what the problem is and yeah you can do that with so called remote debugging so let's say this is a client machine okay then on the client machine you can run the program gdp server and you need the the ip address of the client okay this is my ip address the idea here is that the server specifies the ip address of the client so that the server only accepts connections from that ip but from what i've read this ip is ignored so so yeah i don't know i will still put it here and now you need to specify a port i will just do some one two three four seven whatever and the program that you want to debug okay so now it's listening on this port for our connection and we'll then debug gdb test okay and over here i can then run gdb and i can now connect to that server using the command target remote well it's the same ip because it's running physically on the same machine but imagine this is a different machine and now you would put the server's ip here and the part that you've specified on the other machine okay now you've connected to the to that server here on this on the server side you can see here now remote debugging from this host and just port okay some auto negotiation i assume and yeah now you can again do the usual stuff break gdb test dot cpp colon 10. okay it doesn't have a symbol table simplify gdb test it's now reading the symbols from file gdb test but it's also no debugging symbols found in gdp test okay of course i've i had stripped them out now it's reading symbols from gdb test dbg now i can set a breakpoint and notice the server is running the program and it doesn't have debug symbols okay i've read this this list of debug symbols here from my local file okay that i haven't given to the client okay break gdb test continue now it's waiting for input i guess eight now it's hit the breakpoint and i can look at the back trace and the usual stuff i would just say continue 100 now it's run through and yeah now the debugging session has ended and yeah i think that's really cool because you see now um that you can debug on a separate machine that separate machine can have the symbol table you don't have to give the symbol table to your customers this can even work across different architectures so i wanted to run this on my raspberry pi here but i had a bit of a problem because a raspberry pi has an arm processor and my laptop here has an x86 64 processor and for that you need to to use gdb multi arc and yeah installing gdb multi-arc isn't so simple on a manjaro system so that's why that's why i didn't do that physically i had to do this with two different shell windows here but i think the idea is clear and there are circumstances where debugging the way that we've done it so far doesn't really make sense you know when you have a break point and you hit that breakpoint and then you you stop your program under certain circumstances that can produce really bad behavior you know if you have something that depends on certain timing behavior you know you if you uh stop your program for just a couple of seconds uh something might time out and some something might crash or just disconnect something whatever um so under certain circumstances you really don't want to stop your program when you are debugging and gdb supports a feature that's called tracing now i have read that this doesn't work in the official manual and i couldn't get it to work i've never used it before but um when i prepared for this video i tried to get it working but i wasn't able to but i still want to show you because i've read from people who say they have gotten it to work so so tracing basically says you you want to do something similar like like debugging so you want in certain positions of your code you want to know what has happened um but you without stopping so instead of stopping you can tell gdb to to just collect the data and immediately continue and uh yeah that's called tracing and yeah then it's supposed to store that data and give it back to you later okay and yeah what i've read is that this only works um on the remote targets and even there it supposedly doesn't work but okay so you would run this with a gdp server and then on the gdb side you say target remote connect to it again load your symbols and yeah now instead of saying break you just say trace instead let's say line 10 okay and yeah so so now you have a so-called trace point you know if i say info trace points now there's one trace point and yeah now uh now just having a trace point wouldn't do anything right because uh you hit that trace point you haven't told it to do anything it just keeps running because you haven't told it what you are interested in okay so so to this trace point you can now associate actions and yeah here you can then set certain actions like collect the local variable yeah here we only have w but i don't think that's considered a local variable so i'll just say dollar rex so that it just collects all the cpu registers then say when when you have set all your actions for that trace point you say end and now you can say uh t start for trace start okay and um yeah now supposedly uh i can say continue but uh yeah happens again it just fails here and i don't know why maybe it's something wrong with my installation but as i said i've i have read that this feature just doesn't work at all but i there there are so many features to this trace feature that i really cannot imagine that it just doesn't work at all so so yeah anyhow you would at the end run t stop but uh yeah since uh the trace hasn't even started uh yeah it's uh that's kind of pointless i know nothing happens just quit now before i end this video let me talk a little bit about how this stuff works internally because i personally always like to know at least roughly what happens internally in systems when your program hits a breakpoint what happens internally is that it sends an interrupt 3 or a sick trap to be more precise now an interesting consequence of this is that you can just send this interrupt manually so you can just say assembler int 3. compiler okay and uh yeah now if if i run this now you see i have hit a breakpoint here before this see out i you know it tells me that i've hit a break point here on line nine but yeah i think that's okay and yeah i don't have break breakpoints you know if if i ask for a breakpoint it tells me i don't have breakpoints and still i've hit a breakpoint and that's something interesting for example i've seen that in the source code of google chrome or the chromium engine behind it where they have implemented their own assert function and if that assert function fails it doesn't immediately crash the program it sends this this interrupt first so when you're debugging chrome and it and some assert fails it doesn't crash it triggers a break point so the debugger is then immediately told to show you what happened okay and i think that's really an interesting thing so for for debugging okay but yeah before i wrap this up i have two more side notes nothing spectacular just one thing i want to mention is um if you've seen the movie tron the disney movie um or the sequel tron legacy for that matter i just find it an interesting side note to mention that this name tron actually comes from the basic programming language and it's the the command to turn tracing on you know that's why it's tron it stands for trace on and there's also the opposite command trough which turns tracing off speaking of tron the bad guy in the movie tron is actually called a master control program mcp and interestingly enough that was one of the first multitasking operating systems on unisy's uh mainframes so uh it's kind of weird you know the the bad guy in this movie is an operating system kernel that doesn't allow the poor programs to talk to the hardware directly they don't have the freedom anymore they uh it controls everything which totally makes sense these days you know i mean back back then it might have made sense to write your own drivers for all your devices but imagine having drivers for all devices that there are and you would have to have such drivers in all your applications that would be absolute madness you know having an operating system kernel in between you and the hardware makes total sense so yeah having this this operating system colonel as a as a bad guy who who takes away the program's freedom is uh kind of funny looking at it from today but uh okay anyhow the last thing i want to mention is um when something crashes on windows this doesn't happen anymore much these days because most programs will have some try catch block in the outermost main routine which will then just catch everything that crashes but yeah only a couple of years ago it was relatively common that something crashes and and it doesn't end gracefully it would really have a hard crash so um and you would get this little window telling you your program has stopped working do you want to send a crash report uh send or don't send and um yeah i haven't looked into that but i'm pretty sure that this send button would then collect the call dump and just send it to microsoft you know and so that's how that worked internally okay yeah anyhow i think this is enough for today if you like this video like it share it subscribe and see you next time

Channel: Write your own Operating System

Views: 8,006

Keywords: gdb, debugging, coredump, remote debugging

Id: Rudz-uSdWHM

Length: 54min 26sec (3266 seconds)

Published: Sat Dec 18 2021