- Hi everyone, my name is Simon and if you're here how C++ Debuggers work then you're in the right room. Before I get started,
the Microsoft we do have a survey which would be
great if you could complete and you can win an Xbox One, woo. That's half hearted but
I'll let away with it. Okay so this is a bit of
my background in debuggers so you know that I actually kind of understand vaguely what I'm talking about. So I used to work for a
great company in Edinburgh called Codeplay literally until this week. And there I worked a lot on debuggers so I worked on a debugger for HSA which is like a low level standard for heterogeneous systems. We have gpu's and dsp's
and tiny image processors and whatever. I also worked a lot with open cl debuggers which is another kind of similar standard and I worked a lot with
LLVM's debugger, lldb. There are few people at this conference who actually work on lldb so if you'd like to talk to them come see me and I'll introduce you. So a bit about what
I'm talking about here. Presumably we all have used
a debugger at some point. Hands? Everyone use debugger? That's the right number of
hands, I'm in the right room. Anyone know what this is? The bug, yes. This is the bug. Grace Hoppers bug, yes. Anyone know what machine it was from? No, I believe it was
Harvard's Mark 2 calculator. They found literally a moth. Our debuggers don't
tend to look like this. If you're doing this then
come speak to me after. I want to hear about it. Our debuggers sometimes look like this. Which is LDB running inside a terminal. Sometimes they'll look like this which is GDB running inside Emac's. Yup, some people like that. Some people might use
the LDB cursor interface, it's quite nice. Visual studio debugger. My definition is, I
definitely did not steal this from Wikipedia. An application which is used to test and debug other applications. And the most common actions you will use with a debugger are breakpoints, saying I want to stop
here at this line of code or I want to stop at this function. You could step around,
could be just instructions could be jumping over functions,
jumping into functions. You might want to do
expression evaluation. What is the value of X right now. Or you might want to
say, how did I get here. What's my call stack. So these are the main things
I'm going to talk about and a little bit more
towards the end as well. A bit of an introduction to the platform I'm going to talk about. So I will be focusing on Linux system's on X86_64 with ELF as a binary format you don't need to know what that is yet similar for DWARF, which is debug format and using ptrace as the
main interface library. So most of the concepts
which I will talk about are transferrable to other
platforms and systems obviously the function calls will change you might be working at a
different level of abstraction but the kind of general
concepts will apply. So this is a very nice
look at what ELF is. There is an even more
detailed version of this but it looks terrible on slides. So go to this Ange Albertini's
website if you'd like to see an even better introduction. But the basic idea is
that it's a binary format for executable's and objects. So it consists of a header which tells you about
what this executable is and what machine am I compiled for and then a bunch of code which
is separated into sections. So you don't need to know
a whole lot about ELF but this the kind of main
format for Linux executable's so it's useful to learn a little about it if you want to actually work on debuggers. DWARF, you noticed the
funny joke ELF, DWARF Knut people are hilarious and Dwarf mostly consists of die's not like this, they look like this. This is a Dwarf information entry and these are used to
describe the entirety of your programs from compilation units which is what we have here it's like files with all
the includes already in. Functions, variables, types, pretty much everything you can think of can be expressed someway
in Dwarf information. And this is a standard,
you can go and download it you can have a read. It's actually for a programming standard it's quite understandable. So I would recommend
going and having a look if you are interested. This is an example of
a compilation unit die. So it tells you like that
was compiled by clang 3.9.1. This is the file that is compiled for C++ you can get quite a lot of
information about your programs just by looking at these. And this is what the
debugger will be consuming when it's trying to understand everything about your program. Debuggers can do a lot
of things without Dwarf just by looking at the
Elf file, symbol tables, things like that. But for any like heavy weight debugging you need the Dwarf information. So as well as die's like this there is line table information. So this tells you which
lines of my source code correspond to which
machine code addresses. So you'll see that
these have the addresses on the left hand side and
then you have rows and columns for your source code. And then over on the right hand side there's some descriptions
which are all in nice acronyms. So NS is new statement,
prologue end, end text the end of the program. So this is what the
debugger is going to look at to try and understand how your source code relates to the machine code. Not only how your C++,
like your representation of your program maps to it. But also the literal text in the file. This is Ptrace. Ptrace is, well it's Ptrace. Everyone who's worked
with debuggers at any time if you say Ptrace, their
face will just drop. That's because this is
Ptrace's one function you pass it a request, a process ID and then an address and data. And depending on what this request is maybe it ignores these other two. Maybe they mean something different maybe you just an all
pointer all the time. So a few things you can do as like say I want my debugger to
be allowed to trace me. I want to read and write memory, I want to read and write registers. And so you just pass these
enum's into this function and it will do wildly different things. So Ptrace exists, it mostly does its job. If you want to read more
about it there is a page. Breakpoints. So breakpoints I think are, I mean debuggers in general are quite compared to something like compilers people don't know a lot
about how debuggers work. Compilers there's tons of
books, there are blog posts. Most people who can kind
of do a little bit of C++ will have some understanding
of how a compiler works, right? You get in text, you build an AST, maybe you haven't been
prepped in representation and you get to machine code and their number step to get you there. But debuggers breakpoints are like one of the most fundamental parts and people see it as black magic. This is some box where I say oh holy oracle, please place
a breakpoint at line 20 of this file and by some magic you get a breakpoint at the line in that file. But they're actually not so complicated. So there are two main
kinds of breakpoints. Hardware breakpoints and
Software breakpoints. Difference between these two
is that hardware breakpoints usually have some kind
of special registers which you're going to write values into and the hardware is going to take control make sure that when you
actually hit those addresses that something happens,
you get a breakpoint. Software breakpoints on the other hand you actually take the code
which you're executing in memory and you're going to modify it
so that a breakpoint gets set. And we'll see how these
both work a little bit more. Hardware breakpoints are limited because you're not using
physical registers. You only have so many. Whereas software breakpoints because you're just dealing with memory, it's essentially unlimited, you know, apart from however much
memory you actually have and what not. The cool thing about hardware breakpoints is that you can set
them to break on reading or writing or executing an address. Where software breakpoints
are only on execution. So if you want to say,
debugger I want to be notified every time this address is changed which can be really
useful if you're trying to track down some really weird bugs. But you can only really do
that with hardware breakpoints. Software breakpoints are just execution. So I talked briefly about
hardware breakpoints and we're more going to
focus on software ones. So in x86, you have four debug registers which used to write addresses into these and then when you get to that address the debugger will get notified by means we're going
to see in a little bit. It has a register which
you can read the status of and see like what's going on with them with all these breakpoints I've set. And you have something which
you can actually control to say I want a break on
reading, writing, executing. I won't go into much more detail
about hardware breakpoints especially because they're
very hardware specific. Software breakpoints on the other hand tend to be implemented
in very similar ways so if you read a debugger read the source code for a debugger then when you get down to
actually setting breakpoints it's all very similar on x86
and this is how it works. So say we have some assembly code This is x86 assembly. You don't have to
understand what it's doing. So on the right we have the assembly on the left we have the actual
hexadecimal representation of these instructions. I want to set a breakpoint
at this new instruction Maybe I know the address
of this and I'm explicitly telling my debugger set
a breakpoint right here. What the debugger then does
is it takes this first bite this ox48 and it's going
to set it off to the side remember it for later. What's then going to do
is replace the old value with this special ox cc and what this is int3 instruction and this is going to trigger
a software interrupt. So if our pc is sitting at
the top of the program counter we go down, we stop, software interrupt. Basic idea is the operating system, if you don't know how interrupts work the operating system is going to register some interrupt handlers. And when an interrupt is triggered then the handler for that
interrupt will be invoked. So for int3, which is our one up here then the operating system
will put into place something which will be called for int3 and on Linux this is the
function which gets invoked eventually do int3. So this is the actual
code from the kernel. The most important part here
is that in the middle of this we have our do trap and this
sigtrap thing is important this is debugger magic. So if we look at the man pages for signals sigtrap is the trace and breakpoint trap. Now a unique signal will
get sent to our process and our debugger can then
say okay we hit a breakpoint. And this is how the magic works. So I'll show an example
of this all put together we start off we have our
debugger is awaiting input. Sitting there waiting
for us to type continue or breakpoint or whatever. Our debuggee is stopped,
not doing anything. We have set a breakpoint at
this second instruction here note the cc on the left hand side. So what it's going to do is the debugger is going to wait for us to do something and we type in continue. So it's going to continue the debuggee and then it's going to wait on a signal as it's using waitpid
which is a Linux thing. This is essentially saying okay do nothing until I get a
signal and then I'm going to wake up and I'm going
to do something about it. So we continue the debuggee
so this is now running forget about the debugger for a minute. We go down and we hit int3,
we stop, software interrupt we drop down into our interrupt handler which issues a signal, now
the debugger is woken up because it was waiting on a signal now we've got one. I got a signal, yay. And now there are a bunch of methods which we can use to say okay. Where am I stopped, where
did this signal come from and we can report this
in some way to the user. Does anyone want me to go over that again? Because that's all the magic
of software breakpoints. Yes, the question was about
if we have something like this continue debuggee and then waitpid like what happens if we
have the pathological case where thread scheduler like preempts us right after we call continue debuggee. Debuggee runs and we
haven't hit our waitpid and we miss a signal. I honestly cannot remember
off the top of my head I think it would be
possible to miss signals with something like this unless you did some extra synchronization
and I can't remember what kind of extra
synchronization you would do. So there would be methods of doing this the overhead may or may not be worth it. Source level breakpoints,
oh sorry question. What's stopping the
debuggee from continuing? It's the operating system,
so this ptrace thing is known by the operating
system and interacts directly with the operating system. So when something is halted,
it won't be scheduled. So the operating system will make sure it's not going to run, yup. What was the question, sorry? Okay good question, I didn't cover that but I can go over it. So the question was,
I talked about earlier when the debugger will save out the value that was previously in this cc and then the question was how, when does it get restored
back and continue execution. So there are few methods to do this the most simple is to actually manipulate the program counter so you
go back over the instruction you replace the ox cc
with the original bite you single step and then
you continue execution then after you've replaced
the breaking point. Because you don't want the
case where you continue and maybe you have a loop and it should instantly hit the breakpoint and that's actually
not, this happens a lot. So you want to take the breakpoint off put the old value in single step, put the breakpoint back on and continue. That's the way a lot of debuggers do it. Another possible way is
actually having some memory put off to the side. And relocating that
instruction into the block changing the program counter
so that it essentially will eventually jump
back to the right point. And it means you don't have to single step and replace the breakpoint. I think GDB calls it displaced stepping so it's very useful for
multi threaded environments if you're operating in a
context where some threads are running and some threads are stopped. Does that answer the question? Okay, yes. That's correct, yes. So the question was about the interrupt and so the cc, most interrupt instructions are not just one bite. They're like an interrupt
and something else but int3 is special
because it wants to be used for debugging so there's a special one bite instruction for it. And Linux will register a handler in the interrupt vector
table and descriptor table I can never remember which one's which. And that is the do int. So the bites after int3, after
the ox cc are not touched because when we eventually
get rid of our breakpoint put the ox 48 back, then
we want the instruction to still be as it was,
so they're not touched. Yes. How does it a single set,
well we have slides on that. You'll see in a minute. Yes. When you disable the
breakpoint and not delete it so one thing is debuggers
can just swallow breakpoints. So they can just, a breakpoint is hit but
the debugger continues and doesn't tell you about it. So it could do that or it could. Yes, so in the debugger, the question was data structure set aside
to remember what's disabled and what's deleted. And yes, the debugger will
have some data structures to say okay well the
user set this breakpoint and currently asked for it to be disabled but may want to enable it again later so just remember the
address and then maybe just remove the breakpoint or just swallow any ones that come up. Is there another question, yes. Sorry can you repeat, I didn't
quite understand the question Yes so, maybe this is a
little bit misleading. I put the arrow I was supposed to have an
arrow from the do int up. But it got lost somehow. But yeah the do int will
send a signal to the debuggee and the waitpid is waiting
for a signal on that process. Okay that's the last question
for now, I'll take more later. Source level breakpoints,
very simple example we want to break on main. So what we do is we look
up our Dwarf information we find a die with the right name and we look at the low pc this gives us an address. We can then set a
breakpoint at this address and it all works using
what we just talked about. In reality, debuggers will
skip over the function prologue things like that, but this is
essentially what it's doing. So yeah, going from being able
to break point on an address to certain breakpoint
on a function like this. I mean you have to have a Dwarf parcer, but it's not a huge step. How about we have an overloaded function. So most things I've talked
about so far have applied pretty much to assembly
or nothing c++ specific. In c++ we can overload functions. So what happens if we want
to set a breakpoint on just do_thing double and not do_thing string. So one thing, what's going to be different about these two functions
when they're compiled? Mangling, exactly. So do_thing double will be mangled like so do_thing string we mangled like this. Because c++ is fantastic. So what we can do is look for something if we want to set on just the double one we can mangle the name and then just look for something with that linkage name look at the low pc, rinse
repeat. Same as before. Yes, Victor. Yes, mangling can depend
from compiler to compiler so if you're doing, there
are other methods to do this I just picked a simple one. But yeah, this will only work if you knew that you're working with the itanium api. So if you've compiled
something with msvc mangling and you tried to do this but mangled your debugger mangled using itanium abi then they would miss. Yes, so on Linux, Itanium abi for mangling is like defacto standard essentially. How about lines. So if I want to set a
breakpoint on foo.cpp line four. Then we talked about dies and we talked about
line table information. So this is line table information and we can see there
is line four over here there are fewer entries for line four this could be because there are one line with multiple expressions, which generate multiple assembly instructions. So we wont' set one on
the start of the statement so we look at this NS and
set a breakpoint there. Have the address, it's
the one on the left there Again, going from instructions to source is just a case of looking at
Dwarf and mapping it back. Stepping, here's the
answer to your question So the main types of
stepping you want to do are over single assembly instructions, stepping over functions calls, so you don't want to go inside
a function if you call it. Stepping in is when you do want to go and stepping out is when you want to finish this function or turn out. So on x86 with sufficiently
high kernel version number you can just do this for
instruction stepping. I will go into a bit more complex things you might have to do. But ptrace has a single step enumerator so you can just pass this and it will do an instructional single step. For stepping out, you can
find the return address I'll talk a little bit about that when I go into stack unwinding. But you can find the return address and you can set a breakpoint there. Now if you're stepping in
you want to set a breakpoint at the return address, in
case you're not actually stepping into a function. Actually you're turning
out and you don't want to just continue into cyber
space and never gain control over your debuggee again. So you want to set the
breakpoint at the return address and you want to set a breakpoint
at the next instruction in this function or the callee. Great, simple. Not really. What is the next instruction? If I'm on some assembly instruction, how do I know where I'm going to go? Software breakpoints. You want to set software breakpoints but you need to know where
to set the breakpoints. Yeah, so you might just be
going to the next instruction you know, you're doing an
add, you're doing a subtract, you're doing a move,
you're doing another move. Simple, you just set a breakpoint
at the next instruction. Maybe you're doing a jump, maybe you're doing a conditional jump, maybe you have no idea what's going on. So actually, answering
what's the next instruction requires understanding your actual target. So in reality, you need
to inspect the code to work out possible branch
targets, set breakpoints there. What this essentially ends
with is your debugger ships like an instruction
emulator for your target. It's what LDB does. I'll talk a little bit more about how that makes a mess
of some things, later. This is okay for now. Stepping over is very, very similar set a breakpoint at return address and the next instruction in this function. We still have that same question but the answer is the same. You know you need to work
out your branch targets. Registers and memory. This is a useful thing if you're doing like really low level stuff, you want to be able to write
or read your registers. Reading memory is useful for
a whole host of problems. So reading registers is
actually way more simple than you might expect. There's just a ptrace call for it. So you say, I have this
user_regs_struct thing which I construct and I just pass it in and I get out something like this. Which just has a field
for every single register which I might want the value of. Fairly simple. If I want to then write the registers, then I just do the read, I set some value and then I write it back. Not that magical, eh? For reading and writing memory
it's the same kind of thing I have these peekdata
and these pokedata calls. The unfortunate thing about this, can anyone see something, yes? Exactly, it's a word at a time. So if you're wanting to
read a big amount of memory maybe I have a massive array which I want to show to the user. Or the user has requested
I want a memory dump of like this entire bit of my program. Doing all that reading a word at a time is super inefficient. Because we have a siscall
and we have a contact switch down into kernel mode
and then we come back up every single word. So that's not great. So there are calls like process_vm_readv and write which can do multi
word reading and writing. So this in one case
where you can use ptrace but you don't really want to. Stack unwinding. So stack unwinding can
get very complicated. There is in fact, an entire
talk on exceptions in c++ and how the stack is unwound on windows that's James' talk later today so I recommend going to see this if you want a more comprehensive view. I'm going to cover the
kind of most simple case it should give you a flavor
for the kinds of things which will actually occur. So this is what a stack
could look like for x86 64 given the system 5 abi. So you have all your
arguments, your return address the old frame pointer, so
the pointer to the previous stack frame. And then your local variables. Everyone okay with this, any questions? Cool. So we have a code like this. Bar plus foo, foo, hi I'm going to set a
breakpoint, here on the cout. Then what are we in? We
are in some stack frame we don't really have any
information apart from that. So if we want to show the user how we actually got to this function and what functions did we call on the way then we need to walk up the stack. So what we can do is you can say, okay well we have the frame pointer for the previous stack frame. So we can just go back. Now we found another stack frame but we have no idea what it's for. Like, above this return
we have some arguments and then we'll have some local variables and things like that. But we don't really
know what this thing is it's just stack frame. So we can use the return address to look up the Dwarf information. Because the dwarf information
says okay here's the range that this function's code lives at. If I know my return address I can say okay it doesn't match this function, it doesn't match this function, here's the function we're in. We know it's between this low
value and this high value. And that's how we can find out okay, our last stack frame was bar. And we can then send that to the user. And then we just do the same thing again we go back along the frame pointer, check the return, and we know we've done baz call, bar call, foo. Then we stop when the
frame pointer is zero. In reality above this,
we'd have main and like underscore start and things like that. But for sake of brevity. So this is how stack unwinding would work if you have easy access to frame pointers. Sometimes you don't. Expression evaluation. Now you might notice as I
get further into this talk things get more and more complicated And Expression evaluation could certainly be an entire hour talk on
it, it's multiple talks. So I'll talk about the
kind of broad aspect of it. We want to print my integer. It's a local variable has some value, it's on the stack. Fairly easy to get a hold of. So what we can do is look
at our dwarf information. Dwarf information is so handy and this tells us its names we've found the name,
we might have multiple things named in the dwarf information you have scopes so you can
work out which scope you are in and which is the right one. And this tells us where
our variable is located so this might be, oh
it's just in register n. Or it could be, in this
case, it's an offset from our stack frame. This thing is stored on the stack. So we then need to go and
find where our stack frame is and offset it by negative
8 and we'll find our value. And then using ptrace or the vm process then we can go ahead and
look where that address is and read the value. Yes question. Yes, so the question was
if something is relocated like if it starts off
in register 12 and then is moved to register 17
because the register allocator decides it needs to move things around is that reflect the dwarf information. The answer is yes. There are various different
ways that dwarf information can represent addresses, and
one of those is list of ranges. So it can say between this
pc and that pc it lives here. Between this pc and that
pc, it lives over there. Here it's on the stack,
here it's on the register and that can all be represented by dwarf. Answer question? Yeah. And yes, so I just
mentioned that there are various ways for dwarf information
to represent addresses. So this becomes annoying
for debuggers to handle because, you know, maybe my frame base is just in register 6 and
I can go read that register and I can offset it and it's all fine. Maybe I need to go look a
the call frame information this is like a section in our ELF, which stores information
for stack unwinding for exceptions. This is like the real way which
you could do stack unwinding in the absence of pin pointer information and things like that,
but it gets quite hairy so I'm not going to show that. And this example right in
the bottom shows the ranges. I didn't really expect people
to be able to read this but this just showing
that there are a bunch of different ways in
which it's represented and that's how the ranges kind of look when you dump them out. Getting a little bit more complicated we have some local variable my_int and another one called a and we went to multiply them. So we could kind of do a
lot of stuff in the debugger but what some things do, this
is what LDB actually does generates function, so this
has some special magic in it to say what local variables
I'm using and things like that. It generates a function,
generates llbmir from it. You don't need to understand
what this is doing just note that it exists and this is actually what
lldb actually produces. And then that ir is interpreted
to get out the final result. So the debugger isn't having
to implement its own c++ expression parser and evaluator. If you want to do something
like call a function maybe this function has side effect. Maybe we want the side effects to happen maybe it's going to change some values or print something out and we want this to actually be occurring as if we called the function in our code. So we can create some
ir and interpret that because the side effects
won't be properly represented. So instead we can compile ir, lower to machine code, map this code into the
address space of our debuggee and then execute that function. Just by manipulating our registers. So this is how you actually
end up calling functions within the address space and having all the side effects work. So it's pretty interesting. There's a whole lot of
research in to various ways to achieve this and it's really cool. If you go and Google some stuff afterwards or come talk to me or some
other debugger experts here. Yes, question. So the question was do the
debugger need to know how the debuggee was compiled. I'm trying to think if it would need to do anything to match
calling interventions or abi or anything. Probably not because
you can find the address of functions by looking at symbol table or the dwarf information. So you know where the
functions are gonna live. You just need to make sure
that the addresses are correct. Does that answer? There's another question here I think. Yes so if our expression that we're, like if we're not calling a function here but say this expression is inset or like right into a memory address then yes we have to do the same thing. Why do we not always do that? I think this has changed in lldb sometimes I think it used to do more interpretation than it does now. But I'm willing to be
proven wrong on that. I think maybe it now just always gets it. But yeah I would have to check. Victor had a question. What if the functions in line that's a very good question. So dwarf information
does give you information about in line functions but a lot of the time not very useful because in line functions
are subject to other things other than just in lining because you have more local information. I do have an example coming right up. So just a second. Yes. So I think this is how it gets mapped in they can get calls and m-map essentially and then writes into the map memory. It has to generate a call to m-map that it calls m-map
within the address space and get the address out. So the functions actually
in the debuggees. Yeah come talk to me after. This is getting to the end lining. How many of you have tried
doing something like this? Quite a few, did it work? No? Did you get something like this? Yeah. This happens a lot. Especially with templates
everywhere in c++ things get in lined, you don't have a
definition for the function so you what do you call? It's not there. Pardon? (laughter) Who are you gonna call, yes So what you can do is just write this So you could just write this
in some translation unit output actual definition
for this in satiation I have actually used this quite a lot it's awful but it works quite nicely. So if you want to be printing
out a bunch of expressions which are requiring
functions of class templates which would otherwise not
have definitions outputted then you can make sure
definition is outputted somewhere and then the debugger can call it. So this is like a hacky debugger tip. Multi threaded applications, again could be like an entire course. So you could do ptrace
has options you can set. So you can say I want
to get a trap every time the clones is called. And that is what happens
when you create a new thread. And then there's some
magic you can do to say oh what's this. Sigtrap from a clone call and if it was then you can work out what the new pid is and add it into your
internal data structures and things like that. So this is like the thousand mile view of what the setup is for dealing with multi threaded applications. Shared libraries, Oh sorry, Victor, yes? Yeah that's the question I
didn't want to get into much. (audience laughter) The question was when you set a breakpoint do all threads stop,
does just one thread stop the answer is complicated because debuggers will
have different modes which allow you to have non
stop mode or all stop mode and some dealers do it better than others. Yeah it's another big topic which I didn't quite want to cover. Yes. Isn't it wrong to stop only one thread? Maybe, maybe not. (audience laughter) Well if you stop only one thread maybe your gooey can keep running. Right yeah, if you have
a multi threaded server and maybe you're debugging infraduction. (audience laughter) Maybe you're a terrible person. Then you can do this, yes. Yeah come talk to me after. Don't debug infraduction. Okay shared libraries this was something in
which I didn't understand for the longest time. Because it's really badly documented until you find the header file which explains everything and it's great. But if you wanna trace
when shared libraries are loaded and unloaded because maybe you, if you use shared libraries at all you've probably got to the
situation where you want to set a breakpoint on something
in the shared library and the debugger says
oh we couldn't find this do you want to wait and
we'll try and set it if the library turns up. This happens a lot. This is how it's implemented because it has to know when
something is being loaded. Because maybe you just hit
a breakpoint on function you continue. If shared library has loaded
with that function in it you want that breakpoint to be hit rather than just waiting for
the death of the universe or the program terminates or something. So somewhere in your program there is a way to get this somewhere. There is this kind of data structure which has a bunch of tags and information about your program. This is like dynamic process information. And one of these entries points to something like this, r_debug. Now the most important things in this are this link map which tells you every single shared
library which is loaded, its name, where it's loaded. So this is the kind of
thing we want to be, it's a link list. So we want to walk the link list and find what our shared libraries are. The really cool thing, at
least I think it's really cool I'm really nerdy. Is this. So this is an address of a
function which will get called every time a shared library
is loaded or unloaded. So what can we do in order to
read the link map every time it's something our debugger can do. Set a software breakpoint. So we set a software
breakpoint at this address and every time the function is called we know that the library
is being loaded or unloaded we can then walk this link map, see if there are any differences. There's some ways for the
operating system to tell if there are changes, things like that. But we can essentially walk
it up our data structure, set break points. And this breakpoint is
hidden from the user. So the debuggers are setting
breakpoints all the time without you actually noticing it. But they're doing it
to do things like this. Trace, shared library
loading and it's all hidden. There are some ways to get the debugger to show you these things. I think lldb has a option to
print internal breakpoints so you will see things like this being set which is pretty cool. So this kind of how it goes the debugger sets
breakpoint, magic function, when it's hit you walk
through the link map, update data structures. Remote debugging. So this is kind of how a
remote debugging session looks. You have the debugger on your host, you have a debug stub on the target. Now this debug stub is supposed to be, supposed to be, a very thin wrapper around the operating system, debug interface. Like ptrace. In reality, this thing is massive. Like, lldb debug stub for x86 ships like an entire instruction emulator. So yeah, it's not really
a tiny thing anymore which is a bit frustrating. The idea is that the debugger communicates with the debug stub,
the debug stub operates with the debuggee in some os
specific manner like ptrace. And then just everything
sent between the debugger and the debug stub. Now, the interesting
thing is your actually using this all the time. Because this how quite
a lot debuggers operate, just doing local debugging. Because it means it's a whole lot easier to implement a bunch of this stuff. You just always have a debugger
connecting to a debug stub. It just so happens that the debug stub is on the same machine as
you run on the debugger. So this is how lldb operates
all the time for example. I think GDB does as well but
I don't know GDB as well. Yes? That's a very good question. What if the debug symbols
are not available? And yes debug symbols should generally, are not available on the debug stub because the debug symbols information is high level information which only the debugger has access to. The idea is that the debug
stub should just be handling addresses and super low level stuff. So generally the debug
stub does not have access to the debug info. It's just the debugger
works out all the address and sends it to the debug stub. Okay. Yeah, so I'm going to show
how it sends request just now. So there is a remote protocol called the gdb remote protocol. That's what gdb and lldb use. So it looks kind of like this. This is just blown up a bit, so I can explain all the different parts. So you have packet start, some types which is a little identifier. Arguments and then check sum. So this is just sent over a network or over any kind of communication channel. So for example, this is a debug breakpoint packet. Which sets the breakpoint at some address and there's an architecture specific thing I can't remember what that's
actually used for on x86. And that's how you write, or
that's how c++ debuggers work. Thank you. Okay I have resources
there if anyone would like and I will now take questions. Yes. Why can't we properly debug over fork. This annoys me too. GDB has a very, very good, well comparatively very
good follow forks mode lldb does not. So the gdb follow forks mode has, in my experience mostly worked okay. So it's just something you can set and say okay when my program forks,
I want to follow my child rather than the parent and maybe you had bad
experiences with that. You want it to follow both. Right, yeah. I haven't thought much about debugging both at the same time. Maybe modern GDB has more
support for that or something. Yeah GDB just connects to
the child as soon as it can rather than doing something nice. I know some people who have worked on follow forks mode stuff. But can't remember the details. I'm not so sure about that part myself. Other questions, yes in the back. Sorry can you come to the microphone. - [Man] How do watch points work? - So watch points you use
the Hardware break points because you can set them on
reading and writing as well so you say I want a set
watch point at this variable I know where that variable lives because I have the dwarf information and then I can set harder
breakpoints to trap on reading and writing. - [Man] So you can only
do that on registers then? - No the registers are having
addresses written into them. So you're watching addresses. Other questions? Victor? - [Victor] So you have a
protocol between the debugger and this stub. Is that standardized or. - Yes, well it's partially there is a document which shows all the different gdb remote packets, I think I have a link
to it in my resources. In reality, debuggers will
have their own extensions so lldb has its own packets
which gdb doesn't support. The packet format will
have multiple versions so if you're using
different versions of gdb or your debug stub, then
maybe there's a mismatch and things like that. So it's not like a standard,
but it's documented at least. Or at least most of it's documented. Yeah. - [Man] I think somebody
over there was actually trying to ask this earlier. Maybe I just didn't
understand the response but some debuggers have
a pause button, right? How are you inserting a breakpoint
wherever you happen to be at the time. - Okay, so the pause button on debuggers usually just sends like a
second to the debuggee process. So it's not like taking a
breakpoint wherever you are it just sends a signal and
that will stop the process. - [Man] Question about
core dump debugging works. We don't have a process that will work. - Yeah for core dump debugging, I never looked into a whole lot. My naive understanding
is that you just have your core dump will
have enough information to give you some experience and then your debugger is responsible for using that information
as best it can to provide. - [Man] Petrie skulls to
get registers to get memory are replaced by reading
from core dump, right? - Right, yeah, as I said I haven't looked into core dumps much but I would guess that's what's done. So if you evaluate an expression which results in an exception I guess when you're getting
the call to the function then the compiler which
you're using, would have to I would put code to handle
to deal with the exceptions. Like report back to the user
that an exception was thrown and not just end up
terminating everything. Sorry can you stand
closer to the microphone? How does the debugger know
whether it's an exception. Well I mean the debugger is
using like a full on compiler so the lldb calls aren't
inclined to actually compile the expressions which you use. So it's not one of these things where it's just hacked together. C++ subset is like actually using clang so it has everything in clang available. Sorry can you say that again? - [Man] Do you know why the Linux kernel only got single step support recently because x86, I think has
single step hardware support. - Yeah do I know why ptrace
only got single step recently I mean it wasn't recent, recent. This was a number of years. I can't remember exactly what
kernel version it changed at and I'm not sure why it took longer maybe it was just ptrace didn't have that actual part available. Before then to do single
step you'd have to do something like the branch target. Finding and then taking breakpoints there. - [Man] When debugging application
that was built with clang sometimes I can't see the
internal representations of memory but I can with gcc. Is there anyway I can
instruct the debugger what I've built with so
that I can give it better, rich information. - Sorry can you say that first bit again - [Man] Oh so sometimes when we build an application with clang you can't really see the internal data but we go to gcc and see your
strings for whatever you need is there anyway to instruct the debugger like hey I'm building a clang, use clang internals to represent this data - So yeah the debugger will
know what you've compiled with because it will be set in the start of the compilation unit
and it said which compiler is being used. So will have that
information available to it, it's just a case of what it does with that and whether it implements
all the necessary machinery to do that. Thanks, one more question. We have seconds left. - [Man] How do you debug when
the variable is optimized away - When a variable is optimized away, then the debugger will just often say this is being optimized away. So if it's not there,
you can't visualize it. - [Man] So then do you
add printer statements for the variable and not
make it get optimized - Yeah you could, in your program, try to do things to stop
it being optimized away like stick it in a volatile
or something like that. Okay thank you very much. (applause)
I watched this one earlier today, I have no practical use for the information but I found it technically interesting and in depth.
After watching this, I now thoroughly understand the comment from Greg Law's "Debugging Linux C++" talk about the ptrace API.