Arguing with Linus Torvalds - Steven Rostedt

okay right good so welcome everyone back in the main track of room a with the incredible opportunity to meet a person that you must probably run cold by and actually somebody who wrote an important parts of the Linux kernel that in this very moment enable the streaming work on some of your mobile phones and many other applications so it is again with great pleasure to introduce Steven roster who is going to talk about the side effects of the spectrum mitigation and technique a technical specifics of how this was approached and what it takes to mitigate a hardware problem that is so important and so only present that basically every device that runs an Intel CPU reason one is prone to attacks so I would like you to make a very warm welcome for very important person today Steve Ralston okay thank you very much for a nice introduction again Steve Rasta I currently work for VMware I started my first kernel development was in 98 going for my Master's my first professional work on working on Linux kernel was in 2001 with the time sis kernel porting their Linux real time kernel to that various architectures I then worked ten years at Red Hat and I'm back at now I'm at VMware for the last three years my claim to fame is the real-time patch which is about hopefully make it into mainline soon I i among a few others were the original developers for making linux into a real-time operating system among that I also became in I created F trace which is the official Linux kernel tracer of the Linux current or official tracer of the Linux kernel I have code all over the place inside the kernel so that's truth so yes my code if you're running an Android phone you're running some of my code before I go on I have a tradition I always do it all my talks which is I love to do selfies but I used to do self my first selfie ever did was in 1984 with those little yellow throat disposable cameras how you go crank crank crank crank take a picture and then three months later after you send your film away you get to see what it is so that's why I started with and so I don't this is what you use a real selfie with it's not it's almost I want to get some real film camera but I'll go with this but he knows smile everyone there you go anyway so let's get on now this talk I'm giving today I it's about what happened earlier this year March May March timeframe I was we're talking about some code and leanness didn't like what the approaches we did and I came out it was several of us having this argument with leanness and if the thread went on quite a long and somewhat someone from the links foundation Shoah Khan was a fellow at the links foundation she's also a kernel developer she was CCD on this and she said Steve this is a great topic come give a talk at the open source summit in North America about so I said sure and I waited to the last minute of course to write it up which I finally did and I really wasn't thinking much and I gave the title I gave was problems emulating function calls from the entry breakpoint in the kernel and the crappy solutions we came up with I just realized that that talk was a good end up being an awesome talk but the title was so boring people fell asleep before they got to the end of it and when I was asked to give a talk here and I wanted to give this to talk again because I think it's a really great talk because it covers you could be really you don't have to know anything about Linux kernel or as long as you have an idea about programming you could understand this talk but also if you're really deep and your major kernel developer you know everything about it you'll learn something from this talk as well so this is a wide rain that so I think it's perfect here so that's why instead I changed the title to just arguing in Felina Stovall's this okay who here I want to see who here does not know what this is I raise their hands does not know what this these are symbols for specter and meltdown okay who has not heard of Spectre and meltdown well in January of 2018 so I add it on my slide so I think it was this year for some reason Spectre a meltdown were announced in the public in actuality it was discovered in the summer of 2017 and went through a whole bunch of things and wasn't until 2018 that we got to even know about it I mean there was there was a lot of speculation on what the problem was going to be and these are hardware bugs they're not software bugs they're Hardware bugs what does that mean it doesn't matter what software you're running if you're running Windows Linux Mac you're affected by these because it's a hardware bug and I'm more talking about I'm not gonna talk about meltdown i'm gonna talk about a little bit more about specter and i'm not going to really go into the true details of it that's because at linux crow recipes in 2018 palo bones indie gave a awesome talk and it's online so if you go to kernel kernel - recipes org in fact i'll post these slides and where does the link to where to get to this talk he goes into the details about the intel architecture that describes why this is a problem basically all you have to know is that memory is slow and the cpu is fast and there's something that they call the cache in between and a cache can't you can't cache all memory only caches you like hash part of the memory so you try to use the memory that you're going to use the most pulls in the cache as the CPU is cuck the engines calculating it will look ahead it'll what we call speculate to see where is it going and try to pull in the memory before the calculation algorithm or machine engine gets there and what happens is the logic says it's not calculating it's making several guesses it's speculating which way it will go and by these speculations you what we've learned is you can actually get past various checks like if you have a check that this person doesn't is not allowed to do something that you don't go this way you could fool the speculation to say no I have permission to go this way and go that way and then you can modify the cache so when the engine gets there and sees oh no we speculated wrong it flushes everything out and goes the other way but it doesn't reset the cache which means you have this thing called a side channel attack where you could just go into you you could allocate huge parts of memory and then train how the speculation will go and I'm not going to go into the details about that this is talked about this these talks and then what happens is you run the system call and it will speculate it correctly and then what you do is you just read the cache or read the memory and you time okay read this read this read this and it'll be slow slow slow fast slow slow slow fast slow slow slow every time it was fast you know that the cache pulled in and now you could determine oh wait this jumped here and it will only jump so basically let's say you have an array of 256 cache lines and your array or your what you could find a way to speculate on some password and you go to the first character and based off of what that first character is you're going to jump into this array and read some data that's this array happens to be in your memory space so the speculation goes in and does all this work that the kernel will never let you do but the CPU will do it for you and then you go in and it will read say based off of this character say a which is a skis just I don't know it's 60 so in your array 60 it will go and read that spot of your array and now after you run the code into the kernel and it doesn't do anything all you do is go back read your array and when that the 60th element goes fast you know if that password starts with an A and then you go to the next letter next letter next letter so you could read anything in memory by teaching your system of speculation david Woodhouse then gave a talk about how we stopped this there's lots of tricks that the software has to do there used to be able to joke when I worked at Lockheed Martin I still work at Lockheed Martin too when I worked at Lockheed Martin we say software was made to fix the hardware and today we still say the same thing he gives a great talk about how the lantern 'el mitigates all these issues he showed this slide called ret police how many people have ever heard of a ret spleen okay a few hands okay don't worry if you don't know what it is you'll learn today what a ret plane is because I I knew Bret please I kinda had an idea but once he showed me the slide don't worry about the slide now I'll talk about it later I went oh crap this affects my code big time and I'll get back to this by the way like you said if there's something I say you don't understand just get the gist of it just get the idea of it don't don't know everything this is just more of a fun so what does all this mean it means dynamic functions have overhead if this is your thought right now in your head you're keeping up just fine normalcy okay how many people are programmers here most people hey good so all of you should know what programming is and so this were this is probably a very common thing hello world so then we have something called a white collie their dynamic dynamic funk I call them dynamic functions but their function pointers or indirect calls all the same thing Q sort is a you know quicksort algorithm in the G Lipsy library that you don't want everyone you know you learn how to write Q sort but honestly right now today if someone asked me to write Q sort I wouldn't remember how to write it I just know there's a library that does it I know that's fast and I just used someone else's algorithm I have things I have to work about that I have other things to worry about so I just always use Q sort the thing I want to talk about is that compare function that compare function is what is that what I call a function point or a dynamic function so why do we have dynamic functions the same thing with Q sort you have say two arrays and you have one function that could sort to differ arrays one of integers one of strings obviously they're not going to be compared the same way so you have to tell Q sort how you're going to compare these so you're right to string function you write two functions one is called int compare one that's called s compare can't use string compare directly that's because the the way that what's passed in our it are the address of the strings not the strings themselves so you had to put in a wrapper around string compare so those are the two things you have to that care about now if I was to write a sort so if I had one with numbers and one with strings and I would write my in compare my yes compare this is just showing it without to get into the details of this and then I go look at Q store and I look at the source code by the way because open source it's great you could download the source code that you know gee Lib C library and look at the code and basically this is what I did for the talk and I kind of pulled in and I found exactly where the Q sorts implemented and ideally it's this function right here you have the pointer and you have you see the function is being called that's whatever I passed in is going to be called inside so if I pass in in compare it's like I just had in compare in the code and if I passed in as compare it's like I have s compare in the code but again this is a function pointer it's not a hard-coded it's not a direct function call and this is a big difference and it matters for Brett points so let's look at let's do our own little function call and I I just say let's go I created a function called Goldilocks that you pass in a bed and the bed function will take one parameter and that parameter is going to be 23 but it could be anything I wanted so the first one is minus 3 will be minus 3 is blah next one is plus 4 next one is a is just right and then I call that same function over again you could cut and paste this put it in and compile it it should work and if I run the code you get this minus 3 is 24 27 23 is perfect ok one of my talks while ago I did I had something and I was told that it was inappropriate for him and he said I don't want like you should have warned to be first so the next following slides might be inappropriate for some members of the audience they contain assembly you've been warned okay I take the dynamic function and I come I compile it and I do obstinate I look at the code itself so the first thing it does is this subtract eight of the stack pointer I don't know why it's doing that okay it's giving a buffer on the stack pointer for some reason whether or not I need it or not I don't know the RDI I know is the first parameter the register RDI is what gets passed into the first parameter and then I move hex value 17 which X is 23 into EDI why is it RDI EDI does anyone know why it's not RTI again well I'm not going well I'll tell you right now because I want get going it's because it's an integer it's only 32-bit EDI is 32-bit RDI 64-bit this is x86 64 machine so I passed it it takes the RDI which is a took for some reason it took a 64 bit and then it's converting it to a 32 bit and then it's calling it calls RA X whatever is in our a expose member the first parameter gets put into the second parameter a void pointer 64-bit that's why it's our di it goes to this into our ax and then from here I call what's in our tra X and then I put the stack back in return so down at the bottom you saw a call Goldilocks that's a single command and it jumps to Goldilocks that's called a direct function call if you look at it if i dis compel this disassemble this and actually with the the machine code there will be a hard-coded address that takes you right to Goldilocks they call our ax is called too indirect function call and or and what this is doing it's calling something that's inside a register so whatever our ax is it's going to call you don't know the CPU can't know what it is this is the problem with where Rep leans south remember I talked about how you could train the code you can train one of the things that the code does is on indirect calls it will cache what's in this call the first time it runs something you'll say oh it's going to call here well you could since the call memorize the entire you codebase he only remembers a section so some calls are will be mapped with other calls so if you know we're in the kernel one call is you could actually map it into user space and train the caller to jump to a specific spot then call into the kernel and then you could make you teach you get the speculation jump anywhere you want including user code or to user space code so speculate this is why this is so dangerous to have these available so we have to stop that this brings us to that slide and when I saw this slide at kernel recipes it freaked me out because this is where I said oh my god this is going to kill performance let's go back to golden box we have our call there this is compiled without rep elites I enabled rep lanes which is GCC option to enable a rep range and this is what Goldilocks looks like this is what happens that simple call there's no indirect call here the first call from rep alene it calls this rippling thunk re X which is a single trampoline what a trampoline happens to be is just some code you inject that usually it's not part of the seed code it's just some assembly code you're going to inject somewhere that you jump to to help you out later on so if you hear me use the term trampoline all it means is some code that was written in assembly puts or someplace that we could jump to so GCC creates a this rep lean thunk re axe it has a rep lean thunk RDX and a bunch of registers and it places it here so everything that called the head RA X as a call it would jump to this rep vote this with this little trampoline to do it it's a direct call that the speculations way gum in and jump into this the first thing it does in that trampoline is it calls set up target which jumps down below and if you look at it if the call way a call does call just two things a call will push the return address onto the stack and then it will jump to the locator that move the instruction pointer to the location you want to go to the return will pop whatever was on the stack and jump and jump to whatever location that was returns have issues too but let's just worry about this one thing at a time anyway what the speculations going to do when it reads us it doesn't know about this little trick you see that move right there at the they use this right the move right here is put is basically taking re whatever's are in re X Murrow we put 7th sorry where's re X called here we put the RTI RTI which was my function pointer into re X call this rep Elaine rx right here we put into the stack pointer we just over wrote the return address and then we called return now that the speculation code doesn't know about these is like I said it's not doing mathematical stuff it's just speculating calls are specially knows what a call is it knows that's going to be pushing whatever's on the stack and it will come back so the speculation thinks it's going to go back up to that capture speculation part so the speculation when the CPU goes ooh and speculates it's always going to go to that capture speculation and then what that does that elephants is basically flush everything so speculation stops right here which means that every time we have a call rep lean we slowed down the CPU because we're not speculating which way it's going we're basically telling the CPU stop your speculation right here and don't do anything else so that's going stall to seep every time that's it if you have a function that's going there doing this and just going down it's going to kill the performance the CPUs not gonna be fetching anything once it hits that little reppin stop we trick the CPU and to stop it right there and then we're then where we have to redo everything wait for the memory to come back in and all that what does this mean dynamic function calls hat or overhead my code which I'm the f trace the official Linux Ike tracer or of sorry the official tracer the Linux kernel I have this all over the place I have this call this is a snapshot of what's in the code it's messy just look at that that is a dynamic function call and it's all over the place I have this and I went oh crap I but the thing is what's lovely about Tracy is fact that doesn't change much it's you either turn you manually turn it on you manually turn it off so what does this mean again I did some measurements to make this is the actual first thing telling you what I did I ran hack bench with and without it so with wit so I basically ran hack bench with tracing then I ran hack bench without tracing then I turned on rep aleene's ran hack bench with tracing and then turn off without chains so what I did the comparisons here so without rep leads versus no rep lanes with no events which means that this is actually I'm not tracing this is just the overhead of rep liens that it's gonna affect you no matter where you are it caused 4% overhead so hack bench is just this little stress CPU stress test it's not the anything big but when I ran this it caused a 4% overhead rep run e-rep points when I turn on events it caused a 14% overhead 10% more than what I would expect so how can I fix this I need to get rid of the indirect function calls or a direct pointers but you said this doesn't happen often they say the user enables tracing and disables tracing this isn't something that's done thousand times a minute so I could do something big the function tracer which is another tracer within a at which were basically f trace has been named after which also turns no-ops into calls and it does dynamic changes as well of the code and it doesn't do this because it doesn't do it all the time here there so you see where I'm going with this if we have this instead of putting in a rep lean what have we put in this just call some function directly and then when we want to change it we just change it to something else now how does this work we do it for function tracing we did and it's happens with breakpoints I'll explain that right now so when you compile your kernel wave function tracy works is we add two options in the compiled wine we add a dash PG and a dash MF entry and what that does is it puts in a special code at the beginning a bit a special call at the beginning of every function in your in the kernel and then at boot up we convert it into a no op so that way there's no overhead there's five bytes there there's a five basically a five byte buffer at the beginning of every single function and that is reserved so we can modify that into a call if we want to trace it or do something else with it so first thing I do is I we put in it we replace the first byte with a break point okay a break point is something you're familiar with if you ever use gdb and you put in your break points and this is how gdb does it it uses P trace and goes in it actually puts in this CC annex ad and this is x86 terminology obviously on other architectures there's different ways of doing this it'll put in CC and when the code hits this it causes an exception goes into the kernel and the kernel will say okay we hit a breakpoint we what do we do with it zero user space did it or the kernel did there's kernel code or use basecoat so this is what we do in code so once we have this breakpoint in F trace for converting to convert this to from no up to a function player we could hide the system because we just changed just go ahead and just change the code without having the CPU skip it if we change the code on one CPU and another CPU executes it I'm not gonna go into the details why you've been talked before but it can crash the kernel so we have to make sure all the CPUs don't see us modifying this so we put in a little breakpoint we change the code and then we remove the breakpoint with a new thing and we are able to modify running code and I give a talk about this before in fact at the kernel recipes this year you could watch that so how does this work when we hit the breakpoint it causes an exception what the exception just means is that the CPU is going along and we hit the breakpoint the CPU is going switch mode jump into the kernel and jump into a vector and call another kind of a trampoline and says what do we do this well we have a handler we have a do in three handler inside the kernel and inside this kernel I mean this is overly simplified version of it it will store all the registers of that of it so basically registers are the state of your code as you're running your application is running the register are happening when interrupts happen everything else we just store all the registers because that's the state of your the processor and do whatever we want then we restore the registers and go back but we are the interrupt handler or the exception handler is given a pointer to the state so what we do here is we the IP is the instruction pointer and we add 5 to it remember those 5 bytes and then we return it returns passed it so we actually never execute the code so basically we just emulated a no op no matter what was there we're able to emulate a no op and the the kernel is all happy about that this is great when you're going from function to know up and from no op to call function by emulating no op you're emulating one of the states of the transition back or forth so when you're going on a function if you say I want to make it a no op and you put the breakpoint in you just automatically went to the next state we don't care or if you're in a no op and we want to make a function you put the note no up or the break point in your emulate you know if you're keeping the same state and when you return back you've switched to the next state all that works because the know up is part of the state transition but what happens if we want to go from calling foo - calling bar there's no no up there so if we were do this if we're calling foo we have to switch the calling bar by emulate by putting instruction pointer five points or five away and then coming back it doesn't work so what we must do is emulate a function call from that end three Handler and this is where the fun begins because a call doesn't do one thing it does two things I told you earlier it pushes the return address on to the stack and then changes the instruction pointer to go to where you're calling and the return does the opposite it pops from the stack and goes so we need to emulate to emulate the function call we need to push the return on the stack and that's where things get tricky so this is why when we're running and we hit that in 3 remember I told you that we say well we save a bunch of State on the stack before we ever get to the end three the hardware will put in the stack the stack segment the stack pointer the flags all your flags will be done they're like you know you're the state of your flags so you know whether or not you have a compare you do compares those are flags that's all stored it'll tell you the code segment and the instruction pointers all save the hardware does this this is even user space over this thing the hardware does this then we put our own status actually we collect then we put our own register state and then we call this guy but if we want to go to foo and emulate it we need to put a return address on the stack as well so you see the problem here where we need to put the return address it's exactly where the hardware had put its own stack address so it's not just trace events that are problem we could have if we have when you register a single call back to one of the function tracing it will call your trampoline directly what happens is if I say I want to do function trace on a single function it'll create dynamically a trampoline and call that and so it's all doing direct cause there's no indirect calls it'll create a trampoline that calls your callback directly and then it will have the does what's called the no op at the beginning of the function like you're trampling I'll call your trampoline and then do whatever you want then if you say yep two functions and one calls one and one calls too but now you want button them both to call too so you actually now you want to switch it but you don't want to ever have a know up there so the translation of the transition between those two states is no longer function to know up its from function to function the no op is not in the state transition so it's it's the same problem as we had with the function call before but do we care about function tracing yes because have you ever heard of live kernel patching it's a big thing that yep live kernel patching is something that is used by Red Hat and susie-cake graph - Suzy K patches in Red Hat its bigger their systems they what it does is basically you have these high availability systems so it's doing a lot of running you can ever you can't shut down the machine but then you find one of these major bug reports or bug bug issues and you fix the code and the only way to fix the kernel is this is if you reboot the kernel so what they do is actually make another function load it into memory and then chain make it call this guy how do they do that with F trace so remember we have the call they have a live patched trampoline that F trace will call it will go up there and we call the live patched trampoline and when it basically though it calls its live patch call which changes the instruction pointer remember I said before the breakpoint just goes past on the return we just make it skip the breakpoint handler just makes a skip well the F trace handler will actually have the return jump to a new function so say if the schedule is broken and you have a new scheduler it will load the scheduled fixed program in there when the schedulers hit it will jump to the trampoline and then when returned back to that trampoline to the other scheduled fix so this the buggy scheduler code never runs but let's say we want to trace the scheduling as well if we want to trace the scheduler it's going to change to a generic trampoline that's going to call our loop function that's going to loop between the tracing as well as the 5 patch code there but you see we went from calling a fixed bug to a generic trampoline but if we were to do the breakpoint Handler between that you'll see we just call it we went we return back to the bug leaf scheduler so during this transition we're going to run buggy code so we can't do that we have to make we have to go from the live patch trampoline to the generic trampoline so solution number one so it's to have if someone came up from the live patch people I think it was a Suzie groups notice this so see in fact actually I've studied all the other ones I forgot this one because the one that we didn't throw we threw away oh yeah that's right they had a purse tasks we had basically a protest descriptor with different levels you're in whether you're in normal level soft I refuse hard I recuse or ennum eyes that's because when you're in normal context you can get preempted by the soft irq which could be printed by a higher IQ which could be printed by EMI so we had this stacked up it got really ugly and stuff like that but anyway the way it would work is you jump in you jump to this we return to this special trampoline that would then do some case don't care about this because really Lina's hated it it so did everyone else so we just threw that out forget about that solution to what if we could just move the stack so remember I said we had that problem with the stack was there so instead of doing this and have the stack frame we would actually move the stack frame at that point before we called the handler this way the handler could just put in the return address and when we return we're we're right there and that we have the return address simple solution very simple we all loved it but there was a problem with 32-bit the 32-bit Intel does something's different 64-bit memory stores everything and our old the old stack pointers right there well 32-bit because back then you want to save stack and all you don't have much memory so remember 32-bit goes way back to almost the original until the i386 it didn't have a stack pointer and stack pointer was right actually at the it just left a stack pointer since says okay you just know that when you had the exception you just subtract something so you would see where the old pointer is so if you want to see where the stack pointer US was for the task that was running you have it you just offset it you could look at the offset from where you came and entered where you entered into the handler the problem with this is if you what's it called it's just so well I'll make sure yeah on 30 on 64-bit to get the stack pointer member you get that read the regs of descriptor they can modify the instruction pointer you also have access to the stack pointer and there's a lot of cases you want to read the stack pointer as well to see where you came from if you want to do a back trace or whatever so on x86 or 64-bit you'll see that all you have to do to get the stack pointer was you read the regs indirect pointer of SP but if you're on 32-bit and you do this say if you had on the old stack one and then you went to go read the stack pointer stack pointer equals one worse yet if it's an unallocated page so basically or it could be times where you could actually take an exception where that where your stack isn't even allocated yet there's nothing there if you actually were to try to read the stack pointer from kernel space you actually crash the machine and this actually has happened when people forgot to do this so you can't do that we actually have to at we actually have to read the address of the stack pointer to get the stack pointer so there's two ways of getting the stack pointer in x86 depending on whether you're 32-bit or 64-bit so we have a macro that wraps this and this is one of the biggest causes for bugs that go on for it's been going on for ever just every 12 you'll see a crash and that's because someone was developing on 64-bit forgot to wrap this how to access a stack of the stack pointer and when 32-bit ran it crashed so what we want to do is when we take the entry like I said we're going make it work for the same both time so we're going to actually since we have to move the stack frame anyway hey why not on the 32-bit code move it and put the stack pointer information so now we get rid of that hack through out the kernel and make it look really really nice here's the problem everyone loved it we got great reviews from everyone leanness was absolutely against it so what do we do I think the reason why it was against it is this was the last remaining code from 1991 we were getting rid of his baby and he was dead set against us from doing this like I said this is really really old code and aah and we're because we were we didn't realize this because we're all going why is you know why is leanness being such a hard head on this I mean this is it's obviously this is a good solution and the solution he came up with not so good so he wanted these Pursy P variables again this is almost like the first patch and so what is required was we had to make a special trampoline for every synth type of context we were kind of in we had to make a trampoline whether we had interrupts enabled and we had to make a trampoline whether we didn't have interrupts disabled and I think if my part is Lena seems excited about this hack because it was really a hack the way this looked was that in your in you had to check to see if you're an animai and you used one trampoline if you weren't in an animai you had to see if interrupts were enabled and you used another trampoline if you weren't if you're in another animai you were in another trampoline and what the trick was is that you had to jump to a trampoline - so basically remember the stack frame there well when you returned back to the trampoline you actually had you're returning back to a trampoline now you got rid of the hardware stack frame and then you could do this trampoline could add the return address onto your stack frame so you had to use a per CPU variable of where your stack is jump to this guy now if interrupts were enabled when you'd this happened if you jump back an interrupt could come in and screw everything up for you so what he said was well when the exception happens it automatically disables interrupts so what you do is you return back to the trampoline with and keeping the interrupts disabled so you you modify the flags bit to say okay jump back with interrupts disabled still and then modify your flag or modify the stack and then enable interrupts from this trampoline and then jump back he thought this was better so this is what the trampolines looked like you had to enable calls it's like that well leanness obviously liked it everyone else hated it it had locked up problems interacted we have shadow stacks that's coming soon to prevent stack corruptions well it's a hardware feature so like that that you can actually can create a shadow stack in if you if anything ever modifies the stack without doing it a normal way using you know if there's a bug something like a usually a lot of times you'll have a lot of security features done by array overflows where you go off by one bugs or you overflow usually get into the stack modify the stack and do something different there's the hardware's going to a shadow stack if anything like this modifies the stack it will give a fault so you wouldn't be able to do these things so that's exactly what this is doing this is kind of like tricking the stack so we'd have to there are ways around it but it's gonna be complicated that way when that comes Leena's answers was we'll deal with it when it comes to it and then we had to have a trampolines for every time we did it and the thing was Lina said I'm not going to take your solution too and to do solution 3 so I actually worked really really hard implementing solution 3 to make sure leanness was happy with it even though I hated it and I was hoping the other solution would go which was the first time I ever worked really hard on code I was hoping the other solution would win and but the thing is I couldn't even make you know say okay slip this in because leanness was analyzing and leanness is damn smart so if I tried to do anything to trick it he then he wouldn't trust me so I had to keep his trust so I really tried hard to make it and I even asked him for advice I'm like I'm stuck here can you help me he came back and helped me with it so I got it all working good and finally when we put the two up what's called uh he finally said yes number two was better because the code was so horrible no matter how hard I tried to make it look good it was just so bad compared to solution 3 but he wasn't done yet he came with up with solution for ah so only modify the end 3 says you know let's keep it's only it three that's our issue let's just modify that basically he was trying so hard not to modify 32-bit code he started to make the 64-bit code look worse so well he wanted to do was instead of this little trick where we had remember we had this issue where we got we have the flags on 32-bit he said let's put extra space after the flags and then when we do our do it we have twos we have a special do int 3 caller and it would return what's called it would return the basically the regs that was passed to it back and then that's what it would change the stack to so what we want to emulate a system call it would return it would actually do it the call that would be happening would be but let me go back yeah let's see yeah so when you call do in 3 you had the handler move to stack frame not to see code that's an assembly code the assembly code before we had the assembly code move it before we called the C code he's having the C code moved the stack frame come back and now we could return and change our call stack to jump back now we're here and we go back again leanness obviously loved this approach don't worry if you don't really understand it it was really really ugly but it was leanness love this approach but i said to him you know you just really made it just put all this ugliness into the x86 64-bit kernel why I'm like I said to him why are we putting all this ugly code into the 64-bit kernel we're putting all those ugly code into the kernel or into the architecture of the future to maintain the architecture of the past so where we end up well x86 I said we're not doing that he finally said fine ok it's ok with the 64-bit code and the 32-bit was still not implemented so we may just let it die but this isn't true actually something did happen I found out this week I was talking to Peter Zylstra who was implement was when implementing the other code and he said he finally convinced leanness to our solution how the DIF stat if you know what diff stat means how much lines code you added deleted or changed the diff stat of solution to had all this deleted code the diff that of his solution added a lot of code and one thing leanness has been saying he just said it now I really love deleted code so when he got to the point we showed him the diff stats he finally went okay now it basically showed it was him accepting that he's he was fighting because to this personal belief which to me it shows Lina says Perez is a human we all have our code that we love I'd say code is art it's a reflection of ourselves and down those sistas like i said the last remaining bits of code that he wrote from 1991 I mean it's been there forever and he just didn't wanna let it go I can't blame him but he couldn't come up with a technical you know the brilliant Minister Vols could not come up with a technical answer to beat solution to thank you [Applause] right so that's all when you get such a deep insight into how the Linux internals actually work and what it is to implement them and maybe there are engineers here who would like to ask questions I'm now an hour if we have time now or we can move back it went maybe we have five minutes now so if there is a question that somebody wants to ask in front of everybody else and there's two go to one of these microphones please do so and afterwards Steve is gonna be around for more questions right over there in the room do you where is the lecturers corner so here we got intermediate questions now they won't understand anything I talked about raise your hand if you understood Hey okay some people I hate to say how brides are here if you had no idea what I talked about all day okay so don't be shy maybe maybe you may ask a question that is not purely technical yeah yeah okay so we give another chance to is there anyone willing to daring to go ahead and ask this person who obviously wrote a lot of code that you guys all of you yes and if you don't have a question you can just say thank you for your code which is still makes sense okay so just go not there that's in front of you with the chance of you being able to understand by the way this is my fifth talk this week I gave I gave four I'm not sure if this makes sense but there's this time how affect the top a little bit away from the mic cause you're a distorted Aloha does this time how affect like the performance like example jam lick the bite bones oh wait you're saying does the breakpoint thing have a performance penalty on this yeah no well it won't all the things we're okay let me take that back if we move do the shift it's gonna be a slight performance it will be a slight hit of the exception handlers will take a slight hit for that shift but it's going to be within the noise I mean yes there is overhead depending on that page faults may feel because it affect the page fault so it affects every single what's called it affects all exceptions which includes page points it's a slight change just removed it but you only we're moving the stack frame before we store the registers so it's only a little bit but it cleans up the code between the two arcs a lot if actually we're able to make things a little bit easier so there's some parts it's one of those things makes a little bit more expensive here faster over here because we don't need to deal with two different ways of handling the same functionality so yes I know thank you may I ask a question sure if I understand a bit of assembly and what the registers are 64 bit 32-bit if I know a bit of C and I've read through some of the kernel code is there a way to approach you guys that from our perspective you're somewhere high in the skies like shining stars is there way to approach you in to contribute and to help with this development or even to learn and to get involved in some way because mine like my sixth sense is that there may be people in this room or probably people watching this topic and they say are they so cool like these people do such a cool stuff how can I start be doing this well there's three people in this room that I talked to earlier that are going to be yeah they point them out but there's going to be shiny stars they'll be up here in ten years so yes come with me I'll come say some of things I told them Linux weekly news that net lwn net is a great news source to know everything else it's cheap it's easy to go to try to find local communities there read that if you want to know about Linux kernel go on the videos kernel recipes I the videos they showed you there's awesome talks that start from low to high and most these people like I mean just kinda have to read like latest weekly news find things there and eventually if you look into at the kernel and you find a bug report a bug report it directly to the person who wrote it if you do find anything and say look I found a bug here if you find a stack trace and do okay like if you ever write a bug in or ever run the kernel you see a stack trace the code inside the stack trace will show you functions and if you run those functions like a look in the kernel you might feel fine who wrote that code send that person an email saying hey I found this bug it has your code in it they might tell you it's not mine it's theirs but then you send the other person there just listen to it then you that's no I was just tell why I told the people is why first started here I was you I was no different in fact I even said to someone like my first I worked Red Hat my first joint I said I don't I don't think I'd keep compete with these people and the person say come on come on in and in five years you'll be one of them so I'm exactly like the lifetime I've been doing this for a long time I'm only here because of alright done before so if you have an interest and a compassion that's the two things I interests in passion anything's possible anything's possible maybe another question no it's not it's all functions those no ups are in all functions and if you an ape if I if you wanna go back there and show you a quick demo if you were to go into as root and you mount a bug FS or trace FS directory there's a directory you just echo function into this file and then you can see all the functions that are going on inside the kernel on your laptop right now okay yes the - pg - MF entry does that every non inline function that's there's things in the kernel that we actually turn it off for because it has some issues so there's things that we actually turn off for so it's not all functions but the last time on my laptop that I had it was 50,000 functions they did it to within the kernel yep good any other questions maybe someone I have one more okay that's okay the work that you did for this mitigation was it sponsored in some way because obviously the world that interest vote Specter in the processor was sponsored more or less with the work that you and your fellow programmers and the people on the current developers group was it possible somehow by no VMware oh maybe right out of someone else the answer is yes VMware pays me I mean I am an employee of VMware this work I'm doing is all copyright VMware so my work is you know VMware lets me do this so I work on it and has nothing to do with helping VMware VMware knows that helping the community the open-source community is going to help them back so I'm just out there to help it out we had a Red Hat person doing something we have someone from Google working on this we have someone from Intel working on this and we I think we have someone I forgot from one of the banks you know the security actually so there's some people are working on this that they don't you tell me who their employee employer is okay so basically it's a work that results in open source software but is backed by the industry it's the open source community which is not in the old days used to be mostly volunteers people who did it on their own time now companies are realizing that they need to make sure that this works well basically spectrum you heard out scary Specter Mel Thomas and Linux has runs on Apple or Apple the Android and all these machines and banks banks use Linux raspberry I got those the bank's everything else and you saw how easy it or not really easy but you could read passwords and private information because of the hardware so obviously a lot of effort was done from a lot of the community because we need to close these holes and the sad part is this line of attack is new and people are finding out more and more ways so it's going to be something that's why I haunt us the spectre will haunt us for a long time to come ok so was it really in your opinion design mistake or something that was overlooked or put there on intent to be able to be exploited oh no no no all of work in there of course some conspiracy theories that I've been hearing that you know all the little books were intentionally left there no the obviously not I think what it is is the fact that a lot of designs today are all well you know to make things fast and now that you have SP and multiprocessors and some like that you it's a problem with performance versus security if you want to be really secure turn off your cache but now you're gonna have a machine that runs like the 64 or commodore 64 or back in the day so it's all these tricks you guys do all these little tricks to get your machine faster and that just happens to open up the door for things that people like oh I didn't think about that complexity it's the complexity yes okay and one last and then yes because we're doing that there's a lot of link time optimizations the whole kernel it's filled with link time optimizations so what yeah but the linked link time optimizations help for speed up but does it solve these bugs all right oh thank you very much Steven the Great East Side [Music] you

Channel: Openfest Bulgaria

Views: 13,408

Rating: 4.7903929 out of 5

Keywords:

Id: 0pHImHVrI2I

Length: 54min 37sec (3277 seconds)

Published: Tue Jan 21 2020