See what your computer is doing with Ftrace utilities

okay it's time for tea welcome my name's Steve frosted this is the title slide that LCA has told me to do but since VMware paid for me to come here I have to do this to let everyone know that yes I work for VMware and you might say well why is being why I'm working for VMware but they're trying to become a kind of a directive from the CEO like they want to be more involved to the open-source community so I was hired I was ten years at Red Hat came over here I'm my job is to be open source advocate both outside and towards inside the company itself so I'm actually pushing open source procedures within the company and also I'm to come out and still be paid basically full time to work on open source projects and because of VMware I'm able to do this presentation and I'm happy for the support they've given in the support of my work so anyway this is a tutorial and before I go I have my own little thing I do selfies we have to see and I've been doing selfies since 1984 my first selfie with one's little yellow disposable cameras that used to crank for those there's a lot of people way around my age you guys remember those and I was in Hawaii and I kept taking pictures of myself my dad's like you can't take pictures like that gay arm is not long enough bla bla bla those are my best pictures I took so ever since then I always use selfie with a real selfie camera I got a new one here so smile everyone let me get to let's see I get everyone there yeah this one has yeah well if they play yeah that's good I'll post on my Twitter account so anyway I said this is a tutorial so I've given this talk a few times but I've never really had it where I really guys want you guys try to play along ideally you have I mean you don't have to you can just sit back and watch but if you can get the most of this if you can open up a root shell I know if you don't want to do that you can do pseudo on everything but I prefer Isaac screw it just do root because f tres let me ask who here has not heard of F tres okay few people Wow quite a bit good it's the official tracer of the Linux kernel allows you to see events that are happy within the black box of the kernel you get I'm hoping those that say they never have F trace how may have heard of s trace yes okay s trace is the thing you see the system calls which is a communication between the application and the kernel but before F trace you were not able to see what the kernel was actually doing that me slope you might get a print K they could look at D message and see what happened but F trace is actually made to see what actually happens to the kernel came from the real-time patch the real-time patch is something that we really had to analyze the systems because we're trying to find Layton sees and the real-time patched about determinism and we had to be able to make the system completely preemptable and whatever we had along latency we had to find out exactly why so what we did was we did a lot of tracing utilities and this and people asking why can't you bring this to the mainline kernel I rewrote the latency Trace write my own little trace and I put them all together and this became F trace and it suddenly boom it became a whole beast of itself and instead of being a real-time maintainer I became an EF trace maintainer so anyway if you can't I've already mentioned this is the this is the link on my USB stick if you want yes you have a question or you I pass it do to make GUI if you can as well make a pile the GUI the question was can you compile the GUI on this compile the GUI for those on the watching this that are not here this get clone of my USB disk won't be very helpful to you so for those are watching this either on YouTube or live streaming or both you would want to go to get that kernel that'll work search for my last name you'll find that slash trace - CMD get go to that they download the code there do it just basically make make install make goo or install GUI you might need dependencies within the kernel shark directory there's be a readme file look at the readme file to tell you what the pendency would need to build the GUI and you can do that now where you're doing this so where do we find if trace well f trace is basically on every computer here that's running Linux keep it on your phone leave it or not Android actually uses I have friends in the Google community that tell me that they use f-trains all the time and when you root your phone you actually could almost you actually do everything I'm doing right now I'll show you on your phone if you have it routed it's going to be most abusive distributions mount the debug it bests directory which will be insist slash kernel flash debug you can also if you don't want the debug APIs you don't have to compile the debug vests directory because that also adds a lot of debug code which can make your system vulnerable so if you just want Tracy and you could go to cysts lash kernel slash Tracy and you go mount the trace the best directory and this is the command you would do so if you want to do that or okay so let me see how many people who are kind of following along here okay how many people are now with a root root terminal open okay can you see or do LS cysts last kernel slash tracing is there anything in that directory no okay do that command upside wrong button do that command [Music] that one mount - t trace of s slepe snow deaf cysts Colonel Tracy it's probably in your this directory already there's gonna be a debug slash Tracy's directory if they'll use that what I want you I want people to learn how to mount a directory amount the directory so this is a tutorial you should learn so mount dash T trace of s no dev slash slash slash kernel slash tracing got it still nothing oh you gotta be roots you are root oh you got likes SC Linux or something going on there oh really or what is that a Mac okay unfortunately you might have to go look at someone else oh they can't play I don't have time to debug things maybe I'll come with you later after after after this you'll get a free tutorial by your apt if you have time for lunch okay if you I really want people to be able to do this so if you have questions please stop but if you look in the directory this is what you see something kind of daunting there'll be a lot of files I highlighted some of the files that we will be talking about within this tutorial but right now I gotta move on you'll see a readme file Greg Hartmann was actually quite shocked and pleased that he wants to my name goes wow the tracing directory has a readme and it's actually quite a formative the very that way first when we first created this the very first thing that the readme told you was how to mount the trace the festival actually is debug this file system which was kind of like catch-22 you can't see the readme unless you mounted it so when you mount it you learned how to mount so but the one thing we actually tried to do is we actually use the readme file to let you know what features are enabled because traits that are F trace has a lot of features that are config some things you don't want to run on a production system so you'd have it enabled on your debug system so to know what stood able to disable do a lot of times we try to make it so we could just look at the readme file and figure out okay this is what's ain't able to what's not and there's other ways to find out what things are enabled if you look at there's a file called available tracers cat it see what you got you'll see various things you always have the no op tracer the tracer is kind of like special come that you could do or make the tracing system do the know up tracer just basically is a tracer saying I don't want you to do anything or just put you back to default that's the default tracer the function tracer which just function don't enable that yet you could if you want if you enable the function tracer you will trace every single function within the Linux kernel function graph tracer makes it pretty the block B okay I'm not going to talk about this it's how the block trace works that's a completely different maintainer for that so I'd like to talk about that I do want to talk a little bit about them and my old trace how many people okay how many people are kernel developers here just case two couple okay embedded developers or how about people that will like to reverse-engineer binary modules got one are you a familiar theme mio tracer you're not familiar with them and my old tracer okay I'm glad you're here I I did create them my own tracer someone else created it and I thought this was awesome if you enable them in my old tracer first saw it will put your system into a uniprocessor system it'll shut down all CPUs but one and reason why is because all the code that just does the guy did what it was sorry so complex he didn't want to make worry about SMP gameplay it's just like turn off all CPUs we're a uniprocessor no SMP I don't have to worry about locking or anything else for synchronization or racists and no races what it does is this if you're familiar of how a device is when you plug in a device or whatever you know usually way a driver will talk to that device it'll take a part of it or map part of the address space to the actual device memory so you can actually control the device the NVRAM or whatever through this magical address space and by reading and writing to it what the end of my old tracer does is it makes that magical space disappear so when the driver goes to read or write from it it causes it to fault it takes a page fault the page file exception cam and I will trace injects into the page file handler looks at what that drivers writing or reading does the work for the driver records it in the buffer what it did it tells the driver everything was okay go continue so you get this nice trace of all the interact between the driver and the hardware will that be useful so that's all I know about the mio tracer I just know it exists I don't maybe find bugs I think there's another maintainer for it but I I think it's a cool thing so that's why I always bring it up other available tracers that might be like in Fedora has some of these enabled Debian didn't that I'm using a Debian box right now and that's the HW tracer which basically cause it will just put your system into a spin with interrupts off and searching for estimize we're basically the like bias will take over the Machine and it will it might do something for a long time reason why we do this is for if you're doing real-time processing you wanna if the biases hijacking the computer the kernel has no idea about it so you've got these huge a agencies and not know why but it could be because your your laptop has decided to you know check the lower the fan system down or clean the memory we have all things that the bias does and some of those things would take a long time I mean like a millisecond so hardware latency detector h-2b lap is a way that just it'll tell you what it finds Layton sees we also have a wake up lazy tracer that you could enable that tells us the traces from the time a process where go to sleep and everybody started when a process were woken to the time actually schedules in and it traces everything that happens in between the wake up does I always tries to do only the highest priority tasks but if you only care about real-time tasks you would do the wake up - underscore RT if you worry about deadlines scheduling tasks A Sketch deadlines don't worry about if you don't know it I've given talks on that it only does those tasks i recused off you get to trace how long interrupts were disabled and find out where inside the kernel interrupts were disabled which means that you can't do anything the preempt I ramped up is when preamps is disabled doesn't care but I always use preamp tire cues off because that's both preemption or I accused as I worry about scheduling and if interrupts are off preemption is disabled by default but that's not what the preempt off records but pre-emptive cues off records both we won't talk about these but they're good to know current tracer now okay let's start to function trades just so you could de idea just that I mean you could do it and then just to see if it works all you do is bash command echo function into current tracer very simple turn it off echo no up current tracer I mean how many people had their laptop hang no one no one had your laptop had good it works yeah yeah pause yeah that's fine that's as long as it comes back to life it's good then go back to no up anger we'll talk more about that later but actually let's talk about it now I did these slides like last week most of them and then I kind of finished up at the end later on so actually I forgot I haven't looked at my slides again so this is kind of a surprise to me as I go so the echo function there you dip cat trace let's do that now echo function cat trace see what you get you see all these functions I'll use my mouse because I think it's yes those are the functions they're happening on your computer right now so you can see is quite daunting again remember to turn off because Tracy and functions can add your to overhead run benchmarks and then try and enable function tracing and run benchmarks it's funny is you can actually enable several functions I mean I've enabled over a hundred functions Tracy and my benchmarks don't even show it it does eight it's in the noise so it's actually very very fast usually a function trace happens I think I'm down to about 200 nanoseconds per trace or some like that or maybe a little less than that tracing on is a file is a file that you'll see there all it is is an on/off switch whether or not it's basically make the the ring buffer writable or not if you echo zero into the Tracy on so you could type you know echo function then you type Tracy on equal to let's do this enable function tracer okay equals zero to Tracy non maybeyou cat trace and cat trace cat trace you'll see it never updates everyone got that I know I'm gonna be going kind of fast but I have a lot of means I have like place I have a hundred twenty slides or something like that that's okay Rory on twelve I haven't told I speak kind of fast so if you everyone sees that you're you see on so that's much show that Tracy on stops recording what's not in this tutorial and I'll say right now if there's another file called trace pipe that's actually okay tracing the trace file when you read it actually pauses it stops the recording while it reads it because it's an iterator but it's not a consumer you could see it over and over and over again there's a trace underscore pipe file you read that that's a that's a consumer so if you were to do tracing on equal or echo zero and to tracing on then cat trace pipe and let it run because it might take a long time it's it's over it's like 1.4 megabytes per CPU you have that's how big the buffer is and only if I waits pre-allocated it's when you boot up your system it's down to like only a couple K but when once you use the tracer it allocates all that memory for you so now you're wasting your memory as well so if you echoes zero then do trace pipe and then cat again you'll see that the buffer is empty that's how they kill it by the way if you want to erase though if you do Tracy on equals zero I'll give you another Elmo if I mentioned this in my talk but I'll tell you right now just writing it to the trace file it's probably the next thing will the right ear to the trace file will clear it but I think off top of the layer caution make sure you have a space between the zero and the one when you do Tracy I need echo stuff and don't do with forget the space ok quiz who knows why redirection yes you just redirected standard input and output into the thing so I've had people actually complained actually someone that was a very experienced kernel developer say that's not working and I go there put a space like oh yeah crap so yeah it's a common mistake so I always have to tell people that trace marker is kind of cool because like I said this is all the trace Tracy is done within the kernel well let's say you're doing something in the user space and you want to synchronize with in userspace in to trace marker or into the ring buffer so do this echo no op into trace current tracer which will turn off the function tracer and clear the buffer you know make sure you're tracing ons on you know echo 1 Tracy on by the way I will upload these slides will be available so I you could download it now echo something into trace marker ecla hello world or whatever you want in cat trace you should see whatever you wrote is now in the trace buffer everyone got that everyone ready okay cool if you're writing C code or something like that I have several C code I enter that I had this don't worry about it right now you said it'll be in the slides or maybe I'm way have this is in the slides but basic ideas I'll have like the setup trace marker and I will make this global variable called like marker FD Hale said it's a negative one and then I'll just okay in the beginning of the function I'll open it and have a bar file descriptor for it then I had this other function that's like right trace marker which basically checks to see if the marker trace marker is you know less than zero so I have to be route or something to do this because if it fails but if it fails I don't want to I don't want any error messages you know I just like okay just don't trace so all the traits markers will see it markers like not it failed so just ignore the function but other than that you'll actually write into the ring buffer so like cyclic tests which is one of the utilities we use the real-time kernel to do to see how much jitter a the wake up Layton sees everything has that actually writes into the ring buffer with this when they finds a failure actually go right into the ring buffer saying we found a failure here so at least I could we could find that within the trace and then go back to see why it was delayed so folks are tracing is great need to see all the tracers if you see if but you'll see a file in there called available filter functions that shows you all the available yes oh yes okay the question is can you do the same thing for stopped racing actually in my know same files in fact actually in the cyclic test I have the exact same thing except it says Tracy non ft so I do the exact Chua open I should put that in the slide I actually do open of a Tracy I don't just open trace marker I also open the tracing on file and the first thing I do it first I do is I write the trace marker the second I do is I write 0 I echoes or write I do the ASCII 0 you have to make sure it's asking 0 don't to write the binary 0 but right ask a 0 into that and tracing stops too so I don't over I don't lose the data great and everything I just extract the yeah yes to say that's a good point so basically the idea is don't just do them tracing on download it's in the RT tests it's I think I curl that organ thing we also have our t-test or something that has cyclic test in it and it actually has code it does up some of this so function tracing all available filter all the function key traits are in the available filter functions and this means that you could enable and disable if you only want trace one function or 10 functions or harder functions all you have to do is write into the set a trace filter and there's also set F trace no trace which is basically don't trace these functions hello so Moses M be a message which is probably why shouldn't have here sure are you not there we go ok so I'm going back to this that's one make sure I might it might be later in the talk because I forgot about it for what this F trace no trace will do sometimes I'll enable all functions but there'll be a lot of functions I don't want to trace that happen a lot like locking spin locks which will get traced I don't want trace spin locks because that just adds to the overhead so I start throwing a bunch of stuff I don't want to trace like spin locks RCU code so I'm like that stuff that I'm at interested in I'll throw in to no trace but I want everything else traced and it would be a lot easier to pick functions not to trace that to pick every function and filter out what you don't want trace to so you do that the way you enter you would echo the function name into just like normal bascomb an echo foo it's a sec trace if you want to two functions you just put a space or white space between it so in fact you've actually cat available filter functions into it but you got to be careful it does have the module name you gotta cut that off otherwise it'll get error so I usually use a cut command but anyway if you want to add say if you have functions in the set after each filter set after it's no trace you want to add to it you just do the concatenation of the greater than greater than and so just enable the normal bash command how you can add and remove files so and they takes wildcards so yes we wrote a wild-card parser within the kernel that was interesting anyway like for here I just want to echo this in to set F trace filter you could write that write command if you want to see what you get so echo quote you know question mark locks tardy end quote greater than F trace filter just to see what you get and here I can so Jo you do you actually use grep if you wanted to you something super special you could grep and this is where I said use cut command so I use the cut command to chop off the module names of the functions so I only have function names and then I grep for like say I want everything that has ipv4 6 in their name and I pipe that or I redirect that in to set up trace filter yes there's a question back there with the question was was it showing just export symbols there's this showing symbol as internal to modules anything it's everything in K all since so it's every function that's in kale stems which is basically every single function yes function name you label both the question is what happens if two functions have the same function name you enable both of them now it's actually kind of interesting is and I did put it in here I'll know if I have I maybe I did something special you can actually there's a special commands on here and you could read it because there's actually documentation in the kernel source code that you could read that it's in the documentation slash trace /f there's a bunch of files there and one little tricks we do is like echo : mod : module name into this enable everything just for that module even if there's head so it doesn't care about names just everything for that module new label you actually could do that before you load the module and when the module gets loaded then it'll apply it then so you can actually trace it on module though that's that's actually recently done that's a recent kernel like four fifteen maybe they said here's the cut command so I did lock set F trace filter I did let go clock because they don't want to trace clock functions cause lock saying I want all lock functions I get all the clock functions with it just because lock is within clock so I've said lock and then put clock into no trace so I did have to filter out anything and then I got by everything I traced you could write that if you want it's no big deal we'll move on because it's a lot to go function graph tracing this is actually something it's really really cool it's like function graph transient but prettier not only just trace the entry of a function it traces the exit of the function and this way allows you to see how functions who calls what and it gives you a nice pretty display and the way we do this is kind of what we do is when the way it's implemented just still-life quick it's kind of cool so when a function it gets entered it traces it will trace it normally like the function tracer does but then it looks at the return value or looks at the in the stack the return address or if it's the link register or whatever does it hijacks that it takes it out and puts in a new address for the return code that jumps to a trampoline so when the function runs it when it returns then it pops off the stack the return address where it's supposed to go it doesn't go there it goes to our magical trampoline that does a recording of the exit and it but it saved the we're supposed to go puts that back on and then jumps directly back so there's a lot of put this way tracing makes the security folks pull their hair home this is what function graph trace it looks like so you do it now echo function graph into current tracer and look at it you'll see much prettier and easier to read be warned although function graph tracer is going through a rewrite right now I'm rewriting the code because we're going to try to make a faster a little bit quicker and also more flexible to allow multiple utilities up systems to be able to reuse the function graph tracer it's ok function graph traitorously slows that your system down by like you know X amount well function graph tracer will slow your system time down by 4x that amount so it's a much higher overhead although I think we fixed it we got it better on x86 it's actually much the best but used to be PowerPC into my PowerPC machine died so I can't support PowerPC anymore and I have much arm ports but I never got around to it and other people know are much better than I do so I let them maintain that so I usually do all the work in x86 but I would love it to work in other architectures as well and work with the other architecture maintainers to help them implement the best features to make things more faster here you'll see the timestamps which shows you not only that it is records the timestamp when it enters and it uses that timestamp tafero ok how long did that function take be warned because of the overhead of function graph tracer when you enable the function graph tracer if you have if you're looking at like the move down here you'll notice this thing there we go protection with the mouse yep if you here there's several other functions it works called all these guys were called these hold the overhead of these guys so this is an exaggerated version but where you see a single function you'll see a semicolon it actually traced both sides but the tracer will know that oh there's nothing in between these so why put a little bracket bracket we just make it look like a normal function this is actually very very accurate because they only has part of the overhead and overhead still less and I'm like much less than 2 microseconds still in the nanoseconds so this is usually about and this is like what I said it's telling you that this function took point 0 4 5 nanos or sorry basically 45 nanoseconds so that's basically part of the overhead mostly that's probably overhead of the function graph tracer these rice talk about don't trust in the nut x and all that basically I just this is everything I just said so I'm like I jump over oh except the last bullet point the trace filter in the set of traits no trace effective function graphic tracer as well so basically again if you only want to trace one function all you have to do is put use of function graph tracer or the set F trace no trace or does that have trace a filter and it'll only trace that one function so what I'd like to do is this I echo do irq and you could do this just slowly this is kind of a cool thing you do echo do irq do and cap its case that's stiff so do underscore caps irq and pipe that or a redirect data to set F trace filter everyone got that cool now echo function underscore graph into current tracer and after you do that cat trace this shows you the interrupt times how the latency of all the interrupts are happening within your system and it's pretty accurate like I said the function graph traits are still in the nanoseconds so you can see some big whoppers I mean I've done this once I found like something I took over a millisecond I like what the heck was that and it being the Wi-Fi card see it's great but it doesn't show much data both the function of function graphed a search those new functions it gives you very much a lot of information gives you timestamps but I don't see variables I don't see parameters I don't see things that are else a little bit more interesting and looking at the code what's happening within the kernel so what comes about is what we have is called trace events trace events are built on top of a something called trace points which are in with our positions within the kernel there's like over a thousand trace points within your kernel that have that allows you access to specific variables and data within the kernel they're grouped by system there's a lot of them by the ones that I'm mostly concerned about so you'll see scared irq net syscalls module KVM exception cgroups and many many more it's like but wait there's more okay do this LS events those are all the systems within of trace events within your kernel that's running on your machine right now the trace event since the actual trace event is with grouped within those systems so for example the sked group you'll see sketch switch get waking and a bunch of other things process fork that's actually when a process actually Forks you know the exits to the old process exit exact pretty self-explanatory when tasks migrates from one CPU to another CPU when it chases priority we have things so if you do let's do this LS events slash good these are all these scheduling events that you have you'll also notice right there I just highlighted it point to it here the enable file that's how you enable trace events you know like I said here we go echo no up to current tracer turnoff for your function graph traits or everything else you might want to echo one to Tracy on the case you've turned it off and you forgot about it bullets echo one to events /schedule a keen slash enable use your bash completion please tab tab hey I do it for sched which as well and after that cat trace heals yes does a interactive EPF a PF AP okay there's a whole nother talk but these this came way before EPF and EDF has hooks on to trace events yes this is actually how this is this is how you PPF oh no it won't affect EP be f EP effects the tracing the way I wrote the trace event system is very much a abstract module system anyone could plug and play and use it as a utility to subsystem the Tracy is a subsystem so you've got multiple people using it it it'll just it just basically tags it makes it a little loop and and right now I'm actually working on that there's only one attached to it just jump straight to there using some magic you know modify the code like I said the security folks hate me I do everything that root kits do yes so enabling trace events so here inside this whole mess of stuff you'll see I notice here you can't it's not the best like said contrast you uh catch isn't that bad here's the comp force and controller I have no idea what this thing was but I just said here's where woke up here's where or sorry yeah here's where it was scheduled in fun that's just a lot of data by the way you want to enable a full system a group of these events I can say screw it I want an able all skin events just echo one into the enable file and try it it events / get such an able cat race and you'll see everything a lot of data and if you want an able everything just echo 1 2 / enable events / enable by the way you could do it the other way around you can enable everything and then go through equals eros in two things you want to shut down so basically equity one nickel you know and if you cat the enable tell you the state and it's funny because if a group if the events like a lower grab like so if say if you echoed 0 two events enable that enabled sket events or any skin of it and then just cat the events enabled it will show a big capital X means Inc meaning that something is enabled but not everything so what's in the event now you can actually see the event and this is information that we will be using later on which will be so if you cat events scared sketch which slash format take a look at that you'll see this wonderful design the first part of this design of the output okay the first thing obviously is a name a ID which is what's used in the ring buffer and the format the field the first part with before the space this is the general all trace events have this exact same common Vence you'll get the the first one would be the common type the common type is basically just saying what type of n am I what will be there will be that ID so inside the common type will be 312 on this event what's the flags the flags are basically weather internets were interrupts interrupts were enabled or disabled preemption will tell you if preemption was enabled disabled there's also more information in there like are we which context were in are we in interrupts context are we in enemy context or whatever after that and the process ID after that you'll see that we'll have the previous calm and this is the specific data for that specific event and underneath that it shows how do you make sense of all this yes what's that because it must be signed in the type must be signed variable whether or not that so actually that it's a good the question is why is the PID signed up here you'll notice up in the common PID and even down below it says PID says PID signed and the way it does it actually does a little you know is one it has that little trick camera the exact thing like is zero graded less than one or is this type of 0 it does that thing to determine whether that's a 1 or 0 so that's actually automatically done by the compiler so the compiler said this is a signed variable so why it's signed because the kernel made it that way and then this horrible thing that you see is how to print it I pray I made it a little prettier for your viewing pleasure and this is what it looks like you'll see the previous comedy if you look at this is the actual data that comes out this is actually the code that is run executed in the kernel this is actually the printf to display a reads the buffer parses it like you saw there and then it actually throws it in that wreck is the structure based on the way the information was done and this is how it prints it out the funny part is this looks really really ugly but it's because user space doesn't know macros or doesn't know in ohms doesn't know things like that so before it goes to user space because tools need to parse this and but you don't know how to parse like this because no one knows what task top is it needs to know what a number is so believe it or not basically what I what this output is here everyone compile with - capital Yi GCC - Kappa Lee yep a few people it basically runs the preprocessor code so you get to see the output the preprocessor code before it gets into the casual compiler so all your macros are expanded that's all we did we just expanded all the macros and all the enums and everything that's why it looks so ugly it is just because the expanding cuz tools need to know what it is it doesn't know what this is for humans so if you actually went to the kernel code this is what you would see Oh TMI too much information it could be worse than not enough information sometimes you have all this signal or noise or yeah the signal-to-noise ratio is really bad so we want to just see specific things so that's why there's a tracing file called filter so another thing you use for human people is to read your format we don't care about the printf we just want to see what variables are there and now I could do something like okay everybody here let's try this it's kind of a long command just to give you idea echo quote preb prev underscore comm space equal equal space quote bash if you're running bash I'm assuming everyone's running bash if not whatever shell you're running quote space you know two ampersands previous state single ampersand zero X zero to quote greater than and do the old event scheduler here's your bash tab completion actually I should make this I should add a completion so that you actually fill out the other part - it actually would be able to do that with the completion things I'm sure I could write code to do that and they didn't what yeah well no it's not just well no though I'm saying for the command the proof can you has to parse the format file yeah so I'm saying you could write a batch completion script that you could be if it knows that you're in this in the trace FS directory you get it will actually know it'll fill it for you but then you need the quote which then kills it so I don't know but actually later actually I could do that well when we come up to the second part of the talk so everyone try that hey echo one boom well you can't your trace oh by the way did echo greater than trace that erase that clear so buffer if you just write into the trace file with a great design it clears the buffer for you so that trace and you'll notice that you're only tracing your schedule or when your task runs in the the D State the D State means it's blocked on something so that's zero to is I want to be blocked the D state you have to kind of figure that out by looking at the code and that crazy thing everyone try it guy go to the next screen and with it no okay the the time stamp is defined by the I don't think I talked about here but if you see in that directory on the top-level directory is the trace underscore o'clock and that tells you what time step you use you'll see local global x86 TSC raw here's even I think there's even a counter it's just one two three four five six seven eight nine ten for you to bet it right now it uses the sched clock which basically is what the scheduler use and the time stamp is basically the uptime the sec of that's in seconds and step time and the actually resolutions down it doesn't show it only shows microseconds but the resolution of the time stamp is nanoseconds know will not be not because t message doesn't use the skate clock using a different clock there is a boot clock I think when one of the clocks is the like you said you can switch which clock you want to use and when the clocks is the same type of clock as like the from K o'clock I believe everyone ready okay so that's what you get just to show you what how you can filter something that's why I want to just get you but then let's do something even more special let's do event triggers so what does a trigger do trigger is actually let's put action on to it well even using EPP F but you actually make an action happen on your trace trace event so you could turn on tracing turn off tracing make a snapshot what snapshot is basically there's a second buffer you'll see a snapchat file in your directory that's a trace file that's basically a snapshot it's basically when you've enabled snapshot and actually you could enable that's out of the scope of this tutorial but you could actually automatically do it snaps at any time which basically makes the two buffers swap so one is always static and the other one's always be written but once you get to snapshot it swaps it to the other one becomes writable the other ones static so then if you just want to constantly look at things like give me a snapshot boom and you actually have a trigger event happen to trigger it snapshot you can do a stack dump so if you want to see where something is you can see where how you got to that event you could enable and other events you could disable the other events those last two I could talk to you get a little bit complex so say where you don't care about this or the state you see you got okay up arrow says everyone wrote that command go back up arrow and do your previous command make that door if you still have it don't do anything I want you to have that filter enable but let's make a trigger with that filter as well so whenever that happens we ever schedule kind of you could do a cut and paste from because you would need that same filter or that same action I want you to echo stack trace all one word space if space that exact same command you just wrote and pipe that into events schedule so if you're good with bash and you know how to up arrow it this should be quick now triggers are not affected by tracing art or maybe it's not affected by tracing on or whatever the pentane enabled if you enable a trigger this will have the trigger triggers so you actually need both if you want the event to show up and the trigger you have to enable both so they're there too they are not related to each other so the trigger event will happen then the trigger will happen and what you should see here is I what what's that did you do it just like this echo let me take a quick look I see what you did okay if comm equals right here a valid argument oh you've left to em out - I'm gonna wait good hopefully I'm hoping there should I see how many yeah so everyone got that you know okay how old is your kernel what and what's with you do it you name - are ah yes that's that's like five years old so some of these the features have been written in sorry I'm stood it said I said a recent distro so we're all sending a question or was it for the filters you just echo zero or back go right into that thing you also do it higher I'll get to that too after the break sighs so I might screw up your computer so you have to come back and you should see that I move audits we're go we want to do a break and I am guys still got more to go how did disable it for the trigger trigger special I know it's okay we're a little inconsistent but Wes curl-ups kernel developers are sucky at ap ideas API designs API decides so echo echo have to put the quote make sure you have the quote you don't want that quick don't want to put a exclamation point in bash shell without quoting it so echo bang period bang stacktrace bang just have you want to put the whole filter just put the sack trace and into the events sketch trigger and that turns it off so I'm almost ready for the break wait did you need this this yeah just the trigger that you just write to the wherever you wrote to the file to the trigger just write whatever just cat trigger there and look at it and you'll see it in there by the way if you have the source code download it and the Linux kernel the Linux kernel if you go into tools / testing slash self tests /f trace some long thing you could remember that it's their search for F trace and the tools you'll find it there's a F trace test that runs all these tests but if you look at it's all bash scripts and it basically shows you it enables and disables this so if you forget something and you can't understand the documentation you want to actually see an example the examples in the F trace tests code run it because then if you find a bug please report it so okay go on everyone cut there so f tres is a tool with anyone that has busybox so embedded people love me you don't need a tool you don't need to you don't need a compiler you don't need anything but a way to write standard input standard output into a file and read a file from standard APL but it's extremely simple it's the quickest way that's why well the first time I routed my phone I was playing with that trace like crazy because all I needed was a terminal had echo and cat and it worked I'm not allowed to root my phone anymore because it's got work on it and they don't let me do that so but it'd be very tedious a lot of things to remember it's not very intuitive it takes a lot of painstaking stuff to redo this so this is where it's good to have a break and after this I'm talking about the tools that makes all this that basically this was just to teach you bout F tres after this it's going to be a way of making it much easier to do the X Act but this is hopefully could get you motivated to come back okay so ten minute break be right back thank you so obviously using echo and cat and all that is awesome because it's so simple but it also sucks because it's so simple so a long time ago I created this tool with an awful name called trace - CMD there's history behind that name and I would love to change it but it's too late everyone says they use in like they have it in tools and scripts so I'm sort of stuck with it it interacts with the trace of s files to some effect actually if you were to unmount the trace set of file system right now switch to root and run you know trace command show it would actually mount your file you have to do is root it would actually mount your trace the best file system for you and do that so you all everything I just told you about you could forget now well it's good to know you know but everything everything I did in the first half of this talk was basically is basically how trace command interacts with F trace but now you know you could forget about the trace of that file system you only need to worry about the tool so it uses another file if you were in that trace at best file system it uses a per CPU directory because the the buffers of F trace are all per CPU for efficiency and it will actually you'll see a trace pipe raw that's how it reads the data directly from the ring buffer and it just raw binary data which is why you need that the format file that explains how to parse the binary data and how to present the binary data that that print that horrible print format thing that is almost human unreadable but very tool readable it uses a splice to system call how many people know this place to system call okay sort of splice to system call okay how many people are C programmers here yep okay this place I have to tell you everything about this place system call it's an awesome system call it works with pipes but say if you have copy you want to copy a file from on to copy a file from one file system to another file system and what you do it is you would open do a copy open F T the naive way of doing it would be you would open up a file descriptor on one for the file you want to copy open up the right file descriptor for where you want to put it read from the one right well what's happening you're taking data from the kernel we're copying it up to user space and then copy it back to the kernel very very slow and basically wasteful what you could do is instead is create a FIFO or PI a pipe sorry not file a pipe for input and output and touch the file descriptor of the input file to a pipe they create another pipe where you connect to two pipes and the second pipe writes to the second file and the way you talk to the pipe is you use the splice system call and what you're telling is tell the colonel hey I want the data to go from this guy - that guy it says your page cast and everything and the data is already in the page cut so basically it copies from the disk into page cache and then it takes that same page cuts or writes to the new file never going to user space much much faster you could try it write your own copy version and use splice and see how and do the copy gigabytes and you'll see the difference here - - yes here - why was it it works well yeah okay well then you figure it out but basically this is what trace command uses to write from the ring buffer into the file it just uses the spice and just whips it in zero copy from the die so it tries to keep as fast again to keep up with everything so they say forget everything you learned so far now CD to a directory that's not in the pseudo directory because you're going to start writing to disk and if you try to run trace command from debug of this or trace efest filesystem you'll get an error because it will say you can't write to this file system even if your roots I'm already believe everyone installed everything oh there's a make install dock by the way I was very out of it to make man pages so if you do if you've did the make install dock you could do man trace command which gives you all the lists now you could do it's like trace man trace command list J man trace command record all that is in there and I have batch completion as part of it as well so if you didn't make install trace command has bash completion for most of its I was supposed to do more I forgot the speakers dinner went off for too long I was going to write a better that's completion but it still pretty works so let's go back to what we did before and let's try to figure out the one little thing here which just okay here it will be a little bit typing so tres - sandy space starts space - II which means event scheduler the same thing you'd wrote before quote previous calm - MS equals quote bash the SAC same thing and quote continue the line same line don't I mean whatever I put the little slash there just in case so I only had so much room on my side - capital R which stands for trigger and this one of these things trace me I discard the the start and record has got option bloap so I'm running out of letters to use so some of them I I don't like doing long I do have long commands that represent it but I still like to do short commands and then the same thing stat trace blah blah blah blah hit enter trace command show and you got the exact same data yes you're right that was a cut and paste there you know actually it's funny because I did that and I got the air and I fix it but when I did the cut and paste who's the broken reverse it because I was trying I did a lot of cut and paste here and get any things here so basically I could I did all this on my laptop well just the last couple weeks ago everyone got that no come on you're just uh peril to what you already wrote before and just cut and paste what you've got no output oh wait you got no uppal from the previous one oh are you writing Bosch Oh okay if you have to see me you should talk me through the break I would have played Tilly so it would be any errors see me afterwards and we'll work on it I want to make sure you guys actually everything works and this is great because it's not working for you I'll take it back yes hit event oh you know what because I'm actually either funny part is I'm triggering off of when your task is blocked so maybe you have to do something so you just say enter its if your systems not do anything maybe it's never hitting the blocked event so holy the event ever actually occurs well maybe I should we just have you guys removed the previous state you could see everything or maybe change that make make previous state equals zero that's so you could switch instead to a previous state and zero to make it previous state equal equals zero try that see if that does it so for both sides so it's Oh see yep you actually both those are doing an event you've been this because I probably should have said this is the block state if your systems never blocked you won't get any output something that wasn't a good example okay here's something is much very useful and it's not fully developed but it's getting there it will be there more so please keep updating your system do treats commands stack this shows you everything you forgot to reset now do trades command reset and then trace command staff so this cleans up all the dirty crap that so those I didn't come here that have that all that stuff on their system still running slowly their systems down they don't have trace command reset unless they're you the watching the live feed Daniel okay by the way trace command reset does something that if you ever do trees come in reset and then you jump into the file they start doing stuff and like why am I not getting output trace can't reset did something I I kind of regret doing but I'm afraid to not do it cuz I'm very Marie I'm likeliness Torvalds don't break backward compatibility I very much have strong rules with nope backward compatibility so I don't change things very easily if it did it one way it kind of always does it that way even if I hate the way it does it treats me reset well if you do trace command stat you'll noticed there you go straighten it's disabled so that means that means Tracy and on file has zero in it so a workaround if you do trace command star because fishnet start resets everything before it does something and then I say start the no opt tracer if you matter what that is do nothing you'll notice that Tracy is enabled so basically what start being no OP is it's just a hack that basically uses the fact that the start resets everything and then they start nothing so record and start so start enables Tracy just like you did it does a cat show you can see basically everything that we did up today record is something new record is saying I don't want you just to start something I want you to save it so this way I want to record gigabytes of data and you can do that really really quick like basically in half a second with the function tracer so start is compared with show record is with report they use basically almost all the same commands so start it record are almost identical for almost every single command e is for events P is for tracers Y first rope trace command we took all the tracers plugins so I use - P I wish I wish to use - T but I'm sorry I said I don't break bread' backward compatibility because of - P it it's we have plugins has another meaning now but historically we're stuck so now the title of the after the break and everything Allen gave to what the title of this tutorial is about let's see what our computer what the computer is doing let's write ourselves a bash script everyone let's okay open up your favorite editor then Emacs Nano Jo - and echo and let's just write this hashtag bang slash bin slash bash make sure that's all together tight no spaces you can put a space I don't care what you do new line multi new lines where I just echo quote hello world bang quote anyone got that do I need that if we got it good so I talked about s trace let's run s trace everyone should have s trace its installed if you don't you should install it someday and if not just watch what everyone else is doing so when we do s trace hello and you do ampersand greater than everyone knows about and person greater than or yeah bless they are yeah I agree that just basically redirects all standard input and output to one file so you have to do the magic stuff that's a bash enhancement and then cat out and or vim out or Emacs out Jo out and a no L and what you'll see here is the exact V where X occur execs hello does Lowe's a bunch of libraries did it it uh it Maps stuff a lot of more M app calls accesses more libraries reads this elf file where the hell that elf file come from war libraries biz waited mal file more libraries more library or more mapping continuing on it uh open at now it's like opening this hello thing I'm like wait why is opening a hollow now and does a whole bunch of other stuff reads writes here's our hello world Pope there so here's that we look at open app and the read you see our file there on s trace so it's reading or hello it reads the bash care there and explore so this is basically our bash stuff going on right here but weird what's all this other funky stuff that's going on so how do bash scripts actually work on Linux I'm sure several people already know the answer but let's take them for let's do a deep dive into the actual kernel so let's have a look at exactly which is how you exec you know basically it's a way of changing the world of a program from one thing to another thing so first thing I do is I want to find the exec V system call which in the kernel now the kernel could call it anything I could call it Bob you know the function but luckily the system calls are sort of mapped so let's do here do this trace command - the list or that's lit or sorry trace command list - F which means function exactly E and see what you get because that you'll get different things depending on which kernel you have because they've changed this throughout the time so I don't know what you guys so search for that sis what you'll get nothing so trace command lists - f right here some of them you'll get capital S little Y capital S yes this is new that's why I yes what kernel my running is the latest Debian testing kernel a what I mean I did a apt-get idea app to to update what oh ok yeah oh cool I had it up there for 4.18 so that's the Debian testing kernel like I said I want to do this on a distal kernel so I run Debian testing on my laptop and that's what I did it on so but I know sent away oh it's a sin - lets guy believe or is he still you up though yours set to s is probably I don't know what you got on that one but that's why I told people to do this oops Ted go back there's one make sure yep so again for more information man trace man list if you want to see what that does but we don't details that we just know well we found exactly so let's G okay actually you may not have to you actually get this is case-sensitive be careful actually it may not be enema tripe trace khmer record despi function underscore graph - G quo star the prefix you had capitals s little Y a couple of whatever that prefix was underscore exec just exe see if you want star quote and your hello script that I hope you made executable because it says command not found or something like that you forgot to put change mod + X hello everyone got that one did that was it alright okay any problems let me know everyone got it everyone ready yeah we're not ready did you change mod or invalid object oh wait what colonel do you have and you've got an invalid object for you've got the trace memory chord SP function underscore graph did you have a function graph tracer enabled yep - G space - co-star but if it doesn't it did it'll be a valid argument if you don't if you don't put in the right yeah the case in case sensitive on the yes there's K sense that you don't put the right if you don't put the right function name you'll get invalid argument was that what did you type a - G it looks like my point - f okay where's Eddie let's go and they trace from a report so now you'll see the actual function of your you'll see your events you'll see the functioning that started and only that function the dash G means graph that function only and you can put more than one function in there so it doesn't trace anything else only traces but there's a limit of how many functions you could trace or do that to but it will let you only trace that function so maybe I'm only interested in one thing so I look at here and yes it's pretty big lot of it from there my function it's sixteen thousand eight hundred seventy function calls for hello world I'm sis exec not even hello l just sis exact allotted for there but interrupts also happens so if you do trace Khmer report - capital I remember I told you it tells you what context you're in it will then remove interrupts because while it's recording that interrupts are happening like crazy on your machine every time you're even typing or whatever that or timer interrupts or whatever it's going to you want to record that because even though the graphs that function if the inter up happens while it's graphing it records the interrupts and then goes back down so - Kapil I tell us the trace compass it's the data is already on a trace that file that we could actually analyze we'd tell the report I don't care about interrupts or soft interrupts there's two types or the hardware update of the soft erupt so so if you do the trace of a report - capital is piped at the WC - l to see how many vents it should bring it down quite a bit and we're okay with that so let's do tradesman record function graph let's what's that oh it's no it's no there yeah well mine was over ten thousand yeah add this unfortunately this is a long command but I think it might wash the place it might work if it was installed properly so it might work I don't know if you do - - max - graph maybe I'll fill it out I'll know batch completion works or not it'll be my first Detroit it worked put five and then run it again run the record again by the way a feature in this will be it I should be having to write in the record again the report should be able to do this itself but I have implemented that yet so that's on the to-do list if you run it again it drops down by a hell of a lot what that means is only go five levels deep because I don't care about all the other crap I just want to see what just let's see what it's doing at five levels deep so 347 events is not bad shoe I've lost a output anyway yeah cuz I think it might have been in the very beginning yeah you'll see that a lot of these things what you'll notice is if you look at your call you might see something like condition reschedule also kind of busy I don't care about condition rescheduled like all over the place so if you want to remove something if you put dash n which means no trace which is basically the no trace file you could tell I don't trace condition Reece can't either so if you do this whole thing some by the way some kernels mites that not supported cuz it uses no graph trace doesn't really do there's no trace use the no graph trace so might not work on some current older kernels - and you could make it even smaller so I got I've lost another 100 functions and let's just do four let's just do it to four and doing what so basically this is you'll have to do it I'm going to show you what I did so I did a report - cap is and I'm just looking at these functions I'm like okay we have the exec it gets the name across the file and interrup happened but you know what the flight part is reason why SMP irq work interrupts this function gets called because I was tracing graph the functions it gets called but before the interrupt the kernel could set the context so the current inside this it's inside the irq enters where the context could set so actually you'll still see interrupts happen but the context for it is not recorded in the buffer until the colonel says it's the context of switch so we're actually record there's a little race window there that we're recording events when an interrupt happen beep like from when they interrupt actually happens to where the context is Colonel says we're in interrupt context well we get the functions that are called between those two points that's why you'll see that you'll see these in there so user share files stuff like that that's the interrupt let's see here you've got a bunch of open execs gets which Malik and keep down copy string fun stuff here we go you'll get to search binary handler thing hmm and then down here you'll see load script this is where it's doing something about loading scripts that sounds like what bash would do yeah you get there yeah it did a lot of work didn't it's amazing so let's see here what is this load script doing let's wreak let's do the record again but instead I just want to graph load script max step 3 if you want I don't know doesn't matter and non so you could record that so basically just up arrow delete everything just let's see what the load script is doing and hopefully your kernels have load script on it everyone's got it set go guy oh look at it so I look at load script you get all this lovely stuff F put it's just basically putting a file descriptor away whatever copy some strings more things does a search bodies thing and and here's something very interesting I noticed it calls load scrip again load script calls load script now you realize inside the kernel that it has a fixed deck we are we I saw this and went wait a minute that's recursive call we don't do recursive calls in the kernel because recursive calls can easily blow away a stack easily and cause very bad things I might oh my god this is like bash it's calling most great let me a little look at what it's doing I'm like what you know is this a bug can we exploit it so I looked in the source code if you have the source code available you look at it could do a search for a load script I believe it's a crime for which files in but anyway you can look at it and I said hey look I found look what it's doing right here in site load script it searches for it checks the I believe this is called the binary parameter so that's what the beep erm that I I'm like that's one of the worst acronyms ever to have I'm like you see this beep erm all over the place in the kernel exec code and I'm like what the heck is its binary something what does PPR mi thought binary parameters anyway well the buffer that's the buffer that's the file that basically that it's mapped it's the beginning of the file and it checks is the beginning of file that you're about to execute you say I want to execute this file it reads the file what's the first couple characters and say hey you got a pound negative sign here you know if it's not that exactly this so we're loading a script here and then it calls later on down here search binary Handler so I'm like hmm can I repeat this you could try this now this is going fun create that create a file called you know bang pound slash temp slash blah and type that in and put that into the file called slash temp slash blah so in other words I'm telling the the loader to load itself again which means they'll load itself again and load itself again I'm like I'm going to crash my system are you nervous change mod X and run it let's see who's brave got it you everyone get this error too many levels of symbolic links telling us symbolic link it's screwed that's that's a bug incorrect Erik's actually that's probably whatever they encrypting bad in bed interpreter that's that's true that interpreter so what happened let's record it so let's look at loads let's graph load scrip and on blah so I go there and this YC trace came a report - capital o you could do to you should be able to do your bash completions to see what - capital is like basically some of the events interpreters there's like plugins real plugins which means how do you read this event so you can load a plug-in to say how to read in a particular event so the function graph tracer is a plug-in and the plug-in has options and one of the options is tail print which means I want you to not just give me a squiggly on the exit I want you to tell me what the function name of that function was called so I get the get this I get these loads scripts comments at the end of the functions without that without tail print this will just be a bunch of little squigglies so if you want to see what the function is I was it so this is what you see and that's how many layers so glad said one called one well maybe this is the same level so one two three four five six six layers it kind of died there so let's poke a grep - see - I'm going to give a little more information to see what's going on here and it does a lot of work here and down here you'll see it does something different the first one does search binary handler I like wait what's this thing doing so I go there and look at this recursion depth greater than 5 there is a check to make sure that it doesn't kill itself so so this is interesting entire able to you it's just the events please I don't know what I'm talking about let's see the world so if you want to trace everything that's right this is about function tracing let's see events so let's see all the events so if you do trace square - e all it will in sleep 10 you get this everyone got that do trace command you know record - a all this is the command what I'll keep it up okay Jason we have record come on type as fast as I talk so here we got it okay tres come in re tap tap evasion should work yes wait was it if you just said if you just type record without the sleep ten just without putting I sleep ten just let it go it will say hit control-c when you're done and I'll just keep going into your Philip all you to space yes well it just keeps it try to keep up it's it's a producer/consumer you lose events yeah and you'll see inside the thing event lost yes it'll tell you and sometimes because of other since I'm very much backward compatibility I can't change the way it did and I want to add more information about it if there's space in the end of the buffer because it's done program by pages and there's space to play like if there's four bytes space to tells you how many events what's missing otherwise this has a flag it says events we're missing I'll just say dropped events which means there wasn't room to put how many but if it has room will tell you how many events were the dropped but you always know that events were dropped so you get this right really informative if you're a computer how about this the Contras may not be the best but hey welcome to Colonel shark 1.0 well it's prettier chrome shark was created in 2009 the gtk version I created because I had a month to play around something and I just and I was a dtk programmer before I came a carnal programmer way back mid-90s and my kid I'm gonna brush off my gtk I haven't done it in a decade so I would really urge gtk and wrote this thing and but then I started going back to worker then and curl shark became my idol tasks so I only worked on it when I had nothing else to work on and then one day someone said you know I have to see a commit in two years well show you how busy I was again talk about working for VMware my dirk hondo who's here who's one of my it was the vice-president here he don't mean Steve is there anything you would want you know it open-source world to work on like I need Colonel shark working on slow a developer name you're done I might but pronounce his last name because they always mispronounce it from Sofia Bulgaria was hired and he's done a fantastic job he liked cute cutey and I told no what I've been wanting to rewrite Colonel shark for a long time because I did it just kind of rust it and I'm like okay I found out they did a lot of things I didn't like so let's just throw away the code and write from scratch so for a year he's been working on it so now just this yesterday I finally to lead to the gtk version and now we only have the coupe version and it was written scratch from Q by you're done I've been hello I've been mentoring him he's been doing a fantastic job so actually I've introduced 0.9 should be nine eight actually I'm afraid to do 99 congratulations who are all my beta testers Bugzilla kernel.org please play with us report any bugs you find yeah who likes something wait was someone here he got a core dump awesome originally okay I was really going to say I was originally going to say let's call it 100 but your honor I was like waiting oh yeah well you and I've been playing with it you know someone else was might break it I'm like okay captive audience if you go to plug Zillow that kernel org you see tools utilities you click on that you'll see trace command you can report bugs there's a Linux - trace - - valid wait there's someone send me an email about a bug about compiling on like where they heard um Ken I think it took it was you okay yeah so you had issues yes I got your email I think a chance to reply yes thank you yep yeah that's what she's got past that but I was like that's right you're using slack we're right wow that that's that's why I started on but hey anyway one point always kind of coming out time we call it 1.0 because that's what my slides were originally everyone so dude racing my record what do we do that whatever you that all record you did that thing right just right Colonel shark did you get something like this awesome well like hopefully you don't have to yeah well it's not much information about what you do it for did you do this race communal let the trace command record all oh yes make sure you do a trace command record - II space all I'll go back to let's see that one what we don't have anything for remotely yeah I mean you could do it over X Windows I mean I I've pulled it usually s state to a machine and have it pop up but I just use it like the SSH tunneling okay yeah you might we don't have we don't have that working yet that's that will come there are this will come this is where I wanted you here oh okay well you know afterwards we'll work on if you don't get it just watching will work our day afterwards where I do oh this is this right here on the top you'll notice that I said tasks he was so if here you hit plots tasks hey you'll get this window it's a time for some reason has a bunch of MySQL demons running so I just pressed it hold press the left mouse button hold select them all you have to hit enter this is one of the things that this little difference between the two but for biggest it's a cute thing I don't know but the quiet objects I always select them and then hit apply and nothing happens and I'll go by Mike oh I forgot to hit enter because you could select him but if this doesn't actually select them so when highlighted they're not selected only if they have the checkbox are they actually selected so we have to highlight hit enter selects is everything a highlight or if you just click the checkbox that works too and then you hit enter and you get well oops oh wait yeah that's how you play wait since I've switched we have less than ten minutes I have this chart go ahead and let's do Java how many people are Java programmer I feel sorry for you I'm not I was just pretending this is my this is like the first Java program I wrote in probably two decades everyone typed this just copy it from someone else then and say you did program in public space class place capital hello capital world [Music] Open bracket public space static space void space main prin capital s for string left Breck right back back args and paren squiggly bracket this is why he had a Java capitalist system dot out dot println and this kind of like why I'm not big Python fan I tell people I'm afraid of snakes I like oysters see how many people got that one : hello world well and everyone got this yet I should have put a prerequisite you had to take typing one on one before you came here I'm telling everyone's got it or not no no no hey anyone else not ready hopefully you have Java installed because if you have Java installed you just wasted all that time it wasn't a prerequisite because I was afraid I want people to show up for my talk almost got it or could someone just show them as screen so I could go on yeah I got five minutes so you just put up to five minutes like that's why Mike gotta go we're almost done okay did you get it well actually I should put it on my USB disk and just set here open the file clear just give him the file I tell you think about doing that actually probably throw it in I'm gonna have to go I'm sorry yeah okay couple more seconds because it's not a big deal it's it's if you do it because I'll do it with you afterwards I gotta help you anyway I try to be nice got it okay I'm going to jump ahead okay now when you have you got that good now go wherever you save that file remember where you save that file so you'll need a full path name for it I believe if you go up into tools go down and hit record oh yeah don't worry about it so hit record by the way you could have been non-root by the way now you don't have to be root anymore in fact actually don't be root because if you're not root it well now kernel shark will now ask you for the root password you type in the root password you hit enter and actually we have a separate code to do the recording that's very thin line so it's more it's going to be better that we could vet it so you're not running the GUI anymore I mean it's it's a small GUI you're not writing the full GUI or running a very small GUI as Roo it's a separate actually it does a separate execute a separate application to do the execution with a special pipe between the two so once you get that you should get this file which is separate from the actual application so so we try to secure it a little bit better it says it's still we're still working on it we still have to vet it it's still not totally secure so but this is vehicle people who ride his roof and you'll notice this is still this is kind of we're still working on it it's going to be much more feature ful soone but this is the old Colonel shark didn't have this done very well we'll do this so you come in here and you'll see you know you export your you could do a bunch of things and then export your settings and we'll save it where you import settings they'll pull it back so whatever you do not do it twice you click all events it'll select everything which I think it's a default or you could go unselect it it will unselect everything and then you could kind of click in and out picking events well let's do all events I want you like to get to this no you just had Carl shark oh you take currents are getting hit tools record and then it gives you this and then gives you this and then you type in here Java whoops let me go back here this is where the output file will be right now it's my home directory type Java space where over the file full path name to your file then hit apply don't hit just hit apply because apply will show you will come up here and put everything it will put the command the trace command record up here and then you could go when you click it has a read only thing you click that to turn off the read only and then you can add in let's do - capital F - see this is because those things that we're going to add later on - down here but for now we let you modify you could put anything you want in here this is actually a command I'll be executed so you can put any parameters capital F means follow this function only trace this function don't trace anything else but it will trace wake-ups which means it'll trace events that woke it up and - c means spot record all the children for that as well depending on certain kernel develop kernel levels you might have different ways of does it they will either use P trace poorly or newer kernels have Colonel support for it which will make it it will definitely work very well so I'll create your kernel and once you let it go and hit captures and we're by the time I got to get going go down here not hit capture yep one minute welcome to hello world this is just helloworld.java cord um you get that too which is a bug which they said bugzilla report cross them but anyway this is the output of hello world of Java and this is everything you could do in here thank you I don't think we have time for questions or anything so yes yeah I I'll stay after well go round here I'll hang out here help anyone that wants to do anything any other questions feel free and we stay out here for for all of lunch or something like that people go get lunch or something come back but yeah I'm here to help thank you thank you very much

Channel: linux.conf.au

Views: 17,614

Rating: 4.9657145 out of 5

Keywords: lca, lca2019, #linux.conf.au#linux#foss#opensource, StevenRostedt

Id: 68osT1soAPM

Length: 91min 5sec (5465 seconds)

Published: Thu Jan 24 2019

Reddit Comments

[u/foxes708]

if i had better knowledge of bash scripting,i would attempt to rewrite some of Brendan Gregg's utilities,because unfortunately,they don't utilize the capabilities of the tracing subsystem to their fullest extent

[u/Galaruss]

Any other talks from this or other conferences similar to this? Follow-along plus presentation style?