Real Time is Coming to Linux; What Does that Mean to You? - Steven Rostedt, VMware

so fellas here this is about real time is coming to Linux now you might be all laughing about that because we've been saying that since 2004 ah maybe I 2 times before I think we were saying it about 2008 so as a whole joke was you know this is a year you know real time it's going to be merged into Linux it was like saying this is a year that you know Linux is going to rule the desktop was equivalent this time we actually mean it and how I know we mean it and how I know we mean it this time is the fact that all our focuses our talks our meetings are not about what do we need to do to get real-time into Linux all the meetings are now like oh crap we're getting into Linux the job's not done not the fact that it's not done that there's not more work to do to get into Linux and saying that our job's not done once it gets into Linux there's a whole new set of problems that come up one thing is it's now needs to be maintained kernel developers can't break it if it does it's our responsibility to teach them not to break it again so before I go further ado I have to get my you know selfie oh shoot the battery died No oh crap I can use my phone but that's cheating so no selfie today the battery just died as our first okay so who is this talk for well it's mainly for a Linux kernel developers I mean how many legs Carl developers are here okay how many people are not letting skirl developers well 5050 I'm actually pretty impressed pretty good so if you do core kernel driver codes file systems they'll pretty much anything it touches on Linux kernel you're this talk is for you it's also for those that are just curious about what is this real weird thing we call preempt RT you know what makes it different it's also for those who want to see how fast I can talk so what is real time it's kind of like the question of saying what is your favorite color so the term is ambiguous I always hate that term so I looked it up lesson this morning actually started writing these slides at 5 a.m. seriously and I went there I'm like woman I talked about real time let's see what the internet says about it so I went to one of the most most common sources of information urban dictionary and I got there it says real time instantaneous taking place at once as all other things are also in progress when I survive the situation in real time there were only four people who met the qualifications or surveyed the prison sorry I survived surveyed instantaneous imitates blah blah blah okay I was actually pretty impressive from urban dictionary you had something decent then I went to what is calm it says real time is a level of computer responsiveness that a user senses as sufficiently immediate or that enables the computer to keep up with some external process I name definitions for that it continues real-time is a adjective pertaining to computers or processes that operate in real-time I love definitions that include what you're trying to look up is included in your definition and they actually had more real-time describes the human rather than a machine sense of time actually I think that's the most accurate so that's not what I care about I hate to term real-time because it has all these means you have real-time tracking when you get ups or your go to Amazon and you get your package and you hear all this real-time but the question is what does it mean to us the preempt RT aka the real-time patch by the way that's another hint why it's giving its Linux we no longer call it the real-time patch because it's not going to be a patch anymore and we call it preempt RT now means determinism has nothing to have speed has to do with knowing what's going to happen so we care about latency we always calculate what's the worst that could happen because I always say it's knowing what will happen when it will happen so that's what really real-time is to us I always say there should be not called a real-time operating system it should not be an artist it should be a deterministic operating system so what's our strategy to making Linux into a real-time operating system well first thing is we have to make it as peremptory as possible we don't want we want any time a high priority tasks wants to do something we let it do it as quick as possible so let the user determinate what's important and as soon as they're defined tasks of importance wants to do something it should be able to do it and give it as much you know power as possible and try not to interrupt it in any way so we try to remove preemption interrupts everywhere that we can so we always have scheduling as powerful or everywhere so always let the most important highest priority tasks run when they want to run that's what we try to do so if you when real time comes into the Linux kernel and you go do make menu config and go to processor type some features there's going to be a window that pops up in the printer will be a preemption model before the first three is only what's there today you'll see the first three no force preemption volunteer of voluntary kernel preemption and preemptable kernel Louis lead us up to new things are headed and for full preempt RT that's what we get so no force preemption is the old days only notes anyone ever run with no force preemption without config preempts none anyone do that a couple people I almost want to say why voluntary preemption is probably it's like that's mostly the default Red Hat I please lose sloths Debian voluntary preemption basically anytime you hit like a might sleep within the kernel it will say hey we're telling there was a debugging option put called might sleep to trigger if you had a case where you're in a critical section and you called this function and it might have a path that it might sleep because a lot of times what would happen is you could run your code and everything's fine but there might be a function with a path in it that will actually go to sleep and you'll sleep while holding a spin lock and your system will crash and like oh no and these bugs get boiling up so we throw out the kernel we put these might sleeps throughout the kernel so that if you hit a function that might have some if conditional some strange conditional path to asleep you'll know right away that oh we can't call this function from a critical section let's call something else or let's fix up our code so that solved a lot well what they realized was all these might sleeps that are scattered out because these functions can sleep we can sleep there it's this also told us a dysfunction even though is one little path that might sleep it means that we can sleep so voluntary preemption took up on that and said hey when we hit this might sleep if there's a scheduling something needs to be scheduled let's schedule right there so that's what voluntary preemption is it took this debugging app option that helps save us from no calling these might slips and just use it to say we can schedule because we know this function can sleep it's okay to sleep is it okay to schedule out that's let's use it great it actually helped a lot I I don't think I would love to get rid of server nun because it's pointless I think that might sleep ones are perfect then preemptable kernel is what we have today Lola a CT desktop that's where basically spin locks become preemption points so we grab a spin lock it will disable preemption and you could sleep almost any time you're not holding a spin lock or don't specifically say I don't want to sleep so that's throughout the kernel so it's actually very very already very reactive it's it's basically a soft to real-time system but we have two new things so why don't I kind of ignore is just the basic the first one basic RT that's more for testing purposes on us but the next one is where it's full preempts RT and this is why we call it preempt RT because that's the config option so if you're wondering why we always say hey it's the pre up to RT patch or preempt RT now and not the RT patch this is why because it's going to be a config within the mainline Linux kernel so interrupts as threats you can always do one of the things to help get a pre-emptive kernel is to have interrupts be able to threads you know long-running interrupts you can't do anything what it's going on so we got to be able to preempted so the only way can make it printable is it has to S has to have its own scheduling context and be able to switch so request our key or to request threaded irq it's been there since 2009 how many people have used requests threaded irq hey good thank you appreciate it awesome because that's even a mainline kernel your thread is or your interrupts is not keeping important processes from running that's that's nice of you also if you ever want to do it there's this thing called threaded IQs which was added as a debug option to debug certain criterias where was really because we want to get our RT clinic our real-time patch into the kernel so a lot of times we would dress it up as a Trojan horse and say hey guys this helps you so much this is something we always had rationale they had all the real-time patches that may or all the real-time stuff for all the patches rk4 the real-time patch we're basically hidden as gifts to developers locked up came from the real-time patch mutex came from the real-time patch let's see generic I accused the IRA cubes making all the IURC you systems basically into core architecture it came from the arty patch this was because we couldn't do our tea without it but people liked the code like hey this makes things cleaner because - one thing is real-time patch requires clean code so our gifts are cleaning up code to give to you so but this one was kind of like hey you can debug your system with you know making all your threads or all your interrupts threads that way when so if you have something crashing you can debug it easier that was a rationale that's what we said but almost all threads so only becomes threads except for maybe no if you Mart especially market to say no thread and those are timer ups IPI processors don't get saying I this my interrupt is so important I don't want it ever to be a thread because people will hate you for that just lets you normally door Pamela is basically so we have a high priority task this is why we don't like interrupt handlers that are not threads because you'll cramp to the task the handler has to finish and boom so when you request an RFQ you'll get you're given a single function called hand or handler which is a function that's going to run when your interrupter is triggered when you do a request threaded irq you're given a handler and a thread function now way that happens is when your handler takes execute it'll go up do the handler in this way if you want to show off your device because say if you have a shared interrupt line then you have like a big log if you have a one of your threads where your handler takes a long time shared with other devices you may want to run it as a thread this way you could go acknowledge your device and then go away and just do your handler mazie enough I searched the kernel and very very few people do this which is actually a good thing and I'll explain why in a second when you do force to thread it interrupts which is basically when you to enable preempt RT it does force the force threaded interrupts becomes on and what that does now is there you have the arab triggers you can't stop interrupts you know the hardware that's a mechanism of hardware so you trigger there up and you say okay all it does the interrupts says okay we're going to with disable that interrupts line and then wake up the thread for the interrupt thread to handle everything else so basically I would just turn off the line so no more interrupts will come in and we wake up a thread and that's all that I I would have made that smaller they interrupt smaller but then the word interrupt couldn't fit in there cuz this really actually this isn't this is not to size it should be a sliver so interpose off Boop right back so it's really really fast you know a microsecond at most then it runs to your high priority task and we have priority tasks you schedule now here's the kind of interesting thing you'd be used both I say if you have a top half and a bottom half now it's I'm like I'm sorry but you're gonna have to scheduling search because your top half is still there run as a thread and your bottom half is still gonna run this right so we're working on ways to make it run is maybe a single thread you know may below then get rid of the schedule switch we have to we're looking at ways of fixing it but for now yeah so we enabled preempt RT here's the big thing this is the big hammer this is what's getting into the kernel and this is exactly what we mean by real time is coming into Linux it's the it's the option we've been all dreaming of turning spin locks into mutexes they're not really a spin lock anymore are they so they don't disable preemption they don't disable interrupts even if you tell it to disable interrupts if you do spin lock irq or spin lock irq safe nope it's the same as a spin lock same function boom so how do we do this why do this well the reason why spin locks disable interrupts if you do smell like irq safe or spin lock on your cue the reason why you do that is if you have an interrupt or a spin locks being that it's being shared between thread context and interrupt context so if you were to take a lock and not to see if you later ups and your process gets preempted or not interrupted by interrupt the and Lauren repeller runs and it grabs the same walk boom deadlock don't want that so this is why we don't need to disable interrupts because everything is a mutex everything is scheduled so you take your lock your handler I should have made that smaller again there is handlers less than interrupts I could've made that shorter okay I was lazy remember I started this at 5:00 a.m. so he takes off the handler only tears wakes up the thread say this time it's not a high priority process going on so the product the thread the inner paler goes right away and it takes the lock but the thread that had it the thread that it preempted or yeah it's yeah preempted now not interrupted printed and scheduled had the lock so it has to block so it schedules out it can enter up handlers can now schedule out you run your task it once it releases it releases the lock scheduled back in your thread so priority inheritance right now we have priority inheritance within the kernel but it's only for a few ticks fast user fastest pace music's when you'd use pthread mutex attribute so if your user if you use pthread mutex within your application and you were to say turn on the attribute prayin haratz you actually get priority inheritance and I'll explain them preempt RT does that inside the kernel to all this mutex is all this sleeping spin locks and to the mutex so my talk can't be complete without me doing these slides I've did this this slide is actually for my first time I ever talked about real time it's been every single talk of mine so it's like me taking that picture where the battery died so normal the best example of where this could go wrong came in 1997 the Pathfinder flying up to Mars they got it started resetting it would suddenly would reboot and they didn't know what was going on you know got to Mars it would reboot every so often reboot and you're like what's going on this thing's just normally going on it would they lose communication for some time and then suddenly it would reboot and they would get communication back and this scared them so they took all the code and they they were able to simulate the exact same things I was going on up in space in their labs and they found out what was happening they had this process a very very low rare process seldom ever ran and let's say it's process see that's like bondage in and this process would just collect meteorology media or article I can't pronounce word meteor data and it would wake up collect some things to put onto the bus and then go to sleep in the meantime the bus manager highest priority process in the in the on the system would be accessing that bus and would have to grab a walk before he would touch that bus and for some reason priority inheritance was turned off on that walk because I think they did for performance reasons and they didn't they forgot about this little lowly task that was sharing that lock and what would happen would be the process will wake up get preempted the bus manager would go run try to grab the lock okay I got a go sleep go back to sleep the the little vlogging data continued and then there was an intermediate process that came in that one runs for a very long time and it started up and it preempted the guy pre-empting aim and now when a was stopped that all bus management stopped and the system had a watchdog timer go off and it did a reboot boom he says things are not things are locked up we got to reboot the system so the fix was simply send in a pet or send up a little command if they're able to upload a little switch that turned the bag to a priority aversion task or a lock and they never had a recent the resets stopped what happened they so what happens is sees goes once a runs in blocks it gives a priority here in inherits C inherits a priority of a so it runs a so when B wants to run it can't so it has to wait so C releases a lock loses priority a runs it sleeps beacon L run normal our W locks has been our nemesis for some time we've done things I've actually done I actually wrote code to make multiple priority inheritance no priority here since people don't like priority inheritance because it's kind of complex but I actually even made it more complex people I love you know me but I kind of like code that's complex I usually complex algorithms which require complex solutions I love that I revolve around it so I made multiple things and Thomas looked at it says Wow get rid of it so what do we do about reader/writer lock speeds rewriter locks everyone loves because you think Reena Roy blacks are really really good for little multiple CPUs because you can have a reader multiple readers gret grabbing something custom read and write locks are basically d buddy you know do a lot more reads than you ever have to write so if you sell them right you just do a lot of reads you want that all in parallel that's great so you want they all going on so reader/writer lock perfect so each time a reader goes to get to look at the data and read it grabs the lock fine multiple readers multiple CPUs Oh fine and dandy a writer comes along it's got a block and wait for all the readers have finished depending on whether it's a fair lock or not that could have issues and but finally when all the readers are done it will get to lock and do its data and then or do its updates and finish it all the readers come back in again it sucks for multiple CPUs don't use read or write or locks well you don't have to use RC you get rid of the reader a locks for those of you your guys are Hardware guys this is ELC here I'm sure a lot of you LC people here cache line bouncing will kill you I could show lots of graphs of performance when you are sharing the same cache line that is now the bottleneck that we're hitting almost everywhere reader writer locks that's a bunch of readers hitting the same cache line we've actually made spin locks special to do the spinning outside off the cache line on local memory so we wouldn't have the cache line bouncing that was killing everything so a lot of works been done to try to make the cache line better because that's what speeds things up again if you use reader/writer locks it's not much faster you're actually it's kind of it does that more CPUs to get it flattens out so avoid that with more CPUs with our CEO it's still exponential not really but pretend it is so we compromised we just said reader/writer lock sir mutex is still so even if you do a reader are double lock it turns into mutex but that's fine they just won't any priority inheritance so avoid it for another reason you can got get priority inversion with reader/writer locks now on the RT patch so avoid those paths try avoid it we're trying get rid of it so that's what we're going that's what when we analyze real-time systems we look at the device drivers make sure we try to avoid all real reader/writer patches past that or at least make sure that there's no writer involved in any of the real-time threats or anything but writers do still have inherent stencil in themselves their normal mutex basically the try lock issue the try lock issue okay how many people have ever done something like this a few hands you're afraid of raise your hand you should be this is usually what happens is you've got this mixture of work where normally let's say you grab lock B first say you grab your locking orders grab be grab a and for anyone that's not a computer science and you're just here to see B talk which I think I'm talking very slow right now if you grab B and a always in that order you're fine you're not going deadlock but if you ever have any place where you grab a then B you get deadlock because Co the mixture of be a be a up we're dead so we always grab a one order so one of the ways around doing this because there sometimes there's times there doing some data that requires protection of a but then you I do something that requires protection of B and you don't want to drop your lock and redo everything so okay we'll just do a try lock and if we get it great otherwise we'll drop a and go back and try it again because if someone's blocked on a once we release it they get the lock and then they'll release it so when we get the lock a we could try it again and do this this is great okay problem solved but does it work for you no it's great for spinning locks but not for me fixes so what you have in a real spinny lock you have CPU zero CPU one so the try lock you know the CPU one takes its lock to lock a CPU zero takes lock B now you want to try lock-on B so you actually go into this little loop this is you spin while CPU zero has locked B and you're waiting for lock beam so what you do is you you know you really say you grab a again you grant try B fail repeat now wash rinse repeat boop-boop boop-boop boop-boop get nice pretty colors on graphs and finally cpu zero releases the lock hey we're done we got B we go we're all happy what happens when you do it with sleeping spin locks for mutexes on a single thread and say if the green guy is much higher priority than the yellow guy so the yellow guy grabs suaq B we go try to take or we get take lock a oh we're going to try to get B so we go through that whole magic of release a try to grab a try to grab B again da-da-da-da-da-da-da-da and guess what you preempted a or b are the owner of B you'll never get that lock so you'll just spin forever so one solution that's a hack best is don't do the trick but if you haven't figured out what to do yet here's one hack that will work for you and that is to grab B after you released a and then release B and then grab a again so how does that work so the task says be a B test the first task gets B and then you grab a so they try to get B and you don't get it okay so what you do you really say and then take B again no so you're going to block your garrote almost guaranteed to block now because you tried a try B it's not there you if you preempted got your guarantee the block boom P blocks not only that priority heritance kicks in Boop you just boosted that guy to your priority so here if it's of a lower priority he gets here inheritance he'll run once he releases B bump you'll get it and go on it's a hack it works try not to do that passages avoid that little trick next thing is per CPU variables well this is what you have a person view variable that's only associated to your CPU this is something that we like to encourage people to do use per CPU variables yes is awesome like I said we don't want cache line bouncing to scale you need to do everything as much as you can on one CPU without looking at any other CPUs that's great we love it well how do people usually protect per CPU variables take the stable preemption and some time they do it for a long time and what sometimes they say hey spin locks disable preemption so I'm like grab a spin lock and doom I perceive you variable stuff and then release the spin lock hey great I just protected that preemption there well guess what we don't disable preemption anymore but so this is what you'll see if sometimes you see spin lock you know let's say I have two per CPU variables I want to add them together and so arm to a third spear CPA variable so I grab a or X Y Adam to Z boom right spin locks do disable migration so this still does work this is okay you're not gonna change your plants prehaps Archie comes in it's still okay to do this we just save up for my migration so as long as you protect that data by that spin lock if it's a per CPU spin like that we do have per CPU spin locks that's okay we love them too but don't do both so if you had task a that said hey I'm doing spin lock here or one guy does a spin lock I'm disabling preemption here no you're not and you do your little magic and test B it has another thing where it doesn't need a grab a spin lock so like we only get grabbed spin like we'll just call preamp disabled and preempt enable and you do your same little magic and what happens here is you get this so that operation supposedly has to be atomic so you got to do all those three commands without anyone jumping in between well guess what you start yours you grabbed your spin lock but we didn't disable preemption you just stuck on that CPU so say you have a higher priority perhaps cut process comes in you get scheduled out boom primp disable great now you turn off preemption you do your work preempts enable fine so this is bad but primps primped disable itself is not bad we don't want to tell people it's bad the only thing is if you can't see why I tell you this if you use a preamp disable and you can't see the preempts enable on that same screen that's small on your phone you did it too much something like this I could pull out a slide say it say that to yourself could I put this code on a slide and project it for people in the back row can see it fine okay I think that's good enough almost don't call functions there's no functions in between in there anything don't do this this is something we've seen just because they did something like oh I need malloc something and they just happen to be in a cramped disabled location and they don't really want to reorganize their code so they're kind of being lazy and so I'll just allocate right here bump and now you're now you all sense a jiffy atomic so you have to use the atomic which is bad bad in itself cause Co you don't want you know came Alec to schedule out or anything problem with this is team Alec called spin locks it's been luck sleep now so that's not good we don't do this do this do here your allocation upfront if they fail don't disable preemption do your work it takes a little more organization guess what though it's cleaner code it does you it's makes your code much nicer much easier to look at people will be pleased they'll read your code when they go to bed at night do this with GFP kernel so now you do a normal malloc cuz you can preempt you can sleep great you don't have to do these crazy things and get special memory and all that so print-disabled is not bad if it is short keep slow operations out of print this table this is not only good for real-time but it's also good for just to reaction in the system because it's good for mainly to disable interrupts so avoid local irq safe most likely if you do it using local Eric you save it's a bug there's a lot of times I've seen in old drivers local Eric you save where they just figured they were written when everyone had a single CPU it needed a spin lock I think we got rid of but that was a lot of things but you really shouldn't have to disable interrupts and also your being I say you're being greedy you're being selfish whenever you use this because especially for a device driver they say I use the local Eric you save your disabling Arabs for everyone else not just you okay I'd make a political point right now about my country's leader right now but I don't want to go there I'm American product this clicking so anyway that code track there is a case where if you have per CPU data that's shared between actual interrupts and you don't want to grab a spin lock you could make it we have something that we're we have in the real time patch something called a local lock I didn't add it here because Thomas doesn't want to add it he's like I'm trying to get find better ways to write code so we don't need it but we haven't come up with the situation where yes you could have you know shared per CPU data between the thread and inner panel err so on main line the best thing to do is look like you saved because you stopped the inner panhandler going on that way but then again you are to stop being everyone's in your panel or you're not being your again you're being selfish with that too so best to rewrite your code maybe try to find another new texture or spin lock we have this thing called a local lock which is basically just a bookmark you save without or preempt disable but without a mainline but on on Rt it's a mutex so it's fine to do and we also what's also nice about that is you're annotating why you have this local irq safe because a lot of times well we've actually had this where we go back to you guys developers and say we've studied this for three days we have no idea why you have a local req safe here and the developer went oh I forgot about that we don't need it anymore we just deleted that was the solution so reason why is because they had no they didn't they didn't know why they had that look like you say because there's no annotation about it it's just a little guard cue saying I needed to stop interrupts only but they change their code head spin locks they look like you say it was not needed anymore don't do this we've seen that some people say hey I could be fancy I could put a local req save do a lot of work and then spin lock and then I'm spend a lot and do a local are to restore no those look are acute that's just spreading the pain out to everyone don't do that you want if you do look like you save you want to keep it with the spin lock if you're not move the spin lock out you're not saving anything it's not that much you're not saving any benefit for trying to be clever with where I just say will end up somewhere I grab locks don't do that misses a party because that's low car IQ saves a real disabling of interrupts and then if you do the spin lock guess what your schedule so soft I recuse there are one of the biggest penis to mainline as to us they have a long history I'm not going to go and talk about the history what they do is they rate you raise or something you raise it from the dead and then you're basically asking it to run and when it will run when it can and right now the way it does it does it by interrupts and it's interrupts softair cues are in interrupt context most of the time so if you write an irq or software eq handler and or a task lit or something you think that you're always going to be an inner of context you're not because there's times we get too many software cues we kick off case off to our acuity runs in a thread and runs your handler as a thread so we have this thing called local ph disabled and spinlock ph spinlock ph does actually do something different than it does our other ones i think yeah because it will kind of do a special lock for soft area cues so it's a Sables preemption on preempt RT but doesn't but alright sorry nan preemptory kernels but it stays preemption stays enabled but migration is disabled so you can use per cpu variables that's fine so currently a mainline software queues are indiscriminate and oh what they run so when you raised one although you might raise the networking software EQ if there's like a block itself Derek you that wants to run it will run - and you gotta wait for that as well so there's no way to change the priority between them there's no way to that make it deterministic so soft.i recuse really do suck on mainline and they hurt people actually are working on fixing it now and mainline it they're using our tea ideas to do so so we do things differently result our computer are accused and we went through several iterations to get this right so almost every debate from probably every three years rewrote the soft req lodge logic completely different so if you go back like well not now actually we've been consistent ever since the for blow curl so foreplay oh girl I think is the time we went hey we got soft i recuse finally right before that it changed all the time is we had no idea what to do because we we wanted to separate them change the paradigm because we want every soft Eric you to have its own priority so you can run the networking part of networking software queue at a priority of like the networking stack let people change this we had soft our queue threads but then they cause deadlocks because it changed the paradigm how things work and and because soft are cute soft I recuse could run on any CPU at the same time but not on the same CPU at the same time so things are really really weird and what we finally figured out what we could do is just let whoever said I want the soft Derek you to run run it run the soft Erik use well whoever raises it that's the priority you know what the priority is so if you're high priority test raises the soft IQ it runs the soft area logic under its own task works great but main line is currently suffering from software accuse from starvation like I said the networking software cues can't run because the Block software cues are going to long and it's they're having issues with network performance and everyone wants real good response time even on mainline Linux so there's work right now that's going on that's taking ideas from Fredrik vice Becker who also came through the real time guys he is influenced his work has influenced right now an L KML the first series I wrote is not the best and he it's going through you'll see soon what we're going to do is now start doing annotations you could actually say I only want to run like when you enable or raise a software IQ when you with the local Eric you save and restore it's going to record okay we know what you saved and when you enable software accused it will only run what you enabled and let everyone else go to case off dire QD that type of thing so we only those that are important like Darcy you case soft req Dean we don't that no one really needs to run that there are you one that's a garbage collector he could run whenever it gets a chance to run as long as the eventually runs that's fine but things like that don't need to run and those could take up a long time because those are running like the RCO cleanup code which could take forever so you don't want those things running so we're actually working to get mainline to say hey just pick which software queue I'm going to raise and it's going to be all done magically for you hopefully we'll see we'll find out what the patches from Frederick come in but the way we do it now is once you do the race it sets a flag in the task struct saying okay this guy raised it so he wants this run we run our his priority he asked for it let him take the penalty for it so soft.i queues are non real-time looks like this high priority task has it raises a soft IQ Arup handler goes off and then when the handler before it goes back to user space it runs soft air queue handlers all of them are see you whatever it has it could go off so this high priority tasks has to wait for that so what we do in the real-time tasks is say the intro goes off and triggers a soft req within the handler it will come back and the handler itself actually I should have I just forgot I'd put it softer but this slide is a mistake the green should have ran first then the handler swap that just in your head put swap schedule there and then it runs a soft archaeologic within its handler itself finally while i talk about this it's getting close to being end rossmann locks this is actually our gift from leanness back in 2009 ross pin lock has been introduced to the linux kernel you probably how many people have seen Ross pin lock in the kernel just people just tired they don't want to raise their hand yes this has no meaning to mainline zero none Nick we came up with excuses but they're all BS this is just a way that we could differentiate what when you have a spin lock that can't sleep for an instance the scheduling spin locks the timer spin locks those can't sleep you can't the scheduling spin lock schedule it kind of gets into an infinite loop that way so there are things you actually have to have as a raw spin lock well this is it Ross spin lock scheduling Pope we have it in there don't use it if it becomes a problem for you see mix because it makes order important yeah this is what I wanted you can only use them for critical usually I would say hardware or scheduling activities or CPU going down the major events could use them not your normal device driver not saying that oh my code is special don't do it if you do it it could cause problems and actually it defeats the point of being a preemptable kernel fully preemptable kernel do more spin locks at Ross relax you add the less real time it becomes it's less deterministic it changes time it's horrible don't do it so your lock is not as important as you think it is questions what an anti climatic ending okay I think we have like four minutes if there's any questions there's microphones because I've justice is being recorded so come and show your pretty face to the Q TV OOP got someone coming up here well thank you can you go one step back please regarding if you cannot figure it out yes actually we had a problem and it was what do you write a sketch even while atomic I had a lot of error messages in my kernel log and it was actually because we ported from 4.9 to 4.14 when the irq safe flag inside the HR Thomas tract has been removed oh and yeah we got it compiled again but then we saw a lot of scheduling while at Thomas form a minor bream party a preempt arty patch sir have you reported that to Thomas no please do because that could be that could be the case where you said we switched to higher Q what that means what he removed our EQ safe flag from the inner up handlers what that means is he didn't think there was any path that could get there that's not from an actual hard interrupt we are using it in hot interrupts sorry turn around stop it to run I started these slides at 5:00 a.m. okay no I meant the other way around he thought there was no path that it would be used from a heart interrupt it contacts he thought and to get here it had to be threaded and if not something about his assumption is wrong or maybe something at your assumption about using that function is wrong so that's where it's a communication thing I would highly recommend reporting that saying hey I found the path when you change this to that this broke what can we do to fix it or should that be an eye or Q safe you always set a patch and see what happens okay this is somewhere else yeah what about near the beginning you mentioned were you number two there who's yeah where they stood me oh geez I'm like I I thought like you patient me so I could take part Oh just that the speakers over here I thought someone was yelling okay I beg your pardon sir you talked about basic car T at the beginning and then the the do me full arty yes what was the difference between those two oh I believe it's been a long time since we used it and we kept it in and some of those things that they think breaks we come back actually what I believe it turns on everything but the spin locks sleeping spin locks it does everything else it does this thread the thread our cues everything else is just to say basically waves is it the bug because of the sleeping spin locks there's a bug for something else basically the singing policeman locks is the big thing that we want everything else is kind of like there might be a bug there too so it's when you have both thought it's hard to differentiate I've never used it Thomas I can see still uses it because I said should we get rid of that because he he doesn't expect that to go into mainland that's the one thing he wants to rip out before actually makes it into mainline I only put it up there because if you download and install the preamp 20 patch you'll see it I just wanna explain what it was anything else yes yes yes sweet comments you put on some sweaty users you if you can I could correct it if you may because I see you only four high priority tasks better to use mutexes yes I would say I would say you can and may or to differing some reason why I say user see you when you can which means that it's not going to suffer the performance than everything else because yes our CEO is extremely fast for writers extremely slow for reader or sorry swap it I need more coffee extremely fast for readers horrible for writers so yes if there's a case you could do it but yeah it's going to kill your performance because you have a lot of writers that's a case maybe you could try to rewrite your call guys I've actually worked to try to rewrite and get rid of the writers or put them elsewhere and try to constantly piggyback and do use more RCU because that really are see scales are to be mutex does not okay thank you and one more comment about driver sent actually this way rose locks we did several patches to fix GPA driver because they implemented our cubes so in some drivers it's only way okay I'd like to say something reason why I said it so strongly is I don't want people I've seen it thrown too quickly there's been a few cases where we said okay they grabbed a lock that's only basically it's the same thing as the print preempt disabled thing if you could see on your cell phone or on a slide with no function calls just grab a lock or this law Rossman lock and only a few things and it's I think slab we might do this too because it never grabs it and does anything for more than a very very short time that's perfectly acceptable but anything more than a screen full or function calls with in it no does that make more sense yeah but this GPR drivers is basically a yogas register so nothing else we have to we while we're slowly fixing thing one thing at a time what is in place what is in place and being put into place to help catch regressions early on priam thority are you coming to plumbers crap anyone coming in plumbers Lakes plumbers we actually have a RT micro conference that's actually one of the agenda how do we catch the regressions because I Kai said well by the beginning of the talk I said we've doing we got it in great oh no we have a lot more work to do that's one of the topics we have ideas we have answers and not saying anything yet because it's still a work in progress what about sake looks this good for a tea bad for a tea try to avoid them if you can they're not good for a mainline no they are they're not there there they because they're like a repeating lock you know like oh we did I mean it mattered okay matters how much contention there is but that's the oils things that's okay I think RT if I haven't looked at the code recently I think we do make it more deterministic because right now that's a non-deterministic functionality it's depending on your load or something but most the time like in time catching a timestamp so that's usually a that's usually not something of a probably most paths I hit that are not usually real-time tasks so if it's not a real tight task we really don't care about determinism but that's one of the things they have to audit so yes see quacks or yeah anyone else yeah we're here are we are we still trying to get rid of the semaphores the traditional ones I don't think we've seen much left what's left no they actually getting more Oh what what December of course they had that getting added faster than that get removed we might actually what down to the thirty I think at some point now we're over 100 good no no thank you you know that yeah we would love to give and have some ideas about what how we can improve that are you gonna be a plumbers yes I'll see you then any what else well you gotta come up and get a bike it's being recorded but if you all this is a short question I can repeat the question okay the question is if we get preempt arty into the kernel what will happen to the patch set will it go to zero hopefully yes we might have a niche patch set for things that there are some things in the kernel that may not got get into the mainline that's for some device or for some weird situation we may have a patch set just to do that it might just be like a staging to try to get some a might come up with a new solution new that but the big hammer is that sleeping spinlock once that's in that's real time that's everything else is sort of one of these little fixes everyone says looks at the patch say well you got three hundred bytes or you go three hundred patches in there 290 of them are basically little fixes like that so and it's like once we get one thing into mainline it'll take out 200 patches so because those two things are hacks because we don't have a solution for this one thing what's it get that solved bump those go away we may have little things like that but hopefully everything yeah be optional yes yes at that point will be something that you could it's good before those cases where you just have like someone I want to get this in and we say that's not a great solution but we may we may maintain like a patch set for those cases where we don't really have a real solution for yet but those are like that's going to be for them 99.9% of people here you won't care about them in fact I probably never run it it might be this like hey I got this okay we gotta figure out how to solve that here we'll put this patch here before so everyone that want has the same issue could use it shared the Linux kernel really is always forked everyone says you know the people they say lapis of Lynch gets forked I hate to say us there's hundreds of thousands of Linux kernels out there not one of them are like oh those 100,000 are the same they're all Fork ship Android even Debian RedHat they're all Forks bread has its own system they have their own patches Susie has their own patches same thing anything else when will it happen we're okay we were hoping this year it looks like because of because of recent events it's not going to happen it has nothing to do with technology Hales okay thank you oh yeah going yes sorry practical just very practical it's this I mean with all these things are we getting really deterministic clock cycle you know it'll happen within this many clock cycles or are we just saying it's guaranteed 99% of the time that kind of thing and what's the what is the time the latency that were guaranteed to the jitter that is not determined by the RT kernel it's determined by your hardware thank you yes thank you

Channel: The Linux Foundation

Views: 26,417

Rating: 4.6966825 out of 5

Keywords: Open source, linux, kubectl, kubernetes, kernel, Red Hat, blockchain, Cloud, OpenFaaS, CI/CD Pipeline, Scaling, security, Zephyr, kubeadm, IoT, Machine Learning, Kubeflow, Embedded System, Cisco, Yocto Project, AI, Data, Edge, Cloud-native, Buildroot, FfDL, OpenCV, U-Boot, Edge Computing, VMWare, Microsoft, Microservices, container, virtualization, NVDIMM, OpenOCD, DevOps, KVM, eBPF, XDP, Hyperledger Sawtooth

Id: BxJm-Ujipcg

Length: 51min 7sec (3067 seconds)

Published: Thu Oct 25 2018

Reddit Comments

[u/2cats2hats]

FAQ for those who know nothing about it and prefer reading over a video.

[u/ImprovedPersonality]

How do I find out the worst case execution time with an RT Linux kernel? Are there any guarantees?

[u/jcelerier]

in my experience, RT patches have a negligible effect when you want to maintain a deadline (on a desktop computer) when comparing to disabling intel sleep states and turbo boost.

[u/kommisar6]

A processing that changes the screen must be completed within 16ms?