How to Build an OS With Erlang: A Whistle-Stop Tour of HydrOS - Sam WIlliams - EUC17

hi everyone um so yes I'm Sam Williams I'm from the University of Kent and I spent the last four years building two operating systems with Erlang and so I'd like to talk to you today about how you can construct these kind of software projects but rather than just using my own experience I thought it would be interesting to reach out to some of the other people that have built Erlang like or Erlang related operating systems as well so yeah I've integrated some of the experiences that the Erlang on then Erlang on run run people have also had and and what I really want to do this talk is essentially put most of the design space information that you might need in order to build an airline operating system in one place so you can give people a kind of head start because there's a lot of duplication unfortunately of mistakes that we've made in the past so hopefully we can yeah make it a solid foundation right and critically what I want to do here is discuss the design space so to map out what are the problems that you're going to have to encounter if you build a system like this and what are the potential solutions to those problems and architectures and this other stuff ok so first we're going to look at why you might want to build an operating system in Erlang to begin with and then the existing systems that we have will go through in turn their architectures their yeah their strengths and so on and then we'll look at uncharted territory so potential ideas for future projects will map out a couple of those as well as looking at the challenges that are associated with it so the questions you're really going to have to answer if you want to build a system like this then the lessons that we've learnt as a community that's far with these kind of projects and then we'll look a bit at hydrops which is the second operating system I built and we'll use that as a kind of lens through which to look at these problems and then a quick demo and acknowledgments right so why would you want to build an operating system in Erlang in the first place and there's a number of reasons but the primary one from I think is fault-tolerance because obviously when you're building an operating system normally you use C or assembly and and recovering from failure at that level is is not easy at all so the fault tolerance print primitives of Erlang offer a nice way of getting error recovery into your kernel and operating system there's also descale ability aspect so as we look to a future with hopefully hundreds or thousands of cores on chips the current operating system design the sort of monolithic single single kernel system that uses finer and finer grained locks over time in order to stop multiple cores in the system using the same functionality at once it doesn't scale very well so there's an interesting study done by the people that wrote an operating system called FAQ to do it and they basically found that as you scaled up to you know 16 or 32 cores when you're running a web server say then you end up spending like a very significant amount of time doing lock contention so that's not good so yeah I think with the Erlang we can find ways around that and we have and I'll speak rather later another interesting thing you get with an Erlang operating system is that you get a very high level execution environment with interchange or a nice interchange or data interchange format between programs so rather than just using text like you do with UNIX if all of your programs are written in Erlang you can just pass functions around and pass you know deep lists and structures and things which is really nice particularly the way that you can pass functions between programs is really quite cool at the operating system level so the other thing you get is machine independent programming environment if you're using a VM that is and we'll get to that a bit later so this is really nice if you're building say a distributed operating system and what you could essentially do there is because all of your operating system code is written in Erlang it runs on top of a beam and if you put the beam to arm or x86 the you can run the same operating system code across those different platforms so if you build a distributed operating system you can now integrate the same or you can run the same programs across your cluster even if it's arm x86 or or MIPS or whatever you like so that's quite nice um yes so first we'll look at uni kernel Erlang systems now I can use the word uni kernel quite a lot in this talk so we better get clear what it means just in case yes so a unicameral is essentially a program that is compiled into an operating system image that can be run natively on a machine or in a hypervisor of some kind so the idea here is essentially you take the program and instead of deploying it on top of a traditional general purpose operating system in Amazon ec2 say you you instead turn it into a very small very specified operating system that only contains the code that you need to execute your program and then you run it is normally in the cloud and this has a number of advantages primary of which is that well the images are smaller and the zero configuration you just run them and they work in the in clouds and cloud environments which is rather nice you don't have to move around very large images or anything like this and also from a security perspective it's great because you essentially remove all of the code that's not going to be used so there's less for an attacker to use against you so for example there's no console in UNIX kernel systems so if you get access to the machine well you'll have a difficult time attacking it so that's the idea at least and there's also the fact that they can boot up extremely quickly and this means that you could essentially or potentially boot up a unique earner in a cloud environment in order to respond to a single Network request so the architecture here would be that you have say one unit kennel running it receives the network request and then it can spawn another one that just goes and answers that and that's all of your yes that's all of your environment essentially it's very small very neat and you only get charged for what you use obviously because if your if your VM was only alive for you know half a second well that's cheaper than an hour on ec2 yes so the interesting thing about Erlang on Xen is that they built their own VM and called Ling and I've spoken to Maxim keratin Co about this and he was talking about how one of their primary goals when they were writing this VM was essentially to lower the amount of time that it takes to get from starting the computer to executing the first Erlang instructions and he said that he even he counted the instructions from when the machine booted until those first beam instructions are exit or Ling instructions are executed so I don't think you can really get okay sorry one of the screens change there I don't think you can really get faster time to Erlang execution than this all right so that next we have Erlang on ramp run so ramp run is a kernel it that well it's unique kernel system it's very small it generates about six megabyte images with beam on top of it yes so the difference between early on ramp run Erlang on Zen is the earning on ramp run users to beam earning on Zen users Ling and the next is Erlang on OS v so OS V is the unicameral system that creates fat uni kernels so unique Ansel's are much larger about 80 megabytes in size and the idea here is that essentially you have a much more UNIX or POSIX compatible interface and that makes it much easier to port programs to the operating system and also to configure it so yes this is Erlang on OS we used it a Jiang so that's Erlang on top of JVM on top of OS v on top of hypervisor typically yes and it's quite interesting and there's also a low s which is just beam on top of them which was my first Erlang operating system which ya best not go there right so there's also Erlang embedded platforms and these aren't quite the same thing that they're certainly related and so the first of these is grip which is Erlang on top of our temps which is a very small operating system or I've heard it described as a closer to a controlled program than an operating system both really well precisely yes and yes built so that you can you can create IOT wireless devices with Erlang among other things and target arm it's also the nerves project which takes a slightly different approach which is to run Erlang or elixir code on top of the beam on top of Linux but it's basically like the minimal amount that you require in order to solve Linux in order to run the beam so it's just the kernel and that's pretty much it and the standard libraries and one of the interesting things they're doing there is they're working to expose kernel functionality inside the beam inside the beam environment and that's quite interesting because a lot of these other projects have worked to move operating system functionality out of the kernel and into the Erlang space this is taking the inverse approaches moving parts of an existing kernel into the beam itself and there's also the Erlang embedded initiative where they they created a smaller line package that you could use on these devices like a Raspberry Pi and also an actor library from embedded which provides a nice way of interacting with like I to see devices say in an embed environment right so the final operating system is Hydra which is as far as I'm aware the only general purpose operating system so written in Erlang and it's for servers and desktops x86 machines generally really and it focuses on providing fault tolerance and error recovery from typically catastrophic OS failure so things like other things that would normally cause a blue screen of death for us just LuthorCorp and then we restarted and it only works and we'll talk a bit more about that later and I also have a demo will they'll do exactly that yes so it's written entirely in Erlang from the ground up or at least as much as we could put in Erlang so there's really very little by way of kernel it's just been on top of yeah they're metal right so here's two ideas that might be interesting for people that are looking at building an Erlang operating system one of them is truly a native Erlang code running as the kernel of the system so the idea here would be that you take hype or potentially the new JIT and modify it such that you can create yet a kernel written directly in Erlang and there's an interesting idea here with how you could use this in a.m. in a cloud environment where you can essentially create unique kernels with this system that could launch other unique kernels as processes so you essentially have uni kernel processes with the hypervisor providing a scheduler to the system and this is quite interesting for a number of reasons one of which is the potential for so I was saying before about how unique annals are specialized well if we had unique kernel processes in Erlang if we specialized them we'd remove all the modules that you don't need in order to execute that process and if you do that you can start to create a sort of disparate space where if an attacker tried to get into your system they couldn't access all your code it wouldn't be in the same environment so even if there were parts of your system that could be used together for some malicious end they would return oriented programming or something like this yes they would have a hard time doing that because they'd have to break out of the VM instances of the hypervisor which is certainly happens sometimes but it is pretty significant task another thing that you could look at is creating a smaller line VM for many core devices and and one of the things I'm really interested about this with this is the potential for Hardware message passing so it seems the Erlang model would map very well to Hardware message passing but the platform's haven't really caught up with where we like them to be so far so and there are starting to be some platforms that allow you to do this so having an Erlang operating system that actually used Hardware message passing would be pretty interesting right so now we look at the challenges associated with Erlang OS design and the primary one of these is obviously if you're going to write an operating system inner line you're going to have to have some way of executing your line code so and this is the the primary architectural decision you're going to make because it changes the entire way that the system runs and and the performance and fault tolerance characteristics of the system so it's very important to get right so will it be native or will it be interpreted or will it be interpreted within another vm or or OS like say the the nervous system would fit into that category and also at what level would be beam compatible so are you going to run beam instructions is it going to use like half a half the airline compiler and then have a new code generator at the end perhaps you do that if you wanted native native code execution or are you going to use a beam code transpiler so this is essentially what link does where you compile stuff into beam and then you take the beam instructions you can pile them or transpile them into linge instructions so this is these are pretty fundamental choices you have to make so now we'll go through some of the potential architectures that you could use to write an airline operating system the first being Erlang on metal this is you get your airline code you compile it's a native code you run it on the hardware it's pretty simple and probably the most efficient too if we could actually get it to work um so the second is the virtual machine operating system this is essentially where you have your Erlang code running on top of a VM possibly being probably beamed yes and and that's how you you organize the system or you could have a multi kernel system whereby you attentional you have one VM per core in your machine and then you have a distribution layer on top that unifies the the virtual machines and this is what we've done with Hydra another potential way you could organize the system is to have airline code running on top of a VM running on top of the hypervisor running on top to hardware so this is what happens say when you run earning on them this is the kind of stack you have or you could get a distributed unique kernel Erlang and this is essentially similar but you have a distribution layer across your Unni kernels on top and this is rather interesting because if you if you add code such that your unique kernels can spawn other Unni kernels then then you can expand the system in the cloud as you wish or yes and contract it - and the final architecture that we'll talk about later in this talk forget time how good unfortunately clock has stopped running thanks Android right wonderful okay so essentially in the multi unique kernel system what we do is we we take the the multi kernel approach which is split every core into or split the machine into a set of independent cause allocate some memory for each of those calls allocate the kernel copy for each of those cores and then they all execute independently but they interact and communicate of the at the ER line level and coordinate program execution but we take that idea and instead of having the same colonels across all the cause we can replace some of the cult eternal nodes with uni colonel notes and and so here we essentially run our C code alongside our Erlang nodes and a run our app across the top of them however I'll get into that a bit more later so another critical decision you have to make is where you're going to draw the lines between what's implemented in C or assembly and what's implemented in Erlang this is really important because it has a number of fundamental impacts on the way your systems will behave in practice and so will your drivers be written in Erlang how about performance critical DUI code here if you do this you're going to take a pretty significant performance hit unless perhaps you're you're using the native code system that I was or a potential native code system that I was talking about earlier um yeah and so in hydrops what we do now that's cutting off the bottom of screen okay in hydras what we do is we have the the local kernel which is written in C which is essentially just a beam with the absolute bare minimum by way of library functions this is required to make it run and and what we do is we try and minimize the amount of functionality that's written at this level so that we can recover from failures better because obviously when we experience failures in C and assembly much harder to recover from the one error in Erlang so we've taken the fault tolerance a fault tolerance approach which has some pretty significant effects on performance but the critical thing here is that we we keep the C code out of all of the distribution layers so the C code runs in an environment where it doesn't touch any other kernels ever and and yet this is quite nice because obviously it limits the potential for failure cascade which is pretty bad in these kind of systems and so yes and then we have our line code on top that runs all of our drivers and into processor communication and all of this other stuff right so most Erlang systems will use at the some C or another language or they'll have to interact with another language at some point so you have to ask the question well okay if you've got an airline operating system whereby you've got the beam say running directly on the hardware how are you going to execute that code and indeed how you're going to load it because it one of the nice things you can do if you've just got the beam running on the hardware is you just map the address based linearly and you can compile the system so that it knows it'll be loaded in hydraulics loaded at one megabyte notes loaded there every single time so there's no there's no loading done at runtime everything is pre computed but if you're going to incorporate C crypto libraries say you're going to have to find a way of doing that where loads and executes properly and yes so there's a problem with drivers if you if you use C drivers so it would nice way to organize the system you'd think might be to have most stuff in Erlang but then your drivers which are high-performance are written in C but then you have have the the problem of scheduling so with hydrops we just run beam on the core and it's never interrupted it just does its thing but if you've got drivers you've got to work out how you're going to execute some or you're going to allocate some CPU time to running those drivers so one of the ways you could look at solving this problem if you did want to have native code drivers is you could if you try and write them in an event-driven way so they only execute code in response to interrupts and this way once you get the interrupt you you registered with the interrupt handler already that you want this piece of code to be run when you receive this interrupt it does its thing you D assert the interrupt you return to the code and and that's how you do scheduling but there's potential problems of starvation there so if you receive a lot of interrupts and potentially a lot of interrupts in a row you might just never execute the encode and it's also there there are a number of devices that I don't think this kind of system would work for at all you have to have some kind of process attached to them but it's something to think about okay so actually getting your operating system to the point where it can execute Erlang code is is non-trivial I mean when you start when you start an x86 machine you have one core in 16-bit mode getting from there to having all your kernels loaded and in memory and all of your cores running and in 64-bit mode is really quite a journey so I would suggest that if you can you use a boot loader a premade one like gravel lie low or syslinux or something along these lines which will certainly help you out but unfortunately for a lot of the architectures that I mentioned in the previous slides there are there are differences that they really make it so that these boot loaders don't offer you very much as a hydrops we essentially ended up writing our own boot loader because you need a kernel for a copy for each core and then you need the page tables loaded such that each core thinks it's using these core thinks it's at the start of memory and there's a there's a number of a complexities of essentially meant that actually doesn't really offer you that much one of which is that when you start additional processes in sorry additional cause in the system they start in 16-bit mode so if you're going to execute your code in 64-bit mode you're going to have to have the mode switching stuff that'll get you there anyway which is kind of half of the job of what the bootloader is doing so yeah there's some intricacies there and there's also the idea of securing and Erlang operating systems so one of the things with the beam well the beam is like an operating system in many many ways it does device abstraction memory management processes management scheduling all of this stuff one things it doesn't do very well is security because it's sort of a that's going to be done by the operating system at some other level and the edges will be protected so if you want to make an operating system in Erlang and you want it to be secure you're going to have to come up with some sort of system for that on your own and if you just decide say that you want to protect the edges of the system then are you okay with the fact that applications inside the system couldn't just interact with each other arbitrarily so I could get a I could run an application and it could get a list of all the pids in the system and then I could send a kill to all of those kids and there's a likelihood that maybe one of them or exit rather there's a likelihood that one of them will be like oh okay so you want me to exit them that's fine so there's no application isolation in Erlang as it stands at the moment so if you want to if you want to have a secure system then you're going to have to come up with some solution to that problem so two fundamental lessons that I think we've learned from Erlang operating system development in general the first is that your operating system is going to have to be placed on the fault tolerance performance spectrum somewhere there is no way you can really get around this and so obviously native code is fast but failed to harder to recover from and Erlang code is slightly slower but you have really nice fault tolerance characteristics and another interesting thing that came up when I was speaking to maxim was the potential for not needing SMP support so having VMs with a single threaded talking to each other at the Erlang level and doing distribution that way and he says this simplifies the implementation greatly and speeds things up the philosophy is that you'd need more VMs to achieve true multi-core parallelism hard hypervisor is the only hardware scheduler Erlang processes a green thread and this is a similar approach to what we've done with Hydra where each of the multi kernel calls runs its own VM and this is also quite important if you're writing a unique Enel operating system that doesn't normal the use multiple threads so something I got by speaking to the creator of the Erling on-ramp run system was that that if you can get away without fork that would be really good and and so with Erlang you can configure it such to you well you can right now but somewhat changes in our twenty you can use the plane system which won't fork allows you to run a single thread Erlang system yeah so now we can look at Hydra so Hydra uses a multi kernel architecture where you essentially take the cores in the system and you split them oh sorry you take the you take a single machine you split it into a set of independent cause and you allocate memory to each one of those cords and then a kernel copy to each one of those cores in our case the kernel copy is really just the beam with tiny bits of library code added to it to make it execute on bare metal and then we have a distribution layer written on top of that essentially and and you can see in this example that we've got a unique Enel which is a multi unit kernel system that took around a second so one of our fundamental design decisions is that essentially we needed failure on one of the cause not to affect failure on or not to affect execution on other course so because the VM they're isolated we don't mind if there's a massive software failure say like a blue screen of death level software failure or potentially even a hardware failure in the core as long as there are other calls available in the system and this really means as long as the failures don't happen on all the calls at once then we can restart the call if it is still functional or we can just ignore it and obviously you can use your you know Erlang fault tolerance primitives to monitor and get messages when yeah when parts when processes that were hosted on a VM good has failed are no longer available so you can recover from it quite nicely OS subsystems and drivers are also isolated from one another in the sense that they're split into many smaller processes so failure in one of those processes obviously doesn't affect the other processes and we can recover from it hopefully and there's also potential for decentralization of the system in the future so we're really interested in making systems that are tolerant to almost all kinds of faults and obviously having a centralized structure makes that much harder but if each of the Syst because in the system could act independently be quite interesting so high cost uses a system of Biss which interact directly with the hardware and essentially what we did here was try and keep them as small as possible because obviously if you get a failure in the bit we can actually recover from some some of these kinds of failures and we use the system whereby essentially if you get a general protection fault or something of this ilk so we that we can't normally recover from while running the bean safe we just kill the process that was running at the time the interrupt happened and then we re-enter the beam main interpreter loop rather than returning to the the previous place we were in execution and sometimes it works sometimes it doesn't but yes certainly there's some there's some fairly recovery there but critically we wanted to keep the bits as short as possible so the likelihood of failure inside them was very low and these are really just things like enabling and disabling interrupts everything above that level we've had a lot of them are just wrappers for instructions everything above that level is implemented in Erlang yes so hydrolysis Erlang only drivers which as I was saying as sets of communicating processes that are distributed across potentially multiple nodes in the system and we use this thing called the i/o a pick which is the input/output advanced programmable interrupt controller it can only be named by engineers I think to route inter around the system then the this is kind of interesting because it's essentially the Rooter of the network in yes we can we can make it so that the interrupts go to a specific core where the main driver is found for that device that it's attached to or we could duplicate all of the interrupts and send them to all the core which is quite nice and we've got Erlang control over this so you can just call the function in Erlang and it'll start routing interrupts in a different way so one problem that we have with this that hasn't yet become a problem but we think you will in the future one problem is the dumb for level triggered interrupts so it's actually what Erlang its effect on the next screen I can show you so the node kernel this is the C part of the system when it receives an interrupt the interrupt handler runs and it essentially just sends a message saying okay I've got this interrupt number two the interrupt dispatcher which is an Erlang process which then has the routing table of who wanted to hear about which kind of interrupts and here in this example we've used the keyboard handler so okay interrupts 33 means the key was pressed so we send the interrupts round to the keyboard handler you would handle it reads the data from the device parses it does you know turn scan codes into characters this sort of thing since keyboard events are people that have registered to listen but critically what's going on here is the node kernel will send that message and then little dear sir the interrupt and return to execution and this means that that message won't have actually got to the Erlang program and it won't be or the reason for the interrupt won't have been fixed by the time the interrupt is D asserted and there are some kinds of devices but if you do this sort of thing they'll just instantly reassert the interrupt which means that you end up back in the interrupt handler so you just send endless messages until this program or system crashes so that's a potential problem we're going to have to fix so hydrops window management one of the new things I've built with Hydra since last you see how am i doing for time folks 10 are great things yes as we build a window management system so essentially you have a set of fullscreen console applications we wanted to be able to multiplex between them but this led down a rather interesting road where we ended up creating sort of composable window interfaces so we have a window process which essentially tute as a buffer and you can write you to it it says ok what location would you like to write and you say write this string and do it in red and the background of eat green or something like that but on top of that we we built a system of more abstract essentially interfaces like a terminal so you can start the terminal process which will then start a window process and the terminal keeps track of where the cursor is the scroll position all of this sort of stuff and it sends render requests in the form of print this at this location to the window and what you can do is you can then just send your messages to the terminal rather than the window and it'll it'll handle all of that complexity for you and it's generic and you can you can build them on top of each other repeatedly which is interesting and we'd like to go into this a bit more with potentially building a forms system so you might have a form running on top of a window and then you might have button processes or what have you running on top of the form and so you get composed interfaces like that um yes and another interesting thing about the window management system is that it requires the window managed apps have at least two processes so you have a process to which all of your keyboard input and mouse input is sent and then you have a process or a window process and this leads to yeah systems are designed a little like this so you have the keyboard driver sending key events to the input process and then the input process sends regular commands to the window and the window if it's in the foreground right to the text buffer so here with the Hydra console app essentially we have going on is the keyboard input comes into the input server and it builds up a string until you press ENTER and it sends that string to the worker process that try to execute it waits for a response sends the the response back to the input server which then prints the string record yet just tells the terminal to print a string and the terminal says oh okay well the cursors here so we want the print the string to be printed here and it does scrolling and this sort of stuff and then yes the render commands go out to the window okay so finally hydrops multi unit kernels so it's a bit of a mouthful so we call them MUX they're essentially causing the system that are running free code directly in real time rather than a normal pie grass multi kernel node so not running Erlang and one of the interesting things you get about this is that you get I don't think you could configure a system such do you get more performance out of it your code will run uninterrupted on a core and you can do whatever you like with it which is quite interesting because say you have a hundred core system and one NIC then it might actually be efficient to write what we call a unit earn or driver which is essentially just a single core in the system that you dedicate to servicing the piece of hardware at the maximum possible speed or throughput yes and and there's the typical library for interacting with Erlang nodes on top of this and one of the things we changed in its last EUC was it's now really really easy to build these things so as long as you can get high drops to build which there's instructions for on the website you just go into a directory you add an add a sub directory and put a C file in it and as long as it's got a K main function it'll just compile and Link and run properly so I made a an example hash farm system that would you could send it a request and it would hatch that request a number of times and I managed to get it down to like 50 lines of C with with no associated make file or anything like that you just put in a file it runs which is quite nice but if you want to build more complex systems you can put in your own make file it'll not use the default one and you can make it as however complicated you like so here's an example multi única or potential multi unique kernel system let's say running a web service so you would have the network attached processor which is just talking directly to the NIC and then potentially a cache in front of that because obviously this is C you think you can do it exceptionally quickly and you could then potentially send a request to so I don't know if maybe it's a it's a service that does are adding filters to images something like this so if you had a filter image request you've then send it to a GPU connected uni kernel which just manages the GPU directly and you could send the HTTP request to the airline program where that's sort of a more suitable way of handling it and so the critical idea here is the use the right tool for the job so if it needs to be fast go with Erlang it needs to be sorry if it's fast go with native or and if it's slower or it doesn't need to be so fast go with the running and you can you can compose these systems to make essentially micro service real time unique kernels so here we have a potential you know sort of Facebook esque system where you've got user account management as well as a social graph search function and file storage and these four cords at the top here are all running potentially different programming language environments even so thank you and so you could port a prologue VM to run on top of a Hydra uni kernel and then you can just embed your Prolog program into your hide roster ployment and again right tool for the job is the sort of thing we're trying to stress here oh and just one other thing a nice thing about this is that you get a Hydra you orchestrate the system for you so if kernels crashed or something like this it can restart them and it can potentially even swap them out at runtime of this sort of stuff and another point is to make very clear is normally unique kernels run on top of a hypervisor but in this system the Unicorn runs directly on top of the hardware there's nothing between you and a hardware which is interesting so security in hydros I'm going to skip this actually but basically very quickly we create trade trees of decreasingly capable processes so the idea is that the root process in the system can do all things but then when you spawn another process you can say ok this process shouldn't be able to do this kind of thing say memory access or any and it's the rich language so you can say memory access between this range and this range say or ports here and here and then all of these the children processes of that new process won't be able to do those things either so you make a tree where they get less capable at the bottom and the nice thing you can do this is you start a console but you say ok don't let it have raw memory access or don't let it have real court access and then somebody gives you a dodgy beam file young I don't really trust this so you run it inside this sandbox console and it tries to cleverly add memory memory access between some addresses that is like to use but it can't do it it'll just fail which is quite nice the other way you could look at it is if you're writing a driver safe that does I don't know pointer arithmetic say it's using a memory mapped device there's potential that your driver will go wrong it'll access memory outside of where it's supposed to access could break the whole system so when you when you write the driver you can just specify ok when you spawn it just bond it with these capabilities I only want to use these two ports so if I try and do anything else you know don't let me and that's kind of nice ok so very quickly so here is hydros booting in qmu here right so here's the terminal and now we're going to show the shortcuts that are available from the terminal so here we're going to start the shortcut associated with f3 which is essentially a ping-pong program which takes its pawns to processes on either one on each core in the system and then starts pinging pong between them pretty simple so we can see it's pinging and now we go to the system log and you can see here that we're receiving and sending messages and then here we're switching the window to another console and on this console we're going to show the the shortcuts again and this time we're going to run s7 which spawns a process on the other core in the system and runs and unsafe interrupts by which we really mean be referencing an exceptionally large pointer in the beam so this this is pretty error written stuff normally there's no there's no coming back from this so we run it and you can see here normally the horrible gray tile wouldn't appear but I've left it there so you can actually see something's happening so otherwise it you know there would be no output so you can see that it's detected that failure it's restarted itself now we're going to go back to the previous console and you can see that the process that was found on the first core of the system is still running it's still waiting for the protest on the other core in the system but it's failed and you can very trivially change the program so that it would detect this failure restart it obviously is detecting the failure because it's it's printing that it's waiting so now we run the init program so the ping-pong program again and you can see that it's pinging backwards and forwards so we've reestablished connection we re-established execution across both of them and that's killing an entire core in the system it really doesn't get much worse than that and we've maintained functionality so yes acknowledgment I'd like to say thanks to Simon my supervisor at Kent and also maxim curtain Cove and Niraj Sharma from hang on then and hang on run run for their contributions I've really appreciated talking to them as well as the hydros contributors so our two students George Bates who worked on the original implementation of the keyboard handler which ended up being quite interesting and George forward who wrote a file system for us there which I didn't get time to go into today but we can now write a file so that's nice and also Anton to Masson who very kindly implemented keyboard mapping for Swedish keyboards he's also now working on EFI support which is great and part of that is also looking at using a proper graphical well its proper graphical terminal to start with but then later potentially more graphical things so proper GUI interfaces and EPSRC for funding me okay so you can find it at hydros - project - log I released the latest version last night so it's out to date there with the development branch and I can be contacted yeah at this address and I'm always interested to hear from people that are interested in this kind of stuff so thanks thank you very much we do have time for a question or two so yeah so first a remark Chris does not target arm budget targets arm and PowerPC and Spock and rips and everything that has sorted to bits at least right runs and everything and then a question isn't I don't I have difficulties getting the the reason for I mean I I know I think I get the reason for running separate VMs on separate course but isn't the actual reason because all the most of the the existing unicron systems don't support SMP also it looks wasteful to me not for us we really care about fault tolerance and we didn't want to have errors potentially happening at the sea level with SMP theme so yeah but certainly for a lot of unique channel systems having a single-threaded things that was better yeah by the way crispers S&P right ya know it but not all certainly yeah absolutely but no for us we were really motivated by fault tolerance that's why we went went down this path to begin with so having the VM split okay yeah well because the errors don't happen in the distribution inside the beam there's you don't have C schedulers so if you lose one of the schedulers in the system well well we do that at the airline level we lose the VM instead and so we can recover from it in Erlang whereas if you lost a scheduler it's written in C in the beam it might have a harder time anymore question I have a short one there on the security that you showed all those off very quickly so you limit the capabilities of any process is there any tool that you use or kind of way of reasoning about whether you limited too much or too little for the safety purpose that you have at hand not currently it's really down to how the user wants to use it oh it's yes if you if you mess up then you mess up that's only yeah it's an act can point you to some type system like things that aspirator did before yes where you kind of have if I the type system for capabilities and then resources yeah yeah there's certainly a lot that you could do with static analysis of the capabilities but we were building an operating system so there's a very very wide space of things to do and not so much time to do it in here so we have to pick and choose appreciate okay thank you very much what's wrong Thanks you

Channel: Erlang Solutions

Views: 2,342

Rating: 4.9024391 out of 5

Keywords: erlang

Id: 8OyRFbf6MDk

Length: 48min 41sec (2921 seconds)

Published: Wed Jul 12 2017