Processes and Supervision in Elixir - Steve Grossi

processes supervision not just for your kids so I'm gonna start by talking about three elixir functions spawns send and receive these are the fundamental ways that you interact with processes in elixir all of the cool Gen Server agent additional behaviors are built on top of these but the thing that finally got me to understand processes was starting from square one so that's what we're gonna do here we'll learn a little bit about agent which is an elixir abstraction for having a process that holds state for you and then agent is built on top of gen server we'll dig into Jen server a little bit and Jen server forms the basis of Jen event which I believe is being replaced by Jenn stage so there's you know all these layers of abstraction for basically just simple processes passing messages back and forth and then once we go through all that will embark on building a very simple reminders application to illustrate some of this so spawn send and receive spawn is how you create a new process it takes one function as an argument you can see here this function just returns okay when you spawn so when you open up a console like this a console is its own elixir process so when you spawn a second process its return value is not available to you that's why when we spawn a new process we get back a PID pit which is the unique identifier that represents that process but when that process itself returns okay we never find that out at least not by default there are ways of listening for the results of processes which do send is how you send a message to a process takes one argument which is the bid of the process alternatively there are ways of giving processes names which we'll get into later and you can also send a process a message by name the messages you can send processes are any elixir I'm sorry any elixir or Erlang term so map structs basically all the basic types any of those can be a message to the process and then once there's process module has some neat functions for interacting with and introspecting your processes like process not alive one thing illustrated here is that as soon as we spawn a process we can see that it's no longer alive and that's a fundamental thing about elixir processes is that once they're initialized they run their function they do a job and then they die of course that wouldn't be super useful if you were constantly spinning up processes that just died so there are ways to get processes to stay alive and listen for instructions which we'll get into very shortly another thing you can see here so as soon as we spawn our friend the new process we send it the message and we'll always see in the console is that we sent the message hi the process itself was dead at the time when you send a message to a dead process you don't find out whether it received it or not that's just one thing with inter process communication again there are also ways around that you can also check into the process that's alive first but yeah so processes died as soon as they're done executing their the function that they're initialized with however receive is a wonderful elixir tool which when called within a process tells that process to wait until it receives a message through through send or some other receive link so here we've got we're spawning a process and within its function we say receive and then you can pattern match within a receive block the message that you get so if we get the message hi we're gonna just output to the console oh hello there and you can see here as soon as we spawn our friend process the process remains alive and again that's because receive is awaiting messages and it will awake messages indefinitely although one of the cool things is that the beam which is the the virtual machine that elixir and Erlang all run on it's very efficient in how it schedules processes and if a process is waiting for a message it doesn't really consume any resources so you could spin up a million processes and they're all waiting for instructions and your system will barely register a blip performance-wise which is pretty nice they only consume resources once they're needed and here you can see when we send our friend to process the message hi because it's kind of matching on it it's going to output hello there but we can see as soon as it responds to that first message it's dead again so processes just want to die they they want to fulfill their function and then go back to the whatever processes come from a great beam in the sky what's that sorry okay [Music] but there are ways around that here we've basically way around that is after a process receives a message tell it to listen for another message right which it can do indefinitely you could nest receive blocks forever but you would run out of characters to type so really the way to do that is to define your functions within modules so we can have a say a listen function in the friend module and because that function which receives a message has a name we can call it from within itself recursively so this is the same kind of principle you know we call friend listen and it will wait for a message if it gets the Adam hi it's gonna do hello there it will return then it'll call listen again and this takes advantage of something called tail call recursion which elixir has which is that if you've got a function and the last thing it calls is itself it does not add to the stack you don't get a stack overflow you know it just sort of loops indefinitely you'll see something here which I'm pattern matching just with an underscore this will match any message that isn't I and all I'm doing is you know I just need to handle that message and that has something to do with the way that process messages works in elixir and early which is that any message you send to a process it's gonna show up in its mailbox which is part of its internal representation and the virtual machine and it'll you know the next time the process is scheduled to check for messages it'll handle any messages in its mailbox that it recognizes and the leave messages that it doesn't unfortunately that means if for some reason your process receives a message that it doesn't recognize maybe it receives them periodically its mailbox is going to get full and you're gonna have a memory leak so anytime you're dealing with processes especially just raw processes and elixir you always want to just pattern match on either true or some unused value so that any unrecognized messages don't fill up the mailbox they get processed nothing happens so here we can see the example we're creating a process called friend but now within within the spawn function calling friend out listen and that's what we define before I get same principle but listen will call itself after it receives a message so when we say you know send friend the hi message hello then send from the hi message the same thing we can do this forever so that's sort of how you go from processes dying immediately to processes dying after their first message to processes sticking around and maybe proving useful so there's also just a bit of syntax when you're spawning a process that is a function defined within a module you can also pass on a different set of arguments which is the module name the function name and then a list of any arguments you might prefer that you can see it's the same behavior so we've got a process that will stick around and that will respond to things that we send it this has a lot of potential for example we could build a very simple key value school as you can see here we've made our friend a bit more useful it's going to be initialized with what I call state but it's just a map of keys and values and now it's gonna receive some elixir tuples as messages so if we send it the tupple with the atom set and then a key in value you know the that's gonna be the API for storing or setting that that key equal to that value within its internal state and if we send it a message with the tupple of gets and then a key it's gonna give us the corresponding value for that key so and again we pattern match on just anything at the end so we don't get our mailbox full we can see that in action now we spawned a new friend and we send the message you know set food bar and now when we send it the message get food we're gonna get far back technically we don't get far back we are I owed put sing bar back it's just for simplicity sake in order to actually receive that message our console would have to be sorry in order for the friend to send us that value the cup we'd have to program the console to be able to receive messages and that just gets a little bit complicated but the interesting thing here is that you know in a dozen lines or so of code we've got a simple key value store of course that's how you provide default arguments so this takes one argument called state and if we don't get it we're going to initialize it with just an empty empty so this this pattern actually is very common and the elixir core has something called agent which is a module that implements this of course with a whole bunch of you know fault handling and it's a lot more or less than than what I had on the previous slide but for example so when you start you start an agent you're creating a process it's gonna return actually the couple of okay and your process ID that process stays alive with the agent update function again you pass in your process and then a function that will make some change to its its stake so we initialized it with a map again and here within that function we're saying you know update the key the map spooky set it to bar you know as you can see that the API is a little clunkier a little more verbose than you know what we had on the previous slide but it's also a lot more flexible so agent is a tool in our tool belt but agent is sort of a special-purpose implementation of what's called a Jen server and when I first heard of Jen servers I'm sorry so agents can be named all processes can be named but you can see here we've got an agent and we're passing in the option of name bond so each bond I really have my fingers crossed I was gonna be a PID double-oh-seven didn't happen but now we can do process out where is bond and we actually get back the process identifier for that process the cool thing there is that if for some reason our process dies and another process steps in to take its place our code doesn't have to change we're not locked into a single process ID by using a name any process that's registered itself under that name can handle whatever we throw at of course you will lose your states if you've got a key value store and it dies unless you're storing that data somewhere else it will happen so again instead of passing in friend again to the process ID instead of PID we're passing in a patent which is the name of that process so yeah agent is built on top of something called Jen server when I first heard of Jen server I thought oh man this sounds really complicated server like web server is this gonna be serving up web requests I don't know I was confused but Jen server is actually just a fairly thin abstraction layer on top of this this process management it's comes from her life it's an airline behavior it's been in production for probably a long time what Jen server does is it gives you a uniform API for creating processes that handle requests and it does a ton of you know syntax sugar and fault tolerance and things like that you know handles a lot of edge cases so Jen server you know you shouldn't hesitate before bringing out a Jen server and all it is is a behavior that you add to your model sorry module so we got our friend module again looking back to what it previously had just this listen loop and then we were pattern matching on some messages of a certain certain format they were all tuples with an instruction and then some arguments and you'll see something very similar with the Jen sir ignore start link for now but you know the set key value and then they get key should look familiar the way a Jen server works is you start up with Jen server dot start link this line here and the first argument is which which module this Jen server belongs to and so by using this sort of double underscore this module identifier we're just saying it's this module the friend module that we're using for a tensor takes an initial argument or state that it gets initialized with here we're just doing a map because we're keeping track of key things and then just like agent it can take a keyword list you can take options one of which is name I could have given the name you know the atom friend or something but you can also just use module names as valid names or your processes because under the hood module name is just an atom and then what Jen server will do is when you send a message to Jen's process you'll do so using really two two main ways of sending messages which we'll see they're calling or casting and the difference there is when you cast a message you can think you're just like casting something away you don't really care casts means you send a message but you don't care about the response you're just sending an instruction but the Jen's ever figure out what to do a call is sort of like calling on the phone where you send a message to the Jen server and you synchronously await a response so you can see here for handle cast when when we're setting a key in value we don't really care the return value of that I mean if we really did maybe we'd want to know we'd get a true back or something that it was successful but for the most part you know if the process is there and we say hey set this key to this value that's enough so handle cast is gonna take the message as its first argument and then something Jen server does is keeps track of the processes state and state is sort of loaded I mean it could be something as simple as a list or a map or it could be some very complicated state in this case the state is just a map and that's you're gonna get that as an argument to any Jen server callbacks and the typical pattern you follow is you take a state and then you reply the new state so if you get an instruction and you have to update the internal state of your Jen server you're gonna then make that change and then pass the new state as the sort of return value of M so when we get you know set key to value and state you know at the start is just going to be an empty map we're gonna basically put that key in that value into the map that's our new state and then with handle cast you reply with a couple of the Adam no reply and the new state and it's a little this seems like it's kind of a lot going on there but it's the same basic pattern that we had for the earlier process you basically just keep looping and each loop you're passing your current state and sometimes you change it based on instructions handle call is a little more interesting you get a second additional argument which is the process that sent you the message again because you need to respond to that message synchronously so when we say get a key we actually want to return that key to the process that asked us for it so the response were just getting the key from the state which again is the key value matter and then handle Paul has to return a couple of reply the response that you send back to the caller and again the new state and those could be different you know we might want to actually delete the actually what is it we might want to remove that value from the store from the state before we return it in which case the state would have to change this sort of a simple key value gen server but we're gonna be doing a lot more gentle server so stick with me but you know following our existing sort of key-value api you can see here a friend start link again because we named that process after the friend module we can use that to refer to our key value process so we don't need to keep tracking the PID we need to start it but we're gonna thank you Joe so again the messages we send are a little more complicated but we're basically saying hey accept food a bar you know the tupple is the list of instructions or get foo and then we're gonna get bar back so pretty cool but where the where Jen server is really shine is that you can layer on a really nice API over your process and these are not required with some sort of convenience functions so within your friend module you might want to define functions that allow you to interact with your friend sort of which I think a better name but your your key value store in a you know less clunky way and so we might define just a friend set function and again that's gonna delegate to the Jen server dot cast function but we don't have to call that from the client side and you know we maybe just want to take the key value and we know we're going to send this this Jen server the set messenger to get so the way that you know right there that's nice you know friend that start link so we got a friend going we're gonna say friend upset through the bar friend back get food get bar back pretty nice it gets better because we can also avoid having to call friend out start link within your application you might just always want to have your key value friend there which you can do through OTP supervision if you have an elixir application which you've created with mix new X if you don't have an elixir application you can pass it the - - soup or sup flag to basically initialize it as supervisor not a whole lot changes but within your main application module you'll just get some super what's called a supervisor specification we basically just say you know this is my app these are the processes that I want running if you already have an elixir application and forgot to initialize it with like I did you can always just re initialize it with mix new dot from the current directory it'll ask you for each file if you want to overwrite it or not and you only need to overwrite mixed up exs in this case the apps called reminders reminders DX so you can see here it makes that exs all we get is an additional option to the application function which is the reminders which is our main application module and then any arguments we want to pass the supervisor supervisor is just as you can see here you know where we're starting up our application by calling supervisor not startling and what supervisor does is it says start me a process but this process isn't gonna do anything itself all it's going to do is manage other processes which in this case they're called children and here the only child I'm finding with the worker function is my friend module so this goes pretty deep I won't cover each each line of this but the idea is that my application instead of just simply interacting with it via the console when I run my application it has children that it's gonna keep running and the interesting thing is you can see the strategy here one-for-one there's a number of different options you can provide there but that represents how your supervisor is going to handle when the processes it manages crash and one-for-one just means if my process crashes start me up a new one there's other interesting ones so in this case if our friend crashes for some reason we're just going to get a new friend and you'll see that in action I think on the next slide so here if we run you know in IE x session but we pass in our mixed environment which is what the - s flag is do wish there were a clear syntax way of doing that without having a call friend at start link we can see our friend is already running the process that we're is and again we've named it after its module so we can interact with it just like we expected we can set the flu to bar or we can get to do back but if we send it the exit message it's gonna kill that process and so what happens when we try to find where our friend is you can see the PID is actually different change from 144 to 150 so because we have a supervisor a friend you know our key value friend died and was immediately restarted though of course its internal state was lost so if you really care about keeping track of Jen servers internal state across restarts you want to persist that to disk somehow or somewhere so that's kind of the basic building blocks that we would need to build a reminders application any questions before I guess before I proceed about Jen servers or odd stuff yes so process that exit takes two arguments the process that you're destroying or you know ending and I think just a reason I don't actually know maybe someone else does if kill is a magic key word I probably didn't need to kill my friend should've just politely asks him to terminate or something like that but great question thank you for clarifying implements what I'm sorry okay okay all right I'll make sure you get any leftovers tomorrow but there shouldn't be leftovers they're really nice shirts I'll be impressed if you did thanks Joey yeah so what will we need for our reminders application it'll be something we just interact via the console so you'll say within the console create a reminder and then it'll send back a reminder at whatever point you want to we're gonna use a gen server to keep track of the state eventually to gen servers and probably is gonna have an API like you know reminders dot client remind me and then something you want to be reminded about and how many seconds in the future you can do all kinds of cool natural language processing to parse dates and things like that but we're only going to deal with seconds in the future because programmers and we calculate everything in seconds and then once once your reminder elapses you're just going to get it back in the console via io applets you know we could beefed it up and have it send you an email or look into the Twilio API and send you a text that'd be sweet but save that as an exercise for the reader so I started with a client to try to just walk through this a lot of it should be pretty similar to what you've seen but you can see in the the remind me function that's sort of our nice API for interacting with this gen server you know excuse me two arguments the message and how many seconds in the future you want to be reminded and then we're gonna just delegate that the agenda overcast to the message new with the message and then how many seconds of the future and immediately handle that via the handle cast gen server callback and then again as with any gen server we're gonna get the message as the first argument in the state is the second and actually going to use a process send an actor which is a way to send a message to in elixir process after some amount of milliseconds so because it's in milliseconds we've got a convert seconds to milliseconds by multiplying by a thousand and they were actually going to send that message to our self so this gem server is going to processes can send themselves messages it'll receive the message of no reminder and then some text and yeah so that's how we keep up and reminder and then when we finally get around to receiving that reminder call back we're just going to hire that puts it with a reminder in all caps to remind you one interesting thing here is you'll see handle info we covered handle call which is for synchronous messages and covered handle cast which is for asynchronous messages where you don't care about the response handle info is the third callback that Jen servers implement and that's actually for regular normal process messages so when we send messages to Jen servers typically it's with Jen's overcast or Jetta server call but because Jen servers are just regular processes regular elixir processes they can receive messages you know that aren't through a gem server so any messages it gets the bat format and lympho is there to catch again but because it's a gem server you are going to get the message and the state that's the second argument so you can sort of change the state so to see this in action you know we just do client that new want to be reminded about and how many seconds in the future three seconds later that shows up in our console so we've got sort of a proof of concept too but we've got some products for example what if the client dies and all of our pending reminders right we had the process and after actually I'm not sure you lose this okay okay yeah because we were back response okay pretty cool kid thank you so this was kind of a contract example you you you could make a small change in order not to lose pending reminders but we're gonna be something else also we can't add more concurrency we can't have more reminders client processes next step that I thought we might do would be to extract a store gen server to handle keeping you know holding these reminders it's gonna be another process and then if our client process dies for some reason and say we don't know pass a none possible string as the message what we're trying to process it it goes down it gets restarted by supervisor but the store as a separate process that just keeps on chugging so we don't lose our data so the store is going to be another worker from our application supervisor you can see I'm just gonna call up reminders dot store it doesn't get initialized with any arguments and all I needs to do is store a list of reminders in its internal state and then any reminders when we ask it for reminders that are sort of past the time we needed to remind someone about it gives us those back and removes them from the internal state so you know it's got two two main functions one's going to be just stored up safe it's gonna take in this case a couple a message and a timestamp and the reason it's a timestamp and not seconds in the future is because it doesn't know when the I guess it could tell when it receives the initial message but anyway as part of this contrived example we're gonna store it as an absolute unix timestamp rather than the number of seconds in the future so we can always compare it with the current time and tell should I reminded you about this or month and then that the same function is just going to delegate to and I'll cast callback which is going to use the handy list prepending syntax of elixir to just put that couple of a message at a timestamp at the head of this list which is the new state that we pass on through gloop it's a little more complicated when we're trying to figure out any reminders that have passed but it's going to just you know we're gonna call it as stored on past reminders pop because that's the elixir I think the name in the elixir standard library for lists and maps when you take something out and return it and remove it from wherever it was so we're actually going to take these reminders out of the store and return them when they're past due and I'm just gonna filter over the state which is the list of all reminders and basically just compare the timestamp if the reminders timestamp is less than or equal to the current system time in seconds then the new state is going to be all of the reminders that we know about - the ones that we just took out because they're past due and then here because it's handle call we need to return three things the atom reply past reminders is what actually gets returned to the process that calls past reminders and then new state is the updated state of the current process that mostly clear the OS looks like it's a does something on it yeah so whenever an elixir you're calling Erlang module you call them with the okay pre-check or sticking uppercase converts all those so you won't happen this right yeah and then to hook up the client and the store up together taking a slightly different approach so rather than using process that send after here I am every second calling a clock function which sends the client itself it tick message basically just like the ticking of a clock every second we check hey it's a second later are there any reminders that we need to send out and so this is using a gen server callback that we haven't seen before in it an interesting thing about gem server callbacks is they're all optional so if you don't define one it's just going to do the default behavior but if you need to override it and it happens when you first initialize a gem server process it takes the initial state or the initial arguments to that process and you just have to return a couple of okay and the state that you want to continue initializing it with but you can do anything in the meantime and so this is going to call run clock which is it defined I forgot to define run clock or I forgot to include in the presentation apologies it's pretty simple all it does is actually a process that send after but it sends itself the tick message every second so when we get the tick message we're gonna loop over all of the past reminders from the store we have any one and we're just going to IO it up what's them like we were doing before and then when we're actually creating a new reminder okay yes so when we call I think I got a little sorry I think I might have goofed up the code in the presentation a little bit but when we get a new reminder we're actually gonna store it here oh yeah so this is when we need to create a new reminder we're gonna call stored-up save and then pass it the message and we're gonna convert seconds from now to an absolute timestamp by just adding that to the current system time in seconds I got just a little bit of data manipulation and yeah it it behaves the same way I wish I'd included a final slide which shows that but again you know reminders client and new or you know what I think that function was actually called reminding me yeah you give it the message and how many seconds in the future and again with the store interact I'm sorry the client interacted with the store it's gonna remind you of that that many seconds into the future so yeah kind of stumbled a little bit at the end there apologies but all the code is up on github Steve Kraske slash reminders and I added tags for sort of each just get tags for each step so you can kind of walk through the evolution play with it there's definitely some possibilities for improving this we might break it up further in some ways we might want to handle what happens with the store dies at some point we need to persist it to disk so we might within the store you know when it's anytime we add a reminder we save that to disk and like the file dot write or something and then when we initialize a new store we maybe read the file system it's okay all the reminders that I had you know all those back from the file system or we might delegate that to another process may be a reminders database in the store you know sort of like you know ecto repos you know the store could be the abstraction layer over the database and the database handles the sort of raw persistent stuff and then we've also kind of locked ourselves into a single reminder client process especially because you know it's cool to name your processes because it means you can reference them wherever you need to but it does hinder your ability to spin up more processes that perform the same role so there's definitely ways we could have multiple clients you know we might depending on how people are using our system we might initialize a new client process for each user you know say there was some kind of web interface or something like that you know within our web server we might have a different process for each user that then makes requests to the store or a pool of stores to keep track of their reminders a lot of room for improvement but I hope this was kind of a good intro to dealing with processes and message sending and then ultimately some fancier layers of abstraction that give you a lot of power thank you [Applause]

Channel: Indy Elixir

Views: 2,048

Rating: 4.8222222 out of 5

Keywords: Elixir, Genserver, Agent

Id: eUxang6_WQA

Length: 42min 25sec (2545 seconds)

Published: Sun Dec 11 2016