Jose Valim - Phoenix a web framework for the new web (Lambda Days 2016)

this is not going to be an introductory talk Phoenix in the sense that you learn how to use it precisely and we are going to build an application rather what I think it's and because this you can find now a lot of material online to get started and start you know writing your application writing your code so what I want to do here today is to focus on Phoenix itself and how it works so a little bit of history and how we were able to to build things so Alex early into the arlynn brutal machine a framework that is both productive and extremely performant okay so a very quick glossary we are going to be using those words a lot so Phoenix is though is a web framework that was written with a lecture a programming language and it's a programming language that runs on the Erlang Rakhal machine and it was with the early inverter machine that everything started so and it started with one case the whatsapp case which today is the most you know talked about and her deployment of large-scale deployment of Erlang and they got a lot of attention right because and this was when Chris McCord the creator of Phoenix they started to hear more about airline because whatsapp was acquired by by Facebook by a huge amount of money and they were always hoping about the infrastructure right and they said you know at the time we had like 10 backends engineer and we are supporting a huge amount of traffic right and they would go and write posts like this that say you know they had one single machine running production and that machine will was able to get 2 million connections ok so I mean I forgot to say well what is whatsapp whatsapp is a messaging application that you store on your phone right and you can exchange messages with your friends and you can install whatever phone you have and you can create groups and talk to our friends and so on so and they would wrote like blog post like this that says you know they have like 2 million mobile devices connected to a server and information is coming and going everywhere and today they have a really big scale where they they are handling more messages than the whole global SMS system and so on they have I think at this point 1 billion users or something like that so Chris McCord the creator of Phoenix he heard about this case and when he was hearing about this he was working on an application that had to have some web real time components right him he needed to connect he needed clients to connect to the server and he had information coming going all the time and the technology that he was using out that at the time was really not enough to build what he needed so he heard this and he thought you know what it's going to be great if I could use this technology here right if I could use the same technology as whatsapp to build this application hello so to build this application that I'm working on right and one of the selling points here was exactly when you go to the article and see the machine that whatsapp was the point of production so they had a machine with 24 cores then six gigabytes of RAM to handle those 2 million connections and even though they were using only 40% of the machine load so it was like well this thing's going to scale really well right if I add more cores to the machine that I'm the point production that technology is going to be able to use it is going to be able to handle even more connections so you know that's how it started so Chris was he heard about this his case he got really excited about Erlang and he found out about elixir and then he says ok I'm going to solve this problem right now I'm going to try to have a good way to write this application and he created Phoenix and he started with something called Phoenix channels ok so at the beginning there was only the mostly this Phoenix channels feature and the whole idea of Phoenix channel is to be an open communication channel between the client and the server right so have a client the clients connected with the server for long periods of time and information can go in an interaction anytime you want and if are talking about the web the most common client is exactly the browser ok so you can think that I have a browser connected to the server and informations coming going so that's an example of how this works ok so this is the little bit of trolling part because the first code snippet we're going to see here is JavaScript and so because if you are talking about the web right and we're talking about browsers if a writing client code we we need to write JavaScript or something that compiles to JavaScript okay so here we have some code that's going to run your browser and there in the first line the first thing we do is that we start a socket and when you call socket connect in the second line there is when you open the communication between client and server so now you know the client open a communication with the server and that thing is established and now we are going to say you know we're going to create channels which is where the message they are going to be sent there they always belong to a channel so here I have the chat lobby channel so if you think this is the chat application that's where our users are going to join they can see who is in the chat right and then you can say well every time I receive a message from the server like a user joined so a user joined somewhere else in another machine or in the same machine but from another client we are going to receive this user joint message and now we can handle it and we react to it in our interface right in our browser now our UI but when someone sends a new message I can also react to that event right so the channel on message allows us to react things that are coming from the server but you can also say you know if this particular client if we put stuff in an input and press ENTER we want to push something to this channel and it's going to be sent to the channel and the channel is going to handle it okay and after we specify all those rules you just effectively join the channel and you're going to get now this information coming going so that's the client code okay it's not really a lot of code and to make this work in the server we really need two files so the first one is mirroring the client is that we need to define a user socket and here's our first Alex your code right so this is a very common pattern in alex here we define a module so in this case is chat user socket and we say use Phoenix dot socket to bring the socket functionality and now I can say you know if the user is sending messages on this chat lobby channel I wanted to be handled by the chat lobby channel module and then if he's sending messages on the room channel I want to handle it by this other module and here is exactly the player can do things like authentication and authorization and so on so if the socket defined we now just need to write the code for the lab Channel and the lab channel is just another module it follows kind of the same pattern so we define a module we say we want to use the Phoenix channel capabilities and we define functions that they're going to be vocht based on what happens right on the client so for example every time the user joins you we're going to call the DEF join function and then we're saying oh the users join the chat lobby and there is a message and there is a socket which is a state and when that happens so that's going to happen every time a new user joins a lobby we want to broadcast to everyone right that isn't that Lobby to let them know that a new user just joined and that's exactly what we do we just call broadcast and we say oh I want to send a user joint message to everyone that happens to be in this channel and the user that join has this username and then we also have handling every time the client sends a message we can handle it here and if the client in particular is sending is Road casting a message to everyone we just call broadcast again and that's it that's how Phoenix started start with this whole channels idea and you can be looking at this now and say like okay that's cool that's interesting but there's nothing really new here right there are other technologies that they provide similar solutions to these kind of problems so where is the difference what makes Phoenix especial okay and in order to answer this question we are going to do two things they're going to take the outside view we are going to see how things work from the outside and they are going to take the inside view and understand how everything is working inside the virtual machine as well so for the outside view what we have is that then we have the server okay and we have different clients connect to the server the first big difference the first big advantage that Phoenix offers is that we say that the channel Phoenix channels they are transport agnostic so it doesn't really mean it doesn't really matter how the client is connect to the channel we can support different transport mechanisms so for example in this case we're using a browser that's going to use WebSockets but you can for example be using a browser like Internet Explorer that does not support WebSockets and then you need to fall back to another communication mechanism but not only that sometimes they are building mobile applications that have specific requirements and you need specifics realization protocol you can just go and bring your own transport and or even more interesting right embedded devices embedded devices we are talking a lot about Internet of Things and we're seeing a lot on this area happening the Elector community and you know you sometimes we have custom protocols for embedded device like co-op or give some messages systems and you can just bring those in as well so that's the first difference the second difference is that Phoenix is going to scale very well both vertically and horizontally so vertically we saw we kind of got an idea of the whatsapp case right they had they had a very powerful machine so vertical is when you start to buy more and more powerful machines so they had a very powerful machine of twenty four cars and it was able to spread and to use all the machines resource efficiently so that's good but phoenix is also going to scale horizontally right by adding more machines to your deployment right transparently because we rely on something called the Erlang distribution so let's let suppose you're dissin air like you wrote your application you put into production and then you're starting to have more and more clients the first thing you can do is that you can say okay I'm just going to bump and get a more powerful machine but at some point say okay I cannot handle all the traffic in one machine anymore or you want to have more machines for you know fault tolerance reasons so what you can do is that you can just put another machine your cluster you have a one line configuration file that tells you know I have one other machine running here and that's it right everything is going to work so now for example imagine that you put another machine into production and then you have the browser it disconnects from this machine and when it reconnect it can reconnect to the other server and everything is going to continue working as before if this browser sends a message it's going to go to that server the server is going to send to this server that's going to broadcast to our clients and this versa okay so you can scale in both directions and it's going to be really straightforward to without external dependencies we're about to rely on the database Radice or or not so those are already two big differences right they're talking about being able to bring whatever you want to play into phoenix and we are talking about the scaling be much more straightforward but things get much more interesting right when we take the inside view because we're going to see the concepts here that are really going to make us productive when building phoenix applications okay so here too trade this idea I have the client here on the on the left side okay and we have and here is what is going to happen on the server so we can think that a client right is going to connect to the server when we call socket socket not connect in the client and then what this is going to create when we connect is going to create a new process on the server and from from now on until the end of the talk every time I say process I don't mean an operating system process I mean an airline virtual machine process which is a very cheap a very lightweight trial of execution in the whatsapp case for example didn't they have two million connections so they had at least two million processes one per connection okay so we really created up a bunch of those processes right we can go ahead and create a bunch of them so when the client connects to the server it's going to create a new process and this process is the one responsible to handle the connection to handle the transport okay and now every time you don't you join a different channel like chat chat lobby or a particular room we are going to create new processes as well and the reason why this matters is because processes in the early inverter machine they are isolated and concurrent this means that being isolated means imagine that there is a bug in the chat lobby if that process crash the one that's run the chat lobby all the other chat functionality all the other channels they will continue working just fine because they are isolated the connection is not going to be affected and this matters a lot in cases like embedded devices alright we're establishing a connection can be expensive you didn't want to do that all the time so the isolation really gives us guarantees for us developers to not worry about you know wait if something goes wrong is going to bring the whole thing now or anything wait continue running its way to continue running just fine because they are isolated and they are concurrent which means if one channel for example is for some reason processing an image or Radeon an external API it's not going to to block right it's not going to block the other channels they are all concurrent between them they are all running at the same time and so those are the guarantees we have a modeling everything with processes and then at the end we are talking about the inside view we also have a pub sub process which is the one responsible for exchange information between machines right so that's the one if you send something one machine the reason why it's going to show up in the other machines as we saw before right is because they have a pub/sub system that is exchanging information qu and the pub sub system use distributed Erlang by default so you don't need to worry about anything you just need to tell where each node exists but if everyone is something where you cannot make the Machine stuck to each other you can rely on something like Redis for example Postgres qu and that's how Phoenix started and it grew up right to be much more than channels today it's a complete web framework and you can use it to to build this web real-time stuff we're talking about but it can also build regular html5 applications or api's distributed systems and so on and the reason why Phoenix was able to grow so much and why it got so interesting is because the Phoenix car team we realized something realize the following a lot of the tools that you use today regarding the web they kind of give us this false dichotomy they say you know you have to choose you have to choose between being productive or you have to choose between being performant right they say you know you can use this particular two or this particular framework that's going to be very opinionated but then you're going to sacrifice performance which means you're going to sacrifice also UX because your application is not as fast as it could be so you're sacrificing a lot of things right down the pipe in for sake of productivity and then you have other solutions that say you know we're not going to be super productive here because we are really offering like low-level stuff right that runs close to the bare metal right and then if they say you need to choose which one you pick and realize that we Phoenix you don't have to choose we can have both right this is this choice you don't need to make this choice okay so that's what I want to talk about for the rest of the talk so we just went through the introduction with how Phoenix came to be and for the rest of the talk I want to focus exactly on those two aspects right how feelings can be performant and how it can be productive and what that means to us so I will start with the performance thing because we can be a little bit more analytical about it and show some numbers and and I and because talking about channels I want to talk very quickly about the work we did in channels performance and which performance I can expect from it so you know we talked about whatsapp case how they had two million connections and the question is can we achieve the same with Phoenix and the truth is that you can so we're doing some benchmarks on our channel system and here you can see a graph of here on the horizontal axis we have time and then on the vertical one the number of clients and you can see that we are pushing new clients to the server at the rate of 10 K clients per second to 20 K clients per second so you can see that we are pushing load and this is a single Phoenix machine so you're pushing load to this machine until we get to the point of having two million clients and the interesting thing about this benchmark is that we could go beyond 2 million we were just we were just not optimistic enough when we were when we were configuring the machine we set the limit in the OS level of sockets to be 2 million but if we have be a little bit more optimistic we would be able to continue going ok so we do get the the performance results we're able to expect and it's really nice because after we got 2 million connections in if you go to H stop to see how the cores are being used because they are just connected there is not really information coming and going you can see this is a powerful machine it has 40 cores you can see all the cars are idle right and then after we have got the 2 million connections we got in Wikipedia article and we broadcast to everyone and it was able to broadcast the art coming like three four seconds to all the two million connected clients and then you could see all the security spiking right starting to do some work so this is really really nice right and really we're getting the performance we expected so that's cool but as I said right it's not only about the the channel stuff Phoenix is a you know full-fledged web framework so the regular HTTP HTTP stack we have it there and it performs really really well too so before I move on I'm going to show some numbers and every time I show benchmarks number I need to do the disclaimer right don't trust benchmarks right use it you know as a guidance whatever need to do you should never base our decisions only our performance numbers per se otherwise may as well write binary code right so so there we go and you can go to this ripple story to find more information about the numbers but I want to use those numbers just to give an idea of what is happening here right so here at the top and this is a machine if I remember correctly with 10 or 20 cars ok so at the top we have plug which is a very free abstraction Alex around the web server and you can see that it gets 2000 requests per second ok 200,000 sorry and what is really interesting about those numbers right is that Phoenix comes right after an interesting thing is that Phoenix plugs a very abstraction and Phoenix is a web framework and even for being a web framework and adding things like Flash session security concerns the the the impact it has in the fruit put it's only off 10% right and this is good it means as you add your own business logic you're not going to see a huge impact in the fruit put and then below we can see things like gene written in gold playing in Scala and in the bottom half of the table we see things like next press clustering no then solutions other solutions and go in some solutions in Ruby and another thing that's nice to highlight in in engels numbers is that for this whole table only rails play in Phoenix they are actual frameworks all the others are like kind of libraries so you need to assemble everything together by the fold they are going to do less things so and we can see here how Phoenix and playing rails doing a lot more how they are going to to behave right so and again so those are numbers take it with a grain of salt it should run your own benchmarks I have an idea what you can build based on your use case but what is even nicer to explore in terms of performance than those numbers is exactly to try to understand right how can we get such good performance and to do that in to understand how that works in the HTTP regular stack I want to go back to the client server view right I want to go back to the inside view so again it works very similar to ad sockets in the sense that every time the client connects to the server we are going to spawn a new process that lightweight route of execution to handle that request and as we have different clients doing requests right we are going to have new and more processes being spawned and again wide using processes is interesting it's because processes they are isolated and concurrent and what this means for regular HTTP requests is the following right so first crashes are isolated we want that right if one request fails it should not make an unrelated request fail as well but not only the crashes are isolated the data is isolated everything that belongs to a process is not shared with another process and what this means is that the garbage collector it's per process right it's you we don't have to stop the world garbage relax we don't have a point where our garbage collectors won't say hey I need to stop everything that's running so I can do garbage collection right that doesn't happen and this really matters because if you're using other platforms right and then this can be anything like Java Ruby Python node whatever you're using and if you start measuring your your system performance you can you can look at the average and the average would say oh you have a good average like 15 milliseconds for example for requests but when you go to the end of the tail of the the response times of your performance you're going to see that the 99th percentile right the longest requests are going to see that there are users that are waiting like two seconds three seconds to have a request fulfilled and that's because those are the unfortunate users that trigger the garbage collector and then there's nothing you can do you just have to wait you have to wait the garbage collector to finish running okay and here we don't have this concern the garbage collectors per process if you have fast requests you can even have a request coming allocating data and then the request is over the process is gone we don't even need to garbage collect we can just reclaim back and all that the process created and as before we can load balance on both IO and CPU right so it doesn't matter if you are waiting on the back hand or if you're processing some image we are going to use all of our machine resources efficiently cool so that was performance right and kind of give an idea on how Phoenix can perform really really well and now we're going to the organic to the most interesting part of the talk which is where we're going to talk about productivity and when we talk about productivity I always like to break productivity in two parts okay we have the short term productivity which is like you know you just saw this talk you got really excited and you want to try Phoenix back home right you want to create a first project and start to get something going so how productive you're going to to do in doing that and there is a lot a lot of concerns here right because we're not talking only about a web framework but very likely you are you have to learn a new language as well which is Alex here so here we need to have a very strong focus on documentation and guides right getting started material books and so on but we also need to have a very good concern in terms of workflows right we need to have a good workflow that at the beginning is going to guide you to whatever you want to build at that moment and help you get your first steps out but more important than the short-term productivity is the long-term productivity right because you have a lot of tools that they sacrifice the long term productivity in terms of the short one they are tools that it's very easy for you to get started but as time passes and you need to maintain whatever it's running production or continue evolving the code it gets really really harder so here in the long term productivity we have other concerns right we have for example introspection because if it's long term productivity it means that this system is now actually running production you need to maintain that thing you need to ensure it's working out so how well can we understand the season that's actually running there so introspection matters a lot and it's also about montane ability right you have the system running there when you need to start evolving changing the code how well can we do this ok so that's what we're going to cover for productivity so let's get started with productivity we can start with the short term productivity and as I said the commentation matters a lot so if you go to the phoenix website phoenix framework org and we are going to have guides that cover from you know getting started installing installing whatever dependencies you need to the point the system to production on your own on your own cloud or whatever you use or to something like Heroku and we also we are really proud to have just we have written a book called programming Phoenix I am one of the authors with Bruce State it's also here and it just went into production this week like two two days ago so we're really proud of that and we hope it's going to be an excellent resource for folks that want to learn more about Phoenix and get started we cover everything from the basic you know MVC up to getting up to channels and doing more of those web real time stuff and talking about MVC right something that's going to help a lot folks coming from other frameworks and sorry Phoenix is that we use the MVC pattern with a slight twist okay so if you're already familiar with that you you have a better idea of how you can organize your code and so on and that's going to help you get started as well and something that helps a lot is that we also have generators as learning tools a lot of tools they have like they have generators code generators and a lot of folks think that you're going to use the code generator to generate pretty much your application right it's like oh are you a programmer no I'm a code generator right I just run the generator and everything comes out at the end and that's not the focus we wanted to take we wanted to have generators as learning too so for example imagine that you just installed Phoenix you create a first application they're like okay I have no idea what I need to do now I have no idea how to create a model how to organize my controller I have no idea what I should put in the view how I should render stuff so if you're building html5 application you can just run this comment makes Phoenix dodge an HTML and it's kind of going to lay out some files we try to minimize the amount of files we layout with the structure that it should think about to start writing something HTML base write something that's going to receive data from a form and put that data from the form into your database and then get data from the database and show it back to the user but if you are building an API you can run the JSON command and it's going to guide you to that direction okay any further interested in the web real time stuff you can try the channel generator so it's really good because you can get there around these commands and kind of having on how you have to say or from that moment on and we're talk about productivity a lot of the tools that you expect to be available in other frameworks they exist in Phoenix too so we are talking about how to get data from the forms right up to the database so we have something called form builders that's going to help with that if you are building html5 applications you cannot escape from it you need to write CSS needs to write JavaScript so integrate with a build tool that is going to take care of this and it in fact uses es6 which is a new version of javascript and which my own pin is a huge improvement to the previous version so you're going to squeeze some extra productivity from theirs as well we have something really nice which is called live reloading and these basically means like for example if you are changing a a template right with some HTML or if your changes CSS file as soon as you save the file we are going to automatically reload the page for you we've you know with the new CSS or didn't if you you just change it and this is really nice because we get you get rid of that side card like we change the file and then you go to the browser and the new refresh it and then you change again you can just see the changes being propagated as you go and then other things like prettier pages so if there's an exception we are going to help you find out what is wrong and come up and solve that quickly this one is very important the first-class concurrent tasks tools because you know today I mean it's a thousand six thing everyone has machines with a couple cars but we are almost all of us running test Suites using only one of our cars right that's not acceptable anymore right for compiling our code running tests running our code everything should be using all the machines resources efficiently as possible and that's what you get with Alex series so we have first-class tests but not only you know first-class tests but concurrent tasks we make it easy for you to run tasks concurrently regardless if they're talking to the database or not and we have a package manager called heck p.m. and it's a community that's really really growing is a package manager that started with Alex here and today it's being adopted by the airline community too and so on and you know we're starting to get more and more packages we already passed the I think two or three million downloads mark so it's growing really quickly so cool this is about short-term relativity right and it's about you know do we have resources harder to get started do we have a good workflow the basic building blocks that you need to build applications are they there and we think they are there right so you can get started and you start building whatever you need to or if it's for fun or it's for our business but as I said even more importantly the short term productivity it's the long term productivity and this part of the talk it has really two aspects to it about long term productivity I like to talk about two things one is the applications which is what we want to cover and the other one which I unfortunately won't have time to go over it is exactly functional programming and since I'm not going to go over it I just want to do a very quick explanation what I mean by that which is to me I mean functional programming when you go to different communities so Alex there is a functional programming language and when you go to different communities different community are going to have different definitions of what it means to be a functional programming language and to me it boils down to you know functional programming it wants you to make the complex parts of your system explicit that's what functional programming to me is about right a lot of the decisions that you see happening functional programming which is like when you get a haskó and you split the side effects and you make them explicit right why'd they do that because the side effects the source of complexity so we want to be explicit about them when we're talking about immutability where you cannot mutate the restructures you need to transform them they are immutable again if if if your data is mutable right we are compacting two things you are complexing they state with time because the states don't change with time behind your back right and then function of program looks at that and say whoa that's that's complex that's going to be hard to reason about so they say okay we're not going to have things mutable by default we want to make those things explicit and as you get more acquainted with Alex and Phoenix I hope we start to see that you think about that how it's making you the complex parts of your system they are all becoming more explicit which means that when you look at your code and you need to reason about it the complex parts they're going to pop out immediately and you can focus on okay so that's one of the circuits I think for the long term productivity in Phoenix but the other one is exactly applications okay and that's the last part of the talk that I want to explore so let's get it started in order to do that we're going to come back to our famous the inside view right so what is happening in our server so we have again the client here on the left side and when as we said when the client connects to the server we create a small process right a small tiny process and then as we have different clients collecting we start to have a bunch of those processes and they're all isolated they are all running at the same time and what we can think here is that you know as we have like multiple processes handing different connections we have something like you know a TCP acceptor the thing that's actually waiting for their connections to start and it's the entity that is going to create a process for every new request we actually get so that's interesting right so now we have some processes and then we have like this process this process here the tcp acceptor that's kind of managing all of those connections so those are some processes that are running our system right when you're running in Phoenix in production that's something that's happening in part of your system but somewhere else in your system we have as I talked about before we have like the pub/sub process which is the process responsible you know if I send something on one machine it's going to be broadcast to all other machines so they can all send information to the client and that's the process as well right it's it's concurrent it's isolated and so on and when we start writing software this way right like we have a bunch of entities that are all running at the same time we start to wonder wait what's going to happen if something wrong know goes wrong with the pub/sub process right because they're isolated so if this pub/sub pub/sub process crashes everything is going to continue running but on the other hand we don't want our system to to run like missing half of its processes right if the pub/sub system crashes right if the pub/sub process crashes we want to do something about it you want to just leave it there right in that state so what we do is that we have something called a supervisor and the whole idea of the supervisor is that it's supervising those other processes to see when something wrong happens and then if they crash the supervisor is just going to notice that for example the pub/sub process crash it and start a new one it's going to restart it okay so why is this idea so important right and in order to answer that we can we can think about an almost daily occurrence which is we are using our computers right and then for some reason things are not working as expected first thing is it just doesn't work what do we do we'll start the machine right like oh my god please not working I want to restarted and then restart the Machine and then everything is working just fine right the issue mysteriously disappeared and why is that it's because we'll restart right when we restart our computer it goes back to its initial state the initial state that we tested right that's guaranteed to work is the initially say that manufacturers whoever built our computer tested a lot because that cannot fail and that's the same with with the code here that's the same idea we are applying to our code so if I have a process like the pub/sub process running some particular code and something goes wrong instead of trying to handle that I say ok it's it's ok to let that process die but I'm going to do that I'm going to start a new one that's going to go back to that initial State that's this state we we have tested a lot right that the initial state our system started so we define the supervisors and then we can thing that disappears the supervisor as well so we have like a supervisor that sometimes have supervisors and what we are building here is what we call a supervision tree and that's where we build this tree of processes we're supervising them we put them into applications ok so what is an application at the end of the day so the applications they tell us how to package and run your code ok so when you go to Alex you create a new project and then everything belongs to an application right and the applications packaging and knows exactly how to run the code and applications that can be started and stopped as a unit so applications they have both the startlogic why don't need to do when you start running that application why do you need to do when you want to stop it and it's important to say at this point that when you when you have an electrical system running production you don't have one application running side by side you have like five ten twenty applications they're all running side by side you can think about them as components if it makes it easier right you have a bunch of components that are running in your system and another nice thing about applications that they provide unified configuration and this is excellent because it doesn't matter which project you're using which project you are working on the way you configure that project and you start up and running it's the same right we have a unified configuration across all elixir applications and again the applications they hold the processes and the state in the supervision tree right so in other tricks so if we row back here because this is our application right we have all of those processes all the process that are running in my system they belong to this application they belong to this tree which means the application can access those processes and because of the code and state belongs to process we can introspect them I could go on and talk and talk and talk about this but I think is going to be more productive if I do a very quick demo of what it means to design our software like these using applications and supervision trees okay so let me just set this up all right so here I have a Phoenix application I just created before this talk and what I'm going to do here is that I'm going to start ix which is interactive Alex here and run the application at the same time okay so now I have my Phoenix application running so I can do you know here and I have the terminal running as well so I can run an elixir command I want but what I want to do here is that I want to start a tool called observer that ships as part of Erlang and the observer two is going to allow us to know it's good it's going to allow us to observe our system right so at least things are very well named right they'll observe observe the supervisor supervises it really helps okay so here is here's observer / running is this graphical user interface and it brings all the information we expect from our system right so which are link or link version it's using the runtime version you know how it was compiled how many CPUs its allocated to the runtime right now so I have four cars how much memory it's using and where this memory is being allocated so this is a great you know upfront view of what is happening right in our system but things got get even more exciting as we see what is happening here so in this case there is not the excess in this application so those low charts are not going to be interesting at all but if you had traffic coming here it could come to the low charts and see how things are behaving but my one of my favorites the other panes here we have like memory locators and so on but one of my favorite one is the process okay and in order to explain why I like this panel so much we need to go back to the Phoenix channel case so remember I said we got two million connections on a single machine and we had to do some optimizations to get to that point and the way we did these optimizations what is we got the clients to connect to Phoenix and I think the first version we got only 30k subscribers right where we were only able to get 30k connections so what we did is that we open up observer in the server and then we came here and we ordered by message queue and then we could see of all the processes right all the code running my system which process was the one that was behind because the prosody the all the information exchange in our system you exchange by sending messages so if something is going slowest because one process is behind it was not able to catch up to its messages in time so I came here we ordered then we could see uh-huh I know which process being slow I could double click the process it would open up this other pane right and then I could see information about the process and why it's not working why it's as low so so that's great right and that's and we did this like three times and then we're able to get too many connections right so we fixed the first bottleneck and they're like okay now we're able to get to 300,000 subscribers okay and then we and then but that's a bottleneck then we came here we try to see where is the bottleneck and then we're just did these three times and were able to get to 2 million subscribers great right for example if you have a memory leak you can come to this panel right to say wait which process taking a lot of memory you order by memory and try to reason about your system okay so that's one nice way to have a view of all cold running our system and gives a lot of introspection what is happening production but the most exciting pain of all is the application pain right because this one is going to show all the applications running our system so here for example you can see that Alex here is an application itself and you can see that we have the airline colonel and the airline standard library so you can see all those applications running here right and all listed here on the left side and you can see that all the applications they have supervision 3 so this is the supervision 3 coming from airline you can see the supervision 3 coming from Alexia right and we can see the supervision 3 for our application the application we just started and what is exciting about this tree is that we can see exactly how our code is structured for a Phoenix application so what we have here as I said before is that we have a supervisor and we have the endpoint which is the Finke responsible for handling web requests and here you can see the pub/sub system and so on but we also have here something that we call the repository because most of our applications they need to talk to the database and what we use this after the database in a phoenix application is something called the repository and we can see that there is a process here this elixir demo repo that's exactly the process response want to talk to the database and so let's let's study here right so we choose this repository here and the way the repository works every time or just after the database what do we do we we create a bunch of collections of the deep database to create a pool of connections and when you need to do a query you go to the to the repository and say hey I want to do a query and when you do that the repository is going to give you a connection back and then you can do the query so here we have the repository here we have the pool and here we have all the collections of the database and now that we have all those things here we can answer that question right which is what is going to happen if something goes wrong with the system imagine that you are talking to the database and this process here - 6 - it goes wrong it breaks right we can reason about this because we can simulate failures I can come here and say I want to kill this process so you send it a queue message and what we expect to happen because we have a supervisory expect the supervisor to notice that that connection crashed and start a new one on its play and that's exactly what happened we can see here that the two six shoe is gone and it created a new process here that we can see at the bottom we have a new PID we've a new identifier right now at the end and we can do other things right like for example what if the whole pool has a bug right so it's going to happen is that we can kill the poo and it should bring that whole part of the tree down okay now we can see that we all got new pids right there are all new connections cool so so I hope these give an idea what our applications right and in order to sum up applications they give us a lot of introspection and monitoring into our system and not only that it gives us all the visibility of the application State right we can go to the tree we can double click processes and we can know exactly what they're doing what are they stick what is they state they have and so on and not only that it's easier to break into components imagine that and this is something we explore in the book imagine that you are working on application and that application keeps on growing because they're adding code you're adding features at some point say you know this is too much we need to break it apart so what you can do is that you can go to that application tree and you can select a sub tree or a branch of that tree right and just move it to another application right with its own start and stop logic so it's very easy for to reason about your software and break it apart later on and it gives us also the reasoning when things go wrong ok so that's what I want to talk about long term relativity and I think one of the secrets for that it has been on the early inverter machine for decades at this point right and we are just leveraging that which is the whole idea of applications how we can understand your system we can understand our software right we can understand how things are run into production and in order to sum up this is the talk about the phoenix web framework if we're excited about it you want to try it out how to phoenix framework argh and you be ready to build your api is back hands html5 applications web real time channels and whatever you want to and if you want to get a more structured look into Phoenix check out the programming phoenix book my programmatic programmers and if you are interested more not only in the framework but are interesting in the and the wrong time you can head out to a lecture - Lena org we have a fantastic getting started guide that's going to guide you for everything you need to know in terms of language and we also have advanced guides about building the systems and the kind of applications we just talked about the supervision trees and so on and we also have a bunch of books on programming Alex here you can pick the one that goes the the banner the best with our style and that's it finally I want to thank my company plataforma tack for investing in an exterior in the whole community for four years at this point and that's my talking Phoenix that I hope you have enjoyed it thank you

Channel: Erlang Solutions

Views: 17,923

Rating: 4.9043827 out of 5

Keywords: Lambda Days, Phoenix, Elixir, Jose Valim

Id: MD3P7Qan3pw

Length: 45min 17sec (2717 seconds)

Published: Tue Mar 08 2016