ElixirZone: Erlang 101 - Processes & Parallelization

oh [Music] [Applause] [Music] one day [Music] so [Music] hello and welcome to the first ever elixir zone i'm james lavin i'm a former ruby developer who discovered elixir about seven years ago and never turned back i love elixir i love erlang i haven't used erlang professionally except it is the foundation of elixir so when you get deeper into elixir you start digging into erlang but most of the code that i write is elixir i love it i don't ever want to do anything else and i want to explain to you why i find it so exciting and why i think it is an interesting language for you to discover as well for me elixir is about happiness i enjoy it i find it uh very easy to work with very easy to understand uh far less likely to have nasty bugs that cause me to tear my hair out and i've been making a good living off of it for the past seven years uh and i think it's something that you would will enjoy as well so i'm excited to share my love for these two languages with you and the ecosystem that's grown up around them which actually is bigger than these two languages specifically and we'll get into that so when i after i discovered elixir i decided to push all my chips into the center of the table and i have actually succeeded beyond my wildest dreams in terms of how the ecosystem has continued to evolve and improve uh i feel like this character from uh one of my favorite shows shit's creek which i highly recommend you watch if you haven't seen it it's hilarious raking in the chips every day i can't imagine life without elixir at this point i wouldn't really consider another job that doesn't involve elixir i just love it that much elixir is often considered a functional programming language i think the main thing which i'll be talking heavily about today is that it's a highly concurrent language but let me start off by allen k is a famous uh programmer software engineer who coined the phrase object oriented programming and objects and uh somebody in quora once asked uh what allen k might think about joe armstrong the creator of one of the creators of erlang claiming that erlang might be the only true object-oriented language and joe's thesis advisors claim that erlang is extremely object-oriented because most people don't consider erlang which is the underpinnings of elixir to be object oriented and interestingly who weighed in to answer this question but allen k himself and alan kaye said who knew joe armstrong before he passed away they were friends they interviewed each other uh i love joe armstrong we lost a great man when he left us recently and he might be right lying is much closer to the original ideas i had about objects and how to use them i encourage you to watch some allen k lectures because he's a very creative guy with all kind he's a polymath he knows all kinds of things and he brings different ideas from different domains together in very interesting ways fascinating guy one one of the interesting things about him is that the reason he loves the erlang model of programming is that it is completely message based and the reason he thinks that's what he had in mind when he talked about objects originally and it's because he comes from a biology background he started off as a biologist he sort of worked as a super computer programmer in his early days to pay the bills but his his love was from mathematics and biology and he thought he was going into biology and ended up in in software engineering but uh the reason the the idea of the erlang concurrency model and processes and objects uh which erlang doesn't call objects but alan case and joe armstrong both think of as objects uh these early processes act very much like human cells in biology where in a in a typical programming language a uh an object could call another object you just reach out push a button basically call a function directly on another object in the real world cells don't operate that way cells don't reach out to other cells and push buttons to make things happen inside the cell what they do is they send messages they um you know they can send various kinds of hormones etc or neurotransmitters or whatever it might be they send messages that may or may not be received by the the intended recipient and the intended recipient can ignore the message or do something in response it's up to the recipient uh what if anything it wants to do in response to any message it receives this is a natural model that allen k had in mind and it it it fits the erlang processes model which i will explain now so here's a simple little diagram of how erlang does process management so a process is completely isolated from every other process inside an online system and again elixir is built on top of burlang uses all the same primitives so i'm going to be talking today about erlang but pretty much everything applies to elixir as well so you have two independent processes and they each have their own completely private memory no memory is shared even the messages that get passed from one process to another are not shared they don't have a pointer to a message somewhere what they have is they the erlang will copy each message when process one wants to send a message to process two erlang will copy the message and give process to its own its own copy of that message so nothing is shared in erlang everything is completely isolated every process is completely isolated from every other process and this buys tremendous benefits one is that if a process dies nothing else is directly affected so if process one dies here process two can continue on um you don't have cascading failures uh a second thing and so that enables recovery from failure so you could have process another process which is responsible for monitoring process one and it detects that process one is crashed and it will restart a new replacement process one to do the same job there's many many many things that airline can do because of this very simple model and because the memory is completely isolated from each process within each process erlang's virtual machine is able to parallelize work across all the cores on a cpu across multiple machines even safely in such a way that if any machine or any single individual process dies it doesn't cause cascading failures let's move on here so this is a slide from martin oderski who is the creator of scala and did a lot of other famous things in programming from 2012 i believe this slide is from and he said that mutable state plus parallel processing is the root of non-determinism and if you're a programmer if you've been programming a while you don't like non-determinism especially we get into things like race conditions and all kinds of very tricky uh problems related to bugs that are very very hard to debug and so his response martin nadersky's response to this problem was to basically try to replicate the the approach that we saw erlang adopt with processes that are completely isolated um scala has what's called the act akka model which i have never used but i've read a bit about it and i'm not no expert on this but it basically is using what's called the actor model the actor model was invented by academics in the 1970s the erlang people in the 1980s and early 1990s who created erlang did not know about the actor model but they uh basically invented that what they built was an implementation of the actor model uh and that's sort of these independent processes uh it's called the actor model because in if you think about human beings for example communicating with each other we don't push buttons on each other to say do this do that we can communicate with one another using messages so we speak to one another for example or we can send each other an email whatever it might be the information inside our brains is not observable directly by anyone else the information in our brains is our own personal state we receive messages we can intentionally or unintentionally ignore them uh we can respond to them however we choose so we are each of us is effectively an actor we're we have self-contained state which is encapsulated and private to us people can send us messages we can send other people messages and uh it's it's inherently asynchronous because it takes time for messages to go from place a to place b even if that is the speed of light or the speed of sound whatever it might be the world is full of asynchronous messaging between actors really and that is the model that academics in the 70s came up with it's the model that ericsson when it tried to build a reliable telecom system in the 1980s basically landed on because they couldn't find anything that solved the problems they were trying to solve and you can see the akka model from scala has borrowed some of its even some of its syntaxes borrowed from erlang but it was very much an attempt to replicate some of the benefits that erlang offers here again this is akka this is an akka actor it looks very similar to what i just showed you with an erlang process uh you have a message queue um i didn't mention that earlier let me go back to the earlier slide so in erlang you have a each act each process has a mailbox so when messages come in uh you could get a whole bunch of messages all at once that you can't deal with them simultaneously a process at any given moment is is processing a single message any messages that come in while the process is busy processing some other message or while it's been put to sleep by the virtual machine which can which can happen that's another benefit of having these individual processes with their own isolated state is if a pro any process is is busy for too long the virtual machine can just pause them the state is all in one place so it just pauses the execution of the process let some other process have its turn and then eventually it'll come back to it or if there's no messages in the mailbox of that process it will put the process to sleep so it's not consuming resources and then when that process receives another message it will wake the process up and said okay here's your chance to deal with this new message but these mailboxes allow for um the ability for messages to queue up and to be processed one at a time the earlier version of the erlang's predecessor which was also in use at ericsson and served as sort of a model for some of the ideas that are built into erlang did not have it had individual processes but didn't have mailboxes so everything had to be every message had to be handled uh as soon as it arrived the moment it arrived and that caused all kinds of problems so part of burling's innovation was discovering this concept of a mailbox um so again akka the akka has taken the idea of an of an actor or a process which has a mailbox and then a process a message handler that will process each message one by one again i'm not an akka or scala person but uh this this article which you can read uh suggests that the akka implementation is not inherently safe as the erlang implementation is because akka does not completely follow the pure private state that the erlang processes have there are ways to do unsafe things with akka synchronous messaging is one of them and the fact that it mixes with object-oriented programming therefore it's possible to expose the internal state of an actor through publicly available methods um it's and that's without using messages so in erlang processes can only communicate through messages and that and the memory spaces are completely isolated which makes it completely safe as i understand it in akka that's not necessarily true you could program that way but akka does not force you to which and if you leave that sandbox you may no longer enjoy the safety provided by the actor model so i'm not here to slam akka or scala i don't know them very well but i just want to point out that the erlang akka the erlang actor model is a very powerful uh paradigm and it's so much so that martin rudirsky a brilliant guy decided to use it to create akka which is in wide use widespread use it's terrific but even their implementation is not as safe as the relying implementation i like this meme i made a variable variable mutable now i can't share it without synchronization and that is that is really the crux of the problem when you have shared mutable state if you have shared state but this state is not changing it's immutable that's okay you can have multiple processes reading the same information as long as nobody's ever changing it similarly you can have a mutable state that's not shared so that's like within a single process where nobody else can reach into the internal state of that process so the process itself can can evolve its own state over time but no other process can directly access that state so that's mutable state but it's not shared the problem is when we get to shared mutable state when you do that you need to use uh synchronization methods to prevent multiple simultaneous interactions with that data and that causes all kinds of challenges and problems and i'm not a smart enough engineer to to figure this stuff out i just don't want to deal with it i like earlying simplicity uh which protects me against all these problems but most languages out there the object-oriented program in languages that are very popular they have all kinds of different ways to try to avoid this problem locks mutexes semaphores and you can do this but you also can end up shooting yourself in the foot in ways that are extremely hard to debug and i just would rather avoid that completely as the erlang team also did so many languages this is hardly a comprehensive list but many popular languages most popular languages use shared mutable state and again there's nothing intrinsically wrong with it it's just dangerous it's it's dangerous by default i prefer the erlang elixir model where there's it's not possible to have shared mutable state which makes programming an elixir neuron kind of boring in that way but i like i like this form of boring so here's a diagram from a book called intro to concurrency and programming languages i want to define the terms concurrent and parallel because it's confusing so concurrency is about processes so could these multiple processes happen simultaneously in the real world or within the virtual machine is it theoretically possible for them to run at the same time and most languages like javascript let's say javascript's event loop in the event loop it it appears as if multiple things are happening simultaneously but what's really happening is that uh execution is happening on one thing and then either that thing ends and something else execution of something else starts or maybe that thing could get paused and then something else happens and then it comes back to that same process a bit later so you don't real you have three different things that in theory could be happening simultaneously or asynchronously but because you're limited to a single thread of execution um you have to split up that thread and share it across all the different things that you want to be doing and so each thing can only happen part of the time even though you know when you're not looking at the cpu level you know as you as a human being looking at your computer it feels like all these things are happening simultaneously but in reality they're not they're just they're have they're being interleaved parallel execution allows you to say okay these things that in theory could be run simultaneously let's actually run them simultaneously in that case you can see that you know this this x-axis is time and so it takes much longer to execute these concurrent tasks in a non-parallel way than it does to execute those same concurrent tasks in a parallel way the problem is that executing concurrent tasks in a parallel way is dangerous if you have shared mutable state because things blow up and they can blow up in sometimes easy to debug ways but often in very subtle hard to debug ways and this is why well so i mentioned the event loop uh here's an example so you can say oh yeah but but javascript or node or whatever it might be can do all kinds of other things as well and and that's true but it doesn't do it on that that main thread or the main event loop thread what it does it can hit it receives lots of requests uh and it can it needs to do all kinds of other stuff and what it basically does is it says okay oh i need to call out to a database now let's have someone else do that work and then let me know when the work's complete and then i'll pick up the handling of it at that point so you can have additional threads doing other types of work similarly io you know printing something to a screen you know saving something to a file reading from a file all these things can happen off the main event loop but uh there's only a single thread for the main processing of information and that's because it's dangerous to do things in a truly parallel way and so instead of doing that they will have a single thread of execution for the main work and use callbacks to offload work to other threads that are safe to do separately like a database call isn't going to affect any data in the system until you put it back into your system so and this approach is fine but we're going to realize that it causes all kinds of challenges for scaling and i'll explain that soon so whatsapp in 2012 put out this blog post and they said that they had a single server that was hosting over 2 million simultaneous tcp connections and they were doing that on the server using only 38 of their cpu which is pretty pretty impressive you know what's app was built with erlang erlang has massive concurrency so you these processes that i mentioned earlier are not typical when you we think of processes we typically think of operating system processes or os processes which are heavy light heavy things that have to be scheduled to a cpu thread and basically own that cpu thread while they're executing the erlang processes are not that they are completely different they are created within the virtual machine which is called the beam but the virtual machine that runs erlang creates million it can create millions of processes completely isolated in terms of memory space uh able to start and stop and pause execution and it's all managed not by the operating system but by the erlang vm and so that's what what app was doing here so they had over two million different users connected via tcp to a single server and each of those connections had its own erlang process under the covers communicating across that tcp connection phoenix is the elixir framework the elixir web framework built on top of elixir so elixir is built on top of erlang phoenix is built on top of elixir and in 2015 uh phoenix framework was able to do something very similar where they had two million simultaneous websocket connections to a single phoenix server and and that's the sort of thing you can do in online it's extremely hard to do in programs in languages like ruby or or python to make this completely safe and if any of these websocket connections died nothing else gets affected the other 1 million 999 000 etc websocket connections would be totally fine because they are completely isolated from one another so if anything goes wrong only that one little process blows up and everything else continues working smoothly interestingly i i don't seem to have the slide here but i know that uh they said that the only reason they had to stop at two million websocket connections is because they ran out of uh i think file handles within linux they could have gone higher than that the name erlang comes from agner erlang who was a mathematician statistician and engineer uh in the early 20th century in denmark and he uh created the field of telephone network analysis and that's where he invented what's called the erlang formula which is a key element of modern telecom network studies uh oh i should have shown that a second ago let's try this again okay and so there's a unit of measurement called the erlang which is what erlang is named after it it also sounds like it ca erlang came out of the company ericsson uh so it also sounds like erickson language and it's an interesting pun but uh supposedly this was the the true foundational inspiration uh but it has multiple uh really has multiple reasons for being the name of the language so carried traffic in erlangs is the average number of concurrent calls see the word concurrent there that's the key measured over a given period off in one hour uh we don't need to go into offered traffic but it's a similar concept so this was named the international unit of telephone traffic in 1946 in this man's honor so the erlang is a measurement of concurrency which is why it's such a fitting name for this language which is all about concurrency this is a speed there's a number of presentations given by the orlan creators uh this is robert verding he's one of the creators the the three main creators are mike williams robert verding and joe armstrong there were many others involved but they were the three uh most famous and most uh heavily involved so he gave a talk in 2016 where he explained the problems that they were trying to solve and he's he's very clear that what they were trying to accomplish was uh true telecom business needs that were uh challenging to ericsson uh they needed huge scalability well let's go through the points so this can these were business requirements that came out of the challenges that eric's in the company faced so actions must be performed within a certain amount of time so you couldn't have very slow things like in in the jvm garbage collection can often be stopped the world i i've been told and uh we ireland couldn't have stopped the world garbage collection for example which causes like the the 99th percentile of of process of behaviors to happen really really slowly couldn't couldn't allow that the system must be distributed across several computers the reason there's multiple reasons for this one is scalability but another is fault tolerance because if you run something on a single computer there's no way to guarantee reliability you need to have at least joe armstrong made the point you need at least two computers to be reliable because a computer can crash at any given moment the system is used to control hardware the system can become very large the system is can be very complicated and have lots and lots of features interacting which is why they wanted modularity so that things didn't complexity didn't explode systems needed to be in continuous operation for years erlang was built so that you could upgrade the software without bringing down the system uh hot code reloading it's called software maintenance should be performed without stopping the system stringent quality and reliability requirements fault tolerance both to hardware failures and software errors so if things blow up either because the server died or because the software was wrong and somebody put a code into production the system needed to keep moving and not fail and the system must be able to handle very large numbers of concurrent activities so they were dealing with they were building telephone switches which would have thousands of users connected through the switch and if any of them any of those connections had a problem nothing else should be affected and uh they needed to be able to scale to have thousands or tens of thousands of users using a single uh switch so this is a very difficult set of requirements and interestingly although it was they were pretty unique challenges back in the 1980s and 90s they've turned out to be very similar to the challenges that we face with scaling web uh web servers etc in uh modern programming which is why erlang is relevant to this day so robert verding he pointed you know he emphasized that they didn't plan out to build the actor model they didn't even know know that the actor model existed they didn't set out to implement a functional languages language they were dealing with a set of very concrete business requirements and they were trying to solve those problems and what they they discovered the solutions to be after a series of you know after several years of repeated uh inner uh testing and development and retesting and all these things they landed on these key principles uh lightweight concurrency so that's being able to have hundreds of thousands or millions of processes simultaneously the the creation of the process the switching between processes and the communicating between processes must be cheap and fast uh communication had to be asynchronous you couldn't have you know one process waiting on another process to respond you can the primitives interlining allow you to are asynchronous you can add a layer on top of that and make things synchronous just by having your caller block and wait for a response from the receiver in a server a client server model but the primitive built into erlang is asynchronous so you can add synchronity on top of it but you don't have to it's it's by default asynchronous processes are completely isolated from one another nothing if one process dies it can't affect any other process error handling when something goes wrong that that failure needs to be detected and handled so there's supervision trees and processes supervising other supervised processes so that if anything goes wrong the error gets detected and handled the system needs to evolve continuously they need to be able to upgrade the system while running and nothing should block things should happen pretty quickly things should never be super slow francesco cesarini is a is a very big uh he's a great guy in the in the field he's been advocating for erlang and recently elixir for many years he's been written several great books on the topic and he said instead of inventing a language and then figuring out what to do with it they did the exact opposite they didn't even they weren't setting out to invent the language well he says they they were actually they weren't even trying to invent a language they first searched for a language that did what they needed and after trying everything they found that nothing did what they needed so then they started inventing a language that solved a very specific problem massive scalability and reliability phone networks were the only systems at the time who had to display those properties this chart will explain why erlang has become increasingly relevant over the last 15 years or so what you'll see is that the clock speeds of cpus has basically been flat since about 2005. this used to be where we got most of the the performance gains is that cpus kept getting faster and faster but they're no longer getting faster and faster because they've hit the laws of physics and the way they've been able to make uh cpus more powerful is by sticking more and more transistors into them but by adding more and more logical cores so instead of having a single cpu thread that's really really fast you now might have eight cores on your cpu or 16 cores or 64 cores on your cpu and each of them may be as fast as a cpu core what a cpu was 15 years ago but you have 64 of them instead of one and this enables if your program can can run in parallel you can parallelize the work and get 64 times as much work done but if you have a programming language that has an event loop or we'll get into something called the global interpreter lock that ruby in er and python have uh you can only use a single core at a time except for the stuff that you're offloading to other uh independent threads like io etc so basically you're stuck with a single thread so your cpu speed performance this is hitting you very very hard because you're not benefiting from any of the advances in microprocessors over the last 15 years in order to benefit from microprocessor improvements you need to be able to parallelize your work and as early out of the box does that so a global interpreter lock is something that languages like python and ruby use to synchronize the execution of threads so that only one native thread can execute at a time an interpreter that uses gil a global interpreter lock always allocates exactly one thread to execute at a time even if run on a multi-core processor so they're intentionally slow and not using the other cores because if they parallelize their work and use multiple cores they hit the problems that synchronization is required to prevent shared mutable state from blowing things up so to prevent shared mutable state from blowing things up they use the global interpreter lock to prevent parallel execution of of their code and that obviously hampers performance of python and ruby in this age where cpu performance enhancements are 99 uh happening through the addition of additional cpu threads rather than making an individual thread faster this is a great article i read this last night it's like 20 pages of the history of erlang by joe armstrong fascinating about the challenges that they faced how they worked through them how the language evolved over time if you're interested in i highly recommend this paper i'm going to quote a couple things from his paper so first he says rowling was designed for writing this is how he describes rowling i think it's a great summary running was described designed for writing concurrent programs that run forever erlang uses concurrent processes to structure the program these processes have no shared memory and communicate by asynchronous message passing earlying processes are lightweight and belong to the language not the operating system erlang has mechanisms to allow programs to change code on the fly so programs can evolve and change as they run brief history of erlang so in 1985 joe armstrong was hired uh and he asked what should i be doing and he was told his job was to solve erickson's software problem which was a little vague he interpreted it as find the language that you know will make our telecom switches work he after exhaustively evaluating every language out there especially the ones with with concurrent in their names or their descriptions they basically discovered no such thing existed so they started building their own oops went the wrong way here um so he started working on adding concurrency to prologue eventually over the next couple of years he and robert verding and i think mike williams landed on something that approximates erlang today they they built they evolved their you know prologue interpreter into what is similar to erlang in 1990 they built a compiler and mike williams built an emulator made it about 70 times faster so it was getting usable at that point uh the next year they started working on distributed multi-node erlang which is something they had always wanted to do but they had never gotten around to to starting it in may of 1993 they wrote they published the first rowling book which came out and they also created the beam virtual machine which was a lot faster than the jam stuff that they had made a few years earlier early systems was created i think francesco cesarini is the the ceo of erlang systems today running systems launched to train consult and document erlang the first distributed ruling product happened in 1995. in the end of 1995 uh erlang was suddenly embraced within ericsson and otp was launched and this happened because of a fortuitous event inside of erickson that uh led basically a big project that was not using erlang called axem failed and at that point the management decided okay we're gonna we're gonna try this online thing and worked really well uh until a few years later even though it was a massive success erlang decided to ban her sorry erickson decided to ban erlang and the reason erickson decided to ban erlang is because they felt that it was proprietary and they didn't want to have the burden of having to maintain this language over time without any external help they thought it was a dangerous situation to be in where they were building mission critical business products on top of a language that they were completely responsible for uh debugging and maintaining uh so the at about that same time that the product that was built with erlang was a massive success it had uh over a million lines of relying inside of it and it had a nine nine nine nines availability which means basically it was off yeah it was it was down for seconds or minutes per year it was a spectacular success running over delivered the product was a huge success uh but erickson still didn't want to use erlang anymore and they they said we're not using any new products and the existing products are going to have to phase out erlang so people within the within ericsson convinced erickson management to release erlang as an open source project which is sort of a new thing back then and the many of the people working on erlang quit and they formed their own company called blue tail a few years later blue tail was bought by nortel which back then was a big another big telecom company eventually years later in 2004 erickson lifted the ban on erlang and joe armstrong and others went back to ericsson in 2006 symmetric multi-processing was added so that's that's the ability to use multiple cores so erlang because it had completely isolated memory for each process and because messages that got passed were passed by copying rather than shared pointers to messages it was able to paralyze work but even though it had in theory been parallelizable up until then they hadn't gotten around to actually making it making the execution parallelized in 2006 they did and that opened up all kinds of possibilities and performance benefits this is where the elixir piece of the story sort of begins the man on the left there is joe armstrong the man on the right is jose valim and joe is sort of the main guy behind erlang and jose is the main guy behind elixir and they became friends before joe's passing joe had tremendous admiration for elixir and what it was doing and thought it solved some of the challenges and limitations of rolling elixir brought a lot of extra tooling and developer happiness to erlang among other things i didn't this is a brief history of elixir i'm not going to go too much into it but he started building it in 2012. chris mccoy and both chris mccourt and jose valim were rubyists uh who grew frustrated with ruby's uh model of concurrency and because it basically couldn't do parallelism both of them so jose went looking for a language that did concurrency better and he found erlang and decided to build elixir on top of it uh chris mccord similarly he built some concurrency tools within ruby that did just didn't work right he couldn't get him to work right and so he went looking and discovered elixir and fell in love immediately and decided to start building a phoenix web framework uh on top of elixir and the first thing he built with that was phoenix channels which leverage uh erlang processes under the hood that's when we get to the two million well websocket connections uh phoenix presence was added now phoenix live view which is amazing but i won't be talking about today but this is a great article that sort of summarizes the history of elixir and phoenix uh how we got to live view it's by chris mccord the phoenix creator great summary that there elixir and erlang are not the only languages in this ecosystem there's also lfe which is lisp flavored erlang there's also gleam a new language that uh it's as it says it's a friendly language for building type safe scalable systems it compiles to erlang and has straightforward interoper interoperability with other beam languages such as erlang elixir and lfe so this is a whole ecosystem there are a number of other languages that aren't as famous that run on the beam well lua lewis kind of famous uh so it's a whole ecosystem now and this ecosystem has its own foundation now uh which is fabulous because it's bringing people together from the orlando elixir sides and the lfe et cetera to all work together and make the whole better than the sum of the parts and this erlang ecosystem foundation has a variety of work groups working groups and covering various topics such as language interoperability observability tools machine learning documentation education and training security it's fabulous the way the ecosystem a lot of relying developers have become elixir developers a lot of elixir developers like myself have been learning erlang and it's just making everyone uh giving everyone more more and more power and knowledge i won't waste more time on global interpreter lock but you can read this it is a bottleneck for languages like python and ruby here's a beautiful uh this talk is amazing and the first thing after you watch this talk if you're excited i highly recommend you go watch this talk by sasha yurich who is a uh legend in the erlang elixir communities he began as an airline developer he does all elixir now but uh he knows things brilliantly and he he has written a number of books on the topics uh this talk is is brilliant one of the things he shows here in this diagram is so he's using a single scheduler so as we said you can parallelize work in elixir and erlang and that's true but you get some benefits just with even before you have multi-processor parallelization you get some other benefits so in this example he intentionally codes a bug and it causes the process to go into an infinite loop and the infinite loop is just consuming that one cpu thread that he's artificially limited the erlang virtual machine to use but all the other jobs in the system are still succeeding at the same time so we the scheduler is pegged at 100 the the cpu core is pegged but all the other jobs other than this infinite loop process are still half are still getting processed properly the reason is because uh the the erlang vm is repeatedly pausing execution of this infinite loop process and letting other jobs have their turn so even though a process is in an infinite loop and there's only one scheduler everything is still working perfectly you wouldn't even know that there's a bug except unless you looked at the cpu and saw it was pegged at 100 percent so the ability to really isolate each process and what it's doing it's its memory and its execution and pause and unpause its execution creates tremendous benefits even if you have only access to a single cpu thread again this history of erlang paper is great joe said what we would like is a way to explicitly manage the set of remaining goals of a busy process so that any we could suspend or resume the computation at any time our reduction machine works by successively replacing the top of the stack by its definition or if it's a primitive by replace evaluating the primitive the point of the reduction cycle is that at any time we can suspend the computation if we want to several parallel reduction engines we arrange to save and store the states of each reduction engine after some fixed number of reductions and because we have multiple processes uh even though we have an infinite loop when we try to execute this another process can detect the error and handle it appropriately so instead of this display blowing up another line process has detected that there is an error where that the the process that's doing the actual calculation is not returning a value and so it can display an error message so this is how we handle error message we handle errors by having relying on multiple processes so if any if something goes wrong with one process other processes can still recover from that situation so this whole display doesn't blow up because you put in an input we're trying to do an infinite loop and we're never going to get to the the end result you can still handle that software bug these are the three creators of erlang robert verning mike williams and joe armstrong they've given a number of talks i highly commend those talks to you i'm going to start showing some hands-on examples of erlang and how we use it in later videos i'll focus mostly on elixir but for the first few videos i'd like to talk show you some erlang i think it's it's valuable for elixirists to see some rying and feel comfortable with it in part because there are all these great books on erlang that uh tell you teach you things about otp which we haven't talked about yet but there are things that you would want to use in elixir as well but the original documentation is written underlying so being able to read the documentation underlying is tremendously beneficial similarly you can use erlang code within your elixir code so there's a lot of benefits for elixiris to to understand erlang so we're going to do some simple erlang coding soon and but this is the this is a from a famous uh video that they made at erickson back in the day where they uh they called each other because it was a telecom con telecom company and they were doing network switches uh they called each other and said hello robert hello joe hello mike it was actually pretty funny um they didn't intend it to be funny they actually thought it was a very serious thing but it's become sort of a a felt feels very monty python-esque uh in in hindsight everyone has a good chuckle over it but we'll implement this in erlang uh i don't know i get that in there too uh let me start by introducing a few simple concepts here so this spawn is a function uh that will execute so f is a fun an earlying function so this is a spawn is a function call which is receiving a function uh so f capital letters are variables in erlang so this is a variable holding a function we are spawning a new process which is supposed to execute that function that's what spawn f is saying is create a new process brand new process and have that process execute the function f and then the result of calling that spawn is it will first of all it will create a new process and it will return to the the code here the process id of the new newly spawn process which is basically the address to which you can send messages so it's the mailbox address for that process then this bang operator is allowing you to send a message so m a capital letters again are variables so we have a variable m which in this case it can be any elixir any erlang data structure it can be sent as a message so this is just some arbitrary earning data which we are sending as a message that's what the bank operator is doing to this process id that we just captured here so we created a new process by spawning that process is running the function f and we're getting the address back for that new process then we are taking a message which is a data structure interlang and we're sending him it as a message to that process and then within the process itself this is sort of part of the the function f in this case so we're saying start the new process and run this function and part of this function would be this receive what the receive is doing is it's saying okay wait until you get a message if you get a message now start trying to match those messages if you receive a message that matches this pattern then do this thing if you receive a message that matches this pattern do this thing if you receive no messages after time however many microseconds has passed then do this this thing here so this is what's called selective receive so the process is waiting for a message if it has no messages in its mailbox and it has received no messages the virtual machine will probably put it to sleep when it does receive a message the virtual machine will wake up the process and say okay here's a new message for you try to match it and it will go through and take actions according to whatever type of message it received so with that i think we can move on to actually code some erlang okay so let's start learning some erlang as you can see here i've prepared a couple of small erlang programs the first one being called joe.earl erl is the only source file extension so let's open this up and see what's in there it's going to look early syntax is a little odd most people find it unintuitive and that's one of the reasons why i never really took off in a major way uh and another reason that elixir was sort of built on top of our lang nothing wrong with rolling syntax it's just very unfamiliar to people so let's let's work our way through it first of all everything ends with a period uh you can if you have a thing so here's a function receive loop that ends in a period but it's broken up into smaller statements which are separated with commas so that's the first thing to notice so um at the head of your file you're gonna name the module in this case we're saying this module is named joe which is why i called it joe.earl the next line is saying that we're gonna export two functions so that's going to make these functions callable so start the zero here is the arity of the function so a zeroarity function is invoked with no arguments a one area function is is invoked with a single argument so we're exporting these two functions in early and elixir you can have multiple clauses with the same up for a function with the same number of arguments and they are considered the same function if but you can also have the same function with different arities and in that case they're sort of separate separate functions you could have a start zero function and you have a start one function and you could have a start two function uh in future videos we'll we'll show some of that and then this thing here is version so this is saying this is version 1.0 of the joe module i'm not going to right now upgrade this but i'm going to upgrade it in a couple minutes and we'll see how you can do a different version so what i'm doing here is i'm going to invoke this it's called receive loop because we will be turning it into a loop in future videos but right now it's not actually looping we have three functions we have call which is expecting a single argument anything that any variable that starts with a capital letter is a variable so this when you call the call function you will pass in a single argument it will then call the io module just like this is the joe module there's an i o module this colon is saying okay everything to the left is the module name so the i o module has a function called f right and since this is taking two arguments uh f right should be able to have a two area version of it the first the first value that we're passing in is this string and the second uh very thing that we're passing in is a list of arguments in this case there's only one element inside the list and it's arg1 whatever was passed into the function when it was invoked uh receive loop is going to do three different things first it's going to create this uh i'm calling it string i think it's called a binary in erlang it's a little different from elixir string we'll get to that when we get to elixir but for now i guess we can call this a string it's really binary uh we're going to turn that binder the met the data that we're going to pass as the message is that binary that's all there is in the message we're going to send that message to robert so we haven't yet looked at robert but there's going to be another module which will be spun up as a process named robert and registered as robert you can you can send messages to pids the process ids or you can send them to registered names so you can associate a registered name with a process and so when we start robert we will get a pid and associate its pid with the the name robert which is its registered name so the first thing when this receive loop starts it's going to create this message send it to robert and then it's going to go into a receive and it's going to wait for a message and once it receives a message whatever that message is because this is this will match anything and we're looking to match things and since this is a uh this is there's no pattern here other than any message at all uh we will we're going to write joe received a message and then the way this works for f right it's going to take the first value in this list in this case the only value in the list and put it in in place of this tilde p and the teal to n is just a line break so it's going to write joe received a message and then put the message there and then a line break and then after it prints out the message it's going to then create a new message which will be a different string hello mike from joe and it will send it to mike and then it's going to wait again and then when it gets another message it will then do another write where it says joe received a message and it will put the message in there presumably from mike and this is the function that we are going to call uh we needed to export it in order to be able to call it so we're going to call this by joe colon start and since it's a zero already function we don't pass in any values and what the then so when we invoke joe colon start it's going to do these two lines the first and again they're separated by a comma and ended with a period uh the first thing is it's going to spawn a new process and the process will ah did not want to do that uh the new process will run this anonymous function the anonymous function is going to do two things you'll see there's a comma there so it's going to call the call function with the value joe has arrived and which will simply print out joe has arrived and then a line break and then the second thing it will do will go is called receive loop which in this case is not really a loop it's just uh sending a message and then waiting for a message to arrive printing it saying sending a different message and then waiting again and when it receives the second message it will print that and then that function will be over and since we're not looping once you get to the end of all the things that you can do as a process you die so we will spawn this process we will grab its pid we will then register that pid as i mentioned earlier that pid will be registered to the registered name joe so any other process in the system will be able to send a message to joe via its registered name or its pid uh either way the message will get to joe and the message is when they cut since joe will immediately enter into the receive loop here it will wait for one message process it send out another message then wait for the other message and then die because well again as i've called it receive loop because in the next round of programs we'll actually turn it into a loop right now we're not actually looping but i wanted to keep the name consistent so we have a mic dot earl and mike.rl is a little simpler because mike similarly is going to start up a new process it will say mike has arrived and then enter into its receive loop function which is not actually a loop yet the new spawn newly spawned process will have its own pid whatever that might be we will then register that uh pid with under the name mike so there'll be a registered name mike and again so once so it will print out mike has arrived then it will go into this loop it will wait for a message once it receives a message it will send out another message to joe so joe has to exist um when we run this we're going to start up joe last for that reason joe sort of starts everything he kicks everything off and if he's uh if he's running if we if we tried to start up um well the order matters we'll see that uh so that's this is uh mike and then there's also robert robert that's not robert there we go uh didn't say earl there we go so robert's very similar uh the only difference so robert's gonna receive a message then send a message to joe and he'll also say robert has arrived if i look in this you'll see here that ignore the ping pong things but each of the three files we just looked at the earl files the dot rule files are the source files and the dot beam files are the compiled versions of those i'm going to remove the compiled version so you don't think that there's some trick up my sleeve that i've already set something up we're going to compile these ourselves right now so the way you start up the erlang repel the read evaluate print loop is simply typing erl um so we'll do that now so now we are in a shell we are in a process an earlying process it started up an erlang system a virtual machine and it created one process just to handle our shell here so we are since we are in a uh erlang process uh we can sorry let's get that we are in an early process we can find out our own pid and the way we do that is by typing self and now we have a self and if we want to save that to a variable we can save we can make a variable whatever it is just needs to start with a capital letter so we're going to save that value in pid if i say self again i i have not changed so it's going to keep giving me the same value now if i blow up intentionally here if i if erlang by default if you if you run illegal code it will blow up and it will kill your process and other processes could potentially be responsible for noticing that and restarting you in a healthy way but for the moment i'm just going to kill myself by dividing by zero which is illegal so you see the exception but it didn't spit us out of the repel what it did was it killed the process that we were in and started up a new process uh that we are now in so if we look at self we're no longer dot 83 we're now dot 88 so i'm gonna store.88 here and again why did that not work oh so this is another thing this is actually different between elixir and erlang in elixir the the variable pid would have been rebound here but in our line there's no re-binding of variables is a truly it's based on immutability so once you bind a value to pid you can't rebind the value to pid there there are ways in the shell to change that but i'm just going to instead create a new variable called pid2 and i'll assign the new value of 91 to that and you'll notice it killed the second process that we ran under 88 got killed as well because i tried to assign to a variable that had already been assigned to so now hopefully if i say selfie this this one shouldn't blow up because it is remaining 91 at this point so okay so so let's send a message to ourself at this point so we can send a message to ourself we can type whatever we want i'm going to say hello shell but we could send any message i forgot the period okay so the when you when you send a message the return value is whatever this message was uh this is not this is different from the actual message being received and processed by the the shell the message is sitting in the shell's mailbox but before i show that um let's send a second message hello let's call it a hello again shell and i again forgot the period so now at this point we should have two different messages in the mailbox so let's let's look at that we're gonna there's a function we can there's an erlang module which has a process info function we can tell it we want to know about ourself and we want to know the message cue length and again i need a period you'll see that there's two messages in our own mailbox and this should be the same as 2 because we said pin 2 is the pid for us we can actually look at those messages like this so you can see that the messages are sitting there in our mailbox and one way we can get at them is by doing flush um if we do flush it it's it just flushes everything that's in the mailbox so right now if we go back to this now now the mailbox is empty because we've flushed everything let's send a few more pro messages to ourselves so let's call this hello number three and i need a period hello number four and hello number five and then we'll again look at our messages okay so the messages are there now we're gonna do a receive uh we're gonna create us because we're in a process we can have our process do a receive and what we're going to do is we're going to get a message out of the mailbox we pull one mail message out at a time and then uh we're just going to return it we're not going to do anything special with it so we pulled the the third message that we put in our mailbox the first two we flushed then the third message we added um we just pulled it out now i would think if i did receive again i would get the fourth message out for some reason it doesn't seem to be working and if anybody watching knows why uh please let me know and i'll post it in in the room uh if i post this uh on youtube but i don't know why receive doesn't work again so instead i'm just gonna show you that the messages are still sitting in the mailbox there's something weird about i can't get at them again so i'm just going to flush them here okay now that we know how the ruffle works let's go back into the rupple and start up our joe mike and robert files so earl okay so first to compile a file so remember we have joe dot earl and it's in this directory so i'm going to say c for compile joe you don't have to type the dot earl part so it seems to have compiled let's check that actually compiled m stands for module so there is now a module named joe and there's some extra information that shows you the exports call start and then module info which was added automatically let's try calling module info see what see what so it added those to joe so we could say joe module info and you'll see that there's a zero errority and a one energy function version so the zero area gives you all the information if you'd like to get a subset of the information you can type in the atom associated with the key of what you're looking for so in this case for example we could just look at the exports there so when i said adam there so if i put something inside of strings that becomes a binary a binary is just a list of numbers that gets interpreted as characters if i don't put the string there if i just say joe that is what's called an atom and what an atom is is that there can only be a single instance of an atom uh and modules are all named after atoms so joe is an atom roberts and adam well they're not yet because i haven't compiled it if i if i actually say uh tell me about the module robert it's not gonna know anything about what i'm talking about um because there is no robert actually did we just lose joe or is joe still there um joe let's find out joe's still there okay so let's let's compile robert and mike mike and robert okay so now we've compiled the files but we haven't actually used them uh so the next step is to start them up so right now there's there's no there's no function so this is sorry there's no processes named mike or joe or robert because we haven't we've loaded modules which is code but we've not actually executed it and started up a process so if i say registered sure it'll show me all the registered processes and there's no process named mike or joe or robert here and we've asked our processes once they're spawned to be named uh registered with registered names so they should show up here if they were running and they're not they're not showing up here so let's let's start them and see if we can find find them so we say mic start so now we started mic and you'll see the string that we asked it to print when it started up is shown there we can do the same with robert now let's run this command again and see if they show up now so robert is indeed here so robert is now running and registered and mike is hiding here somewhere there's mike there he is so mike and robert are there and joe is not there because we haven't started joe up yet but let me also you at any time you can also say processes will give you a list of all the pids that are running um not too useful in that format there are things you can do to get information other information you know say i and it will show you all of the there's a lot of things running that we didn't start up actively but they run automatically like there's logger for example so you can take advantage of the logger to log stuff that's the kernel supervisor we haven't talked about supervision trees yet but um we will in future episodes and it will also show you the initial call that was run for each of these things so the initial thing that this process zero zero zero is a special process the very s first process that starts up um is it calls a two rarity start function um init is a loot it causes a loop and it's a single uh one arity function and the value it takes the loop function is taking is the state and that's a very common pattern in orlang and elixir where you start a process and you loop and that process keeps looping and each time through the loop it's taking the final state of the loop and making that the initial state of the next loop so it's a recursive recursive data data that gets uh saved every time the call the loop is called recursively uh the state is persisted by passing it into the next invocation of the loop function um you'll see that there's a supervisor uh here there's uh there's various things going on let's yeah let's see more there's even more processes running um anything interesting here there's um well i'm not going to go through all these right now but you can get an idea and there's mike at the bottom there mike is there and i have not yet seen robert but robert should be there's robert at the very end um so robert and mike are both listed there as as are all the other processes that are currently running so we can also say where is robert since robert is running there should be a pid associated with robert and we can say where is mike of course if we say where is joe we haven't started joe yet so if we say where is joe we're not going to get he's undefined another thing we can do is regis i should have done this first because you'll you'll see that mike is here for example there's mike with 107 and let's confirm that that's what we found for mike 107. so you can look up all the processes using this red reges which presumably stands for registered processes but let's let's do the thing so we said earlier that joe is the process that kicks off all the messages so let's start joe we we've compiled joe but we haven't started joe so let's start joe and see what happens okay so joe announced that he has he had arrived as did the other processes when they started then robert received a message hello robert from joe because joe kicks things off he sent the first message to robert robert upon receiving the message sent a message back to joe saying hello joe from robert at that point joe once he receives that message sends a message to mike saying hello mike from joe and then mike receiving that message responds to joe and because none of those functions that i called loop actually are loops they were all completed at the point uh when they got to the end of their functions and the function did not loop so each of those processes did the things that they were supposed to do and then when they got to the end they then died so if we now try to say where is mike mike is now undefined because mike doesn't exist anymore similarly joe and robert no longer exist because they have been turned off so let's go on to a second version of this uh where well let me show you the second version first so code sorry so this is a new version vi joe earl so this is a different directory but i've created the same uh modules again and except there's a they're a little different now so instead of sending just a string ah what am i doing instead of sending just a string i'm now sending what's called a tuple so a tuple is uses these braces and you can put any elements you want inside it's like a list but the items don't have to be the same types of things they could be anything so in this case the message is the is a an authored version of the message we were sending before we're just indicating that this is version 2.0 of our program the difference here is that uh when joe sends a message now joe is going to announce that it's coming from joe he's saying this is for me so he sends a message to robert but announces that it's coming from joe and when he receives a message uh the message will have this it will match a tuple because we're now sending tuples from uh process to process we can do whatever we want any earlying data structures are valid but we're choosing here to send two element tuples the first element being the sender of the message and the second element being the message itself in this case so now we're saying joe 2.0 received a message from and then we'll fill in the sender so if it's coming from robert or mike it will say robert or mike here and then the second value we're filling in right here is the message itself again joe is going to expect a second message and record it and then joe will die so and i did some more changes in the others i guess we can look at one more vi mike earl so mike again mike simmerley is going to now be expecting to receive a two-element tuple with the sender and the message and when mike sends a message he will include himself as the sender the first element of the tuple alongside his message so we can let's see we can load this up and we'll compile uh joe mike and then we will start them up start mic that did not work what did i do wrong oh it's mike.start my colon start right of course um robert again i i don't want to do i want to save joe till last because he kicks everything off joe start and we see the expected messages and we also see now that the each recipient of a message is also they're extracting out of the first element of the tuple is the the sender of the message and the sender is an atom uh the message itself is a binary so this is the atom that they're receiving and this is the binary that they're receiving and at this point uh let's see if uh registered uh they have all died they're knowing where is joe because we didn't ask them to stay along after they sent their messages and received them we will next in the next video in the next video we'll go into the looping and how to keep a process alive but i think we've covered enough for today so i will stop here i hope you've enjoyed this introduction to erlang and the history of erlang and the beginnings of elixir i hope you enjoyed the hands-on relying coding exercise i'm looking forward to doing many more videos hopefully much shorter than this one this there's a lot of ground to cover in this intro video but i hope to put out a whole series of erlang and later elixir videos uh that sort of build progressively on top of what we've learned today um now that i've introduced you to robert and michael mike and joe i think you can appreciate i i have a little a small set of computers servers that i use for hacking around and playing with and i think you will maybe now be able to appreciate my naming convention i had named my very first three machines verting williams and armstrong in their honor what they did uh is extremely compelling to me and i i just love erlang and elixir and uh wanted to honor them and it makes me happy just every time i think about the server names i'm so appreciative of what they built and enabled for the ecosystem today i want to share a few learning resources if you want to go deeper on erlang the first place you might want to look is on youtube simon thompson who is a professor i think he recently retired in england taught a series of talks on erlang and he builds up progressively uh their excellent talks completely free on youtube uh also once you finish those you may want to consider university of kent did a series uh you can see three different playlists here there's a rolling master lying master class one two and three it's a whole series of uh videos and that's joe right there i think francesco cesarini did some of these they're excellent uh erlang has super documentation um it suggests here are some some books many of which are free some of which are paid uh the documentation itself is excellent there's long they're basically online books uh explaining how erlang works i maintain a page on github that i call my tech resources and there's various pages but i have one page elixir which is full of alexa and erlang resources and other a little smattering of stuff for other beam languages and finally i would like to end this with some unsolicited advice i'm hoping this will become a regular hopefully much shorter series of presentations on erlang and elixir uh this intro video obviously had to cover a lot of ground so it's much longer than i intend for future videos but at the end of each i'd like to share some advice that i've picked up over the years in this industry um that isn't necessarily related to elixir or erlang and uh since this video is so long i'm gonna keep this really short today my advice is to not drink soda or tonic or pop or whatever you want to call it depending on where you grew up i drank way too much soda for the first 25 years of my life and my teeth suffered for it as did my stomach and uh in the middle of a run one day i couldn't i couldn't run any further i literally could not move another step and i went to see the doctor he told me to stop drinking soda i did the next 25 plus years have been way better my teeth are happier my uh my body's been happier it's unbelievable how much soda forget the carbonation the the acid is horrible for your teeth but the sugar alone is horrible for your body and uh the apparently there's like 29 different sodas at least that this article points out all each each of them a can of it has more sugar in it than four donuts uh so i urge you to try to reduce the amount of soda that you're drinking uh and ideally stop drinking soda i like to keep a bottle of water with me at all times and uh if you have whatever you keep within arm's length of you whether it's like m m's or healthy stuff uh that's what you're most likely to eat the harder it is to get to it the less likely the less you see something um things that are bad for you you want to keep out of sight and out of reach uh things that are good for you want to keep within sight and within reach so by keeping water or i drink coffee and tea those are great i was drinking soda from decades for the caffeine primarily i get all my caffeine now through coffee and sugar and tea and i drink lots and lots of water and i'm happy i haven't had soda in decades and i urge you to do the same it's a great way to be healthy um so thank you for coming i look forward to doing more of these rolling elixir videos and hope you'll come back for more cheers [Music] you

Channel: Elixir Zone

Views: 271

Keywords:

Id: EZY9W-3D5qY

Length: 84min 29sec (5069 seconds)

Published: Thu Nov 11 2021