What every Node.js developer needs to know about Elixir - Bryan Hunter

all right perfect so awesome to have you here thank you thank you we're going to talk about what every nodejs developer needs to know about elixir today there's me on Twitter I respond on Twitter so hit me there I love carrying this conversation on out of here and if you have any questions always follow up with me there there's no expiration date on that so I want to start off this illustration appeared in The Guardian shortly after December 25th 1914 and this is an illustration of the Christmas truce where the German and British soldiers that were a trench warfare they would this excuse of coming out and burying their dead they got together and they were civil to each other in the middle of this horror that was going on they stepped out and they acknowledge each other's humanity and in technology tech we often break off into camps and we don't do that so well a lot of times we don't reach out and so for the node folks here I really appreciate you all come into a functional programming talk and for the functional program people I'm really glad you're here to to think about this node side of things and so at the point that we make problems visible then you can do something about it you can improve the situation but until you make them visible it doesn't really happen and so thanks so this idea of civility we might hate a technology there might be problems with the technology we might know that the technology is dangerous but you have to understand how people got there and have empathy for them and and on both directions so my path had the sort of long road you can tell by my gray hair that I began writing software professionally back in 1994 and so I was writing C and V B so the old VB not V V net but kicking it old school and then Pascal at vbscript jscript javascript and then c-sharp on and through that path you know I ran into lots and lots of problems I created lots of problems you know like like we all do but I probably created more than my share of problems and through that I I might not have been the best developer but I was awfully good at seeing the my mistakes and trying to learn from them and seeing what the pitfalls were that led me there and through that I worked to try to help other people avoid pitfalls and that empathy that I tried to use in in this land of imperative code in O code during the 90s and the odds interestingly it had this reputation for being able to see these problems and help and that led me then to Norway 2007 so I was here was brought over for a large architectural risk assessment project so here in Oslo there was a company that had a huge codebase I had 20 million lines of managed C++ and C sharp code that these 12 architects and we worked on this risk assessment of how is the software going to destroy the company project over the 6-week period and through that all of the problems that I imagined would be in the codebase you know those were there its you could almost get Robert Martin's book and just hand that book over before you even went in and looked at the code because those problems are common across all imperative and OO systems those code smells are there and so that was expected that side of it amazing team but do you expect the problems one thing I didn't expect though is there was a system in that in that shop that had been up and running for four years without a millisecond of downtime say all the other sec all the other applications they were crashing maybe multiple times a day it was embarrassing was causing trouble you've had dependencies within the code that made it hard to upgrade you had the mingling of state data and behavior like we're told to do by Martin Fowler we had that all through the code and that caused all these defects in the software that made death it was really hard to solve problems and we had to event storms because it was you had concurrency involved in this concurrency is hard to do if you have mutable state problem problem problem but this system for years without a millisecond of downtime that was unusual and so I asked more and more about it and I actually learned is even more remarkable there had been two years previously there had been a major code upgrade this compiled language they compiled and they had pushed new updates to this running cluster without a millisecond of downtime then so I was like what is the spooky magic and so I learned for the first time the word Erlang I never heard of it I was off imperative oo guy you know and and I was happy they're sort of happy I knew I had problems and I knew there were problems but but I was comfortable there and so I dug in and I started learning more about Erlang and my code changed and I have built systems that people tell stories about now that you know they've seen these good things and I couldn't have done that before before I run into the Erlang VM and so it changed my life and so I'd like to come back to Norway and you know share you know return the favor so I'm very thankful for this city and for for what it's done for me and all the defects it's kept me out of so here we have the hero's journey this is sort of the arc that all novels and movies and everything follow this arc of the hero's journey and so that was the beginning of my hero's journey and hopefully you know you this will be the beginning of your hero's journey as you go through and you pick up these some of this knowledge will go through this ordinary world and will progress so for the node folks here I imagine you came to node for some of these reasons it's really a welcoming community you can come in without knowing much about programming at all and you know you're welcome to N and that's awesome and it's easy to get started in node and there's no doubt about that it's easy to get started and you can you can be working in angular uh Durandal aurélia one of these frameworks and if you need to have a back-end you need to stand that back up in quickly well you can do that and node it's it's very easy to to kick that up everyone sort of knows JavaScript no one really knows JavaScript everyone sort of knows it then people also come because they've been hurt in other stacks there's problems of the tools being too heavy or not being able to have enough control over things just various problems and there is a concurrency story on node it's different than I think a lot of people think it is but there is a story there and then people know that concurrency is important so if that brings people and then there's good support in Visual Studio and and on Azure I think of this this sort of progression in node and I have lots of friends that went through node I started working in node when it was brand China new because I was interested in the story of it I pretty quickly then dropped it because I had already learned Earling at that point and I saw it's like this isn't really filling the bargain that I thought it should be and so but this is this progression we start off with folks that are junior and they're full of optimism you know they they haven't even heard the horror stories yet they haven't heard any of the bad it's just this welcoming community they come in and they're they're excited about this and that is that is a beautiful thing to have and at some point you move on and you then have your sort of engineers and they have heard the problems maybe they've seen a few problems but the thing is is they can they can say I'll be a craftsman I see these problems they're documented I'll know how to actually be diligent and not run into these problems myself and that's that peer this is the second stage of no development and then you move on to the seniors and the leads and I have friends that are in the spot right now they bought into this they brought their companies along and they've heard the problems they've seen the problems and now they're like oh my god what have I done you know they got themselves in a trap because they used it for everything they used it for things far far outside of its sweet spot and they're in a terrible place and they're having trouble shipping code that's rough that's a rough spot to be in and those folks I'm excited about where they're at I'm excited about all of these these groups coming in because it's a great feeder into elixir I believe but the folks that are senior they're looking for escape pods and a lot of them are joining up at the elixir user groups around the country so around the world so you'll have this mix of Ruby people and you'll have this mix of node people and then if you scattered FP people showing up the elixir meetups and so I'm really tickled about this this linkage in this connection when on node when people talk about node they often use the word scalable and it's sort of a problem I believe to use the word scalable in the context of node well scalable is a rough word it's a sort of a buzzword marketing word it's a word that managers like to use even though they don't know what what it means and it I feel like you could say oK you've got a fast car describing a cars a fast car you've got a car that goes and flies through the guardrails and falls down the bluff at terminal velocity and you could say that that's a fast car but you're missing an important part of the story by calling that a fast car you know there's other things going on and so when you called nodes scalable there's an issue in that that it can handle the 20,000 connections that come in it can handle the 20,000 users as long as you immediately pass that off for something else to do the work because if you do the work you then have got this issue of where you're blocking you're blocking your other 19 thousand 999 users so the issue then is like okay we hand off the work or who's going to do the work then and what do we write that in and there are all sorts of solutions every time you run into a problem there are there's a lot of ingenuity in the node community and they have solutions you'll have things like queue and you have these ways of just passing things off these libraries have been written but it's always hard manual work and a lot of learning and discipline involved a lot of folks you'll see a lot of blog posts that will say node is not a silver bullet and I think you could say that about most technologies you know it's not the thing that solves everything but there's another way of actually looking at the problem of silver bullet and in this sense I think maybe node is a silver bullet in that a silver bullet has a really narrow use case and there's a sweet spot for a silver bullet and that is to kill werewolves and day to day there aren't nearly as many werewolves running around Hollywood would make you think and so the sweet spots that you would fit there aren't as many solutions where it does map up closely and so more more of an uglier view of this might be it's maybe not the Silver Bullet maybe it's more like the full moon that brings out sort of some crazy in the environment and I'm kind of off of this part of the talk but I've seen so many of my friends hurt by this and I believe that this is the most dangerous technology in the world and and so and I hope that you take this in the spirit that I'm offering it that that I'm truly trying to help and I'm trying to keep problems from happening so let's now move on to elixir and how it came to be so we've got this history of a character named Jose a villain who is here at the conference this week to speak and so he was a core member of the Ruby on Rails team and he he was trying to solve problems on Ruby although you know he's a good developer he's focused on these issues it's hard to solve problems on on Ruby like people would ask for scale they would ask for concurrency that ask for performance and that's a hard thing to deliver on the on the on Ruby and so he was battling that battle and he was then reading seven languages in seven weeks out of curiosity who else heard this book awesome awesome yeah and so he got to the chapter on Erlang and he's like whoa he was kind of like everyone does when they first hear about what this thing does and like that's weird I haven't I heard of this thing and so he was reading about Erlang and he saw that it fit these problems of fault-tolerance concurrency distribution this scale thing that he had been asked to solve and so he started looking into Erlang then it's like first like everyone does when they first bumped into Erling they think okay can I steal part some Erlang and bring it over into my stack and their reasons you can't do that that we'll talk about briefly it's really hard it's a part hard problem and so he really he's a smart guy he realized it pretty quickly you can't do this it's not going to happen so he thought maybe I'll become an Erlang developer okay Ruby is known for one thing and that's it's it's about developer joy right it's a path it's a happy place via developer they hug each other you know it's like a good good spot it's not necessarily known for you know those other things we're talking about but it is about developer joy now you could say the reverse about Erlang so it is good about solving these hard engineering problems you've got these crusty engineers that created the language two of them are here this week so we have Joe Armstrong Robert Verdun crusty serious engineer guys and and so they weren't focused on the problem of dev joy at Ericsson and Sweden in Stockholm they weren't they weren't focused on that problem and so josei decided I'm gonna take I've got another approach instead of trying to rip off Erlang what I'm going to do is I'm going to write a new language that targets the the targets the Erlang VM and that's the path he went down it's like I'm going to build developer joy good modern tooling all this on top of this VM that has this amazing track record and we're going to talk now about the sweet spots so elixir it's also approachable and it's productive not just in the short term but in the long term so it's approachable productive and you have modern tooling so first about approachable we can look at you go to elixir laying org and it's a useful website you know it's not just a thing because you got to have a website it's really useful of learning the language and so if you jump in there and you go to the getting started guide down on the right here you see one through 21 and you actually have a really good handle of the language just by following those pages you can start here instead of getting a book and you'll do fine then as you're going through that you can also go to this other link here and find all the meetups around the world and you'll see a bunch of Ruby people and a bunch of node people and functional programming people scattered in this is the pattern I've seen city by city you can go to github see the code see the language and contribute and you'll have feedback from Josie right on the side so you have this you get a pull request and he's sending hearts all over the place and so this is a nice thing to see this Ruby love showing up inside of this list language he's a great guardian of the language he's a great protector and he's also very interested in bringing in ideas from from all outside places another part of the tool story is hex you have hex PM is the package management system it targets so you can use this for your Erlang code which you don't have because there aren't any Earling developers in the world there's like maybe 510 you know there are a few of us and so there aren't many but this lik sir side there are a lot and and that's a growing thing and so we go off and we see some of the libraries out here you won't find libraries that do like three lines of code libraries these are libraries that take on more serious things that people need all the time but you won't have your six lines of code on a package and also if you go to something like ecto and look at you'll see that the that the level is pretty good here this is the documentation off of a package that you pull down from hex hex is fast all sorts of good qualities about it packages don't disappear on you on hex it's this immutable store of packages packages disappearing on you this of course has been in the news this year poor fella over here a sir god bless him and so uh you know so he took a beating and there's a lot of beading that went on there's a lot of schadenfreude oh there's a lot of there's a lot of negative that happened here but I think it's comment by Tracy harms is really nice this catastrophe fault of a single single solitary developers decisions and if a complete stranger can push some buttons and break your production deployment you have something some thinking to do especially if that was around six lines of code that was really about doing something the language should have done anyway out of the box okay modern tooling so we'll have this sort of retro modern hipster bicycle thing here and so like all modern languages we have the terminal it's the way you do your good work and so so we return to the terminal here IEX the interactive elixir shell when we say IEX that brings up an Erlang node so it brings up an instance of the Erlang virtual machine and loads up all the elixir goodies here okay we go IEX and we are inside of IEX and we can do something like H at noon so help we're going to help on this thing we at dot we tab and we get auto completion of all of the functions within the Anu module or within the Anu now with doing the Anu module and so we say I'm interested in what count one does the count that takes one argument I'm interested in what that one does so we get this nice color we get examples right in the examples and this is the way you can go through all of the libraries and elixir in and you have this help right at your fingertips mix is another tool IX is a tool mix as a tool mix is ripped off from closure community this tool called line again there which you know you can see some of the the commands we can do this you can think of it build code run your test scaffold new projects and it's pluggable so additional things can come in so let's say mix new fizzbuzz we're going to scaffold out fizzbuzz and here we go well you get a readme markdown file which is nice and modern we get a get ignore built in which has our ignores that we would want to have an elixir project that's nice and modern and we are config and so when we have our fizzbuzz module and then we have tests I mean this is nice you know we come in here and this is a nice place to be so let's go into CDN to fizzbuzz let say mix test look compiled and right out of the box we get one passing test so they already started off for us with a template for us to have tests so let's now go in and see what that story looks like okay so I have here our scaffolded out code off of our fizzbuzz this is just what is so this is our one test that you know that would scaffold out for us and that's what we had our one now I have a fully implemented fizzbuzz over here and we can see it's down here we we play to a number and it does a bunch of stuff that we don't have to worry about yet because we're going to see about the syntax in a minute the thing I wanted to show here was the experience of saying mix test okay said we have three tests and one failure so we saw one test a minute ago what's that about and why do we have a failed test let's go in here and see what this failed test actually is so it says here fizzbuzz play 10 to 16 and it expected a fizzbuzz here but we had a fizz and stead so let's go up here and see what that's about so what we have here is this is not a regular comment this is a doc test and so right in the middle of our comments we've included examples of usage to make it easy for people to come in and right here this becomes part of our test suite so we have fizzbuzz play one two five that should get this and that became a unit test fizzbuzz played ten to sixteen should get this and that became a unit test that actually failed on us because we have a bug down here we have a pattern match that's too greedy and so what we need to do is move this code up here do this and then we need to move down here and we need to rerun our tests oops i forgot to save dinner yeah so so our test pass now that was based off of that doc test so I think that's pretty sweet okay so another productive nice sweet part of elixir is the Phoenix framework so this is a web framework that was built by the fellow who just walked in here Joe's a valium created a language and sunny Scroggins back there and Chris McCord and there's a bunch of awesome people chris was here last year gave a talk it's on the on Vimeo and so Phoenix is amazing but I'm not going to talk about it because there's a lot of good content about it by people here later today but this is what a lot of people want when they come to a web framework so elixir is functional and so we've probably heard a lot of buzz about functional over the last few years and so a simple way of looking about functional is you have inputs that are transformed by a function and you have then outputs a goes to B inputs go to outputs you don't have I'm going to come in I'm going to set the table with all my state I'm going to set fields properties I'm going to have a class all this stuff and I'm going to call then functions that are going to mutate state and then I'm going to lose track of what I just did a minute ago and I'm going to have all these bugs instead you can look at the class what came in is the input and know what should have happened inside of the function in return this is important and this is a good way of getting out of a lot of those troubles that I saw when I was my first trip to Norway mutability is another part of functional programming and elixir takes this very seriously it has to take this very seriously because Erlang VM has a hard set of rules that it enforces and this is one of them so it's not a thing you can turn off separation of data and behavior so this is the thing we're talking about earlier about classes with their fields their properties the data in their behavior mingle together which limits the use that you'll be able to handle and use that class in this is an elixir you have modules that have functions you don't have property seals you so you call into a module and you call a function within a module and it's just the inputs that come in and then the outputs that they go so you spawn up a new process and you say this process I need to call into this function passing in this input I'm going to take the output of that and feed that into another function and you just pipeline these things together and that's the way you build programs and side of a lick sir so we've expressive dev Julie let's talk a bit about the language now so language language language okay so we have IEX again and we're going to look at some of the types here so we have integer expected Oh X - a that's kind of nice you know we get our hex right there you know we don't get barked at for having seven float divided by three it just does the thing you'd expect the thing you have type called atoms that are sort of there'd them there the thing themselves they're only comparable to themselves they're just a handle or a constant that even if you have a whole cluster of different machines this like that the atom goat is equal to the atom goat on the other server and it's not equal to anything other than the atom goat strings are real strings inside of elixir you might have heard stories horror stories about Erlang whereas like oh it's bad at strings and it turns out it was a problem with naming was the problem with Erlang and strings the thing they call the string is not what people expected it to be and in elixir a string is what you expect it to be you know you get this utf-8 binary good thing and it behaves like if you would expect it to actually behaves a little bit better than you expect it to and we'll see that in a second we have lists you have lots of list processing instead of functional languages and you have tuples so this is a positional thing let me see that okay let's look at pattern matching for a second so X equal five so we've got our five there I'm going to say five equal x that's a little weird so what we're actually doing here this is a pattern match and it's going to evaluate the same it's happy to say yep five is equal to X you know this is a match and you can pin X down and say pin X equal five and it's happy to say that you know that's it we just a pin X is equal to X plus one it's going to bomb and say nope no it's not thing's not equal to six left hand size not equal to 6 we have then pattern matches it can be against more complex structures like tuples here so we have ABC couple that matches against apple banana cherry and we see that we then bound b2 banana you know and then Apple and C Apple to a and cherry to C as well but that's what we showed you in the shell pattern matching will have a little bit bigger of an example we have list here 1 2 3 C it binds does a pattern match sort of as a way of looking at but we're actually assigning this thing so list is now equal to 1 2 3 and we say case our list and we're going to do pattern matching here inside of this case statement so and we're doing this in the shell here you can have multi-line things you can have complex bits inside of the shell which is pretty awesome so we say case list do we say 11 12 13 and this is not going to match because 1 2 3 is not going to match that right guess is going to match oh not going to match because this is a couple 1 2 3 rather than a list 1 2 3 it's a different type right here we have 1 X 3 and this is going to match and it's going to bind the X that came in it's going to bind the two to the X in the middle and we can then use that later on in the same case statement to do our string interpolation and show this we'll see that happen here so that chasm binds to 2x digit separators this is something that's coming and C sharp 7 I believe so it's good 1000 now so this can say 100 underscore and say okay so just pure candy I mean this is this is about the dev joy side of the language I mean there's just all these things that just make life a little bit nicer you know it's not important but it's it's just awesome though thoughtful you know so here we're going to look at a news and streams so we're going to define an anonymous function so a lot of times in functions if you'll there's a convention of if it returns a true or false you can put a question mark at the end of it and so that's a nice convention you see it you know what to expect so we're going to define this anonymous function that's going to use the rim function so remember it turns the remainder of two numbers you know divided so this is like modular and so so we say rim the first argument so the ampersand one is going to take that first argument that's passed in it's going to rim that number comma two so we pass in a five remainder of 5 comma 2 would be well 1 and so it's going to be an odd number right so here we have a range we have 1 to 100,000 and we're going to pipe this full word so any f-sharp people in the room alright you'll recognize this so we've got 1 to a 100,000 piping full word in two so that each one of those values are going to get pushed in is the first argument to the thing on the right so this functional pipelining a new map and we're going to take whatever the argument was we're going to multiply it times 3 we're going to pipe that forward into a new filter and we're going to filter on our odd function that we defined at the top and then we're going to anew met we get our answer let's do the same thing but this time we're going to use stream map stream dot filter and then we do a new him some so the difference here we get the same result the difference is is stream is lazy and so on our first in our noon stage of that we went through the whole computation and we pass then that on to the next series in the pipeline with stream it's dragging you know one at a time as you're coming through as it's a pull based thing and these both work off the same interface it's called a protocol that they both they both implement the enumerable protocol this is like interfaces and c-sharp or the protocol comes from closure where you know this idea is directly borrowed I mentioned strings earlier this is kind of cool you know we can have straight-up UTF awesomeness right here so all the Norwegians with funny characters in your names you can appreciate this right so person equal so let's look at maps so we got our person named Brian beer defaults I'm only beardy in the winter not beardy in the summer and so we can access it by key like this person name you can access it by dot like this get the same thing we can build a new map based on an old map and just push in the properties are the keys that we're going to change we're going to override and so we have then same thing Brian hunter here okay let's look at strux yes yep yeah I could do person and it's still there it's just we have a new everything's a copy there's you know it's not mutating so good question here this builds up on top of that idea but gives us some extra goodies so we're going to define this module called person here we're going to put a def struct in here when we say name and we're going to give it a default of empty string and we're going to say beardy and it's going to default to false and we then have this data structure that we can start using so I say a person I get my defaults I'm going to say Brian is equal to a person named Brian so we see person B R defaults named Brian I say a person named Brian bar defaults boom there's no such thing as bar D so we got protected by using the wrong key there there's no such thing as bar D there's beardy and you know so it blew and barked and kept us from getting in this all sorts of trouble that you've probably all experienced with JSON already versatile this is a really crazy thing so the language you expect certain things out languages here we have something you would only expect to be able to do in C so we have so we have the bit syntax and we're going to run through this really quickly we're going to grab okay bin data and we're going to read this file so if we look back on this image this is a five pixel wide black red green blue white and this is the data below so we're going to read this file capture it we captured bin data we captured the binary data there to the bin data variable we're going to set up a pattern match where we're going to catch on the Left using this DSL called syntax we're going to catch on the left the data if it matches against Ben data here and so we'll see how that worked so here's our data again our left two bytes has to be be m and a bitmap so you can just go through and just think about data struck by Nerys out there and you can just implement using this DSL and be able to process like whatever your your your protocol is that your you're trying to implement your your format so that was a match we're going to throw away 64 bits we're going to capture a 32 bit little endian into a variable called offset pixels throw away grab 5 height 1 size we're going to throw 16 bits away we're going to make sure that the next thing is a 24 here and then we're going to throw the rest of it away now we're going to say that offset 2 pixels this told us where the actual pixel data begins and we're going to remember that and we're going to throw away that many pixels and we're going to capture the rest of it into a variable called pixels we do that so we saw our values that were here so we have this for comprehension where we're going to capture we're going to shuffles in 8 bits 8 bits 8 bits into blue green red and here we go we've got our data so that's one line one screen worth of code inside of the shell and we've you know parse bitmap which is kind of Awesome so something borrowed a lot of things borrowed one things borrowed is Erlang VM our Lang VM grew up out of ericsson doing awesome serious hardcore telecom things we're focused on fault tolerance so to have fault tolerance you have to have concurrency if you don't process dies you're fully down you have to have distribution if you don't box catches on fire you're fully down so those things are important and you see the track record of in fact your phone works and that over half the world's mobile traffic is going through Erlang which is pretty incredible the data text voice more recent story is whatsapp with with their victories on the Erlang VM you know so just amazing amazing numbers here a billion active users a month and so on and another number that's important to the business is 19 billion dollars they were paid by Facebook there's 10 engineers building this mighty codebase we look at what they're laying VM is in context we have the runtime we have OTP we have languages of Erlang elixir list flavored Erlang there's a talk here this week on list flavored Erlang by Robert Verdi and so here's why people can't just grab the Erlang VM like Erlang bits and pull it into their language and it comes down to this thing as an operating system and it took a lot of time to build it it's a serious operating system it's not a general purpose operating system on Linux or Windows it's a special purpose operating system about making a safe place for code to run 250 plus years our development went into it you can see an example of it being an operating system on Erlang Gon's in the zurich demo and this boots up what would take 300 seconds to boot up a Linux VM on ec2 here they boot up into an instance of bare metal Erlang running on the Xen hypervisor it boots up and mill milliseconds brings up Erlang VM runs a web server processes things kicks out your request and you have this result in 0.3 seconds pretty awesome so look at the job of an operating system which are linked VM is process management interrupts memory management file system so on and so on and think about what our code does our c-sharp our C our JavaScript as a job and that job is to eat as much core as it possibly can is trying to do oh it's only worried about itself it's not interested about the safety of the rest of everything running on the machine and so the operating system kind of has to hate you it has to hate your code it can't trust it and this is the place where it's really different on the Erlang B M is because the Erlang code cannot do things to hurt the stability the Erlang code itself has to play by the rules that they're laying VM setup for it are laying VM doesn't have to support C in Java and Ruby and all these languages it supports languages that only are going to do things that they're laying VM allows them to so that Trust is nice when you can cooperate you can get a lot done when everyone's fighting against each other you don't get a lot done and so this is a big part of that story so we got our Shaun of the Dead reference cricket bat processes so processes is the way we win here so concurrency concurrency is not the same thing as parallelism a lot of times those words get used interchangeably but concurrency is about the structure doing lots of things at once and parallelism is about the execution of lots of things at once this comes in on a story at Ericsson so when they built Erlang in 1986 they weren't targeting multi-core machines in Stockholm then they didn't have any tips this year this is this wasn't a thing then and so they built concurrency and they didn't build it so they could go twice as fast they built concurrency so they could code without getting twisted up into these horrible nests of callback e weirdness and into thread weirdness and they wanted to make the code simple to read and to write because code that simply read and write is less buggy and that helps your fault-tolerant story so they did this massive moonshot from 86 and then they turned on multi-core in 2006 and code they've been written a decade before it did then run twice as fast on two cores four times as fast on four cores and like 2010 or 11 there was a test where at linear scaling up to 40 cores on properly written Erlang code we get that same thing in elixir because it's the same VM all compiles down to the same beam so let's look at elixir and the Erlang VM and the actor model so every actor gets some props so our actors are processes so every process gets these props one is memory so each process is one kilobyte on a 32-bit machine it's two kilobytes on a 64-bit machine instead of this memory here we have a heap and a stack and it's that processes only no one else can reach in and grab a reference to memory inside of that process it's isolated and it's also immutable right because it's it's functional and so garbage collector easiest job in the world be a garbage collector on the erlang p.m. cuz you know no one's only one person's looking at the memory you can't change it once it's set and so you each of these processes they get their own isolated dedicated garbage collector so there's not going to be a big stop the world GC that happens each one gets garbage garbage collected independently which is pretty amazing and that's done for low latency and it's done for deterministic scheduling our mailbox this is the only way our process talks to the world so last for York so we we talked to the world through a metal mailbox this is the only way a process can reach out and so we send a message out of our mailbox we address it to another process it makes it innocent box when they go into a perceived block links and monitors is the thing that we build up on to create some fault tolerance goodies this is at this OS level this Erlang VM level where we get links and monitors links basically like I'm going to link to josée over here and if Jo's a dies I'm gonna die with them okay Sonny back here there's a different sort of thing like so so Sonny I'm gonna monitor to Sonny back here so Sonny dies well I don't really exactly want to die you know but I'd like you to tell me about it you know tell me if Sonny dies I actually care about him but I dunno I don't want to die and so so it's a different level of commitment there and so that's these two tools you have out of the box so process scheduling on the Erlang VM we have a CPU core we have a single scheduler that lives on that CPU core get three processes up and each one of these processes is going to get 2,000 swipes at the core mm mm mm there's nothing in the world they can do to eat more core than mm swipes they can't block and hold up the show everyone gets even is pre-emptive scheduling here so this is another way of visualizing it they each get this much and they get thrown back into the rotation same thing happens if there's multi-core if there's big multi-core yeah all happening so we also have this the supervision bit off the links and monitors is talking about we have a supervisor and a worker the worker something happens the network cards down the hard drive fail whatever we have some weirdness the supervisor can then restart the worker and this is important because what do you do when you call tech support about something the story is rebooted right that's the first thing to walk you through and it always works and you're like God I worked again and so this model is built in into Erlang it works so well for tech support that they included it as the part of the language so process running the process running in boom he falls into a receipt block he's going to sit here forever and ever and ever blocking waiting for a message to appear in his mailbox no one's talking to him so he's going to see here well does that wreck the show have we now blocked and wrecked the show well no we haven't because the Erlang VM schedules him out of the mix and then we scheduled in across the other two remaining cores back and forth and we don't bring this guy back into the mix until the postman the Erlang VM the operating system until he knows that that process actually has mail when it does he's brought back into the fold part of the rotation this includes things like file i/o so any sort of i/o that we have goes through message passing like this so even though I write file read earlier on that bitmap file when I said file read I was talking to another process and then I waited for that process to send me a message back you know I didn't have to do this as a developer it's just part of file and the read function on there it's part of gen server and so at the point I asked for that file immediately when I asked for it that processes over there doing the file reading and I'm removed from the rotation until I have a message in my inbox so now I'm blocking i/o that you don't have to worry about you can treat like sequential code it's really done right here four cores four schedulers you got processes and we have this game of balancing compaction so that happens and so busy scheduler busy scheduler I'm not so busy I'm going to take some work from this guy move it over here so that this scheduler can get sleepy and the whole core can go to sleep and this will happen so you get this compaction across your cores and in a server closet you'll have half your cores being able to go to sleep on if they're running early which is pretty amazing for power savings this core memory locality things are building up here he's getting hot hot this one is us more stealing take it make it even Steven balanced so at this game where we we play and we get massive concurrency pre-emptive multitasking soft real-time low latency and the low latency is valued more than raw throughput which is an unusual characteristic for a language scheduling parallelization and fault tolerance an ode because you pretty much left to your own devices here I mean things you know you're gonna be you know you're gonna build build your own build your own operating system every project or use something out there that someone else's had to do so on node we have cooperative multitasking and on elixir we have pre-emptive multitasking so cooperative multitasking it sounds good right but what it means is you have to cooperate and if you don't you can wreck the entire scheduling system on elixir you cannot do that single threaded event loop single-thread event loop cooperative multitasking it's like they read my mind what we do when we work you know we get into this trouble with callbacks and I know that this is one that you've heard a lot about and there are a thousand blog posts out there saying there's no such thing as callback hell that's a myth and all this but a lot of people are talking about it so it's it's some people experience it but you there's discipline you can follow all these paths you can go down this route and you can solve your callback hell problem and have flat code by hoisting things around and flattening things out and you know on and on and on all these patterns if you're a diligent person there's a funny use of words and node but we'll talk about that in a second but so no jest is not aa mcli manage the number of workers for you however it's your responsibility to manage the worker pool for your application needs so know it's easy that doesn't sound easy distribution tends to be very unbalanced due to operating system scheduler vagrancy vagaries loads have been observed where over 70% of all connections ended up on just two processes out of a total of eight well that would suppose to be easy that doesn't sound easy to me okay here let's talk about elixir and simplified distribution and I think I actually do have time for this demo which is really awesome and surprising okay so here we have just a simple loop called blabber and I'm going to move over here and then use I've got three consoles a terminal window to set up and I'm going to start up an ie X and I'm going to pass it s named cat with a cookie of taco here I'm going to start up a IX with the name of dog with a cookie of taco and here I'm going to do bird with a cookie of taco so I've got these three named virtual machines these three nodes brought up here okay and if I say node list I don't have any friends yet so say node connect and I'm the cat so I want to connect to the bird and it says true so if I say node list again I see this the bird he knows that something's happened to and he's like oh hi cat knows about me so I say node not connect and I want to bring the dog into the mix because the dog always you know saves the cat right it saves the saves the bird from the cat so here I go back over here do my node list Oh so we all know about each other at this point so we have a full mesh between these nodes and that's all it took to do this this can happen across multiple boxes across heterogeneous hardware so 64-bit windows 32-bit Linux a raspberry pi on ARM architecture and we can have code this cluster formed and we can deploy code across these so the decoder I'm interested in deploying here is this blabber server so we're going to spawn this process start we're going to call start which is going to spawn this process based on this server loop here this is a really short piece of code here we're going to respond to receive block we're going to wait for someone to send us a stop message if we don't after 200 milliseconds we're going to say nice we've got some uptime here and then we're going to tail call loop into our self passing plus one so that's our logic so let me LS make sure that that code is there I'll say you compile blabber just a barber blabber just to make sure that I've got a fresh good one here okay and I'm going to say I'm going to store off a pig equal blabber start okay so I've got my process ID that was returned from that and I'm here I'm seeing nice zero years of bug free up time on cat at Daedalus okay I'll go over here that's like I'd kind of like to do that too so I say pit equal blabber start da I don't know anything about that because I'm not over here on bird or it wouldn't exist there either but what I can do here is I can say NL and I can say NL blabber and so what happened then is I just deployed my code to all my connected nodes so over here on dog I can now say pit blabber start and it works and I can go back over here and I can say I want to spawn this process on another node let's say pit 2 is equal to node dot spawn and I'm going to spawn that on the bird node and I'm going to spawn it bypassing the node name I pass in the module name and I'm going to pass in the function name to call and I'm going to pass in the arguments which is an empty list right and so now this process is running over on bird we've got the one running on here but we're actually getting the IO piped to us it's running over on this box if we kill this thing we would no longer be getting this message on the left saying six years of bug free up time okay so that's pretty cool I think so here let's fix some code we've got a bug in here don't we where's our bug here we're uh we don't have enough gust oh it's our problem our bug here on this line is we need more gust oh we need an exclamation point here okay so let's go over here we fixed our bug we're going to ship this code and so we're going to first recompile it and so we get this our next tale call into this our bug is hot code deployed on our running server and I say NL to deploy this code to my other servers and notice what's happening there over on our other cluster we are on another node we have that picked up and it deployed over here to this one as well because in its output is being piped back to us so this is pretty crazy awesome is what this is and and it's a sort of examples I can show about this is a basic way of showing how you the foundations of how the real stuff works okay so we've got to now hurry because I took the luxury of a sweet sweet demo okay so let's talk just briefly about OTP so this is where the safety and fault-tolerance story really comes in and it's built up on these basic things that we all get by using they're laying vm by using elixir we get applications we get supervisors and we get gen server's gen servers is where we do the work and these all play with this nice system and this is proven at Ericsson over this decades on these systems that have nine nines of uptime so we're talking milliseconds a year and so you know common sort of thing if you know everything has to be carrier grade for it to be Telecom which is five nines but you actually beat that with Erlang because of these amazing bits here so application supervisor gin server this is sort of looking at the tree of it so an application spins up and brings up all of the things that need to run in the context of that application so you'll start up you'll have super vision where that super vision is then over here watching you know our gen server right and if that gen server hits some weirdness and dies it'll restart if this gen server down here causes some trouble or any of its peers we've got a bunch of gen servers being supervised by this supervisor if any of those croak this supervisor is going to restart them if it happened so often we meet some threshold the supervisor here could actually be killed and if that happens this supervisor is going to bring that back up and so you have these whole trees of these things depending on the what you expect to happen in the code and how nasty you see the problems being so in elixir we code the happy path we don't have a bunch of try catchy stuff worrying about like oh what's going to happen because you never can really predict it you can predict something but you should maybe just coat it differently but what's easier is to code the happy path things go weird just let it crash let the process crash and leave that problem of handling the error to supervision on this side different story unhandled exception means your application by extension node.js itself is an undefined state blindly resuming means anything could happen and this is right off of node.js org and this terrifies me the safe way to code for massive concurrency is to code sequentially the easiest code in the world is code that goes top to bottom that you can just read instead of callbacks are not really easy to grok so you actually code top to bottom in elixir code you've got massive concurrency your coding top to bottom it's not your job to think about all of the plumbing of how the concurrency works so think about this you've got a hundred thousand connected users they're making all these callbacks you're running on the single threaded event loop on this single process and you don't really have a supervision model so one of those dies so you've created this attractive nuisance here no node is this attractive nuisance where people come in and they can just break their ass you know you can really get hurt on this thing and we we have all sorts of tools that we've been talking about so far in elixir where you you get out of this you know pit of failure and you end up in a place where it's just actually easier to build systems at work it's easier to do your day-to-day coding one of those things is being able to code with a powerful repple the shell there so when I'm building my elixir code I try it out in IX I can do all my things I can see at work and when it works I push that up and put it into tests to protect me against regressions so this is like TDD the part where you actually change your interface by the way you call it you're getting that costly inside of the repple in it and then you can then voice that code out to protect protect you against regressions so this idea of testing is big because it's big in the Ruby community it's made very easy here in elixir made easy by having good docks and then dock tests like we saw earlier website called the node way talks about you know this is protecting you from trouble kind of website it's a guidance website no jess has always been easy to grasp but difficult to master their common fit falls never been well-documented it's up to you through painful trial and error you know on and on and on and it's like oh my goodness you know this is the thing this is the tool that's supposed to be easy this is what was going to make us be able to write code more simply and deploy more simply and it's just not what happens because we are required constant diligence there and we shouldn't do this to ourselves we shouldn't do that we shouldn't allow tools and frameworks and languages to put us in that spot where we're constantly having to be on guard because we're fallible we're human so at this spot we have now traveled through our hero's journey here where you've now had this introduction to this thing and it's kind of up to you to go off and and pick it up and do awesome things with it and I think you're going to be stunned about what you can pull off and how it's not just easy at the beginning it continues to be easy it's easy as it can be should be I guess and you know so hats off really for josée over here for this thing he's built it's absolutely stunning and so at the end of this I love this on the hero's journey that they actually end with this line it's returned with the elixir so hopefully you all will go and return with the elixir my to closing bits here our elixir it's a you know pure sustainable joy is I think what you get on a lick sir and you get something maybe a little bit different over here you know it's popular it's crazy and it's dangerous and and I'm going back to this country here at the end of the week and so that's the world I live in so so thank you all for being here we have this over here so please vote and tweet and and go off and install elixir and have fun with it and then also come right back in here in a minute for our next session with yan cui and i saw an f-sharp in the real world and so FP track will be in this room all week other than the workshops which is the hour after lunch and for those would be in workshop I think it's room 10 but it's workshops in its where you can just hang out with all the speakers in FP track and have them walk you through things and so we'll have josée we'll have Joe Armstrong we'll have Robert Verdi will have Mattias Brenda vendor who ran the f-sharp machine-learning workshop with Evelina and so awesome bunch in there I hope you join us for that and other than that actually I cannot believe it I have a minute and a half for questions is anyone yes what would switch I think so I think it has that sweet spot it I mean it definitely has a sweet spot of standing up a website quickly that can handle lots and lots of things but then you know it's like you've got to then have someone actually do the work and yep they would there I mean that would be it's easy to handle a hundred thousand users on the Erlang VM this is a this is a sweet spot of the language that fault tolerance concurrency distribution that's what it was built for so back in the 80s that was what it was built for a hundred thousand phone connections coming in and people expect their phone they did expect it before mobile phones with apps you expected your phone to work and so you had fault tolerance so I remember as a kid when a squirrel would get into the transformer on the power lines and go and in the house all the power will go out we would pick up her phone and we call the power company and that just tells you about that sort of fault tolerant story and it's real there and it's real because of these tools and so again that Erlang VM is why that's happening like over half of the world all of the traffic for mobile is going through this so massive concurrency easy to do I mean it's just the sweet spot as a language a place where I would use node 4 is my think maybe what you're getting at place where I would use node is here where I am using node this is Adam I think JavaScript is a great language for the client you know it I mean JavaScript obviously works well in a browser and you know it's it's a very good thing to have on the client it's ubiquitous it's out there people know it you know the problems with it and you don't have a hundred thousand people that are getting it hurt if my atom crashes and so that that idea of atom and electron there talks here this week on electron I believe and so you it's a good thing to have it there it's really nice the tooling they built up but I wouldn't want to use it on something where I had to count on fault tolerance and I wouldn't want to have my company building up on it for business logic and I know companies that have done that they got lured in their business logic everything they ported from c-sharp because of problems that they saw there they didn't like what the experience there they moved over to node and now they're trapped you know they've they're they feel like they're in deep they've started bringing in micro services which is basically the the plea for help so like when node shops start talking about microservices what they mean is like we've screwed up we're going to have this tiniest bit of code part so and it's like maybe this will get us out of this trouble we're in well you think about what functions are in processes with a message passing that's that's micro-services that's how you want to handle that problem and so any other questions yes yeah well this is a really interesting one so you'll never ever hear on the Erlang VM anyone talking about monads but you have something that so what is I uh monad 4 so we we know in Haskell if we write in Haskell that we want to have all of our code be sane and we want to build a reason about the world right but we still have to we have to have side effects right if your code doesn't have if it can't write to the monitor if it can't write to the hard drive it can't write to the network card you're not actually going to get anything done so you wrap that then inside of the i/o monad well on elixir in Erlang the equivalent of that would be port drivers so all of my elixir code my Erlang code it can only talk to other processes through message passing right can't reach out to the operating system can't poke through do anything it's all in that safe safe world so the way that the the ugly scary world is hooked in is through port drivers those are written in C and they meet a certain interface that makes them look to the Erlang VM as if they are Erlang or elixir it you know they look they look native on that side but you got the scary scary world there and the only thing that then can touch the outside world is a port driver which is a process which can be supervised and so and you don't go off riding port drivers all the time because you know there's the file system there's a network card they're these basic number of things that you want to talk to on the outside world and the way you do that is through port drivers but I love that question I forgot to actually mention it earlier so thank you yes so I'm always surprised it it you know people talk well our lengths not a great language for doing machine learning that sort of thing well it turns out is actually is pretty good at machine learning and here's an example of so you see the F sharp digit recognizer project so the thing it's not good at so I'll say that you can actually do this and I explored this path of using it for this thing it's not good at is it doesn't have a static type system so there are going to be errors that you don't catch at compile time this is a place where F sharp absolutely just beats the hell out of us over on a lick sir because static type system you catch all these problems it depends on the kind of code you're deploying and why you're deploying it so if you're going to a mobile phone for example where you have to go through the Apple Store for a week review and all this and you can't change your code and and you can't get users to install the new one you want to catch every problem you can at compile time so that's why I really am excited you know like Sam read which I have my Sam are insured under here you know you have this F sharp support and there's also a talk on F sharp and mobile coming up a little bit later today but that's a place where I would use F sharp for sure over you know you don't even have the option of using a lick sir on a mobile phone but I would use it for those places now on the server it's actually a lot easier you think about this we did this hot code deployment it's easier to deploy that new elixir code to a server than it would be if you had a static type system with dll's and all that you know I mean that was just crazy easy we changed the code deployed across this cluster and everything worked and so it comes down to what your what your deployment story is so I think in the short run like if an app has to work really really well immediately and it's going to be really hard to update you know that's a static type system problem if it's something you know this other side you have to have a system that runs and runs and runs for years with no downtime the static type system protects you in that first hour you get your maximum then it's kind of like term life insurance that sort of thing you know there's different thresholds when the most valuable static type systems are helping you the most right when you deploy and that first a little bit is live something like elixir Erlang they're helping you if you need that thing to be running for years with no downtime okay and I think I am now out of time so Yan will be in here so don't get too far away because John will talk about F sharp in the real world and he is awesome he's an amazing presenter so and thank you all for being here this is awesome

Channel: NDC Conferences

Views: 21,568

Rating: 4.860465 out of 5

Keywords: NDC, NDC Oslo, Elixir, Node.js, JavaScript, Funktional Programming

Id: q8wueg2hswA

Length: 65min 40sec (3940 seconds)

Published: Thu Jun 16 2016