Billions of Events Per Day with Elixir - Danni Friedland

ah okay awesome so I will give you also a quick introduction about myself the aim of the talk I was kinda in between sorry beard hair in the mouth anyway so I was I was kind of in-between showing you guys some code but I decided that you got or it is the chance to see the code but I would rather talk about some of the concepts behind OTP elixir and the way we use them to scale our infrastructure applications etc etc and why I find it so exciting overall a bit about myself I will run through that quickly I am currently the CEO of Jack analytics where we basically record everything the user is doing within the application without any choice we just collect every possible event from the user obviously it leads to some cool technical challenges been developing for the past 17 years started off with during the happy days of SP if you guys never seen this exception then you're missing a lot and was really happy days where the only possible way to debug things was using these kind of exceptions moved away to start using as bit of net sell for the first time that I'm this kind of real developer doing real programming work except that it always felt like I was trying to achieve something extremely easy in not always the most correct way right so join the startup got introduced to the old concept of Ruby anyone here programming in Ruby awesome so got really excited and for the first time I felt like there can be something that is beautiful to write and is also easy to maintain and you feel like you're building something real and not just a script kitty just playing with some made code but the problem with that is that it is extremely slow which usually is not a problem but sometimes it is during the time all that hype or anti hype of nodejs began and I jumped on the train in my search between basically the ability to write large application with the ease of use and what's really excited about the whole concept of using JavaScript on the client side on the server side was promising to be really fast and I was for the first time really really excited and happy I felt like I found the correct balance between ease-of-use and just writing real applications so a bit about Jaco what we do is as I said before we collect millions and billions of events we are taking the approach if you don't need to define anything and that leads to the basically us collecting all possible user events or user moves the mouse we collect that your clicked on something we collect that you receive something on the screen that's an event and today as a small startup scale we are looking at about billions of events per day which is quite a lot if you're trying to butcher something how we got started and that's where it's starting to get interesting we start off with this single nodejs app and the reason for that it was it's extremely easy to start and usually that's what you will do you will start with a single app you define film API and you are ready to go the problem with that is well you almost immediately when you have some real performance and real grill traffic coming to your application you hate some prefer performing dialog button lengths and the main if you if you look at performance the easiest way to overcome performance is scale so we started scaling and we started scaling that this huge monoliths application the problem with that is that not all endpoint in your application we're experiencing the same traffic so we had this huge event collector handling and thousands of events for a minute and we had this you know user trying to see something on screen which happened like few times the minute an hour or so but because we have the huge one endpoint with the huge traffic spike everything again everything else came in afterward so and the real problem with scalability and we just scaled horizontally everything and the problem with that is that we had this usage problem and again we need to throw a lot of money into scaling this kind of infrastructure so we started we decided we need to split up this monolith and with this there is lots of talks here on the tech world today about nikkor services and things like that it's all the rage I remember it was all the rage back then with soft services and things like that so if we're kind of completed a full circle here but we're excited to split your application into the small micro services and we immediately hit the huge technical challenges due to the fact that when you're a building and whether it's Ruby whether it's no jet or any other framework you have when you are deciding to later on place your application you have almost zero tools to do that you have to really understand and dig into the code to try to split the situation into multiple components and the reason for that because when you're building single application everything will touch everything and unless you are this super well organized programmer trying to think about the architecture things and either on you will need to split things you will end up with this super mangled code where everything touches everything and splitting components out of that requires a lot of really gentle work also move the docker that's another lecture at all but it was extremely fun and and we got the first taste of ear lang and that's how we got introduced into the whole country like we just introduced revenue at the message queue within our architecture and up to this point that basically led us to explore elixir in Erlang and after this point it's still one of our like steadiest components in the system and we felt like why isn't our system as steady as this thing and how can we build application that will be as scalable and just work and we'll manage all this traffic so with other cells can we do better and this whole concept of nipper services is so fun and it allows us allowed us for the first time to experiment with different architectures and different languages for each component and we thought what within it actually create this robustness and this ability to scale so let's break down feel the key concept of this thing called mirror services and I promise to you that it all will connect to elixir and OTP later on but for me it's like the breaking the idea of mitosis and LTP just connect so beautifully together as one thing so where are we cursors is basically small services that do one thing and you know when you're starting it's like the mantra of programming just build small things build things that do one thing is set for et cetera everybody like to talk about that nobody is able to actually maintain that because applications will grow and you will have more requirements and your both will tell you yeah okay so this I don't know if you the funniest thing I read is that if you do like the main pages of LS or cat and things like that you will see that it does like millions of other things it's a full-blown application and that's like you can take that to any application that you build requirements will amount you will build more stuff into that and it will grow it will grow it will grow and then you will hit the same point where you need to split things up each service should be self-contained hopefully with its own well its own states that given but it awfully would its own database and everything should be self-contained within each maker service so no shirt stays between services and the communication between services should be via some protocol or message passing or something like that hopefully you're starting to see the connection if you'll ever read about to be the concept beginning to be kind of similar right it's all coming together each service should have some sort of work you you don't want to be basically if a service can handle one something while you need to work per second and for one second you get two units of work you don't want to crash but you want the ability to handle traffic spikes and things like that and that for like the larger architectural rather than to fit into the picture and the last piece of the puzzle is each service should be able to restart on its own and monitor itself and has this resilience to failure because services whatever whatever application that you are building services will fail there so the two like truth of software things will crash and things will grow and you have to deal with that and the only way I think up until this point people learn how to deal with failure is just to start over either it's it's a life model and a programming model cool so we are at an elixir conference so let's talk let's talk and let's try to connect all this beautiful concept into OTT and the goal of this lecture is not to show you how exactly oh TP works but more of the things you can take through your co-workers and your bosses and your teammates and to basically explain to them why is this such a cool concept and why you should build everything with it except your creeps or something like that but other than that every large thing I think should be built using these concepts whatever it's given elixir but you should keep that in mind so if you see this about elixir its language it compiles to the bean BM which is kind of like the Java VM same thing as your line and the cool thing about that because they're both compiled to this lower level language you can use them with one another without any performance cost because they're all like languages and doubles smaller subsets of language it's functional which means that you cannot change state and it's awesome and it's kind of scary for me what's one of the things that was really hard to convince my co-workers to start using that and I will touch why it's such an awesome connection to web application especially and highly concurrent we will talk exactly about that it has better matching which is awesome and unfortunately maybe I will show some slides about that but maybe not it gives you the ability to do meta programming but all of you are from familiar with Ruby here it's not the Ruby meta programming where everything can change on the fly and you have this awesome power of changing everything and also the ability to shoot yourself in the foot while doing so it's more of a static meta programming where you can change things while they're compiling and adding some capability on top of that it has polymorphism and therefore a code and it has the extra model which is given through OTT and it has sure nothing concurrency which basically boils down to the fact that you cannot share state between two different components or services or processes in putting your application which is also awesome the other part of the equation is OTP and OTP is a huge base that I remember starting grading off and went to the Wikipedia page and the first thing you see is that it's open to communication protocol or something like that and you need we get a big scare I would I would focus on the lecture here on three main things I feel composed OTP and the one thing you need to know is that OTP and elixir are one thing unless you are building this super simplistic application you will need the concepts of OTP to actually build something real that communicates with the world so it's not two different things that you can choose to use just one thing that just work together and it's also kind of for me it was a mind-opening idea where a language is also so intertangled with framework which I think up until now for me it's the only language that does just that so let's learn a couple of concepts of OTP basically for me the three main concepts that is important for anybody coming to OTP are basically this you have your application which basically if you if you're coming from the node.js role it's kind of like your package JSON plus some convention of putting things in index digest and things like that but it's so much more it's a well-defined concept I think well the best the best thing I can think about it's like main let's see and things like that it's just the core point where your application starts and you define how the surger application what the dependencies are and everything regarding your application at all so very cool and a great concept you have a supervisor which I will talk more later on but the key concept here you have this small unit of work you actually it actually does nothing except it you attach processes to it and all it does is whatever the process fails or crashed just restore the process so no logic it's well battle-tested OTP concept Oly does is just listen to stuff whatever something crashed you had you can say wherever something crashed just restarted or you have very strategies on how to restart things and the service server process I'm using them interchangeably to convey a point but basically in elixir its process you it's wrapped up engine server things like that but basically it's a process that hold states and actually does some work that's usually where whatever you need to do some actual work that what you will use eventually and those three concepts work together pretty well and the one thing to note here if you are really new to elixir is the when I'm using the word process it's not an operation system process probably heard of that it's basically like green thread you can launch millions of them tens of millions of them on your regular PC without any performance overhead I'm really permeable also very interested concept I really welcome you guys to see some performance metrics of people launching millions of process on their machine it within few milliseconds and it just created to see that so just to show you it's tiny bit of code it's less the point of the lecture but so we are all well aligned is this is taken directly from the elixir manual but let's go line by line but about what it does this is the most easiest way to start a single process this is the model and elixir I'm defining two functions here the start link function creates a process and links to their current process certain links and execute the other function called loop and passes it and an empty map so if I call the third link I get a tuple in return which gives me the process ID and and okay meaning that it successfully started the process what I'm trying to achieve here is a basic key value stores on defining two methods on the store we have a guess and we have a food so once we started the process we immediately executed the loop and we immediately blocked on their receive call the received call all it does is just each process in the lick sir has a mailbox which also connects to what I said initially that each process has own mailbox and it's a concept of like separating between the work load and the actual work so receive all the does is just go to the mailbox and says give me a message whenever you have a message currently when we started we have zero messages so we just blocks so we stand this the first message we call sand which is just a built-in command we call send we give it a PID and we give it to pool what happens is that the OTP puts a message on the process mailbox the process then the received gets the message and it GU does pattern matching on the message value which for this case is this tuple the first thing is boots and the second Clause matches what happened next is that we do we call master food which essentially just put some value in the map and we call loop again so the really cool thing here is that we're doing nothing we don't mutate State ever we just do an infinite recursion where we pass the state on and on and there is actually zero magic here and it just works the thing to note here is the beam of subverts tail recursion optimization so you don't need to worry about if you come from JavaScript or like me you don't need to worry about instance recursions and things like that it's cool to do that and the next thing that we do is we send another message we send again we do call send PID we do but this time we'll call again and what get does it matches this first well and it's called the PID that I passed which is self which is my PID the IEX PID the process of the i-x we send him a message with the value that within the map any questions wearing that all clear okay cool and we call loop again so this infinite recursion keeps on and on and that the way we process messages the way to receive messages from is that just a small picnic allottee just called flush it flushes the mailbox it prints on the screen whatever you have currently on the mailbox which is the world keyword value that we put before in elixir and in OTP in general but an elixir we have this very nice syntax in a listserv we have the concept of general server and general server and gen server is basically a syntactic sugar on top of what I showed you before it all boils down to the same loop with some conventions and that that's basically it the syntax is extremely simple what you usually do is that you separate your server into two parts you define your client set API here which is not a really important again it's a syntactic sugar and cup of that and you have a few methods that are built in within the gem server the main difference between the two a method call basically expects you to handle value and return something while cast is like an UDP you just send something and it might work it might not work that it might never get value at all so if you look at that the key thing that the really cool thing about that is that you are just building the API so this is the smallest service possible with just a simple stack service but your while you're building that you are thinking about a client-server API even within those two methods you are thinking do I need to handle the response do I care about availability of the service etc etc so you are building this application from a very very different mindset you are thinking about servers who are thinking about services and you are not just thinking about models and lips and just business code so how it all works together basically and as I said before the application is the main starting point of your application and what it usually does it starts a supervisor and a supervisor can start more application or it starts more actual services servers processes and that the bottom of the list basically you are creating this huge tree of Supervisors services etc etc where the list of the tree is always the services what's so awesome about that actually before that let's take a look let's take a deeper look into how it looks actually so you have the supervisor and you have it services and each service has its own mailbox as I showed you before and those services can just start communicating between one another and you have few very cool concept between OTP to actually store the process a-b etc etc but essentially it's the same and if I were to show you a network diagram of large-scale application framework sorry no tremor but large-scale application it will essentially look kind of the same you will have those processes or micro services communicating with one another but what are showing you here is just a normal elixir OTP application so the cool thing here to know regarding this architecture is that whenever your service fails and it will fail as any other application nothing in your application should fail except from this small service so it's it's such a huge concept because if you are I don't know any any of you like to remember the good old days of Windows 95 whenever small process failed or worse stock their entire application the entire system operation system are just completely stuck and that due to the same ideas not implemented each process were interconnected to any other process and just whatever something failed everything failed and that's true for I don't know Ruby on Rails no J's application anything whatever something within your application cells no matter how my news and in important of these everything fails and it's always worried in the case of building your application within the supervisor tree and the OTP method what you get is this ability to automatically be resilient to fader so whatever something fails it just get restarted and you are ready to roll and you're like your entire huge application this is like the most simplistic thing you have on the three sources but usually you have this huge tree and whatever something some small leaf within the tree fails your application is still up and I think I've read like the coconut telecommunication industry boast about 32 milliseconds of downtime per year which is just crazy if you ever build a software that has 32 millisecond or year for me this was just understanding this concept and grasping my mind that you have the same concept of building large scale applications and boiling down to your most basic applications or smallest part of your application is the same is the mind opening ID behind OTP and the cool thing about that is that whenever you have you are now building your application with those small services so the cool thing about that is that you can now just you have this service that you see that it needs to scale for example or you need to I don't know and rewrite it in some other language you need to for some reason you need to build it in C because it's need to handle all this that traffic and all this performance I don't know what people build stuff and C I don't really don't know but they do but because you build your application in this way you can easily decouple this component from your application graduate it to a full-blown application put some communication mechanism between them whether it's messaging XML I don't know HTTP or whatever and you are ready to roll essentially you're building your application to scale from the smallest component to the largest component possible and it goes on and on like this so we have reached the full circle essentially we have started off from talking about how we scale this monolith application to start using meter services and then within those maker services we're using meter services to build the maquette services so it's Turtles all the way down we have small services that do one thing the only way in elixir to hold States is to build those small services that do one thing you have to constantly think about the API of your services each service is self-contained true by definition you have no way to share state between components you can overcome that if you are whatever you introduce the database but that's a whole different thing each service has its own work use so it's resilient to actual performance spikes and things like that awesome you get that built in and each service can be restarted on its own so each service is managing its own life cycle and you know defeats from the outside but essentially the same so why is it so great as I said before you're building for scalability without sacrificing your and I would say coding economics your developer ergonomics it's not like you are now building using all this crazy concept it's still awesomely simple you're building using very cool tool set of tools and you're building to scale so for me I think I think that even now when I'm building I'm still using node.js and still using Ruby from time to time I'm constantly thinking about those concepts I think if you can take one thing about from this lecture is just this way of thinking is just it's just great because we keep reinventing the wheel in terms of nikkor services so applications etc etc but the core concept is essentially just the couple things and deal them in a way that they can scale and they're self-contained and whenever you have a framework that forces you to do that it's even better so I couldn't it's all enthusiasm about elixir I just want to mention a few things that I found missing as dialog opening in the community basically the three things that we found missing in the elixir ecosystem right now are the the ecosystem is not there quite yet if you're coming from Ruby or you're coming from nodejs you will find lots of projects with like in get up one or five stars and things like that so it's really hard to know whether a project is active and things like that elixir is doing an awesome job in that regard but it's still I think it's up to us to actually try and contribute to the community and build stuff and you know start start stuff in github to get people more excited and show that the community is alive for me I for me in the node.js days any project without one get up star I usually don't even consider implementing and I look for something more alive so it's we need we need to get the community going in this regard and just a general question if any of you has a chance to build with docker and elixir I'm happy to learn how you did that we I think the sense ODP was built before this concept of docker they aren't really working as well together as they should be and some of the concepts some of the great concept supposed to be like host called reloading and things like that just I haven't heard of somebody being able to implement them using docker and the whole approach of docker to building application so if any of you had the opportunity of doing that I'm here and basically height I think the the whole point of this lecture is get you guys height up go to your teammates and just tell them how awesome this thing is and why you should build almost anything with it as long as it's not I don't know I heard there a few talks about client-side programming for using ALM and stuff like that but except client-side and except probably scripts I would say that it's an awesome experience altogether in summary we're happy if you couldn't tell and there is a tool for each problem and tools do evolve but I think as I said before it's an awesome tool and all tons of stuff that I haven't covered and I'm living room for questions to discuss just that thank you [Applause] the OTP I think definitely I think that the for me what I've learned from my experience developing is the one true rule of software is that things will grow and there is a lesson that you're building something that will just you build and never touch again which usually is not the case so whenever you're thinking about things growing essentially you will hit the point where you need to start separating things and you just need to maintain things and so I think that's whatever you're using that something that will grow it's a great way of thinking about software of decoupling things and thinking about just this mental mind thinking where your status what are your services within this state etc etc so yeah definitely and microservice that is the internal will one day be external or you know it's external to your teammate so it's kind of the same yeah definitely should I repeat the question everybody heard that yeah so the question is there is this whole talk about moving away from liquor services to umbrella implications single to Gatien and whether what we're thinking about that so I think it's kind of the same I think the only option to do that correctly is with same OTP because OTP provides you with this all concept of single application composed of small services but you are still in the realm of small services and if you are at a point where you can be completely immersed with elixir Erlang and OTP then great by all means you can do that and you have the toolset to do that within the LTP you have all this kind of service disturbed discoverability you have the way to communicate between nodes etc etc all the things that I haven't touched but what you what I like and it's my own personal opinion is to experiment and I like the fact that if you're building your application with small components you can do one and elixir and you can do the other one and go and you can do the other one and Ruby leeandal just whatever the team feels most comfortable to do but essentially the point is that it's still the same and whenever you need to scale something you are still talking about nikkor services in OTT it's just combined under like a single application but it's still small vacation down there yeah straight to that yeah so so the reason we chose docker and the question was whether we thought about directly deploying - you're talking about the docker container directly deploying to that so the whole the whole premise of docker is not doing that and that's my I know people are experimenting with that but the whole premise of docker is just put your container in the trash and put a new one and that's where you guarantee that you have this clean environment and you're not constantly updating so it might work it kind of goes again it kind of goes against the approaches docker teaches which is just one time thing and we never need to update something you just throw that away and build another virtual like machine container whatever so without docker so yeah so the problem with that is that we we're using AWS and for us having we have in one machine intensive services so having one machine per service is an overkill and it doesn't allow you to scale in the way that you want so we want to say like we have this consumer doing something what a boot like 100 is more of that and we don't want to actually boot machines to do that and one like five minutes later pull it down back to ten so that yeah what way something about doctor this may be helpful here Christopher Michael John Pickett John Rogers yesterday gave a talk on assisting people default load loss and while academic in nature he mentions that they're spitting up tens ten twenty four cords or whatever on kubernetes and ec2 or Langley dollars in fifteen minutes kind of interesting but it sounds like there is some work on getting the beams running on docker I don't know where it's at exactly and I don't know what's happening more on the early side versus the elixir science doing is actually different sizes goodness all the same Vienna's may not it's much of a different steps there may be people here for we're talking to about that and I would go to him particularly just to be happens in that mentioned it but maybe has other folks around also can tell you where the fact this state of that actually is failing you know state of the art and I thank you any other person one thing I was just wondering I I guess just you were talking obviously more generally as he told me you would before I've just used how we in terms of processing these billions of the then Oh on how many servers does that hate what is the architecture of like country league so briefly we're thinking about server site I don't know but we're speaking about hundreds of docker containers just running around doing work and RabbitMQ is the core processing the messages another and that basically allows us to scale almost indefinitely so whenever we have the bottleneck in one of the small services we just we have this policy of auto scaling so we run another 15 and 100 and whatever so it just works really really well so they're not connected is no and the reason for that is they basically some of them are no jet some of them are in elixir some of them are some of them are in Ruby so the only way for them to communicate is via those messages via relevant queue thank you

Channel: Erlang Solutions

Views: 3,702

Rating: 4.5333333 out of 5

Keywords:

Id: 6maj55n9dAo

Length: 41min 34sec (2494 seconds)

Published: Fri Mar 24 2017