Elixir/Erlang OTP in Microservice Architecture - Thomas Newton

our next speaker is Thomas Newton from McKinsey Thomas is going to talk about some of the patterns in micro services that are going to apply to the open telecom platform Thomas thanks very much thank you okay so thank you for having me undeniably micro services are changing the way we view the world like I love jaysis talk on how we can take these big complex problems and break them down to smaller and easier to solve problems I also love the thought of taking micro services in the mindset and scaling that across the organization and scaling teams and you're unbelievable levels but like I think a common thing today is micro services come with our own tax like working in a distributed system it's really freaking hard it's you know you have to think about designing for failure you have to think about these new architectural patterns you have to get your developers used to these things but what if what if in a industry that's well regarded for its high availability and fault tolerance they've always been working on this and they've been working on it for 20 to 30 years what can we get from that what lessons could we learn it turns out this is true and to do that let's do that let's go on a journey go back in 1998 to see what was going on oh sorry guys liner interruption geez look at oh there we go okay here we go so we're in 1998 uh-huh Monica and Bill are all over the news Brittney to introduces her first album hits us all one more time Titanic one like every Academy Award half-life was introduced when my favorite games and I had a glorious head of hair but more importantly erlang and OTP is released open source and let's take a quick look at this historic announcement could you help me play that real quick declarative programming languages have several advantages over traditional languages for example programs in such languages are considerably shorter than the equivalent programs in imperative languages here for example is a program in c and here is the equivalent program in airline all right this is an awful I just love this it's such a classic way to iterate the language like we still do this today no certain method that this is in 1998 this is actually 1990 Erlang had been in development for about 10 years prior but it was just released open source in 98 so that's us and we've got your good 20 to 30 years of history behind us exactly what is our line and I think a lot of people heard the language but it's a functional language in my most functional languages it enjoys immutable data high concurrency Headley's its pattern matching but what makes it a little unique than other languages as an Erlang everything's a process and these aren't just normal processes so you tiny tiny lightweight processes you actually write in earlier you write your programs in these processes and they stay running so it's a little different than just using concurrency the whole model is just like microservices you have these small programs that talk to each other and this be in Erlang and the type of concurrency these small processes are called actors and this is known as the actor model so with any language that's been around this year it's also got 30 years of tooling behind it and you see some of these things that we're trying to do pull all so for example like with with Erling can fire up these these consoles and these dev tools and you can see exactly how your processes are running you can even see like how they talk to each other excuse me you can look at the messages and how they flow you can find bottlenecks in the systems you can look at each process analysis how its resources are being utilized and what's even cooler about this is this is all exportable so you can actually use your laptop and connect to a live production system it's the all this flowing and real-time to help a diagnostics and troubleshooting and support but and if you look at it hourly a typical Erlang application in a modern micro service application like it's the same blueprint but in fact it looks so close it's like and in Erlang and in micro services you want to scheduler regular kubernetes or mezzos and then same thing you then use the virtual machine to schedule all of its lightweight processes they both have a registry so you can figure out where everyone is and find them and they both have a bunch of little actors running around doing their job and I love this model like you can look at this this really embodies what the same neumann a lot of other people and building micro service love you other people call you know choreography / orchestration there's no big brain in the middle just telling everyone what to do and once it's kind of working to their own dance early it's a little cooler like if one of these services crashes and falls then another actor will come along and pick them up and dust them off get them back to work right but the concepts are the same it's like ease it's basically micro services but within a programming language and these things are proven and I'd like today I guarantee someone will use Erlang right like you're making a phone call sending a text message the proven in the craziest environments in the rule world so these guys have really started figuring out fault tolerance and especially fault tolerance in a distributive system in fact what's happened it was purchased by Facebook they don't really talk about this anymore have a pre-purchase but at the time of their purchase they had four hundred and fifty million users in only 10 Erlang developers they were doing 19 billion messages in a day and so you have button to other of your great things have couch rabbit amazon simpledb also all based on early so why is going to use everywhere why don't we all just Earling developers and citizens you know it's a functional language right and my most functional languages has a barrier of entry right like the these they're immutable data structures just recursion there's these things that usually take a seasoned developer to get used to is they're sometimes you're ten years into your career before you start picking this stuff up and then the other problem is that the Erlang had kind of a bad reputation of having of obscure syntax right and so the famous quote in Erlang community is makes the hard things easy and the easy things hard just like a quick show of hands like who here knows JavaScript yeah and your form conference all what I expected and here who is proficient in Erlang yeah exactly so that that's basically a problem that's that's why it's not everywhere so then 2014 elixir can't talk about our loan if you don't talk about elixir so 2014 elector back for you know 1.0 and phoenix are released for those that don't know do they baleen like takes on this challenge it's like I want to take these 30 years of tooling that's 30 years of hard work but I'm going to make it more accessible I will developers to get used to these things and I want to make it easier to use its aspire bike a Ruby like syntax it's not a transpiler like hobby script it is its own language but you know it just makes me seem easier to work with and it's more geared instead of doing with telecom switches and messaging it's more geared towards Web API development things like that and then Joe's a and Chris McCord goes on and release Phoenix which is to help with standard web application development socket programming things like that a lot more modern look but using the same lessons from the language and this gives a whole new next generation so Pinterest right now openly blogging about how they're using elixir to help with them but contributing heavily to open source and then at licks accomplished here I saw Christopher Chris Bell have a grill give this like great discussion of how they're using elixir and fault tolerance characteristics of Erlang to solve this like really hard problem they're taking these online orders from from the lab and trying to get those orders to these point of self systems and these really fragile storage right so on when we had a telephone line right so they're using principles of fault tolerance and all these things are baked into language to help make this a much more reliable and robust system and then obviously at home attack and backyard who are contributing into it so in 2003 so that it's not enough like that's enough about where it is where it's coming from like what can we learn from them looking at what can we aspire us with our own micro service platforms and as someone across this paper by Joe Armstrong who's our star of our movie before and the co-creator of the language he wrote a piece deep thesis it's called making reliable distributed systems and the presence of software errors and it's this isn't just completely applicable to micro services today like I don't know what is really why come across is like finding the lost Dead Sea Scrolls the microservices it was like perfect right sir and its really approachable you know I do encourage reading it it's not like a really heavy hard read it is it is easy to read but he outlines the nine items of the Erlang worldview and when you read these things it sounds like standard micro service architecture it is perfect right so what I want to do is just take a minute just run through some of these things and just kind of compare and contrast where we are with our laying where we are today so I kind of call this the motivational speaker I'm a motivational poster bit of micro services like the kitten hanging on determination so if everything is the processes processes are strongly isolated this is this is really close to micro services today this is you know basically take your big problems break them up into small problems isolate on from each other I think this does kind of beg the question you know it always comes up in a micro service discussion is how big should my micro service be for me and especially after studying Erlang and playing with the language some you know I really think in this kind of conventional wisdom it's like the better that your organization is at handling lots of small moving parts and can do things like fault talks and error handling the smaller your service can be right and in Erlang that's got twenty to thirty years of experience with this so you get these super tiny services like nano services it can be just a couple lines of code but it couldn't be really good at solving this they've gotten really good at moving all these these parts around Nexus processes share no resources right so this is this one's really easy relevance isomorphic to micro-services he is like we had a comment saying of one service one database the exact same thing and other lines you have a process it's running a program that program has no memories of the state it can't share in fact that's why the designers of the language use processes because it make it actually impossible to share state the and this is a again by same thing this prevent tight coupling make it unexpected to prevent unexpected changes from happening underneath you those types of things next we have message passing is the only way for processors to interact again like if your your processes or your services if they are strongly isolated and they can't share resources the only way they can you know talk to each other it's the message passing right Erlang will use a low latency mailbox pipe pattern whereas you know microservices we obviously use they should keep you and Jason a lot but the concept is is basically the same the next one it's process creation and destruction is lightweight so you aren't showing again and some of his talks he'll go on to say that a Erling process it's so lightweight compared to an operating system process like a grain of santol Boulder and I just love this visualization as I like saying it's easy to move around it's malleable you can do lots of things with it you spread it across lots of services right a boulder is hard way I'm doing this big thing around and and this this inspiration seemed following in and micro services today like docker is a great start to all this like dr. les you package it up and you can move in and ship it it's like great for our schedulers and it's certainly a lot better than the mountainside servers that we used to deal with there's still a long way to go right the lighter we can get these things the better we'll be able to work with them the better we'll be able to use them in our micro service platforms and so when I saw the announcement that docker bought you know kernel systems like this is even is like it starts all coming together it's like this is even better with container sizes are getting smaller maybe we'll at least get down to Boulder size and not only be can we move things around a little bit better that way and then then okay okay processes have unique names and if you know the name of the process you can call it and again like this is super straightforward and Erlang and elixir whenever you create a process with your code that's running in it when they be there for your program all you have to do is it generates a pit and it's letting you know that pit you can just call it it can be on this machine it can be our machine another part of the world it doesn't matter all you need to know is the name or the pit and you're good to go and then micro-services space like I still feel like we're making a lot of traction here but we still a long way to go right we have it mainly just cuz it's so confusing like we have great products like sed console zookeeper to help us keep track of all these services and then we have other great products to help us figure out how to route the traffic between them but trying to put it all together and your own platform and this is what's great about like talking about pies and stuff like that to help make some of these decisions for us but putting it all together right now is just extremely confusing and a lot of overhead and a lot of thinking that needs to go into it and the Nexus processors do what they're supposed to do or fail and then an Erlang they'll call this programming with intent or let it crash as another common word that she's for and I've also heard let it die and microservices but you know the the idea is really focus on the logic of what you're working on right so like don't overdo it with trying to imagine every possible error case like just program with intent your code has just what it needs to do to solve the business problem and then if it can't do what it can do is it would video it crashes it just died I do what you need to do or just die and in fact in Erlang the compiler is optimized for this you don't you'll see they use pattern matching very heavily to figure this out so you'll see like do this do this and if it doesn't match the pattern the thing will just die right and then what happens when you die but you can't just let it die right and so that's the concept behind air-handling is not local so what we'll do here is when you die you make a phone call and you call your supervisor and your supervisor comes to the job he does she off field skips you started back he may restart to you he has all these strategies get a message this sent to him that has what went wrong and why and then he can use that data as a separate program to figure out what to do to get the to get the original worker back to his job right and this is the whole concept they can Prairie be played armed with elixir early he's seen or heard people call supervision trees that's basically the concept here is you'll have a supervisor who's watching workers and he gets them back to work if they happen to crash or dive it but there's a strong link between them and then the tooling example I was showing before that's how you did those visualizations and those diagrams and can watch the message flows you build this architecture in this topology of fault tolerance right and you have supervisors and completely separate those concerns and and that's it that's what I have for today so I guess anything if I give them saying a fad if they're looking at how to solve these big problems and you want to do this like look at some these languages look at languages that can support things like the actor model like let's take those patterns in the past and see how we can apply onto our microservice architecture today thank you so much for letting me talk to you today at the microphones to pick [Applause]

Channel: NearForm

Views: 5,773

Rating: 4.6559138 out of 5

Keywords: elixir, elixir otp, microservice, erlang, otp

Id: -CIMUwX1OZY

Length: 17min 14sec (1034 seconds)

Published: Tue Apr 11 2017