Dave Thomas: Keynote

is there be having a good time [Applause] excellent well I'm about to ruin that because well first of all I know everybody now is a snowflake so I'm gonna have to be really really careful in the old days as a speaker it's fun because he could play with your audiences and upset them and it was perfectly legal but nowadays you have to warn ahead of time what you're gonna do so this is what's gonna happen in this talk in this talk I am gonna make you depressed I am gonna tell you the stuff you're doing is wrong I am gonna make you so sad that there'll be people crying alright you may have to hold hands if you see your neighbor reaching for something sharp please firmly stop them because it is gonna be really really bad but don't worry because then I will make you all happy again and by the end of this look at this we started with a unicorn and we end with a unicorn and that's just the way you were supposed to be so that's what we're gonna do I'm going to tell you that the way you're writing elixir code right now is wrong and the way the core team is writing elixir code is wrong everybody is writing elixir wrong I mean then I'm gonna show you how I think you should write it correctly now just to start with the depression I'm going to use the world's most overused quotation right those that cannot remember the past are doomed to repeat it I have been guilty of using this many times on slides because it's an important one I hate the fact that people don't know their own history they don't know the past they don't know what other people have done in the past in computing and therefore we keep reinventing mistakes so that's a really big deal honor your history however sometimes history is just a drag history holds you back history is kinda like a dead weight that you're carrying around with everything that you do so in the real world there's a balance to be made and the doubt balance is between honoring the past doing what we should do because people have seen it works in the past and creating a brand new future and sometimes to do that you have to ignore the common wisdom and just go for it right now I think in a little community we're doing way too much of the honoring the past and we need to fix that partly this is due to religious heritage and we're going to look at three areas erlang Ruby and rails somewhat contentious but let's look at the overall problem I think that there is one major problem that we're facing in just about every elixir application out there the 18-wheel truck that's holding us back and that's coupling pure and simple it's this design issue called coupling and the reason that this is significant is that coupling is the thing that prevents us from making changes to our code the more things are coupled the harder they are to change I mean you can think about a physical process imagine you know if you have a bridge structure you have all of these triangles why because the triangles couple all the points together and make it rigid you can't flex it once the same is true in code the more things are interdependent the harder it is to change one of them because those dependencies ripple through the entire application resisting change is the enemy it's what we fight against every single day because we write programs in a changing world and if the world is changing around us and our code cannot change then our code is immediately out of date and not just one draft we've written it as we're writing it requirements are going to change so the only way to be agile is to have code that's amenable to change in fact I only have one principle design well design code is easier to change than batter design code that's all there is to it whenever you make a decision say is this doing it this way going to be easier to change or harder to change and if you just keep asking yourself that question then you'll be doing good design you don't need all these fancy you know patterns all that kind of stuff just think about this every time you do something so our mission is to create well-structured and highly decoupled solutions because if we do that I very much hope it's gonna lead to code that is easier to change deploy maintain and reuse and after all that's what we all will want so let's start on the real serious downward slope Erlang Erlang first came about as you all know in the mid 80s as a response to a requirement somebody had to write software that drives this not particular one this is Plessy and they were talking about erickson but I mean the same concept telephone switches exchanges and they had some incredible constraints they had to write software that was incredibly reliable highly available software software that was by its very nature incredibly asynchronous that was doing thousands of things in parallel at the same time each of those littles blades in there I don't know how many processors are in there but each one of them is doing its own thing switching calls or whatever else right it's a really tough environment and that is the environment that Erlang was invented for now let's think about that environment a little bit when you're writing code in that environment you're writing code in an incredibly well understood highly engineered environment you can bet that every piece of hardware has gone through you know n number of design reviews it is SPECT up the but you know exactly how everything is going to work your functionality although it's big is actually largely replicated yes you're dealing with 10,000 calls a hundred thousand calls but each call is roughly like the other calls all the change that you do is manage change which is great and the entire environment is incredibly high ceremony so that is the task that Erlang has been optimized for and it does an unbelievable job everybody goes around to about nine nines reliability which of course is totally and absolutely meaningless as a number but it's taken to mean it's quite reliable compared to most software unfortunately times have changed actually probably a good thing that times have changed since the mid 80s he's making popcorn in his pants however that has not stopped us from copying stuff from Erlang so Erlang good then some parts of it oh no that's not true a lot of it's still good now but we just cargo cult everything because we're told it we have to write everything is great in the airline world we have to copy everything so let's think what we copy number one we copy the idea of application so let me ask you a question what is an application and this is gonna sound like a nitpicking I'm not it'll get into it a bit later on as to why but what is an application is it the thing we create with mixed new well it thinks it is because every mixed new project has a mix of exs and in there is a little def application so obviously there must be an application in there or maybe is it the thing that we build a release for which needn't be the same or is it the thing we run again which needn't be the same thing and the simple answer is sure yeah it means nothing we talk about applications without really knowing what we mean and that is a really big issue because when we name things badly we make bad decisions we use those names to guide our decision-making and the names are wrong they're one of the big problems I've got is you cannot tell from looking at the source code what's going on it is not possible to look at elixir source code and say yeah this is what I run the best you can do is some kind of weird dependency analysis and try and find the thing that has the less things to pick the least things depending on it and that's crazy something else we borrow from Erlang configuration so let's imagine we have a some kind of thing here that makes payments right so it's got access to bank account and it obviously has a super secret password and we stick that into the config and that config is part of that application it's all bundled up all very nicely alright now we come to use this so we have two applications that use this and they both declare our payment thing as a dependency so now where is the configuration I will take answers from the audience all right the configuration is down here not there and so now what we're having to do is duplicate our super secret password down into two apps that don't need it they don't care about it but they need to provide it up to this parasitic thing here that they're using now can we fix that yeah we can fix it we can make payments into a service buy a service I mean here a totally independently running entity right it'll run on its own node it'll be started separately from all the rest of our code and if we do that then it uses its own configuration and life is wonderful so maybe because sometimes we may need to configure this with things like limits or whatever else that come from the applications and we either have an all or nothing we either have this is a configuration or that's the configuration but even if we do want to go the service route route how do we do that the default in elixir the way we are encouraged in Erlang and elixir to write stuff is to write one big application and bundle everything together and it actually can be quite hard to come up with service type architectures that easily deploy and update because the tooling doesn't assume that that's what we're doing right other bad things that we inherit from Erlang gen server yep I said it here is okay first of all in Erlang the standard way I edited Joe Earl at Joe Armstrong's original Erlang book and at one point he had this line of how do you create a Gen server well you're using Emacs obviously and Ewing you know you include the template this way so I went and I include the template 127 lines of code is a gen server that does nothing right now any licks if we don't have that problem necessarily we start to write a firm out of code but not a whole bunch of code this actually may not be true anymore I think we may have to have an init : here now but anyway so here's a standard gen server and this is drawn from the elixir documentation let me ask you to identify the various components in here let's start off by saying what's the API well we're told that the API is the stuff above all the handles right so really I would say this counts as API because we're gonna call stack push stack pop that's our API so if I was a consumer this module that's probably what I'd want to be able to find very quickly second question where is the implementation of the server well kind of they're not quite aligned we have server jiggle so there's kind of spread over there as well now let me ask where is the actual implementation of our stack and it's kind of there because here's where we set the default up here's where we split it into a head and tail that's where we return the head and then here's where we update the state with the new item when we push now go find a really good book on software design if that's Joe tell him I'm not taking calls find a book on software design and you tell me what that book will call this particular style of software development I looked for a long time until I came across the answer it's called dogs dinner everything is just messed up in one place actually looks like a dog's dinner after has been eaten and then thrown back up again it is absolutely disgusting and yet if you look at the elixir source code you'll find thousand line gin servers in there where all of the implantation is messed up down in a whole bunch of hundred xxx calls down the bottom almost impossible to read but we do it because it's Jen server and that's in violet it's the root of OTP what sucks unfortunate Ruby things we copy see I'm an equal-opportunity disser project structure if you bundle gem you create something that looks like this if you do mix and you you get something that looks like this is there a similarity yes first question for you why the hell is it called Lib I'm waiting I have no idea it makes no sense at all to meet lib or lie but some people call it right no sense whatsoever and then here's the biggest question I've got what the hell is that file for in about half the projects I've seen it's empty or it still contains the hello world string in the other half it contains the entire application it's just total yeah doesn't belong there anywhere and here a little application buried down there but it's not really the application file it's actually the supervisor but don't tell anybody so you say okay but that's okay we got umbr.ella projects and i'm Brona projects that I split our project up into lots of small components and that's why the way we should be doing it so we end up with a beautiful structure like this we have a top-level app and then we have the shared dependencies up there and then we have all the various sub apps underneath it so does this solve our problems does this solve the application confusion is it a library is it an application absolutely not in fact it actually adds to it because now we have this dummy application which doesn't actually do anything and isn't actually run but actually contains all of all the interesting stuff does it configure isolate the configuration absolutely not does it allow us to reuse our sub applications not easily it actually makes it harder than if they had been separate applications so is this simpler must be simpler sorry it really doesn't help at all I don't use I've never used on brother apps that's not quite true I very rarely use on brother apps if I have to do something similar I use file dependencies because if I use file dependencies at least that way I know I got isolated components that I can merge together now the big one unfortunate things we copied from rails fortunately the answer is very simple so mm this slide took me about an hour and a half to get right so but here is rails new and Phoenix new dirt the dirt the dirt the dirt the dirt the dirt the DA rails 44 directories 59 files Phoenix 28 directories less directories 42 files getting on there from this we can conclude that rails is 70% sorry we are 70% as good as rails it's ridiculous here is our Phoenix project structure and recently 1.3 we split everything out right so now we have all of our quote contexts our business logic out in a separate place that must be better right sure because what we've done is we've actually increased the coupling in our application by forcing it into that particular place we have our HTTP interface here we have basically our web server down there and that's going to be calling back and forth here they're going to be pretty intimate and if you've ever tried to use ecto both inside and outside if you have a Phoenix application you all know that it can't be done it is coupled so once you got ecto inside Phoenix you cannot send a message to an ecto running outside of Phoenix so you've got coupling everywhere and you still have that damn file there it is the entire whole enchilada just waiting for a dog to come along and eat it so that is the despair part that is why everything is wrong why it's all bad why you all suck so the question then is okay what can we do about it and this is where I get to be positive I'm English that's kind of hard to do but I'm gonna try first of all here is today's number one rule you will take this away you will tell everybody you know about it and they will tell everybody else they know until the entire world knows we no longer use the word application application is banned in the elixir community instead we're going to use one of three words for the things that you create using mix knew the first word is library a library is for those components that have no state they are isolated as their own applications because I wanted them to be reusable but they have no state the second word that we're going to use is component and a component is a thing you create with mix new that does have state it's going to be one or more processes a library is not going to be a process a component is going to be one or more processes those processes will run either on the same node as other code or potentially on their own node so they are inherently going to be accessible across a network and then last and least in terms of effort is assembly which I think I borrowed from Microsoft assembly is the deliverable it's what you actually ship it's what gives value to your customers and clients and assembly doesn't have any code in it and assembly is just a configuration of components so it might look something like this here we have the little yellow circles our libraries the ones in the middle there are components and the whole thing together is an assembly now this is just naming it's just the miniature but it means that we can start thinking about individual parts of our assembly as their own freestanding components we no longer throw everything into Lib instead we throw it into its own project and just combine them when we need to at runtime it is not hard to do and I'll show you how a bit later on one of the cool things about this is that reuse is easy and we can actually do reuse as in the same instance gets reused or just reuse in terms of code reuse and it just falls out automatically because each of these things is isolated freestanding and then composed effectively at runtime into our assemblies configuration we're gonna stop using this ridiculous configuration scheme that were given and we're going to start doing it properly and properly means active configuration is now going to be a component a process it's stateful every assembly has a configuration component that component receives values it receives values from the overall configuration of the assembly and it receives values for each of the components in those assemblies actually not in that order it does the components first and then it does the assembly if a value is set in an assembly and then overridden at the sorry if it's set in a component and then overridden at the assembly level then it becomes the Assembly's value that sticks so when we're starting or if there's a configuration change the flow looks like this our individual components here each have a default configuration which effectively is their defaults which get fed up to the configuration module configuration module then reads the assembly configuration and stores that result and it does that on startup and it does it whenever there's a configuration change now whenever a component needs configuration it's going to use this nice configuration API up here to get it and therefore our configuration is now assembly wide and automatically it handles all the cases that we want where overrides or doesn't override this just seems to make sense it also means I can have a nice API into configuration where I can actually have maybe you know some kind of web interface it allows me to change it on the fly for a running application and because on config change will get invoked will just automatically update stuff last you may say but what about Phoenix applications what was the rule I gave you a few minutes ago no applications thank you they're not Phoenix applications we're no longer writing Phoenix applications now we're gonna write Phoenix components what does that mean here's my assembly for maybe playing poker so I have some libraries I have some components yeah the quote business logic and oh yes I have one more confirm that component down here is the front-end and guess what that's a Phoenix component how much business logic is there in that component 0 it is purely a view layer its handling the channels and the HTML and all of the work is being done behind why because it's the damn user interface you don't put business logic in your user interface you don't bundle your contexts into your web server it's a web server you guys have got more pride than that this is how it's going to be from now on gen server I complained about gen server and I showed you this example and I complained because it strikes me that there is a whole bunch of totally useless crap in here stuff that you don't care about I don't care about God probably doesn't care about but the Erlang runtime apparently does we don't need to specify this right this is a sake okay I've got to be careful here I was about to say this is a sacred cow that we have to slaughter but apparently that's offensive so I'm instead gonna say this is an idea whose time has passed right we need to say yes the implementation of gen server is great at least we're told it's great and I have no evidence that it's not but the expression of GN server in particular this expression of gen server sucks because it totally and absolutely obscures what we are trying to do so I've been playing around with options and I got a module now call component that lets me do this so it's the same implementation def module stack and I have two functions this is a one-way function this is a two-way function the one-way function allows me to push a new value onto the stack and a two-way function allows me to both set the state and also return a value this is the most complex case you can actually do a lot simpler if you're just setting a state or just returning a value so that is a server right now I'm showing you a name server so that's a conventional singleton named serger you can also say component cooled component hungry and component library and in those cases you'll get a dynamically allocated servers you'll get something I call hungry consumer which is a bit like a gents age where your consumers are sitting there saying give me work to some overall scheduler and you let the know state version the library version and you go but you just thrown away gen server no I haven't because behind the scenes that code is generating this code and in this code here we have our API push and pop here we have our implementation of pit push and pop put into their own module which means that I can unit test them in isolation here we have our server code which is just standard gen server code and then here we have our component interface code and the reason for that I will show you in a minute apart from the cool cool fact that I needed to have a function that returned the configuration of this component for various nefarious reasons and then you want the name to clash and now we can have utf-8 names in elixir then component info written backwards and partly upside-down is a perfectly valid valid function name you've been warned so nothing there has been lost it is still Jen server but you don't have to write all of the fluff you just write what you want component structure they look like this right now well first thing we can do is get rid of that stupid format or exs cuz no one ever uses it then we're gonna get rid of config because you don't need that then we're gonna get rid of that stupid Lib directory because dammit I am writing a stack the most important file in this entire thing is that it's the implementation of my stack why do I need to bury it of all things right format to dirty excesses at the top level my stack my main program is buried in something called live ridiculous so there's a simple one we don't actually the mix not exs either if we use sensible defaults which I would like to do but I just cannot see that flying if our stack implementation gets more complex and we'd like to put into a single file then we can create that I don't know whether to call it liberal stack and I'm on two minds about that and we can put stack popper and static pusher in there why do I have that structure and why don't I by default create that subdirectory it's because I don't want you to use it I want every single component that you write to fit into one file I don't want all this extra stuff if you write a thousand line file at that level there I'm going to take you out and beat you up with a rubber hose right that's not the point the point is to write lots and lots and lots and lots of trivial components and then compose them all together because that way you get reusability you get maintainability you no longer have to release the entire thing it's just so much nicer talking about deployment there is no easy way to deploy a component based elixir application what I want to be able to do is to deploy my assemblies I want to be able to say here an assembly consists of these components this configuration etc etc and oh I want to deploy it to a dynamic set of servers I want to be able to I want these components to be able to discover each other at runtime so once they've been loaded and started they need to be able to communicate back and forth and they need to have things that are obvious that we don't have right now common logging common stats common monitoring all of those kind of cool things that every other platform has so let me introduce you someone tell me who that is it's knotty knotty is the node dynamic management system yeah okay knotty actually exists in a very simple form Noddy is the component assembler it takes an assembly configuration first thing it does is it looks through it for a list of nodes and associated we node will be a list of servers sorry all the way around a list of nodes a list of service associated with each server will be a list of nodes to run on that server so Nadia is going to go through and it's going to create as many nodes as is needed on as many servers as you tell it to once the nodes are there not quite once but it's gonna be doing this in parallel it then starts kicking off components so it's gonna cover the components on node a and then it's going to do the same on B and the same on C etc cetera again in parallel and not is going to do a little bit of orchestrating to make sure that things start it rough at the same time it's going to handle things like or handoff to configuration it's going to handle common logging and all that kind of stuff so are not really naughty configuration is an assembly configuration it contains no definitions and all that is is a list of node names they won't just be n1 n2 but it's going to contain a set of server assignments is going to tell us how we assign those nodes to particular servers so in development are probably I probably want to run all of my nodes on localhost and the same in tests maybe but in deployment what I'm going to do something different so and one's gonna run on dot 42 and did you realize by the way that's perfectly valid elixir syntax is that cool because you can actually have : star as a symbol so therefore you can use star : as a map key there you go that was worth the price of admission wasn't it anyway so that's going to start in wall on this node and all the other all the rest on the other node and then finally we're going to tell it what components to use where to run them and then various other configuration stuff so here I'm going to run component 1 on both n1 and n2 and oh by the way I'm implementing it using the module called dictionary and then here I'm going to run component 2 just on n2 so that's going to create two copies of end it's going to create comp on on two nodes and komp2 on just one so Nadi launches notes components provides global name service etc etc I have written both the component library and I've made a start on Nadi the existing one does that it did use a swarm to do pub/sub but I got some problems so I kind of backed away from that and I do have a version that's using stats D and stuff to do monitoring but it's not reliable so it's kind of like not yet there yet and I've got myself a little stack of five raspberry PI's in a tower which I'm using to do testing it's going actually deploy into five servers at the same time and it works is it perfect absolutely not it's crap it really I mean it really is it needs a whole bunch of work it's nothing more than a proof-of-concept it's nothing more than an experiment to say okay it's all very well saying we should do it this way right but in practice it's impossible right no let's see if it's possible let's not go around saying oh you can't do that because of this let's instead say what happens if you take what looks like a good idea and actually try to run with it so both of these are up on github and I would love anybody who wants to take a shot at it so to work it through I got a little road map in my head which I could make explicit if people want but I'm not in charge of this right it's just throwing it out there I think this is a direction in which we should go I think we should make a lick sir easier to do use I think we should make it something that anyone can write a component in three minutes slap it out there and then have other people use it this is the future this is the way software is going to be developed and we have the opportunity to be there so yes we need to learn from the past we need to revere the past but we are not bound by it we need to reinvent and we need to do things in a way that makes sense for us in our current time and then last but by no means least in fact the very most important thing at all is you got to remember to have fun while you're doing it so thank you [Applause]

Channel: EMPEX Conference

Views: 11,945

Rating: 4.9673915 out of 5

Keywords: elixir, erlang, empex, functional programming

Id: 6U7cLUygMeI

Length: 39min 51sec (2391 seconds)

Published: Thu May 31 2018