ElixirConf 2016 - Building Umbrella Project by Wojtek Mach

can you hear me okay I guess we can get started um thank you very much for coming my name is Wojtek mark and i want to speak to you about building an umbrella project I work at collect and at my day job I'm mostly working with Ruby on Rails and on a side I'm basically throwing the elected community with type projects like the github API adapter for octo or the LP library for Alex year and today I'd like to show another toy project of mine with its Bank Acme Bank okay so what's the number of a project before answering this question I'd like to take a few minutes to talk about some inspirations for this talk some resources that I have found really insightful and interesting over the years the first one is a talk by Robert C Martin at Ruby at Midwest Rubicon 2011 the talk is titled as architecture the last years in that talk Robert described a typical dress application to be just that a rails app we look at the other projects talk structure we see the app folder and the controllers and models in views but by looking at just the project structure it's not a bit it's often out a bit hard to see what the project actually does what are the important parts what's important and what's not and how the things related to each other one memorable moment from me for me from Roberts talk was when he said something along the lines that a top-level architecture should scream its intent you should be able to look at it and understand what's going on another resource is a talk by Stefan hajiman titled wrangling large large Reavis code bases at Rocky Mountain Ruby 2012 Stefan talked about large apps and he gave a fantastic piece of advice about writing them never write in our jobs Stephan's showed how to structure our potentially big race project to be a collection of components gems and engines a gem is like a hex package and an engine is like a mini race application that's distributed that's often distributed as a gem what they think was really innovative in Stefan's approach was that he uses used these engines and gems not as means of distribution of the code but as a mean to break up your application into smaller parts each with its own namespace and its own test suite and configuration and so on and then the main race application didn't actually contain any logic of its own it just held these engines and gems finally I'd like to talk I'd like to mention one other talk which is the art of destroying software by Gregory on the premise of the talk is pretty simple organize your project in a way that makes it easy to delete code perhaps there is a part of of your system that is a constant source of box or it's really hard to understand or usually both when you need to make a change in that part instead of patching it up what if you could just throw it away and start from scratch start fresh how liberating that would be but but you need to organize your project to make it possible first and I think it's a really powerful idea to organize a project in that way to make it easy to delete code ok so what's an umbrella project an umbrella project is basically a mix file ok can you hear me ok thanks no pressure so an umbrella project is our is essentially a mixed file apps directory and in that abstract directory we have the collection of the OTP applications for the port for the purpose of this talk let's briefly talk about what an OTP app is per the documentation an application is a component implementing some specific functionality that can be start and stopped as a unit and it and which can be reused in under systems and on OTP application has can have application dependencies so these are the applications that must be started before this application starts there is a way to configure it and we can define the start and stop behavior in this start when we start some applications what we want to often do is to start our supervision tree and depending if we have these supervisors which sometimes can can call an application to be a regular application or if it doesn't have supervisors it's sometimes called a library application a good example of our library application is poison the D Jason library for Alex zero finally a typical application is kind of a standart if you know how to configure an OTP application and like start and stop party if you can if you learn how to configure one you can pretty much configure any of them because they are all similar it's a like a short understanding of how we should work with them the word application itself I think is pretty interesting here because outside of the airlock and OTP sort of ecosystem the word application has usually a bit of a different meaning and when I think about OTP applications and the fact they can be again configured start and stop awarded that comes into my mind is a server or a service and so when we have a big server when we have a big application out of code we could call it a big service but if we have a very small and focused one we could have a micro service right we'll talk about micro services at the end of this talk okay so let's let's build let me walk it through building an umbrella project so let's build the Acme bank so we create a new project we're gonna call Bank platform and we passed - - and Rafa flag it's gonna create a little bit different mix project in the mix file we can see that we don't actually have the name of the application because the umbrella project itself is not actually an OTP application it's just a container for for your other applications that are in the apps folder so let's create our first OTP application and we actually have a pretty interesting decision to make what actually start we want to have a web user interface for our bank but it doesn't mean that we absolutely have to start with the web functionality first I believe in working on the harder hardest problems first and so I'd like to start with building the core business logic of our application first and then keep it the couple from the web stuff and then later on we would connected so let's build the actual bank application that's going to hold our logic we CD into the apps folder and we run an onion create a new mix project we pass the SAP flag because we will want to have supervisors because this Bank application sucks we can maintain the storage as well and and yeah if if we create a new project within the umbrella mix will know that is in fact inside the umbrella and it's gonna change the mix file a little bit as you can see here like a difference from a regular mix file is that we have the build path config path depths path and the log file and in fact all of them all of the applications inside an umbrella will point these directories and files that are in the root of the umbrella they shared the same dependencies and log file and bill folder and all that okay so let's let's build the first feature of our bank let's let's compute the account balance of an account a naive approach to that would be to have an account struct with a balanced integer integer field when we deposit some money into the bank we would just increase the balance however this is not actually at all how banks work and since this is a really important to build a bank let me do a quick accounting one on one any movement of money happens between at least two accounts one account is debited and the other account is credited for example Alice transfers $100 to Bob we would actually write two different entries for that movement of money a debate to Alice for a hundred and a credit to Bob for hundred if Bob transfers 20 dollars back to Alice will have another debit for Bob and credit to two eyes and then maybe he sensed and more so will have the proper entries now to calculate the balance all we have to do is to get all of the entries for Bob and sum them a credit it's going to increase the account balance and the debit is going to decrease it so for Bob I'm sorry so for Bob we have a hundred minus 20 minus 10 so it's 70 and for us we have the opposite I think I have my math wrong sorry oh no it's okay so yeah so we have for us we have the opposite and this is actually a pretty important role in accounting that the sum of all credits must equal to the sum of all debits at all times and what we've briefly described is called double entry accounting system and double entry because we write and every transaction twice one for each and so you might be wondering like why I'm mentioning this as I mentioned is kind of like the core domain knowledge of the bank so it's kind of important but also it's functional the list of entries is immutable we never go back and change an entry or or deleted we just keep going keep adding these entries and actually it has some interesting mathematical properties as well the order of the entries doesn't matter so it's both associative and commutative like it doesn't matter if it's a debit and credit or a credit and debit and with with some extra work we could make it I'd important when we see an entry we just make sure that one entry is only calculated once for the balance and with associativity commutativity and Ida patterns this starts to look like a CR DT and as you know she abilities are the foundation for the Phoenix presence and there's actually a pretty interesting article on high scalability website that basically says that banking systems are not strictly consistent but are actually eventually consistent and this idea of eventual consistent consistency for banks actually is traced back to Renaissance era and in a way Florentine merchants like 600 years ago were using see our duties before it was cool so I think that's pretty interesting and finally like the whole reason to talk about accounting I think this stuff is actually not non-trivial and I think in most if not all non-trivial applications there is always going to be this essential complexity that's specific to your domain and I think it's very beneficial to separate it out so you can actually think about it in isolation independent of some details like like web so briefly I'm going to talk about entities in our bank so we are going to have the account and entry that we just talked about and the customer I want to keep a customer separate from the on account because usually a customer can have more than one account although actually it is just going to be one in this project so in a Phoenix project we will implement these entities as models so web models account and so on however seen since yesterday as you know models are out of fashion and moreover it's not actually on a Phoenix project so we are going to put this in Lib so it's going to be live bank account and so on so account and entry are a little bit different than a customer because they have a different rate of change when we implement our double-entry accounting system like after we implemented we are we should be pretty much done with that code because the accounting rules are very stable like 600 years old and it's like and so that part is very much different from the customer because a customer is something very specific to our application the customer can change as our requirements change in our particular bank and and so I think I think it's it's very worthwhile to to keep that separate and in fact I'm gonna add it to our ledger sort of namespace and directory to sort of show that it's a separate so let's talk about persistence how we actually want to persist this data for simplicity I'm just gonna use echo and now the question is where do we put the repo access so also since yesterday you know that putting the repo accessing controllers is out of fashion as well and again we don't really have a it's not a Phoenix project so where do we put it since the bank ledger account and the ledger entry are in the ledger namespace it's it's very natural to just put the extra functionality in the ledger module itself so we are going to have a function to calculate the balance and function to write down these entry sins and per system and we are going to do the same thing for the bank so there's gonna be a create function create customer function to actually create it and then update customer and so on and so forth so so we have these three functions and they are use cases of our application these are the high-level operations of the describe what the application does and inside these we have the repo access or maybe calling out to different services but on the outside this is this is what our our application is doing and our benefit is no benefit of of that is that we can just call these functions in our controllers or tests or IEX or seeds we can go to we don't need to go through the repository we can just call these very short named functions with very descriptive names and get and get the job done so we talked a little bit about accounts and balances and so briefly let's talk briefly about how do we actually want to store that so in many banks we have to support multiple currencies so let's have so we're gonna keep both the amount and currency on the money on the money record we also will have some operations like adding two different money objects and we usually want to make sure that we add stuff in the same currency because you can have unexpected unexpected results otherwise and also persistence since we start both among amount and currency we want we are gonna need to civilize that to a database column to keep it in in SQL and so actually all of these all of these things that we need to do are actually pretty generic and it they could be reused somewhere else so we don't want to in might be so I don't want to put that in the bank application and I don't took actually keep it separate so later on maybe if I'm gonna open source it is going to be absolutely trivial to just publish it to hacks and by starting from the get-go with a separate application for money which is going to be a trade or application with no supervisors when I start with an OTP application from the get-go I I achieve I'm more productive by doing that because if the code is already at hand in the Android project I don't have to worry about versioning or releases or like coordinating pull requests between two different git projects so it's it's much more productive and it's actually a case for all of the applications in your umbrella it's just so much easier to have everything in one repository although nicely partitioned and have everything at your fingertips in order to use the money application in the bank application we need to set it as a dependency there is an extra in umbrella flag that you can use and then so mix is going to know that it needs to find it this package in the umbrella and not on a hex p.m. we also need to add it to the applications list even though it doesn't start a supervisor's it needs to be there okay so let's talk about the D web interface for our bank this is how we are going to create it we just again CD to the apps directory and we create a new Phoenix project mix will know that we're inside the umbrella so it's gonna create a correct mix file and then we again have to add a bank at this time as a dependency to our Bank web web app and this is kind of important our web interface depends on the business logic but not the other way around the business logic doesn't care about web it doesn't reference it in any way okay so Bank web this is actually pretty boring because since we already have this logic all we need to do is to just just use it from the controllers and views so just just calling these functions so I'm not going to show that but I want to really mention again the separation of concerns that we've achieved by by doing it we can evolve the web application separate from the domain logic and the other way around it's also kind of important when you have to make major changes so I'm not sure if anyone was ever on a big web UI kind of redesign project anyone so I've been approached on a project like that a couple of times and I really wished that I would just start from scratch right because the UI was so coupled to everything else but if we now keep it separate we can again as Greg young said we can just delete the web stuff and just start from scratch or at least we have a very nice sort of we have good places to find the functionality okay so so this was so this was the this is the application for the web UI and for the customers to use and in our bank in production we we want to have some functionality for the operators like customer service and stuff like that so let's add kind of like an admin interface for our bank and connect all the back off is just because that's how it's usually called in the bank industry all you need to do is to just only to have here is just some screens to like see all of the accounts and transactions and stuff and I could I could write it but there is actually an excellent ex admin pockets on hacks and it actually works out of the box with umbrellas and in fact I shouldn't even mention it because there is nothing special about X admin because because it is just oh it's just Oh TP applications they they just work by design because we are using again that shared understanding okay so with the back office which is another Phoenix up in our project we need to set up different ports in development so maybe there is support four thousand for our main Phoenix up and port four thousand one for the for the back office so this sort of mention of different ports is a sort of sign that our deployment is going to be a little it's going to be a little bit more complicated and I'm gonna talk about deployment in a second but I also wanted mentioned the talk Phoenix is not your application violence earlier this year I think this is like having the back office and Bank web in one umbrella umbrella project is a testament to this idea that there isn't anything special about Phoenix it's not in fact your application because you can have many instances of Phoenix in your project there isn't anything special about Phoenix it just is just an OTP app okay so briefly talking about the government I'm deploying it to Heroku just because it's it's easy and in Heroku you can only have one only only one process can bind to a web port but what we have two applications so so what I did is I have a very simple proxy application that just forwards the requests to the appropriate Phoenix endpoint depending on the URL and a big thanks to Gary Rainey who allowed me to use that code it's really simple it's just few lines really and speaking about deployment distillery which is an xrn replacement already has a first-class support for Ambrose so if you wanted to do OTP releases of umbrellas that should be very easy with distillery okay so now I want to briefly mention an authentication subsystem of our bank so authentication is one of these things that all like pretty much always starts very small and often stays small but there are certain cases were in in certain up authentication can can get really complicated so I decided to start with our separated authentication subsystem first so you might be wondering well maybe it's a bit of a upfront design and I get that but I think there is actually married to it if you are building a bank we we are gonna know that we will have certain features that we're gonna absolutely need like maybe there are different strategies like the customers are gonna sign in with the username and password but the operators might need something like at the AP we probably are gonna need to factor authentication we probably gonna need some tracking like IP tracking to get that annoying message that you'll bugged in to your bank from a different country and it's not gonna allow you and stuff so so we have all of these extra functionality that we need that we're gonna need so that's why I wanted to create a separate authentication and in fact the authentication subsystem is going to maintain the user accounts and for that there is a separate ecto repo and actually a separate database because I think that in my opinion each application should maintain its own state so that's why there's a separate repository active repository from the from the other bank and thanks to Chris McCourt for some inspiration for extracting out the authentication into a separate app I also have a messaging subsystem in the bank but I won't get too into that today and and yeah this is our Bank platform we have seven applications of Bank Bank web back-office master Proximus engine and money and each application is an OTP application with very well defined dependencies and because of that we can actually graph how our platform looks like what depends on on what so it's like this top-level architecture that we want to have when we come into a new project we can see what are the pieces and as we as we evolve our project we are going to have even more applications so in a bank maybe you are gonna have some fraud detection subsystem that monitors the incoming transactions or maybe there's going to be a landing page that it's gonna be like a marketing landing page maybe it's gonna become a CMS system of some sort later on so so here we can see like we can manage the growth of our system by keeping the application separate okay so I want to try a quick demo of how it is to actually work with Android project so let me let me is that readable enough in the bug in the bag bigger okay so yeah so we are in the root of our project we can inspect the apps folder we can see or over apps we can run the tests so if you are in a mix project mix will automatically run tests for each application in your umbrella which is really useful when we CD into a critical app we can run the test suite separately so it's really good right we can work on our application separately run the tests iterate and then after we are done we can go back and run all of the tests to make sure we didn't break anything okay so because we have all of the code in one directory in one get project we can actually navigate the project very easily so we can let's see so we can go to ledger we can go to money which is a different LTP project we can quickly jump around the project without going to a different repository which is which is very fast to actually work on the project okay so I have a console that I started for this project and because we have these modules start to describe our use cases we can actually look at them so we have the bank and then we can see there is a create customer function or create episode find customers we can actually look at that like and again like calculating balance instead of copying that in the controller we actually have a place to put all that logic and we can document it so yeah oops I should started so let me quickly show you how the application looks like so this is the this is a very simple user interface for the bank that's running on port 4000 and we have the back of his application for four thousand one and and yeah all of the code is is available in github it's easy to deploy to Heroku and you can see all of the applications so let's go back to the talk okay so as we mentioned an OTP application is a bit like a micro service so there are well known benefits and drawbacks of micro services like there are strengths module boundaries and independent deployments and and the Dropbox are operational complexity so I was planning to do a comparison of micro services and umbrellas but if I told you that umbrellas give you pretty much of the benefits with none of the drawbacks you won't believe me anyway so I will just leave it like that as I exercise to the viewer so weave with umbrellas and micro service it's a little bit like with micro services right they won't solve your problems you still need to architect your system you want to have just the right amount of applications and with the right responsibilities you don't want to have too much or too many and to conclude keep an eye on the big picture and don't write a larger keep keep it separated isolated by functionality and by the rates of change of these components and delete code liberally and that's all I got thank you very much do you have a question in the front so I'm kind of a big fan of my microservices but I I certainly understand those drawbacks nonetheless the big benefits to me are the independence and dependence of Technology and independently being able to scale or release different services a lot of that goes away if you stay entirely in the erlang of lives or ecosystem but most of the time you can't especially with large organizations you're going to have multiple technologies so how do you man how do you map into sure that kind of a service environment yes oh that's a good that's a great question I mean if you're in if in your organization you just have components in other technologies you just can't do anything about that that's just a fact but I think there is actually I mean independent organization but there is a big benefit to stay on the same stack because then you can switch to different teams as far as a liability issues that you mentioned I think it's still compatible with number of applications you have the flexibility of deploying just one application to our cluster of nodes and other application to a different cluster so you have everything in one project but you can just figure out how you want to deploy it you have a lot of options so it's sort of like Google has this huge repository right with all of the code but they still deploy it separately so you actually have an option for that that's really that answers the question you talked about organizing your code in a way that makes it easier to delete are you were you just saying that an umbrella makes it easier to delete or is there more you can say about organizing your code so it's easier to delete it sure I mean like this principle of organ secure product project to delete code it's not sort of specific temperature like there isn't anything special about umbrella it's more of a general principle but I do think that having an umbrella kind of a guides you into that direction but you have these small pieces that are very isolated and independent of each other and then you can sort of take them and think about them separately and then if you just want to start from scratch of a component you can you can do that but yeah if you structure your like big application in a way that each piece is separate any I can continue that but I just think that here with an umbrella you it's all a bit easier to go into that direction where pieces are a separate I'm curious if you can speak a little bit toward whether there are any guidelines for how you would write integration tests for a system like this like it seems like you would probably have some type of top level integration types but I would also like feel a pull toward the rabbit hole of having an integration test but like every yes so does have a good question I well you can approach it like you would normally approach that maybe you would have integration tests for the for the bank web UI which is kind of like the entry point you could have integration integration tests there but I think an another interesting approach is to actually even have I mean I explored with the idea out but but an interesting approach is to actually have a separate kind of project that integration integration tests the entire thing and what's kind of cool about that is that you can even like deploy everything maybe to Heroku like all of the applications and then integration tests that kind of life which sort of compared to micro services its output easier because with micro services you kind of have to deploy everything separately and here you have everything in a bundle but yeah I mean I mean I don't really have any specific guidelines at this moment but yeah I think there are good options when you're using master proxy is there a way to share logins and authentication put your pizza ring when you're using master proxy to you have a separate administrative application another way to share authentication and login oh yeah definitely so so yeah there's the the the authentication subsystem keeps track of the of the login information so you would use the owl application in either of these apps and then the map must approximately does just a kind of like a router it just for us requests so you solve the authentication problem on the on each separate web application anyone else thank you all right thank you very much

Channel: Confreaks

Views: 7,756

Rating: 5 out of 5

Keywords:

Id: 6NTmUQClHrU

Length: 37min 28sec (2248 seconds)

Published: Tue Sep 27 2016