ElixirConf 2021 - Henry Popp - Elixir as Your Development Culture

[Music] all right well good morning welcome welcome this is elixir as your development culture so who is this guy on stage well i am henry pope i am one of the co-founders of codeedge cottage itself we are a software design and development consultancy we've worked with companies of all sizes everything from tiny startups all the way to fortune 100s such as redacted and non-disclosure agreement those are my favorite kind unfortunately we can't talk about a lot of the work that we do but kind of the gist of us think seal team six so we get in there we fix elixir systems we build elixir systems and we're very very good at what we do but it didn't always used to be that way today is a talk of heartbreak of triumph and hopefully a bunch of code snippets that don't bore you to death we'll see so let's get started so the year was 2015 my co-founder and i were fresh out of college we had been doing a lot of ruby on rails work and native mobile development in college for various clients but we decided right after graduation hey let's actually found a company so you can see the photo on the right we didn't have a garage but we did have an upstairs room in my parents house so that was officially cottage headquarters for the first couple years worked really well for us but what's interesting about the codex journey unlike a lot of companies that adopt elixir we didn't have any preconceived ideas about architecture and design we've been building ruby on rails apps but we were effectively a blank slate because we had just graduated college and so i remember thinking around the time i'd been studying elixir phoenix kind of watching it for about a year and it kept i kept hearing this thing over and over again it's like ruby on rails but fast even today that's how i explain it to people when they don't know what elixir is but i really really wanted to try it out on the project and so fortunately we got the opportunity pretty quickly right after we found it so there was this customer who came to us and they needed this prototype built it was a simple crud application with a real-time chat prototype piece to it which i was thinking okay so you've seen so many different like phoenix demos you know build a live chat in 40 lines yeah phoenix phoenix channel seems like a perfect fit for this use case and so we built it was it was the crud portion we did in rails obviously channels for the chat part and the thing worked really really well so mission accomplished we had another project very soon after that which before coatedge i didn't really know anything about the cattle industry now i know a lot about the cattle industry so we built basically ebay for cattle so it was a cattle auction platform that basically and this this happens across the country every day thousands of different cattle auctions so we figured okay phoenix would be a good fit for this as well let's do it um so we did it it worked really really well the code base was amazing complete joy to work with which is what i would have said if it wasn't a hot dumpster fire so our first full-scale phoenix application was really really bad but to its credit it worked it met all of the customer requirements it did everything really really responsibly really quickly and i think this is a real testament to elixir that you can design a very bad code base but it still smokes everything else out of the water our biggest competitors at the time were some php applications and net we were the future of the cattle industry using phoenix go figure but that said yes the code base did have a lot of problems with it had a very limiting database schema and this wasn't necessarily phoenix or ecto's fault by any means it was just more of the one database schema to rule them all we had to generate a lot of reports it became very difficult with a lot of the table joins and everything also on top of that so we adopted the pattern that's common in ruby on rails fat models and skinny controllers that was a terrible idea so it um we basically ended up with these junk drawer modules for all of our ecto schemas is really bad on the front end it was just a torrent of jquery many of you are pretty familiar with this this is was our unfortunate reality before live view but back in 2015 to 2016 yeah it was it was bad so i kept thinking there has to be a better way so who remembers this blog post so the the great jose wrote elixir and times of microservices and this was effectively the holy grail of how to design good elixir applications back around this time so i was reading it learned about umbrella apps a lot of it led me to further read up on domain driven design suddenly this whole universe opened up and i'm thinking okay we have all these cool ideas let's apply them on our next project and we'll succeed well no the cows are still on fire so our cattle client came back to us and they said hey we want another platform built for some different types of cattle auctions so this one had a bigger scope before i didn't know there were multiple types of cattle auctions evidently they they have their own rules so it's a bigger scope than before did a lot more things so we figured okay we'll do it as an umbrella application we'll do it as our first attempt it domain driven design and it was a way better designed mostly kind of it ended up being a configuration and dependency nightmare which you've run into this with umbrella apps before we made the classic mistake of we had an app called repo and that was like the whole database you never do that basically we had kind of built it was still a truly mono repo project but we were pretending like these are separate applications in here it was just bad but to its credit again the platform worked really well for the customer so mission accomplished there several projects later so let's talk about docker for a second who here remembers when docker used to cause random kernel panics on debian yeah the code edge remembers um so we we had been watching docker for a very long time but we still didn't quite trust it yet it was around the 2017 to 2018 uh time period especially when they fixed the kernel panics that we finally decided hey okay yeah let's let's go ahead and start using docker so we had a new customer come to us the project was a customer outreach and email marketing tool so you know how you get spam in your inbox well the great thing about elixir is it allows you to send a lot of spam really quickly so with this project we decided we were going to adopt docker but we had a very unique opportunity here we realized that hey instead of doing an umbrella application let's go ahead and split them into separate repositories entirely because docker makes it very easy to deploy applications so let's go ahead and we'll break it up all of our business domains they'll be separate repositories and then we'll have all the domain again events propagated with rabbitmq in a typical domain driven style and then we do a front end single page in view and guess what it worked it worked really well this was the best architected application we had done to date so we figured okay let's do it again this next project 6dos is known as six degrees of separation so this is a really really cool web app that it's kind of like a contact manager slash networking tool so you upload your contacts and it will basically deduplicate them and score how well you know somebody so let's say like my co-founder in the audience i have his home address got his email got his phone number got a lot of different little pieces about him i know him well and this is important because with these scores that you generate you then join teams of people other networking type people that let's say they want an introduction to someone and they realize oh hey that person knows that guy really well that's you can have an introduction made that's always the best way to network and so 6dos is a platform that allows you to do that so with the architecture on this we did the exact same strategy we broke all the domain events into their own repositories but this time we're going to try something on top of it we threw the command pattern into the mix and this worked even better and so before i get into the command pattern i do want to talk about 6dos architecture just in general um i can legally talk about this unlike the other project which is a big part of why um i brought it up but so you'll see here you have five verticals the on the left side you have the account manager that's like user accounts and team management you have the contact importer which does what it says it imports contacts you have the contact manager which is really where the heart of 6dos lives it does all the scoring all the deduplication it is truly a contact manager product manager is just your billing system not a lot to talk about there and the report engine just generates some reports and sends emails out so let's talk about the command pattern um a lot of you may know this from object oriented it's basically the idea that you build an object out of a command just a general user flow or some sort of like complicated sequence of actions that you want to encapsulate in one place so in elixir we do it with modules and structs and this worked extremely well we were designing modules that did one specific user flow so it had all the pipeline steps in it it had all the data that you needed let's take importing your google contacts as an example so you have to authorize with google you have to fetch the data you got to sanitize it and then you got to prepare it to go into the contact manager well all of this lives in one file within the contact importer service and this makes it very very easy when there's errors let's say with the google contacts you know exactly what file to go look at and we like this pattern so much that we actually made an open source library out of it it's called command x so it is the coolest tiny repo that you probably don't know about so now you know about it and go look it up and there is another thing too so this is really cool we discovered when we started to put things in this style anytime you generated documentation your module names then became a list of everything that your service did which works really cool for service oriented and triple d because then when you look at the repository you just know that hey yeah this is this is straightforward so why do these things work well i'm the type of person that quotes himself in his own presentation it um it took a long time to put words to these ideas but i call it instantaneous complexity and so it's the idea that software should not be too complex at any given scope this is actually something that you already instinctively know it's times when you pull up a module and you're all like oh god that's terrible you know because the complexity just spills out of the screen and it gives you a bad reaction but complexity at any given scope well what are the scopes so we got four circles here i consider software to be very fractal in nature so really these are all kind of the same thing just at different levels your functions are obvious they do one thing your modules are a bag of functions your repository is a bag of modules and your system is a bag of repositories now an important takeaway from this the repository does not necessarily have to be code that you wrote in any typical system i might consider your databases to be repositories obviously hopefully you didn't write your own database but let's say you deployed postgres well that was a repository that somebody else wrote you're just happening to use it on your cluster and so when we look at service oriented architecture and the idea of breaking up umbrellas into separate repositories well what we've done is made the repositories less complex but we've made the overall system more complex you might have seen this before in microservices which took it way too far i don't understand why in the software industry that it seems like when we have a good idea we then take it so far that it becomes a bad idea if you have hundreds and hundreds of repositories deployed on your cluster yeah you've done something wrong really what we've done here is taking a happy medium between the mono repo style and micro services as we all know them and really just deployed services and this isn't even a new idea either service oriented has been around since the 80s but it's funny how it will keep cropping back up is a good architecture strategy and i remember until i really read up on it i kind of thought i'd invented service oriented and then i was all like oh nope so with the command pattern it's kind of a similar thing it will make the modules more complex and the repository less complex it is debatable whether or not the command pattern actually makes modules more complex it does make them bigger but it makes them more focused and then likewise decreasing the complexity in the repository it becomes straightforward just because you have all of your user actions segmented into their own modules so maybe you've seen this graph before this is project size verse complexity for strongly typed and weakly typed languages so you'll see for smaller projects weekly typed is definitely simpler to work with and then as the project grows the complexity airplane decides to take off and then suddenly last minute it decides it wants to go to space so you do not want your code going to space and that's actually why we consider our strategy to be so effective it allows us to put to make our services as big as we possibly can before they become too unwieldy to work with so yeah that is the the code edge dash line of awesomeness but this isn't necessarily a magic bullet for every case everything always has exceptions so i have a flowchart here um it's the handy dandy codedge architecture decision process just three questions so let's start at the top left is it a corporate marketing website now this is one of the most frequent questions i seem to get asked mostly by big companies that don't really know what software is but they know that they need websites so generally for those types of jobs we immediately reach to phoenix now you might be thinking oh why don't you just use a static site generator to which i say that's cool until you need a contact form or maybe a little tiny blog system or hey i just want to like display some banners from time to time suddenly the complexity jumps up it's not purely just static visualization there is a little bit of logic to it which we find that phoenix and live view are just a perfect fit for that but let's say it's not a corporate marketing website so the next question is is it a big scope well if no then you reach back to phoenix again in the monorepo style this is a lot of times things like very very simple crud applications that you know if a user just wants to be able to log in and like i don't know manage pictures of cats or something and that's really all it does then yeah phoenix would be the perfect fit but let's say it is a big scope well the final question here is do you hate your future self now if you thought 2020 was a pretty good year um then go ahead and stick with the monorepo style i imagine i probably just irked some people in the room a little bit but it um okay you can build good systems in the monorepo style but it's very very difficult um this is the main reason we don't want code that's ever a dumpster fire by separating out into your separate repositories if one of them is a dumpster fire not all of them are a dumpster fire but if you keep it in the same repository yeah there could be a dumpster fire there so that's really not what we want so let's go ahead and change gears a little bit let's talk about continuous integration now it is october and i wanted to be a little spooky yeah um so we have the photo on the left here those are two servers sitting on a filing cabinet one says dev and one says prod yeah no it's not what you're thinking they were not client production services running on our filing cabinet this was actually where we used to host our self-hosted gitlab installation my co-founder is the most frugal person on earth so we didn't even pay for those servers he got him for free i kept saying hey let's move into the cloud and he was like nah um eventually i got my way but for the first little while at codedge yeah it ran on that box and so gitlab works really really well for all of our processes it's been so woven into the company because we've been using it since the start um we've built all of these really amazing elixir pipelines we use it for private hex packages basically our code is great because of the things we've been able to automate with gitlab as we do this and for some reason it seems like every big company that we've worked with that uses gitlab their pipelines are just bad i don't know if it's maybe apathy or they just don't know what you can do with elixir pipelines but generally it's just like test build deploy and it's like that's cool but what if we did something better so this is a screenshot of the code edge marketing website we've got 11 stages on it and i will i will break it down some are boring some are interesting we'll run through all of them but you'll see kind of an overall picture of of what we do so we have here the test and lint stage this one is pretty straightforward key takeaways are we use ex coveralls we use mix format check formatted and we use credo all of those tools are great if you're not currently using them that's like step zero to having good applications so i recommend that that you start there first let's talk about dialyzer for a second so dialyzer i consider it the gentleman's type checker it is a effectively you want to be classy in your development process dialyzer used to be very terrible to work with it is a lot nicer now the error messages are actually legible now it tells you a lot and it's really effective to use type specs are hopefully something that you should already be putting in your code once you get hooked on type specs it becomes one of those things that whenever you read documentation you always read the type specs first before you actually read the description of the function weirdly enough i feel like my whole world just kind of revolves around type specs but for the people who don't necessarily like dialyzer i think a big part of the problem is is that they don't use it from day zero of their project dialyzer if you add it in after the fact that becomes very very difficult to work with but if you start it from day zero you catch errors early and often and everything works well so this is a neat feature of git lab it can actually host per project static documentation not a lot of people know about this but this is actually the main reason why we don't use private xpm packages we basically host everything with gitlab and we have all of our documentation in one place so the stage itself is actually pretty straightforward all you need to do is have ex doc in your dependencies you run mixed docs and then you move that documentation into a public directory within your ci stage and then gitlab will handle the rest it'll just deploy it for you and then suddenly you have hosted docs that are most importantly gated behind your authentication so you're not just publicly hosting your private documentation but it makes it very very easy for developers to be able to reference things you can put badges in your readme everything's nice so there's an important piece of this this isn't a ci stage this is more like you need to understand this to understand the rest of the ci stages if there is one thing that gives it away that it's a code edge project that we built it it is the version file it seems like our whole process kind of revolves around this version file and so it's as simple as it's a little text file in all caps version you put the semantic version in there and then we reference that from everywhere so you can see the snippet on the right we read it in we trim it we set it as an app module variable and then it gets compiled into the project everything works great but then likewise having that version file in the project root you can also reference it from ci and suddenly your shell scripts and everything become very very easy to work with there is for a lot of people i call it the 0.1.0 club and so you know who you are that if you initialize a new elixir project and then you never touch the version again i feel sorry for you because semantic versioning is great mix it works so well with it it fits so well with service oriented architecture and most importantly it gives your whole team a language with which to express changes in your code so if you're just tagging your docker images with the commit saw and just throwing them out on the cluster you don't really know i mean yeah you know what was in the change but you don't necessarily know the implications of the change and that's why semantic versioning forces you to at least say something of the sort of oh yeah this is just a patch like don't worry about it or hey this is a major version change we're going to break something so going off the version file um we have the release stage which is kind of the first step once you've merged onto master and you're preparing for this code to go out in the wild so get lab has a really cool releases section and it's very easy to configure ci to be able to tag your commit with whatever version it is unfortunately there's kind of some weirdness you have to prepare the tag before you actually tag it so that's why we have it broken up into two stages here but going back to the version file we only trigger this if the version file changes so if it detects a change it will run the stage it will tag the commit most importantly if you downgrade the version and it runs it again the git tag will fail if you want to just merge the master and not deploy anything well you just don't change the version file seemingly what is just a text file actually dictates a lot of flows and ways you can approach the repository so for building your images this is pretty straightforward we like to build a docker image on every feature branch i don't know how many of y'all do this we find it works really well for our developers that a lot of times yes you are editing docker files you're tweaking things with deployments it is good to just see like okay yeah we're not going to screw anything up here when we go to build an image but with the way that gitlab artifacts work it makes it very easy to build an image in one stage and then publish it to a docker registry in another stage and so that's exactly what we do so before the commit is tagged in the release stage there will be a docker image built and then the image publish is only triggered on changes to the version file after the release from which it just fetches the artifact and then does a docker push and it's got the semantic version in there which once again is catted from the version file and everything works great we have the final ci stage in here this is kind of just a little add-on so code edge loves sentry have been using century for years and years now they have a really awesome gitlab integration [Music] we actually use self-hosted sentry we might be migrating to the to the sas platform at some point but basically sentry does one thing well and that is error tracking it works so well with get lab and it works even better with release tracking so you can tell sentry hey there is a new semantic version of this code it will lump in all of the code changes from the prior commits and then once that version is out running in the wild if there's any errors with it it will detect that hey there was a regression probably caused by this version and then it will actually email the authors of those commits to be like hey go fix your code it's really really cool so what about deployments so let's talk about kubernetes for a second now this was really interesting there was kind of an argument that came up within the elixir community a couple years ago people seemed very resistant to kubernetes because they kept thinking like hey isn't this basically the beam why do we need to run things on cube and honestly that's a fair assessment because they are pretty much exactly the same but you have to think about the problem that kubernetes is actually trying to solve so cube is written and go go doesn't have any sort of supervision it doesn't have any sort of load balancing or cpu throttling so it only logically makes sense that something like kubernetes would exist and so in light of this knowledge i took the liberty of redesigning the kubernetes logo here um been calling it go beam it's uh yeah it's i'm a graphic designer by the way part time at night um but kubernetes itself where a lot of people that might consider it competition with the beam i actually see it as an opportunity here so you've heard this question over and over again who supervises the supervisors well we have an answer for it kubernetes supervises the supervisors no matter how great the beam is it still can't reschedule itself on another cluster you need some sort of external thing that can move the entire running piece of software somewhere else and so this actually fits in really well with my idea within the fractal nature of software i call it fractal monitoring so thinking about the beam in general in an elixir project you have super super responsive restarts schedulings error handling all of the good stuff whereas at the kubernetes level all you really have are your readiness and liveness probes to dictate what's going on so it makes sense then for elixir to be able to monitor itself it's infinitely more responsive but then you also have the fallback of the kubernetes side to be able to handle truly catastrophic failures this combination go can never beat it go will only ever have kubernetes um and so that's why i think this is so amazing that we have an opportunity here because go and elixir have been neck and neck for on so many things for such a long time but we won this battle so why don't we truly utilize the opportunity that we're given and deploy everything to kubernetes so we have here getting it on the cluster um now a lot of people there's really good integrations with git lab to just deploy directly to kubernetes that's really cool if you type things into the google you can figure out how to do that that's not something i'm going to talk about here there's another tool that we actually really really like that we use for all of our internally hosted code edge services and it's called keel so keel runs is a daemon set on your kubernetes cluster very very lightweight all it does is pull your docker registries and then you can set semantic versioned deployment rules so that might be things like hey on patch bumps go ahead and auto deploy on minor bumps require one approval on major bumps require three approvals really really easy really really powerful works great with the service oriented and there is finally so for keel it has really fantastic web hook support i have here a little diagram of our deployment web hook triggers so keel it has a lot of different integrations for a lot of services it also emits just generic web hooks which we actually wrote a tiny little elixir service that will take in those web hooks and then based on the appropriate project that's configured it will notify century that there was a new deployment and that is different from releases releases are just for the code deployments are hey this is actually running in the wild now so it'll forward it on and it will post its sentry hey there is a new version likewise it will fork the flow and also notify discord just so we all get pinged about it which by the way if you've never used discord before i highly recommend it it is the official code edge chat it's written in elixir has really great elixir syntax highlighting we are big elixir or big discord fans so in review here so you've seen our architectural styles with service oriented and domain driven design you've seen how we've implemented the command pattern in here you've seen our ci process that we consider the true foundation of how we build really great software and you've seen how we do things on kubernetes so the great thing about opinions is that there's so many of them and clearly we're right but yeah but this is one of those things that if these ideas speak to you well codeedge is hiring we look for developers who want to try to push the boundaries of design and become the best developers they can possibly be if on the other hand you thought everything i said was dumb well you can send me a very angry email and i'll argue with you on the internet because i love doing that but regardless of whichever way you feel about it i hope ultimately you learn something and with that thank you

Channel: ElixirConf

Views: 716

Keywords:

Id: g6rPd1-DdCs

Length: 38min 13sec (2293 seconds)

Published: Fri Oct 22 2021