The Big Elixir 2022 - The Phoenix Project - Todd Resudek

good morning everybody um just like last time i was here i started out my talk with some music from a native newer new orleanian new orlean and this guy's name is mystical um who's heard of mystical before i guess i guess i'm the only one that hasn't okay um yeah so anyway that's uh he's from here and the cd cover is pretty cool i think so if anybody recognizes that there's uh there was this design company back in the 90s and early 2000s that did all these rap covers called pen and pixel graphics and they ended up being the in-house design firm for no limit records so you can see like maybe this one you had in your collection or this one or this one uh you can kind of see they have a pretty distinct style uh it's pretty easy to recognize they were uh they were on top of the world back in the day um unfortunately i don't think they're doing as well right now this is their current web page i know sorry all right anyway i'm not here to talk about pen and pixel i'm here to talk about phoenix project but let me introduce myself first that's me todd rezadek all right um find me online super simple i'm a staff engineer at jackpocket and as alex mentioned i'm a member of the hex core team all right so a little bit about me i grew up in wisconsin i was waiting for the applause okay nobody finally scott i meant if you're from wisconsin amy come on all right move to los angeles somebody's here from l.a come on albert come on all right do do we understand what we're supposed to do now for the next one okay if you're from this city or you live there now clap okay ready all right new york city there we go moved to new york city and now i live in colorado all right um so this is a pretty serious talk and but before we get started i actually do want to um take a little time out to just mention a man that we lost recently in our community and without him a lot of these talks wouldn't be possible steve wilheit he's the inventor of the of the gif or jif as he would say sorry steve so pour one out for steve today all right so the talk's called the phoenix project live colon live view and the theory of constraints and some of you might recognize the name it's a reference to this book i was also told that the phoenix project is the name of a project in seti for the search for intelligent life and the universe by analyzing radio signals uh it's it's not about aliens you may end up wishing it was about aliens but it's not it's a talk mostly about the theory of constraints and the theory of constraints was originally introduced in this book called the goal and i'll talk a little bit more about how that applies to software development life cycle but i just wanted to call out the author dr ilyahu goldratt heavy drip game this guy so he puts himself on the cover of his book it's kind of crazy but if you're if you look as good as him you would too probably i also make reference to a couple other books this one accelerate by nicole forsgren and team topologies all right uh so this the talk is about a real project i'm just going to call it the phoenix project and by the way we're 25 slides in i haven't showed any code yet and we have about 70 more slides to go and exactly one of them has code um so pick your number try to guess which slide is going to have the code in it and i'll let you know when we get there you can find out if you're right so about the phoenix project so the phoenix project was this admin website for a company now called sports unlimited inc it's uh it was green field work uh didn't have any existing code didn't have an existing team and our ux and our ui were provided by a designer the main goals were ease of navigation speed of interaction as well as getting real-time updates of data so um when we we started out we're kind of planning for the phoenix project we started looking at our previous projects um and so i'll call that project x project x isn't really one project it's kind of a culmination of a lot of projects that the people involved uh brought in from their past so it's kind of it's based on experiences that were common in all of those projects and so project x looking at the team composition usually about four to six back end developers right around the same three or three to six front end developers then he had a product owner an embedded designer an embedded sre maybe a couple of others so this is a basic composition of project x um and so we sort of looked at project x and started by looking at the flow of work so the flow is just basically how work moves throughout your system so you can see in this diagram you know we start out with the requirements and sort of work all the way up to the ui work before we deploy um and sometimes what happens in the flow is you get all the way to the ui work and realize oh there's something that we needed in the api that we don't have um and so in a like lean manufacturing you would have these and done handles between each step you know if something's going wrong at your step you know pull the handle or hit the button whatever the red light goes off and everybody sort of focuses on that we don't really have that in software um so what would happen is you know we'd go all the way to the ui phase we'd find this problem say oh you know i need this field to be this type or i need this extra field uh in the api so that i can show it in the ui later on or i need you to calculate some value et cetera et cetera and so we would have to go all the way there and then create this new unplanned work push it all the way back to the beginning of the flow process say here's the new requirement and kind of put everything on pause until it came back through the system and so flow the reason why we started with flow is it's probably the most observable way to determine how well or how poorly your system is working you get a really good kind of bird's eye view of kind of where the hiccups are in your system what's moving slowly what's moving faster and ideally your flow would be a very consistent predictable cadence so over time you would you know be able to basically gauge how fast something should move through the system uh it won't be fits and starts um and so when we're looking at the flow then you kind of naturally go into the theory of constraints uh so the theory of constraints is a theory uh it's in the it's a theory in the management paradigm that views any manageable system as being limited in achieving more of its goals by a small number of constraints there's always at least one constraint and the theory of constraint uses a focusing process to identify the constraint and restructure the rest of the organization around it okay so this is kind of a heavy concept like that whole book the goal is the theory of constraints um and so this is just kind of a short version of it but so in any system like i said there's one part that's the slowest and you can think of that usually as being the constraint the process involves identifying that constraint you could call it a bottleneck and then making changes in the system to increase the throughput there now after you've done that something else is the constraint so in a factory you know it's you and the easy way to identify this in factory is basically look at all these different workstations and see where um see where product is piling up so if you're in like a car factory and you've got sub-assemblies of motors or something that go into frames at the next station and now you've got 10 motors stacked up here uh it's a pretty good visual example of like of where your constraint is so obviously your constraint is you know right after that motor or somewhere after that motor in the system um it's a little bit harder to do in the software development life cycle because we're not looking at motors but there are definitely ways to do it so for visual learners um it's kind of a visualization of how that works so you can see like the intake step and then step one has a lot more capacity step two is our bottleneck um and so you know when you look at the system it's important to know that nothing's ever going to move faster than the bottleneck so even though you've got great capacity in step one and step three the system is never going to produce anything faster than what step two can can handle um so in the the so to deal with the theory of constraints you've got like this five step plan so the first thing is uh you've got to identify what the constraint is so you know step two in this case or where the motors are backing up etc next is to exploit the constraint so eliminate everything in there that doesn't add value so if there's things that you're doing in that in that step that aren't adding value to the customer get rid of it then you subordinate everything else so this one is really hard to explain to managers um but essentially there's no use in running step one or step three at full capacity if step two is gonna is going to be the thing that's controlling your output um and so in lean manufacturing they would actually say just leave those stations idle like they should be producing as much as station as step 2 can handle and no more and if that means that the people there are working half days and then so be it it's actually better for the company uh that they do that than for them to pile up a bunch of assets so subordinate everything else and uh and then the next step is to elevate the constraint so devote resources figure out what can we do in in this case step two what can we do at that constraint to make it move faster uh so it could be you know breaking apart um the work that's done in step two it could be putting more resources in it so in a manufacturing facility it might be like let's add another machine in uh software development it might be uh you know something else add a tool to the to our system um you know lean on uh a sas product or something instead of building it ourselves whatever the case is uh and then once you've done that you go back to step one figure out where your new constraint is all right so how does this apply to project x so i've talked a lot about manufacturing but how would this apply to the software development life cycle so um when we we looked at project x for us the constraint we found was this handoff between the back end team and the front end team and so we started looking are there other tasks in the handoff that maybe aren't necessary or aren't providing value um if uh you know or can we work in smaller batch sizes what can we do basically to to alleviate that constraint so we identified our constraint it's this handoff between back end and front end let's remove unnecessary tasks from the handoff process is there a ceremony that's involved that we could get rid of some meeting that we could get rid of and while we do that you know if the handoff is after the back end work is done can we instead of just having the back end developers be idle are there you know tech debt tickets or something that they could work on instead just uh so they're not working half days but they're doing things that aren't going to hit that same constraint later on do we bring over more resources is there somebody some role that we can add in that exchange that's going to make this faster but what if so had us thinking what if there wasn't a handoff at all and that's where this book comes in [Applause] all right 42 slides in we've introduced live view how's that all right uh so uh so now let's look at the flow of the system again so now we're using live view right we go from requirements to back end work to ui work and that's it uh the flow is quite different in that so if we find this issue in the ui work like we found before instead of going all the way back to the beginning we you know maybe go back to the previous stage or it might be the fact that it might be something that the person that found the problem in the ui can also fix themselves uh in the back end or at least it's in the same group and um you know since the the issue is kind of fresh in their mind that bit of code is fresh in their mind uh the context switching is is minimalized so it actually might even be faster uh for them to fix that than it would be in the in the old type of flow so i'll get back to this in a minute but let's talk about implementation so how how would we do that if we've never done that before how would we build this team um to work in this this type of flow and whenever you're doing that whenever you're proposing any kind of a new architecture you should be considering conway's law um it's not that conway so when you hear conway you probably think of this guy john horton conway a phd of sydney sussex college cambridge nobody from cambridge okay but you probably know him from conway's game of life it's a like a fun little programming project and he was a specialist in the cgt com combinatorial game theory um so that's that's john horton conway but conway's law is actually named after this guy melvin conway a phd from case western reserve somebody just applaud pretend like they went there thanks all right go spartans uh and other than than conway's law he is most uh he's most famous for working on the mumps spec and publishing the mumps manual uh mumps we got a mumps guy see okay mom says uh has any anybody else worked in mumps before very esoteric language that's mostly used in the medical system if you worked at the va or occasionally at financial products so it actually stands for the massachusetts general hospital utility multi-programming system mumps very forced acronym but uh it's pretty cool i actually i got time i got a little aside on this i met a guy once who did mumps and he's like yeah there's like 12 of us that still specialize in mumps and so when the va needs somebody um they don't ask you how much you get paid you tell them how much you get paid so it might be good to get into um all right so sorry let's get back to conway's law uh this is huge so any organization that designs a system will produce a design whose structure is a copy of the organization's communication structure learn that this is a an easier way to put it so if you have four teams working on a compiler you're going to get a four pass compiler so essentially like essentially your architecture is going to end up being set up the way your org structure is set up and who's talking to whom um and so if you're not being very deliberate about this as an engineering organization these decisions are being made by other people so it could be a high level manager heaven forbid it could be your hr team that's designing the teams but somebody's designing essentially designing your architecture for you when they're building these teams out so just an example of conway's law so like we look at four applications here uh we've got four front end teams four back-end teams but they have a shared uh dba and the shared sre and conway's law would say like the emergent architecture would be a decoupled front-end back-end but a monolithic shared database and a shared infrastructure so you would see what you would expect to see out of this would be one database that all four applications are kind of using and one you know kind of deployment strategy or maybe one kubernetes cluster or whatever it is because there's one person that's trying to do all four of those jobs at the same time um so that's when this bad boy comes in the inverse conway maneuver uh anybody ever heard of this one before where's dave okay there he is all right um okay so the inverse conway maneuver recommends evolving your team and organizational structure to promote your desired architecture so if you think of what the architecture is that you want to end up with what you should be doing today is building your team to reflect that your your team structure is your first draft of your architecture so if we do something like this the inverse conway maneuver and we add db resources and add sre resources we should expect the system to be far more decoupled than the than the previous one so for the phoenix project this was our proposed team structure back end db sre and that's it uh we removed the independent front-end team so back-end team and the front-end team were all the same people um won't they just put more work on the back end team well maybe but not really let me explain so communication structures are the reason the team size matters a larger team size requires more communication structures and that's overhead this follows brook's law so brook's law is an observation about software project management according to which adding manpower to a late software project makes it later so some of you may have read or heard of the book the mythical man month and it basically asserts that under certain conditions adding an additional person actually makes the project take more time versus less time and the reason for that well part of the reason for that is this this combinatorial explosion so a combinator combinatorial explosion is a rapid growth of the complexity of a problem due to how many combinatorics of the problem is affected by the input constraints and the bounds of the problem all right this is what it looks like so you've got linear growth this is ideal you've got exponential growth which we all know is like significantly worse and then literally this is what combinatorial explosion looks like um so yeah it it gets out of hand very very fast and so if you apply that to like brooks law you look at the additional overhead of these uh communication structures going from a team of five to a team of ten um so a five-member team has ten total connections and an eight-person team sorry this is a person team uh has 28 connections so you add three people now you've just about tripled the number of communications uh that you have to have to assess and like you can think of these black dots as people but you can also think of these as you know nodes in your erlang application as well and so you can see there's a lot of noise that's created as you add additional resources there's a lot of overhead just having them talk to each other every day so that's kind of why adding an additional person that's one of the reasons why adding additional people to a team doesn't necessarily make it faster another one is dunbar's number it's kind of uh well let me explain what it is so it's a suggested cognitive limit to the number of people with whom we can maintain social relationships so the strongest form uh happens in a an individual like a team that's working closely together usually that group size is around you know five to ten people they what they've studied in software is that numbers around eight people and this follows the uh if anybody's ever heard of the two pizza rule at amazon it's like no team should be bigger that then can be fed by two pizzas um and uh so that kind of follows that so there's an ideal number of people where if if you grow outside of that it it creates more problems so um teams working as a cohesi cohesive unit perform far far better than collections of individuals so you think about it you uh you know you've probably worked on teams where you don't communicate very well you don't work together you don't maybe don't get along with your co-workers and it's just five people kind of doing work the same kind of work at the same time versus you are on a team where you all really mesh sort of pick up on what each other's what each other are doing et cetera et cetera and that kind of team that cohesive team is going to be a lot more productive so um the any any growth beyond this dunbar's number um it sort of imperils the the viability of software by the team because the trust is starting to break down you're getting to a number that's too large where you don't trust the people that you're working with anymore or you don't have that kind of closeness with them and and so that's that's kind of where things start to break down so your team of five going to to a team of 10 definitely doesn't have that linear growth in productivity because of this and part of that is just the communication structures so i think another reason why you can work faster as a smaller team of back-end developers is has to do with queuing theory and cueing theory is a mathematical study basically of waiting in lines uh and it was the origins of its research were done by agner erlang at the copenhagen telephone exchange company so it's a name that should probably ring a bell no pun intended i really didn't intend it i just know okay all right so uh cueing theory explained in a picture so you've got incoming traffic you've got some q system you know uh and then you've got these service channels where they exit and so if we think of that is like in a store you've got like customers coming in waiting in these queues and then you've got cashiers at the other end and so little's law sort of explains this it's it's the long-term average number l of customers in a stationary system is equal to the long-term average effective arrival rate lambda multiplied by the average time w that a customer spends in the system so that's that's little's law um and so you can think of that as like think of a small store that just has one register and then assume everyone that enters the store will buy at least one thing and if the arrival rate is greater than the exit rate the queue size eventually approaches infinity um and if you think about this in the software world it's kind of like your backlog eventually it will reach infinity because things go in a lot faster than they come out so we've applied this to the software development life cycle think of these as like your jira tickets so you've got jira tickets coming in you've got your sprint commitment is the queue and then you've got developers on the other end are your cashiers that's your bottleneck in the system and so jira tickets arrive faster than they can be completed and you start to get backed up now if you have a back-end team that's delivering to a front-end team now you have two cues um so you've got to deal with bottlenecks at the top queue and then you've got to pass that on and bottlenecks at the second uh at the bottom queue as well so now you've got two bottlenecks to deal with um and so that's kind of my assertion on why i think having just one teamwork on everything should be slightly faster than than having two teams working on it in the case of project x like this explains where our bottleneck was um so in the phoenix project we got rid of that and that bottleneck is gone completely another reason is complexity so we look at project x just the overall complexity of those projects and they have this high cognitive load so one thing that's important is cognitive load affects all of us you might think you're like a wave a 10x engineer but you still this is something that's biological uh and cognitive load will affect you as well so it's just basically the amount of working memory that you have some people do have more some people have a little bit less but everybody has a level um and so you've got basically it's broken down into three parts you've got this intrinsic load which is just how difficult is this subject that i'm working on how difficult you know inherently difficult is this project and then you've got this germain load which is the new information that's coming in that you're having to synthesize to work on it and then you've got extraneous load which is all the other things that that are going on uh related or unrelated to that problem so a complex system is just entirely impossible to reason about you can't no no person in a complex system could understand how all of it works um so by reducing the complexities in the system it decreases the overall cognitive load so we've got like this is a the anatomy of project x i think this is super common uh you've got like this phoenix app and then you've got this api code and then on the front end side you've got some http client that's dealing with the api and then you've got this react app or view app or whatever it is and just so much disparate code there that it creates this really high cognitive load in the overall system and just makes it really hard to reason about now if you look at the anatomy of a live view app got a lot less code um so you know you basically just have the the live view code in the phoenix app and additionally like the front end code is kind of similar to the back end code so the con the context switching is easier the feedback sorry the feedback loops are also smaller as we explain like if you run into something on the front end that needs to be fixed in the back end you're able to iterate through that i think more quickly when i said the the code is similar the uh okay who had slide 87 this is our code slide i hope everybody can read that so if you had 87 bingo um so the the code size is smaller there's less code but also like if you look at the callbacks or the lifecycle of a live view process it looks really similar to gen servers um so you're not doing that context switching of you know going into a react state machine versus how gen servers work or ets works or etc the front end life cycle looks a lot like the back end life cycle so that makes it a little bit easier so the callbacks in a live view app are these ones you know handle call handlecast handle info these are all like lifecycle events that exist in live in gen servers as well and they work the same way so just kind of makes everything easier to understand so some other advantages that we found would be just easier contract testing and integration testing instead of setting up these different proxy servers and stuff everything's kind of running together so doing the contract testing between the ui and the back end is simpler you can do it all in one place or the integration testing all happens in the same code base more flexible resources so everybody on the team can work on all the tickets essentially so you've got a ui ticket you've got an api ticket you've got you know data schema ticket etc everybody's able to sort of move around better we didn't need to hire any javascript developers um which if you're trying to hire anybody right now it's pretty much impossible but javascript develop developers especially um it's hard to find and so that's an advantage as well we had elixir developers um and so we were able to kind of use the resources that were already on staff now to be honest there are some drawbacks there's definitely some disadvantages of doing this one is like the pre-built components so in the javascript world there's a library for everything so you're like hey i need a calendar that you know uses the roman you know julius calendar etc and you know doesn't adhere to leap days or et cetera et cetera and they're like oh yeah i'll just download that one uh that's an npm package in live view you're generally going to be building everything yourself uh which is good or bad but it does mean it's probably going to be a little bit slower lots of breaking changes in live view since it's a zero point release um i know that it's coming up it's at a very much more stable version now but when we were working on this the apis were changing every two to three weeks and so that's you know it's a less mature system so you kind of have to deal with that all right so how did it end up so actually pretty well so we were able to spin up the team really fast uh the product ended up meeting all of our requirements for real time and interactivity uh and with the design we ended up being able to use uh reusable live view components and so that helped a lot and also pulled in tailwind tailwind ui uh which able were able to help us out a lot on the design side of things just the overall kind of look and feel of things all right that's uh that's kind of it so thanks everybody [Applause]

Channel: The Big Elixir

Views: 5,655

Keywords: the_big_elixir, phoenix_framework, liveview, elixir, erlang

Id: oVt6rUuzjU8

Length: 31min 41sec (1901 seconds)

Published: Fri Apr 29 2022