From Monorail to Monorepo: Airbnb's journey into microservices - GitHub Universe 2018

hey everyone welcome I see a lot of new faces here today I will be sharing kind of a journey with you guys I will be sharing airbnb journey from mole Arel into a mono repo and this is not just my journey as a developer this is kind of the journey of all of the software engineers at Airbnb and I'm very excited to kind of share with you guys our findings and kind of our progress in migrating from a really big monolithic application into like a service-oriented architecture so before we get started my name is Yance Vander haha I could probably give a different talk on like the intricacies of having a Dutch last name in the US but you can see the spelling up there I'm Belgian I moved to San Francisco in 2013 and I actually was originally a front-end engineer and then I somehow ended up on an infrastructure team at Airbnb if you want to follow me on all the relevant social media especially github you can kind of find my tags down there on the slide so in 2017 after being here in the United States for a couple years I decided to join Airbnb and I can now probably proudly call Airbnb my home and as I mentioned before I kind of transitioned from being a front-end developer into an infrastructure team and it's been an interesting journey so I'm on the deploy infra and you can kind of see the org chart on the right side there so deploy and Fries under the infrastructure team and we're more of a supporting kind of team we don't develop like Airbnb products I don't work on Airbnb comm I support other software engineers my build to so that other software engineers can feel productive and my team actually used to be known as developer happiness and that name really says it all my day-to-day job is really making other engineers in the office as happy as possible as productive as possible so my team's customers are engineers even though I'm on deploy infra I don't actually deploy code into production I merely provide tools for engine to be able to ship the code into production we don't actually go and do like release management or anything like that everyone can kind of ship their own code into production at Airbnb so let's start this journey with kind of like humble beginnings maybe you're looking at Nate Nate as one of the three founders of Airbnb so nate is kind of the more technical founder of Airbnb he was kind of the original software engineer at Airbnb you look you're looking at the very first commit in our code base this is like history right there it actually wasn't good back then it was still SVN we transitioned to get a little bit later and Nate at the time it was 2008-2009 Ruby on Rails was all the hotness so Nate decided to do a rails on it and create a Ruby on Rails application and you know we're still here ten years later talking about like untangling this Ruby on Rails application so it's been quite a journey to kind of like untangle this application so a Ruby on Rails application is called monorail so monorail mono monolithic rail Ruby on Rails application and so it's been there for ten years and we are still kind of like working on like untangling it and making it into much smaller services so what does it look like so it's a Ruby on Rails codebase the front-end in the early days was backbone I'm sure there's still quite a bit of backbone code out there in monorail but in 2015 we kind of jumped on the new hotness we moved to react Redux and so the backends we have some Java services that don't live in monorail but most of our back-end actually also lives in monorail so there's a lot of like Ruby logic that does the Airbnb backend all the way from payments reservations all of that and that's that was actually all done in this small rail application and we did a lot of stuff in like asynchronous jobs let's say sending an email confirming a reservation and all of that so we used rescue rescue is kind of like Ruby on Rails is messaging queue where you can kind of in queue jobs it's kind of a job worker and so you might wonder like how do you deploy this like really big baller l application it's actually quite a quite an interesting story on how we deploy that so you're looking at the life of a poor request at Airbnb or at least up until maybe two years ago and this is a monorail pour request so we kind of have these steps that a developer goes through on their day to day life so it kind of starts with this CI and code review so this is actually before you merge your guest FPR into master we kind of call this the inner loop like developers are like making changes that are committing the code but they're not merging it into production yet they're not ready to ship it this is like the inner development loop and I would say this is about 80% of our developers day-to-day life is like kind of spent in this like inner loop so we do a bunch of status checks before you merge your poor request we do linting one of our linters is actually really well known or javascript es lint configuration we run unit tests we check for code coverage you know all the good stuff to make sure that whatever you merge into master isn't like completely broken code and once you kind of get through that phase your PO request gets approved we have a code review process as well your pull request is actually emergent to master and then we rerun CI again why don't we rerun CI again so mo Lorella is a really huge repo so while you were running CI on your pull request other people might have first code in so we want to make sure that whatever you merge is actually tested again so we run the whole test suite again we run all of the linkage again so we just want to make sure that whatever you did on your PR other pull requests haven't kind of affected the changes that you made in in master and so eventually we deploy those changes to next so next is kind of our internal staging environments it's basically a complete copy of Airbnb but it doesn't take any production traffic and so that's where developers will kind of go in and kind of test at changes make sure that whatever they did is actually good to go into production and then eventually changes go into production the live and visible to all users with a caveat a lot of our developers actually put the changes behind a feature flag and so they'll actually launch it to maybe a subset of 10% of all users or 20% or something like that and they'll eventually roll it out to all users so you kind of see this pattern of most developers putting their changes behind a feature flag and so at Airbnb we have this kind of unique system in the industry of democratic deploys so what does it mean as I mentioned before deploy infra just provides the tools we don't actually go and sit there and kind of deploy code for engineers we're pretty good at like continuous delivery so that means every engineer that comes in to Airbnb can deploy into production on day one even an intern can deploy two airbnb.com as soon as they get their laptop basically that also means that every engineer has great power and great responsibility so that means we don't really have a dedicated QA team for the airbnb.com website so every engineer kind of goes in tested their own changes and then we roll it out into production so we didn't have a release management team every engineer is kind of their own release manager every engineer is kind of their own QA team and so you'd the delay between like merging your PR and two-masted and shipping your code into production is usually only about an hour and people ship multiple changes a day we ship about 200 changes a day so we're like we have this really high velocity of changes going on to the airbnb.com website and how do we do that well my team built this internal tool called deploy board you're looking at a screen shot of deploy board right here and it's kind of our own internal web app it's also building Ruby on Rails and the deploy pod UI shows you all of the commits that are coming into the master branch and it shows you CI as well so you can kind of see all of the green checks which means CI has passed and if you look good you can actually see the end the yellow and green pea so the yellow n is next so you can see which code is running on the next environment at that point in time and you can also see the P so what we do is we deploy a batch of changes to nexts and then we kind of allow people to test their changes and then we deploy production to whatever next is that and we kind of do this cycle all day long we recently also added a canary target to that but it's similar we basically do next and then production next production we do this like 10 12 times a day we ship about 200 changes to mo larell daily and you can see what a deployer looks like and deploy board so this is a while ago background we only had 1,800 servers I believe we're at 2500 so you can kind of see when someone deploys you can kind of see all of these boxes so these are actual ec2 instances that are being deployed to so deploy board is connected via WebSocket and it kind of shows you this real-time update of what we are deploying right now and it gives developers kind of the tools to deploy into production role back kind of like mitigate issues and stuff like that so all of that is built in to deploy board it even has some monitoring built in to make sure no disaster is happening when you're doing a deploy and that worked well for a while and around the time I joined we were starting to see like a pretty rapid growth of our code base so as you all might know Airbnb has been around for about a decade now and so in the early days when they spun up this Ruby on Rails application had scaled pretty well we had a couple of like listings in the United States everything was fine we could like just horizontally scale a Ruby on Rails application and we were fine for for a couple years and then eventually you know like 2017 a year ago Airbnb is this like global brand everyone has heard about Airbnb you can use it all across the world it's really big there's four million total homes that you can book on Airbnb there's about 65,000 cities one 191 countries there's only maybe one or two countries where you cannot book an Airbnb so we've kind of become this global brand but we're still on this like small on this like Ruby on Rails application so it became problematic just to give you some context we have about twelve hundred and fifty four unique contributors to monorail last time I checked so if you were to take everyone in this room multiply by five and then put everyone on the same Ruby on Rails application and commit every day and that's kind of the situation that we're dealing with so it was pretty crazy so we shipped 220 PRS a day in master we have about thirty thousand database columns that active record kind of manages to support a Ruby on Rails application and up until like maybe a couple weeks ago we had merged a hundred and fifty five thousand PRS just in monorail so it's it's a beast it's such a beast that one I wanted to get bitter information and github I was looking at this for about 20 minutes but eventually it showed up it just took a while nobody was brave enough to actually look at that so it probably had to do some caching so you can actually see the growth so you're looking at like number of contributions a day so at some point we were hitting eight hundreds and you can actually see it going back down and I'm here to explain you why we're how we got there so that we actually have less changes to mole arel nowadays so you're looking at some pretty big numbers three hundred and seventy thousand commits it's probably the biggest Ruby on Rails application out there by far and at first we were like you know maybe we don't need this microservices thing maybe we can just kind of build tools around molla rail and we can cut we can keep scaling that way it's a huge investment to move into micro services we were growing the company at the time we didn't really have the manpower to kind of shift into micro services so we thought you know why don't we build some tools around this we can do we can make it happen so the first two we built a scram subscribe is kind of an automated automated rollback incident prevention tool and it's kind of an internal tool to Airbnb it's kind of like a canary tool so every time a production deploy happens we consume the errors that are coming in so we're using a tool called sentry if you're not familiar with sentry it catches an exception and then kind of bubbles it up to your back-end and then you get all the information about the exception so wasn't as a deploy is going out we're basically running like two different versions of our code during the deploy so we can actually compare the exceptions coming in for the new version versus the old version and so we kind of use some clever anomaly detection algorithms to see if an exception hasn't been there before and so if an exception hasn't been there before and like the anomaly detection detects that it's like big enough of an anomaly then we can like safely assume that it's someone somewhere introduced a regression and scram will actually roll back the P R and so scrams been really good it catches most of the errors before a human eye can even see them on a graph or something with that like scram will immediately roll back it's it's really good so this is what scram looks like so as you can see the white as a char that we were running and at some point we initiated the deploy and so the light gray is a new deploy going out and so you can see all of the airs coming in on the left side if there's an error that's new then scram will flag it in kind of an orange color and it will initiate a rollback and that worked well scram definitely got us another 1 or 2 years out of monorail but eventually we had to build another tool a tool that didn't make my team very popular it's called merge queue so you see this picture with zombies in the backgrounds so what developers are kind of like zombies they kind of want to ship their code as as fast as possible but everyone wants to ship their code before they go home so we kind of saw this pattern where everyone trying to ship their code at 4 p.m. and so people merge 6070 PRS at 4:00 p.m. one person broke it and we had no idea what was going on so we had to like introduce a queue I'm kind of an orderly queue so that we could have some order in this system so this is what it looked like without merge queue everyone's kind of scrambling together everyone's trying to get their changes into production and so what we did with merge queue is we had like an orderly queue it's kind of first-come first-serve and so we built this functionality just to make life a little more organized make life for Mahler al deploys a little little easier and so how does it work so merge queue is actually integrated with github so as soon as your pull request gets approved you get a message on slack and you can type a rocket emoji on your PR which basically means like I'm good to go you can also write the NQ keyword and so what happens you can see it here in this animated gif I'm basically typing rocket and then my PR gets and queued I get a message in slack giving me an ETA of about when my mind might change will be merged and then once my change gets merged into production I get a ping back on slack and that says like hey you should test your changes on next and then I could test my changes so this is great because it used to be kind of a whack-a-mole everyone was just trying to hit that merge button 50 changes might not at once so we can kind of use merge queue to enforce batches so I showed you the life of a pull request earlier so we added this extra step where you're kind of sitting in the merge queue before your PR gets changed the great thing about merge queue is also it's non-destructive if you are in the merge queue and you need to run into a meeting you can just DQ again come back later i'm--you're for request hasn't been merged yet and we even have this UI and deploy board that shows you this date of the queue so you can kind of dig in all - and into all the PRS that are currently in the queue we even have this mechanism that allows you to flag your PRS ready did basically telling the telling the merge queue that you're at your computer you know afk because we don't want to merge changes where the developer is another round so if you're not around we kind of recycle you at the top of the queue and then eventually DQ you if you're afk for too long and so we built this whole queuing system just so we could deal with monorail and things were okay with merge queue and scram we definitely got like maybe one or two years out of molar l but things weren't okay for too long and then we started seeing all of these graphs we started seeing people in slack panicking we saw a lot of incidents we were not in a good place and so we had to make a more more drastic change meanwhile this is kind of my team we were just like trying to keep things alive like alerts were going off left and right everyone was kind of panicking we did an internal survey to ask people like hey how satisfied are you with like the deploy process at Airbnb and only maybe a very small maybe 10 ish percent of people were actually happy with the deploy process everyone was like extremely dissatisfied or dissatisfied so those are not the kind of numbers you want to see from a team called the developer happiness the developers were clearly clearly not happy so we had to do something about that so we did it we actually did move into the release management I know I know I said you know democratic deploys we still do democratic deploys but it didn't work for Moeller l it was just too big we couldn't we couldn't do it so we introduced release management's so release management is kind of a team that kind of keeps this monorail on track and they basically go into the merge queue they handpick a batch of 20 changes they merge those in and they kind of Shepherd the deploy so we actually ended up building some tooling in to deploy board to kind of make sure that the release managers could kind of Shepherd the deploy themselves and this was actually a pretty dramatic cultural shift for Airbnb we had some like please senior software engineers who really liked this like continuous delivery Palen they really liked shipping their own changes and moving into a release management system definitely was kind of a kind of a big change for for a company but we had to do it that was we had to do it kind of to keep the lights on we only do the lease management during business hours so if someone was to ship it change at 1:00 a.m. they're on their own it's back to democratic deploys so there's only a release manager on call during business hours it's a volunteer rotation we didn't want to like have someone sit there at 2:00 a.m. deploying changes that doesn't sound like a great experience either so we kind of found a middle ground there and the results were promising we were able to merge more changes our velocity went up we merged more pr's a day and as you can see from the second part chart people were much more satisfied even though people said like you know democratically deploys is all the hotness and the greatness they like the idea of it but it didn't scale well for a monolithic application like monorail but obviously we needed to do something else just like our company is kind of moving into like different businesses for example trips our infrastructure also needed a more long-term vision a more sustainable plan like what do we do with this monorail situation we double the engineering team every year how do we not run into this huge fire where nobody can deploy changes anymore so we came up with this long term vision called monorail no we didn't want to have a mall around anymore and so we said you know a larger solution is needed will kind of have to split this monorail in two separate pieces so we're actually doing a code freeze so starting 2019 so that's coming up soon we actually will not allow developers to contribute codes into mall RL anymore unless it's like maintenance code you will not be allowed to add new features into monorail and this is really to kind of force people to adopt this micro services palette and so we're investing very heavily and services oriented architecture but we have to make sure that we do this carefully we have to make sure that we don't impact company growth in the meantime you know the company is growing we're adding more engineers we're spinning out new your businesses left and right so we want to make sure that we have a sustainable plan that doesn't impact company growth in the mean time so how do we go from monorail and to SOA so this is monorail and so what happens is we're moving into this future now where we have molar el Hyperloop and treehouse I'll talk about Hyperloop and treehouse and a little bit and then eventually once we get completely rid of monorail we'll just have two mono repos Hyperloop and treehouse so let's talk about these two these two different multiples one might ask why do you have two mono repos we decided to kind of split hermano repos based on language so we have a front end mono repo Hyperloop and then we have a back-end mono repo in Java which is three house so we can kind of build tooling around these two different mono repos we found that that scale pretty well for us so what is Hyperloop so Hyperloop is our front at mono repo and all of our front end codes all of our ER be Ruby on Rails code is actually moving in into no Jess so as you may or may not know know we have this kind of server-side react strain work called hypernova that we open-source and so it's based on hypernova and so Hyperloop basically does json over HTTP and so it basically allows you to package a smaller front end components with an OG I surrender and kind of like host it by itself and so you can do independent deploys so if you look at Airbnb dot-com you can think about the different pages so every one of those pages is actually its own Hyperloop that kind of uses this framework and they all live in that same Hyperloop mono repo so we can put shared javascript libraries and stuff in that repo in and you can just kind of depend on them so this is what IBM B's front-ends kind of looks like nowadays so you can see that the blue part is just one of the many hyper loops and so you can put we can create more loops based on businesses so if a B&B ever wants to dive into a new business we can just spin up more Hyperloop applications and then we kind of have this layer of like shared services what we call like kind of a middleware layer so we have some of those services for example hundred two as our authentication service momento those sessions Krakken which is our API gateway so we kind of have a few dozen services that kind of live in this network middleware layer so basically from that developers don't have to worry about those kind of things like authentication and sessions and stuff like that so that's our front end mono repo obviously a lot of our code also lived on the Ruby side of a Ruby on Rails application we had a lot of back-end code in there so what do we do with that we moved it into another mono repo called tree house and so why tree house so story goes that our most popular Airbnb listing is a treehouse that Airbnb listing is on the island of Java or mono repo holds Java code so that's why we call it tree house that's what the legend goes so why don't we choose Java debatable but it's pretty close to C++ and performance it's definitely faster than you know Ruby and also static typing we found that we really got a lot of benefits from having like static type checking analysis on the back end and a lot of our treehouse applications are based on this open source framework called drop wizards so drop is it this kind of a library that allows you to make these like micro restful java applications and so it's not the same as a hyper loop so hyper loop communicates JSON over HTTP but treehouse can't kind of holds our internal back-end services so we don't communicate with JSON we tried JSON for awhile but we figured out that it didn't work it wasn't type checked so you could kind of send any data to any service and then you have to do some validation on the server side to make sure you get the appropriate the appropriate data and the didn't scale well so we changed our data format for enter service communication into drifts and so drift is basically this binary format that allows you to do schema validation it's standardized it's open source standard an apache license and we actually built our own compiler on top of drifts so internally we have this project called sparse um so sparse um as a C++ compiler that can compile drift to make it even more efficient and even faster and so how do these services talk to each other they talk to each other by doing RPC calls over TCP so there's no JSON over HTTP between services and so basically this also allows us to do cross language communications if we ever wanted to introduce a cutland service or a go service we can just kind of talk over our PC using drifts which is kind of this language that all of these programming languages can share and how do we do the communication so anything that's taking more than a few milliseconds is a synchronous communication you don't expect a result back let's say that I wanted to send an email I can just say like send an email but I don't expect the backend service to send this email immediately so we use Kafka's or a message queue so you would basically just in queue a message just as like send an email and then we have all these workers that are kind of like consuming these email jobs and sending them out so you're not kind of waiting around for that job to complete so anything that's asynchronous anything that takes more than a few milliseconds kind of goes into our message queue so we use jitney which is our internal wrapper around Kafka and we found Kafka to be kind of a beast to manage so we have to write a client called jitney and that's all fine and dandy you can have this like SOA world but if you don't have great tooling around it it'll be a disaster so I just wanted to talk a little bit about the tools that we build to kind of deal with this Hyperloop there's treehouse and of course like the pieces that we still have left of morale and so we build some pretty interesting tooling to kind of deal with this SOA worlds so the first thing we did is we build a tool called cube gen so we're actually moving all of our back-end services into kubernetes so basically if you want to make a new micro service you just do cube gen Ruby and it spits out a Ruby on Rails application integrated with all of our services out of the box it has jitney it has all of the gems so you can make a service within seconds and it comes with all of the tools inside we found that it was really good and it gives a lot of people like good velocity to spin up new services we wanted to make sure that if we're telling people to make micro services that we had good tools to actually create those services another part is like service configuration it used to be where if you wanted to spin up a service you have to go into the AWS console you have to database you have to create an s3 bucket you have to set up a github repo so you're kind of all over the place setting up dashboards and all of that so we created this project internally called OneTouch and so basically with a single touch you should be able to change not only your applications code but also your infrastructure so any of our internal applications has an underscore info folder and so you can put a llamo file in there for example deploy Bordeaux and it will kind of describe what the deploy process for your application looks like you can put data doc in there anything that's kind of infrastructure lives in this folder if you want to add an s3 bucket you would configure it in that infraorder so once you check in your changes your infrastructure will kind of grow together with your codebase and it's like just all uniform it's all in just one application it's all easy in one spot and we actually allow developers to kind of extends this underscore infra concept so if someone wanted to like add Redis support to it they can kind of extend this concept so one touch is really good because it allows you to kind of like put all your configuration in the same repo as your code and so we're moving to infrastructure as code so basically developers no longer have to care about hardware so we're moving into kubernetes we're packaging all of these micro services into docker containers we're scheduling those on a kubernetes cluster so if you're a product developer you never even interact with any of the instances and the under that and so we put some sidecar containers next to your base container for some shared logic like service discovery and stuff like that obviously we could manage all of our infrastructure using kubernetes we really rely pretty heavily on AWS so for example Amazon s3 I am roles load balancers those are managed using terraform fun facts we even think about managing or github Enterprise instance I'm using Terra forum to kind of set up all the repos and the organizations and stuff as infrastructure as code we obviously also need like good monitoring tools so we built this new tool called watch point we realized we had a lot of like different monitoring to we were using data doc we were using New Relic we were using Cabana but they were kind of all over the place and so we couldn't really find a tool that kind of aggregates all of them so we decided to build it in-house and so what point is kind of this observability tool that allows you to kind of observe the state of the world and an SOA world I'm so it's really built for a service art oriented architecture from the ground up and it has this like service graph so you can actually see which services talk to other services because if you're dealing with like thousands of micro services they're all communicating with each other and you can do like things like cross service tracking so if you're sending a request you can kind of trace it all the way from our load bouncer into our front-end and then eventually into all of the three health services and then back so you can kind of trace a request through our infrastructure we also kind of aggregate our logs and Kabana and we add links to Cabana in watch Point and so it a grits from data dock a new relic and also like our in-house aggregation service called up shop so we basically use watch point as our kind of visualization tool to see it's kind of our pane of glass into this micro services worlds and you can even add s loz so an SLO as a service level objective so basically we basically tell people to set up these sort of contracts if I'm building a micro service I need to have a contract that says like okay I'm promising my customers 99 percent uptime if you're building an email service and it's unreliable we need to be held accountable so we're basically tracking all of these as silos in watch point as well so just to give me and my team a pat on the back this is the SLO for deploy board so you can see we're at 100% uptime so we're doing pretty well and so we're tracking all of these SL O's in watch point and you'll get an email if you're not kind of breaking that SLO so this basically allows us to kind of enforce these contracts in this micro-services world if we're talking about thousands of services we need a team to be responsible we need some kind of like contract between service owners and last but not least we're also moving in to deploy pipelines so it used to be where anyone can kind of deploy any piece of code to any environments right away so you could basically grab a piece of code and deploy it into production without deploying it to staging first so we wanted to kind of create a dependency graph that you kind of have to traverse and that dependency graph basically is a story of how your code lands into production and so it consists of stages and every stage consists of like steps and then every step has tasks and every tasks have operations so you can kind of describe your whole deploy process as this kind of like a dependency graph you can even mass pipelines into other pipelines and so this is really useful let's say for example that you wanted to spin up a micro servers and you wanted to deploy it into staging down in Canary and then only when certain canary metrics are good then you want to deploy into production and then you want to do a database migration and for a developer to remember all of these steps is just not doable so just having all of these steps kind of written out as a deploy pipeline and a llamó configuration is just a very useful tool and so to do that we're actually moving into spinnaker as some of you may or may not know spinnaker is kind of this open source continuous delivery tool it's built in-house by Netflix they open sourced it Google used it for ways and other companies are kind of also jumping on that so with Airbnb we're also looking at moving into spinnaker in the future and so you can see a very basic deploy pipeline right there so we're basically deploying our OneTouch config we're waiting a little bit we're deploying to staging we're waiting a little bit longer and we're deploying into production that's just a very basic pipeline but you can create this really intricate dependencies for example if you had three services and you wanted to wait for another service to come back online you can kind of link pipelines together so spinnaker is really the future for us for deploys is really this powerful tool for us to kind of ship code into production so in conclusion so lessons learned for us it's been a long journey we've been dealing with this moral mono repo for a very long time so we learned a lot of lessons along the way I would still say if you're a new company don't bother too much with coming up with this fancy service-oriented architecture focus on your business first make sure you have customers make sure you can actually scale your business first are we angry at Nate for spinning up this Ruby on Rails application and kind of like giving us this year-long migration no we're not he did what he had to do he was able to make a B&B into a sustainable business he was able to you know make money and like he was actually able to make sure that we are still here 10 years later so you want to think about service-oriented architecture and later why and my personal opinion service-oriented architecture requires a lot of expertise you have to have a lot of like really good infrastructure engineers and if you're a small start-up you might not even have like a dedicated infrastructure engineer and so like setting up all these tools it'll probably be like 80% of your workday so don't bother just make a quick and dirty rails application and then take it apart later I also want to emphasize automation anything that can be automated should be automated let's go back to some examples for example deploy pipelines like people have these like really intricate deploy processes where they're like spin up a load balancer they'll spin up in s3 bucket and then they'll redirect traffic and they had these like documents where they would basically list all the steps and then they would use like checkmarks to go through all these steps you can actually automate that by using a tool like spinnaker and you also want to automate anything that like interacts with your infrastructure so use tools like terraform to spin up s3 buckets rather than like manually going into the s3 console that'll save you a lot of time that'll also give you kind of like a backlog of what happens so to come back to that point immutable infrastructure you really want to make sure that if a disaster happens you can kind of grab this bomb repository that holds all of the configuration of an entire company and you can basically spin up your entire data center by just looking at your terraform and kubernetes config so immutable infrastructure is really powerful concept that basically keeps track of all the changes to your infrastructure as if they were code it basically allows you to think of your infrastructure as source code and it's a really powerful concept let's say that you had a point to server out there and someone installed like an Apache on it and it wasn't tracked you wouldn't really know how to reproduce that server if you have a mutable infrastructure you can always reproduce any piece of infrastructure just like you can recompile any piece of code another important point is people always talk about s away from backends but we took it a step further we actually also did s away for front-end so we came up with this kind of Hyperloop concept to build like smaller independent independently deployable front-end applications about 50% of our developers are like front-end developers so we wanted to make sure that even though we're like splitting out all of the Ruby logic out of monorail we want to make sure that they also had a good story in their day-to-day work and that they could deploy changes just as quickly as a back-end engineer and last but not least you need really good monitoring tools especially if you're moving from a monolith into an SOA monitoring a monolith is like pretty straightforward you just have one application and you just make sure that it's up and you can kind of build your tooling around that but suddenly you're find yourself in this like microservices world and now you have dozens hundreds maybe even thousands of services that are communicating with each other and something goes wrong you don't know where it went wrong so having really good tools that kind of allow you to visualize I visualize how a request kind of travels through all these dependencies you really need good monitoring tools before you embark in this journey of micro services if you want to ask me some questions just feel free to hit me up after this talk and I'm just happy to share this journey with you guys I hope that you've learned something today and that you can kind of take what we learned at Airbnb back into your own companies and as you embark into this journey in micro services maybe learn a lesson or two for us from us thank you [Applause] [Music]

Channel: GitHub

Views: 7,026

Keywords: git, github, github universe, github universe 2018, collaboration, programming, version control, open source, software development, octocat, innersource, enterprise software, building the future, airbnb, monorepo, monorail, microservices, microservices architecture, airbnb engineering

Id: 47VOcGGm6aU

Length: 37min 33sec (2253 seconds)

Published: Fri Nov 09 2018