droidcon SF 2018 - Scaling an Android App from 1 to 100 developers with modularization.

everyone my name is I'm an Android engineer Airbnb and I work on the native infrastructure team and today I'm gonna be sharing our story of how Aaron be scaled from an application with just a couple developers and a single module into one with over 200 modules and roughly on the order of a magnitude of 100 developers contributing it to this year so let's start off with a tale of two apps it's back in June of 2011 and there were two Android apps being built the first you might recognize it look a little bit different back then but it was this app where you could rent out a room in your home to a stranger it was being built in San Francisco in an apartment on Rush Street and it was created by three co-founders who came up with this idea when there was a design conference and people needed a place to stay now all the way across the country in North Carolina another app was being developed and let's get the real effect go in here at great visuals you know great sound effects and it was being developed by this really cool guy is high school senior but the crazy thing is when these apps were being built they started off exactly the same way we just you know started a new project in Android studio applied the application plugin but the big difference was the Airbnb app has continued to grow and grow even more whereas spunky's last and the name of my game well it was only ever downloaded by my friends and my parents the Airbnb is organization likewise you know grew a lot and eventually they made the strange choice of hiring the developer of spunky's last stand and gave me this cool drawing there's a you know as you grow there's like this phenomena phenomenon called Dunbar's number which is this fancy term that says that the amount of stable relationships that one person can comprehend is capped at 150 and what organizations do to deal with this is they start to structure themselves so Airbnb broke itself into smaller business units and you can do the exact same thing with your application so you can take code from your application module and modularize it by moving it into a library module and under the hood what's going to happen when you build your application is your library module it's going to have its own manifest file your own resources your own classes your application module will still have all of those but then the build tools is going to merge into a single apk or Android bundle and this talk I have to make a lot of these diagrams so I just want to clarify something when you see something like this we're going to read this as application depends on library and you know again these library modules are our primitives for modularization and Android and the art is you can really do whatever you want with them almost so you know you can have one module depend on many modules you can have a very deep structure the one thing you can't do is have a circular dependency and we're gonna see how we deal with that later but the first rule of module is a ssin is you don't mod your lies if you don't have to so it's something to keep in mind if you're working in a smaller app a lot of the things I'm going to talk about don't even worry about it until it becomes a problem that begs the question how do you know when it becomes a problem so throughout this talk we're going to visit this project structure report card and I've identified a few areas that I believe modularization can help the first are build times so as soon as you add your second module to your project you can turn on this flag in Gradle and this pretty much says you know let me use all the cores on my computer to build my modules in parallel so to see the impact that this can have on your project I created a little dummy project where I just created zuv source files and you know ran builds overnight and see how long it took so you can see that as you add source files your build times are going to scale relatively linearly now when you add your second module it cuts pretty drastically diminishing returns from more modules but definitively mod realizing your code help build times the next big area that modernization helps with is code ownership there's sort of two aspects to this the first is attribution so we've all been there you know some piece of code is not working maybe got assigned some crash well you know what do you do and by default if you're using it we all have this one tool which is get blame and this is really easy to use easy to use you just right-click in your gutter and see who wrote the code but the problem here is it's not durable we've all heard the adage you know what are you gonna do when your team is do you make it it's my boss or you know maybe less morbidly what are you gonna do and they go to the company down the street get blame is not going to work and then you have no idea who to ask so we can move out a layer of abstraction and try to use something else as their unit of attribution so you might consider using file or classes and this has the potential to be really cool you can add annotations directly to your code and maybe build tooling around this but think about how much effort would take to annotate every single class or every single file in your code and farther more like in the event of a reorganization you have to go and update all of these so at Airbnb we actually found that modules make a really good unit of code attribution all you need is one manifest and it to Clank what team owns the code this is really easy to do and in a reorg you just have to update ax you know a handful of modules we've probably all heard of Murphy's Law which is that whatever can go wrong will go wrong I want to go propose an amendment to this which is Murphy's Law of code access and that's that whatever they can access they will access and it will go wrong so to illustrate this example let's say our application is a monolith in samal I'm opening up a coffee shop you know life is good I'm using a coffee grinder yeah what could go wrong well this guy named Murphy decides open a coffee shop and I think we know what Murphy's going to do he's an entrepreneur you know he wants to you know cut corners so he uses my coffee grinder you know serve sneaks into my shop every time I want to order some coffee grinds it up and returns yeah he's doing great but you know I don't love this you know I want to be differentiated so I decided I want to become a bobo shop and i use a building machine and what happens is Murphy now becomes a bobo shop so that's not great and this is where modularization can help we can actually strictly enforce that someone does not use your code without their permission furthermore if you're using Kotlin there is literally a language visibility modifier that operates on this basis and that's the internal modifier so likewise we don't want Murphy to use my Bobo machine we probably don't want them all to know about it either so we can mark that internal and that won't be visible and the final reason that I think you should consider modernisation is if you want to leverage some of the new features with app bundles so in the worlds of applications bigger is definitely not better Google did a study a few years ago that saw that for every six megabytes that your app increases your install conversion drops 1% so that means that like what is the percent that when someone clicks that to download your app that they will successfully download it and launch it it doesn't actually scale completely linearly if you were to go from a hundred megabyte app down to ten megabytes you would actually see your download conversion increased by 30% and the math doesn't actually add up there but it's interesting when I first saw this a few years ago I thought this was like really silly like if I have a 100 megabyte app I'm not gonna be able to make it the ten megabytes but my view on this changes here when Google announced at bundles and the idea there is Google will actually repackage your app and they can highly optimized it in terms of this talk the most important and most relevant part is a concept called dynamic features and dynamic features pre-much says I want to take a few modules in my app I don't want to bundle them with the initial install I wanna make them dynamically available so first let's talk I want to talk about how we might realize that your DB so again we have over 160 modules nothing closer to 200 now and we also have this concept of light belts which I'm going to really dig into later in this talk this allows you to build a small sliver of the application in much less time and you might be like two to five minutes well that's pretty bad my application builds faster than that for contacts the entire Airbnb application on a local dev machine on a you know fresh build could take up to 20 minutes so this is pretty substantial and we jumped over that we learned a lesson so let's see what those are there's no way that I'm going to fully be able to explain how all 160 modules in our app works so let's simplify it down to just two things on the right we have Airbnb experiences this is a product that we launched about two years ago where you can you know learn how to make pasta and then on the left is a traditional home homes product so you know fairly similar pages buy owned by completely different teams so again let's take a step back and to see how our organization structured I actually learned from a talk earlier today that there's even a law aimed after this it's called Conway's law and it's just it's natural you're gonna want to structure your code the way that your organization is structured so what that means in terms of project model modularization is we wanted to modular as a feature before we get into that though it's intuitive that there's going to be some foundation that everything is going to be built on we call this our base module other people call it core and there's a few rules here this is going to be pure infrastructure and what this means is there's absolutely no knowledge about Airbnb and our litmus has to see if that's working is we can ask ourselves could this be open source of all you know this again one of the theories throughout this talk is that every module is gonna be owned by one team as that's our unit of ownership so this is owned by the native infrastructure team and one rule that you'll see throughout this talk is it's incredibly important to keep your base module lean so we do not allow any deprecated code in base so the next layer above base is going to be your features these are again owned by a single team and they encapsulate a single feature and I know that sort of sounds like a tautology so to give it a little bit more flavor there we sort of explained that by whether or not there's a single entry point so if I'm creating a flow with many pages I might not be able to go into page three so each page of that flow doesn't need to be its own feature module but the larger flow should be but generally speaking with odd realization smaller is better and what this you know implies is that a teammate own many feature modules and one tricky thing and one very important rule is feature modules cannot depend on other future modules and we'll see you more on this later and the last layer is some sort of application shell this really should not do anything that interesting so no feature code no infrastructure code the one thing it does do is create a dagger component it's a pretty important job so let's just see if this structure theoretically could achieve the goals that we want with modularization so we have a lot more features and we make some change to the experience module you can see that this is only going to have to recompile a very small amount of code in the app so we can have pretty fast builds with this in terms of code ownership using modules as the primitive for ownership we can mark all these to a team some teams own mode on more than one none that teams can touch other teams code and in a reorg we can just pretty easily remap them and for app bundles we can take some of the features let's say maybe we want to take all the hosting features in the Airbnb app and make them dynamically downloadable and we just move them out like that so overall this looks pretty good but even using this structure as your starting point you're going to immediately run into some challenges the first one that I think everyone will need to deal with is how you how you do navigation so let's start off with a pretty simple example on a listing description page you scroll all the way to the bottom you're gonna see a section called similar listings and this just allows you to see homes or like the one that you're saying at during the same date range so this is code in the same module so launching this for it's pretty easy you can just create an intent for an activity and you can launch it now imagine that instead of showing similar listings we wanted to show nearby experiences and what is really tempting you know if your coding structure like this and you can't access that activity directly is for the homes module to just depend on the experience module you can find that class and it all works the problem is the experience team might decide that they want to show nearby listings and when they try to add that dependency in the homes module you'll get a circular dependency and your build will fail and it only seems fair that it shouldn't be whichever team adds the dependency first that gets to use it the experience team would not be happy if you know they can't do this because the homes team beat them to the punch so what we've created is actually something incredibly simple it's just an intense module it's like a universal navigation directory and it depends on absolutely nothing so this means the only thing that you can give it are primitives like IDs and it returns intents and fragments so the fact that it depends on nothing might be raising some red flags in your mind and you'll see why I mean we're using reflection so that's sort of crazy right like Airbnb he's using reflection for navigation or are they doing it actually turns out to work really well so using that primitive where you can load a class you can start writing code like this that returns an intent or a fragment and I think the power of this model is the fact that the code that you get back is nullable and it's becoming increasingly clear that there are cases valid cases where something that you're navigating to is going to be null especially if you're going to try to start leveraging dynamic features there's a very good chance that what you're having - doesn't exist in your apk and you need to go to the play core library to download it another possibility and we'll dig into this more later is that you're on some sort of debug build that's not building all the code in your application so the page that you might be trying to launch doesn't exist in the flavor that you have so we want to show like a little toast to the developer saying hey download this flavor if you want to go and the last reason is a developer deleted the activity or fragment in the intense module but forgot to delete the references in the code this is bad this is this would be like a mistake in our architecture if we allow this to be a common occurrence luckily this is pretty easy to remedy you can just write a unit test in some sort of module that knows about every other module and if you intended our talk on CI you know that at Airbnb we do not allow you to merge a PR until all the code has passed this tests so it's actually impossible to delete an intent without deleting it from all the places that it's being used so we've never had this problem shipped to production now based on this model so far it seems pretty restrictive like we have a lot of feature modules they're all very small they don't depend on each other we have this base module with only infrastructure you know as a developer there are valid cases that you want to share code and sharing code is a good thing and there are cases where there's just no way around it so consider a wishlist feature and a wish list we want to be able to which lists both homes listings and experiences so what this means is we probably want some sort of wish list manager and if we were to put it in the wish list module this isn't accessible by the other modules for a rule that feature modules cannot depending each other so what's very tempting is to move it to the place that it would be accessible by everyone and that space and this was actually a mistake that we made at Airbnb we allowed this to happen for over a year and what ended up happening is our base module grew and we started having all these weird things that were business specific code in base and this sort of breaks all the reasons that you want a module as your code you have this one room module that's really big that's going to take a long time to build and you have no encapsulation so we introduced the concept of a library module these are similar to feature modules in the sense that they're owned by a single team but the big difference is they have no launch pool features so you can't just create a library module that launches the feature and call it that just because it's a workaround to share code and what we found is that this actually forces better code because when you design a library module you're sort of thinking the sense of this is a an API that I'm sharing with the rest of my team so instead of you know going into some other teams code and writing some switch statements to add my product logic I create that as the library module which is consumed and clearly depended on by other code and I just want to reiterate again how important is to keep your base module lean so from an architecture standpoint it would look pretty weird but your application should be not quite a triangle more like a diamonds like a very thick feature layer and I thin a player and a thin base and live modules help keep that pattern true but even your own infrastructure is going to run into the case where it starts to grow over time so we use guava Airbnb for a while and it was nice but then we discovered Kotlin and Kotlin and pretty much every way that I can think of is just better than guava so we want new features being built to use Kotlin and not guava but remember our rule you know if you can access code people will access it if you can use guava even if you have the option use Kotlin people are going to accidentally use guava so we can use the library modules to solve this problem too we can create a Lib deprecated guava have all the existing modules in the app depend on that but all new modules only depend on base which has just Kotlin so it's a good way to communicate best practices in your organization and the final challenge I want sure today is how to upstream dependencies so to illustrate what I mean we have a concept at Airbnb called a trebuchet this is just a feature playing and idea with a feature flag as you're asking your server whether or not your feature is on or off and we could do this every single time we access one of these feature flags but that means it would be pretty slow we don't want to go to the server just decide whether or not we want to show screen a or screen B ideally we would create one request in our based infrastructure and make it when we start the app but the individual trebuchet keys themselves we want them to be declared with the code that owns them so we want the homes listing trebuchet keys to be declared in the homes module we want the experience keys be in the experience module so now we have this tricky problem where this trebuchet requests that we want to have all the trebuchet keys in the app should be owned by our based infrastructure but we don't know what the keys are in base and we can't just add a dependency on all the future modules because we'll get a circular dependency now we could put this in our shell module but we created some rules that you know there should be no infrastructure in the shell and the shell module should have a single owner this is just one example of why you might need to use a pattern like this but there's many and some of which are feature specific so you know if you were to extrapolate allowing this in the shell module would lead to a module that's very big and owned by many teams so we can use a pattern called a plug-in architecture our base module can declare some sort of interface you know think about it as a socket and this socket is going to be this trebuchet key and the base module also say I know of some interface or I expect to be given some object that is going to contain a set of trebuchet keys I don't know what those trebuchet keys are but I know that I will get some set of them the plugs are the trebuchet keys themselves which are declared in every module and then all the way at the top you can think about it as some sort of operator that tries to find all the modules in your app and construct the actual instance of your set so looking at that a bit more closely inside your application on create you'll want to create an instance of this base graph declared in base you'll want to add all the trebuchet keys from all the modules and then you'll want to pass that interface the implementation that interface back to base and you know there is some code that you have to write here specifically in this set you have to list all the trebuchet keys that you depend on the good news is this is this is not really logic this is just manual work of identifying what modules in your app and creating a set and we use dagger to manage our dependencies at Airbnb and Digger specifically has a tool to allow you to do this with no work so you know talking about dagger in a multi maladroit project could warrant its own talk so I'm gonna keep this at a pretty high level but our rough structure at Airbnb is our app shell contains our Airbnb component this is our route dagger component in terms of scoping this is like our application scope or our singleton scope and every single feature module will have its own sub component and we allow future modules to instruction you scope that sub component to whatever lifecycle makes sense and in dagger that you sort of have this concept of a module and the module is like a glorified factory and it makes sense that you know for the homes listing trebuchet we would want to be declared in some module inside the homes listening module the big problem with this is when we launch the app we don't want to create all the sub components in our application so the homes listing sub component night might not be created which means it's trebuchet you know won't be accessible without creating the component and we can't provide our base module with everything to load so we introduced the concept of app modules these are little plugins separate from your main component that are installed on our main application at launch and this is where you can participate in multi binding which is the term digger uses for this pad and what this means is when we launch our application our sub components can remain unloaded but we are have access to all our trebuchet keys so we actually found this to be a very important pattern and we named the project where we migrated it all of our modules to follow this pattern the unified dagger system and specifically what that code looks like roughly and the purpose of this code again was for multi binding is some base module declares that it expects some set of items trebuchet keys in this case and that is achieved just through adding I multiply ins annotation inside a feature module every single feature module has a dagger file that looks like this now you have the sub component that you can do whatever you want with install it whatever modules you want you own this but you also have this app module and this app graph and these are going to be installed on our root dagger component and again this is where you would actually provide the instance of your trebuchet keys and then the only code that goes in our app shell is this little configuration where we list all the app modules that we're going to use we're gonna list all the graphs and the graphs are going to what allow you to expose the Builder that then creates the sub components and that's all you have to do now what this does mean is we are building one giant component with little pieces from every module but we don't expose that to a given module because a given module only knows what graph it uses and our base infrastructure will just return a little slice of that so we still have all the the goodness that comes from proper code encapsulation so we've come a long way so far we started from this single monolithic application we've broken it down into feature layers libraries our core infrastructure and our navigation library but even with a structure like this it's not going to maintain itself and people you know it's complicated like it's already taken me 26 minutes to get this far people are mean they're just not gonna member to do it so you have to build tooling that's going to maintain your structure so a given module actually has a lot of boilerplate that has this trebuchet key file that I just talked about this dagger file that I explained and of course their own build file and resources and as Android developers you know we get really excited when we see you know patterns like this with duplication and they're like you know I'm gonna go write an IntelliJ again and this is gonna be great I'm gonna give everyone this plugin and they're gonna click create airbnb module and life is good I actually proposed doing something much simpler and that's just using a scripting language so in Ruby if we wanted to create this trebuchet key file it's really just a string replacement we just need to give this file a name this is all the Ruby code that you need I think we all can understand this with just a couple minutes and we can ship this very easily to all our developers you can just give them a script they have one folder in their project where they know to look for all their scripts and so distribution is is pretty easy too I want to conclude today taking a look at Airbnb light and this is this concept that I've been hinting about allowing a developer to build small slices of an application so if we were to continue looking at our clean build times as we scale the number of modules there comes a point where our additional mod realization doesn't really help your build time anymore so this grand project that I've been talking about sort of fails at one of its core missions but what is interesting is if you were to somehow reduce the number of files you were building from you know in half your build time would actually also drop in half and we can leverage the structure be built so far to do that and under the hood what we're actually leveraging our Gradle or Android flavors so if you were to go to the Android documentation website and try to research what a flavor is you're going to see this example of creating a demo and a full app and the reason I think this site is that if you target AP twenty-one your bills will be much faster but maybe you want to ship to more more api's so you can just create a demo flavor and this will open up a Gradle command that builds it with that configuration but flavors are much more powerful you can actually have flavors depend on different code so if we go back to our app structure if we just introduce one more layer a flavor layer we can configure which modules we include in the app so a home's flavor would only include the homes module whereas an experienced flavor the experience module and to keep things consistent our full flavor includes all the modules in our app and going back to build times something I want to remind you is another way to you know the label this X access instead of hundreds of source files you could just call this time you know your application is going to grow over time and the big difference with light flavors is the amount of code that you're building in a light flavor does not grow over time I mean a little bit but relatively speaking it's going to be consistently fast so right now some of our light builds are up to eighty percent faster than our full build and that percentage is only going to increase so for light flavors to be successful Airbnb we wanted a few things we it needs to be easy to create a light flavor you know everyone here like maybe has a few flavors in their applications that you spend a lot of time creating but we have these very small feature modules and we want it to be just a you know matter of minutes so you can select which ones you want and have a new flavor and this is actually easier than you think to achieve you just have a developer say what flavor module they depend on in some sort of like configuration file and then it just takes a couple of Gradle for loops so this one we're gonna iterate across all the flavors that we've declared create a new flavor depending on if the developers said they want this to be installed as a separate application we might add some sort of flavor suffix to the application ID and then the last step is one last or loop where we actually install the flavor module so in the case of flavor homes we will see that this adds the flavor code which then looks at that modules build file and then adds all the modules it needs and again the only responsibility of your app shell is to create that dagger component and it turns out that all the information you need to generate that dagger component is contained in this dependency block so a developer doesn't need to even do anything to set up to Aegir so flavors introduce this possibility that some of the code that you need in your application might be missing and if this was a nuisance all the time developers would hate using flavors and they would start building the full app so our structure actually allows us to work pretty well in terms of dagger if I'm missing modules that have trebuchet keys those just are not going to be included in the multi bound set and that's okay because we we're never going to use the code that looked at them anyways so that set just changes in terms of navigation if I'm in the experiences module and I try to launch a home's listing we have this nullable intent system and we just pop up a toaster developer that this module is not installed but what's really exciting is that flavors can be more than just faster you can make developing on a flavor a better experience so the first example here is creating a custom launcher so inside your flavor module you can write code that's only available for you as a developer and in this case this is what happens when you launch our homes listing module and this has a bunch of interesting things it's like super easy you can jump into a page for a listing without having to navigate through the normal flow in the app or even jump into a page that shows like a translated listing so you can take all the things that you want to test when you build a feature and build out a launcher to handle all of those and the last little tip I want to leave you with is module unloading so when you start building tons and tons of Jules and have tons of code IntelliJ is going to slow down it's going to be indexing everything you're just me frustrated because you're not even able to open up chrome while that's doing this so what you can do is use this structure that we've built to unload all the code that you don't need from your flavor so what I just did right there is I told IntelliJ to unload all the modules in my application I'm then searching for the experiences flavor that I want to load and sort through the same dependency process that we just looked at IntelliJ will just load the modules that actually needs so you can see when this finishes the vast majority of the modules on the left-hand side of this page are gonna go red and that means that they were unloaded so this this literally when you when you use this for the first time it feels like you got a new computer so I just want to give a shout out to Gabriel Peale who did a lot of the initial work on setting up Airbnb light and he's here today at this conference and the stock and just closing you know it's a really interesting time at Airbnb we're going through this you know modularization process you may have heard from Gabe's talk earlier that we have this framework called Mavericks of using crawler features epoxy which has existed for a couple years but it's been pairing really well with Mavericks we're just our ting to look at graph QL and server driven UI so I'll tell you at they have a time for a few questions but I'll be hanging around and if you want to ask me about any of these things feel free thank you [Applause] sure so the question is I introduced this concept of you know string based navigation and why would we not just use app links or deep linking or something that we already probably have in existing the code base that's a great question so we you know do you have a deep link dispatch from Airbnb which is the library specifically for deep linking so we you know leverage deep linking FLE but if you go back and look we're offering behavior that's more than just that we allow you to both load activities and fragments and Airbnb is actually a primarily fragment based application and furthermore I only showed you like a little snippet of what we're allowing you to do but there are things that we want to enable that is more powerful from just deep linking such as you know given a fragment we may want to be able to tell the navigation system to give that fragment a heads up that we might want to be loading it soon and it can preload itself so it's just mostly for flexibility but you're right they're all pretty similar and you know one of our goals is like a page that we can't access with our routing system should be pretty easily accessible from deep links as well yep that's a really great so for the sake of simplicity I completely left out our UI system in this structure so we have a whole set of modules specifically for and sorry I think I forgot to repeat the question the question was where do we keep resources in the base module or somewhere else so again we we have a completely separate structure that looks very much like what you saw today so we have team component libraries using our design language system and those libraries have cases where they need to share code so they have library modules and you can sort of imagine if you were to imagine what this looks like from a schematic perspective we sort of have a duality for both library modules that we saw today and these you know component libraries yeah the question was can I give some examples of what type of infrastructure code is in our base modules so it I mean the one that you saw today with trebuchet requests is one but besides that when we were incubating maverick's like that type of code would belong in our base module and everything from like experimentation to like base logging frameworks and we also have some base classes for our activities and fragments that we put into our base module yes so the question is my answer which is we and I'm trying to rember he said I wish I could give you a microphone but yeah when future modules become too big we break them apart and then anytime that we need to share code that was previously being shared we use this plug-in pattern that we went over today so when we do that we do not have to move code into base necessarily we create library modules and I guess one thing I didn't go into is we do allow library modules to depend on other library modules to solve that case that you're describing Oh Gradle yes it so the question is is great I'll be coming too slow what what part of Gradle specifically are you yeah so I mean as I mentioned like it takes 20 minutes to build our app with Gradle Artful app so that's why we introduced these light flavors which build very fast cool um I can just take additional questions up here at this age but thank you for coming and guide today [Applause] you

Channel: droidcon SF

Views: 6,582

Keywords:

Id: jrnhIgFzgns

Length: 38min 59sec (2339 seconds)

Published: Sat Dec 01 2018