droidcon NYC 2017 - Android Internals for Developers

today I'm here to talk to you about the Android operating system and what is actually managing our application so the first thing that people usually talk about when they talk about Android is they're saying Android is Linux or Android is based on Linux and Linux is an open-source operating system that works on computers and Android took a lot from it and is based on it it took stuff like the kernel and a lot of the core concepts that Linux users and uses them on Android as well but it's not purely just another flavor of Linux or another Linux distribution it's really only based on it and made some extensive changes most of them are based on the fact that it's an operating system made for phones and not for computers and I will also briefly mention this concept later in the talk in various phases I will say this is a concept from the next or this is a modification that was made for Android and we're gonna cover the basic Android architecture in this talk and we're gonna work our way from bottom to top and cover the kernel the init process some service some processes that are important to us as developers the system server and on the top is our applications our Android applications so starting from the bottom and talking about the kernel so what is a kernel the kernel is a computer program that is the core of a computer's operating system it has complete control over everything in the system it's the first program to start up and it handles the rest of the startup it handles input/output from other software or other hardware it handles CPUs memory allocations and the basic definition of a process is that the process is an instance of a computer program that is being executed so let's talk a little bit more about the Android kernel the Android kernel is built on the Linux kernel which means that it does a lot of the things exactly the same it really just copies the Linux kernel it does the memory management the same way CPU management the same way things like again hardware and software requests similarly and there are some modifications that were made on top of the Linux kernel to make it more better suited for Android and better suited for phones instead of computers the reason that Android is using the Linux kernel is that it it works it works on computers and it works well it has proven models for both like I said the memory management and process management and the CPU management it has a permission based security model that is again tested and has been working on computers for many many years and it's open source so it's easy to contribute to and take from so Android has done a few modifications to the kernel to again make it more suited for phones and I'm gonna mention two that I thought were the most important ones and the first one that I'm going to talk about is the out of memory killer the out of memory killer is based on the fact that on a phone a phone needs to always be responsive because it needs to get events get phone calls get messages and things like that and users don't explicitly close applications like they don't be like okay I'm done with this app and I'm gonna go and swipe it away also phones typically have less memory and less tricks to use sort of fake memory than computers so one of the one of the biggest modifications that Android done was being more proactive about deciding by itself when and what processes to kill in order to make sure that we never reach a critical out of memory level we never reach a point where Android wants to allocate more memory and there is no memory to allocate so all processes are ranked according to importance and every time the kernel reaches a certain threshold a certain watermark it basically goes like oh hey I need to clear out some memory this is way before we like I said run out of memory it's some earlier stages and then it just clears out as many processes that our ranked is not important until it clears enough memory to be okay now this threshold has been met in this example I'm showing a list of processes that I pulled off my phone in this case it's 30 total processes and what I did next was I opened up my camera and started taking a bunch of pictures which is a very intensive operation and I didn't do anything else and the out of memory killer kicked in and killed a bunch of processes for me and it did that so I can continue taking pictures and doing whatever I needed to do without running into memory issues the second modification that Android made to the kernel is the way clocks and the way clocks are meant to guarantee that our phone uses as little battery as possible and stays awake as little as possible and that would give the processes they would use less energy and be able to sleep more and the way clocks basically mean if there are no way clocks put the CPU into a dormant state where they can still be woken up by certain calls but it's not working at full capacity and again using less memory so this is what we would normally see if we would query how many wakelocks our phone has at a certain point of time what might be like little Peaks when a new screen is drawn or when a network request is being made but these are like really really tiny and then it would go back to zero if there is an ongoing thing happening on our phone such as playing a video we would see that now we have to weigh clocks that are being held one in this case because it's video so one is for the screen and one is for the audio and like I said there are a bunch more modifications that are made to the kernel I'm not gonna cover them now but these are like I said what I think are the most important ones and the next part that I want to talk about is the init process so the init process is a binary that's picked up by the kernel and is being run on startup and it has a list of settings and a list of instructions a list of definitions to start up processes and it wakes up all the other parts of the operating system so it runs on startup and it's the first user space process it is also the root of all other processes they all created from in it and like I said it spawns everything else some of the processes that it starts we know we recognize such as a DB D and log D and install D and zygote these are all demons which means that they run as a background process the user doesn't have an interaction with them they will not know they exist they will be restarted in case they for some reason crash or die and they just run in the background throughout the life of the phone the operating as long as it's working I drew a bigger circle around zygote because it's the thing I'm gonna talk about next and them more important for us more important demon so what is I goat SCI code is the base of all other applications it was designed to share and save memory and reduce app startup time and what we basically see here is that SCI code is a runtime environment that's ready to just start running an app our apps run inside of a runtime managed environment which is dalvik or R depends on our phone depends on the version and if we were to create a brand new runtime environment for every app that we start we would waste a lot of time just creating this from scratch so zygote is sort of a shortcut in a way it's a process that's really just sitting there all set up and ready to run our Java code our calling code it has like I said the runtime environment pre-loaded resources pre-loaded classes that can be used later to run any app so in order to create a new process on Linux Linux usually uses two commands one of them is fork which means take a parent process and clone it make an identical child process from it and the second phase is called exact and exact is on the child process that you just created delete all of the previous code that was all the previous memory that was there and just start from scratch give me something new to run on this brand new creative process as I goat when it's created when it's creating apps it's being forked which means we get an identical copy of the parent but it's not being exact which means the memory whatever's in it is not being deleted which is again perfect for us when we want to start a new app because now we can say hey zygote please fork this process for fork yourself and run our app in it everything is already there we don't need to delete it we have the runtime environment we have the shared resources we have two shared classes and if we wanted again to create another app a second app a third a fourth app again we would go back to zygote and say fork but not exact again we will have multiple copies of zygote running different applications in different processes running different code run different java code or different calling code this also helps with memory management because zygote and the runtime environment have classes that are shared between them and combined between them so again we can initialize them only once and we don't have to initialize them again and again and this is something that everyone would see when they get exceptions if you dive deep enough into the codebase you would end up seeing some sort of reference to the zygote because at some point we will reach some code that was initialized there or was run there now usually in linux apps run under the same user and user is me or you or an admin and processes that run in this manner have permissions to sometimes share files share memory and things like that and android decided to create a stricter model about processes and user IDs and what they basically said was we'll use Linux user IDs but we'll say that every app but basically every process is its own user will give every app every process and a unique user ID and that way every app will not be able to share memory or corrupt files or touch other processes files or memory and that way processes will be more secure and use still used Linux features about security but change it in a way that suited Android and again this is a list of processes from that I pulled off my phone the left row is the process ID and it's all different so again they cannot be sharing sharing things between them the next thing I want to talk about is the binder IPC I just spoke a little bit about how processors have their own user IDs and how they actually can't communicate with each other and then the next question is but we often do we sometimes have to ask some other process to do things for us or we just want to maybe transmit some information and we need a way for processes to communicate and to do that and the mechanism to do that in android is called the binder IPC and i pulled up this three lines that I think many Android developers saw or other variations of it in this case I am contacting something called connectivity manager it's not sitting in my app it's somewhere else and I'm telling it gets me that system service get me that other process and ask it what is the active network info what is really happening within this case with the network right now but I could make any number of requests and many other processes the reason why I am asking this from someone else and not actually writing this code in my app or why is this code isn't being pulled into instance is that in order for example in this case to ask about network info that process needs permissions to talk to drivers to talk to hardware to ask hardware questions like am I on Wi-Fi and things like that and that would make my app needing more permissions and less secure so by putting this method somewhere else in another process that other process has these permissions and I don't need them so if I go and travel to the source of my get active network info this is what I'll see I'll see an interface an eye connectivity manager member and all my methods are getting cold on this member I don't really see the implementation of the actual call I don't really see the code of what is happening inside this call and if I have a OSP and I search for this interface again this is what I see I see just a simple interface with like methods but I don't against see the implementation of them I don't see it in my code and not there there and then the question is of course why and the answer would be because like I said we are not actually talking directly to that other process we're not really asking it directly for first things we are going through the binder mechanism which looks like that if we go from app 1 we go all the way down to the kernel and then back up to the other app to send and receive messages so I'm gonna go you over each and every component and the very basic one again the very where everything happens is the kernel the binder modal in the kernel it is the only thing that actually knows about process about the other process none of none of the arrows in my previous slide was speaking directly with app one with app two kernel is where this the transition itself is happening the transmission itself is happening and the colonel is serves as the mailman in this example it knows how to it keeps reference to all the processes that are being contacted and from where it keeps their keeps being aware of their existence whether they're there or not and to deliver the right message to the right object so the way that the kernel is getting and submitting messages is through system calls but we are not writing system calls and we're not writing kernel code we're writing Java so there are there is a class in Java that is creating our message and calling that particular system call it's called the binder and it has two main methods the transact is the thing sending the message and the on transact is what will get invoked in case when you received a message and open it up now I'm talking about messages which is a little bit confusing because we're not really sending a message we're not saying hey I hope you're good the weather here is fine I'm saying I want you to execute a function for me I want you to call a method on your behalf on your side I want to ask you a question and I want to get an answer back so this part of creating a message that will later be opened up and be translated into oh you meant to call this particular method this is what is being done by the method transact will take oh you meant to call this method with these parameters turn it into something called a parcel and the on transact will open up this parcel and say oh this parcel means you wanted to call this method I will do that and you back the answer now again if we were to look at this code it's in Java but I mean if we were to spend our lives creating parcels and figuring out how the other side wanted them to be opened up and creating all this like complex extraction and we would just have be writing very tedious and very boilerplate code so that's why we have the a IDL tool that is actually a tool that will create the bridge between the actual methods and a transaction calls and it will generate two parts proxy on the sender side and the stub on the recipient side so on the proxy side it would look like that whenever we want to call this method get' activity network we would create some sort of parcel put information in it this is if you can see clearly the class that is named data we create the parcel with the data in with the reply and in the middle there we will call transact which will call the system call and actually transmit our request and on the other side we would see the stub the stub overwrites on transact it would open up this parcel say oh what you wanted to do is you wanted to call this method get active network and it would do that thing this is the actual concrete implementation in this case of this method so if we go to this get active Network this is what we'll see which is the actual code the actual implementation of this particular method and if we go back to our original request which was connectivity manager get an active network info and we would debug it and see what is sitting there underneath the hood in our connectivity manager we will see that it's the icon activity manager proxy class so to wrap it all up this is how it would look like with the actual calls app one would call some interface and app twos method some method our Java interface classes are a I deal with will translate that into parcels and call transact and it will call transact on the binder the binder will then call a system call to the kernel the kernel will figure out who is our recipient who are we sending this message to and we'll do that actually send a message it will send back a system call to app twos binder class the method that will get invoked is the on transact on transact we'll figure out open up the parcel we'll figure out what actual method we wanted to call on app twos actual object and call the actual method the last bit of the operating system that I want to talk about is the system server the system server is also also initialized by in it and it is forked from zygote so it is also a java application it is the heart and the core of all android it manages all the Android apps and I'll talk a little bit more about that and a little bit and it manages things like location like power like Wi-Fi it has modular functionality it's split into managers little managers inside but it is one big process and it is one of the more core important processes for any Android to run and there are many many managers in the system service server and I focused on three that are most important to us I think as developers which are the activity manager and a windows manager and package manager so the windows manager is the one responsible for our windows and for what is being drawn on the screen by our apps by our activities fragments views etc it's the responsibilities include things like keeping track of the window activities what is being shown what is being invisible transitioning between them giving the user in general good UI experience which means that it cheats a little bit sometimes for example it will take screenshots and display them in certain cases when it's transitioning between one thing to another and it's also handling the system level gestures which means if we didn't handle a certain click or a certain other UI gesture and it bubbled up until it reaches the window manager it will decide what to do on top of it another important manager is the package manager the package manager is the thing that is responsible for looking at our manifest files and doing things like installing new applications and resolving the correct activity when we are looking for a certain activity and it on startup it goes through all the manifest files in our Android phone and it creates a huge list in memory of all activities and all like possible intents in our in older and all our overall in all the phone basically in all our apps and what it can what it also gives us if if we ask for an activity we say we want to start a new activity where where is what can we do is this if this is an implicit activity and we can maybe instantiate more than one type of activity that fits this criteria it can create us that list of give us all the activities that can actually match this this filter match this intent and the last manager that I wanted to talk about is the activities manager the activities manager is the heart of maybe everything it's the most important manager and it's responsible for managing applications so all our applications are being managed by it every time a new activity starts pauses resumes everything it will be the one deciding that that thing has happened and creating activities if needed destroying activities if needed its managing all the lifecycle events its managing the actual user that is whoever's using that phone at at point it is responsible for memory trimming it is responsible for killing off rogue activities that are taking too much memory and handling configuration changes so when we're talking about managing applications the activity manager will pile up our applications into some data structures and the data structure that it uses is it will put a stack there can be one or more stacks for example in case of a split screen there will be two stacks inside the stacks there will be tasks and a task correlates to mostly one process in rare occasions more than one in inside a task there can be one or more activities so I talked about processes up until now but we're not writing processes we're writing activities and how did they correlate with with each other how do activities live inside processes so for every activity that's created activity manager will create an activity record which is just a data structure that is there to keep track of this activity where it is what it is what state it is what window it has etc and in this case it's really simple we have one task and in it one activity record and this task this activity record correlates to one activity that's living in one process if we were to create for this app a second activity we would navigate from activity one to activity to a second activity record would be created in our task and a second activity will be created in our app process so again fairly simple if again another app would be started then another task would be started another activity record would be in it and another process would be started for it so we would have one process holding one activity and then another process holding our other two activities and the last bit is where it gets confusing because say I navigated from my apps activity to another apps activity a very clear example is in the case of sharing I open up my share screen and I wanted to open another app in order to share my content through the other app we would get in this case another apps task we would sorry we would get in our apps tasks we would get the other activity record but the process would be the other process so in our task would be one activity that sort of belong to the original application and another activity that belongs to another application so they would live in the same task but they would not live in the same process so when an activity manager manages the life cycle there are a few things that it does for example so the main top thing that it does is it only makes sure that there's only one resumed app at all time there's every time even if we see three apps on the screen even on split screen even on picture and picture there's always only one resumed app it will pause stop every other activity it is also responsible to deciding which activity has a surface is visible we'll communicate that back to the window manager it's deciding things like thread priority and CPU affinity which is apps that are running right now or apps that are in the forefront will get more resources to run than other apps it will give and take away certain permissions from apps if our app is stopped it cannot do certain things and it will make the out of memory adjustments which is a topic that I'm gonna cover right now if we go back to when I talked about the kernel and I said that every process has an out of memory score and that out of memory score is based on how important the process is and how important the process is is not something the kernel would know because the kernel has no idea about things such as what is an activity or what is the actual activity that is resumed right now what activities are not resumed right now the colonel just doesn't know it and the thing that knows it is the activity manager because it manages all these adjustments so activity manager is responsible for deciding and giving every process an out of memory score that later will be used by the kernel to decide which processes to kill so again if we're looking at the list of processes that are also conveniently sorted from most important to least important we can see that there are different categories in which they are sorted by and the top most important one is the system the system server the second three are persistent then we go to the foreground which is the process the application the activity that is actually being used by the user at this point then it's visible maybe things that are partially visible or if we have some foreground services perceivable and on and I didn't put in the rest of them but I think that's that kind of shows the point and inside activity manager there are two methods that will be applied and compute out of memory score every time anything changes really any time an activity is being started stopped resumed created stopped user started the service anything like that this method will these methods will get called they will run through all the processes and give them a current score activity in case the kernel actually woke up and killed a bunch of processes it will also let activity manager service know and it will remove these activity records and remove all the redundant information because these processes are no longer there and there's nothing nothing left to wrap up this talk I wanted to go through the process of starting a new activity and show you a little bit of how it looks like in the activity manager service and show you a little bit of the other concepts that I talked about here so I'm picking a path that will show you the most of these things there are many paths the paths that can be that can create a new activity you can for example already have a process to run this activity in you can actually find out that this activity was used before you don't need to create it and things like that so I'm like I chose the most like everything is new path and in activity manager service we will see methods that are pretty straightforward start activity is user which will call something internal that is called start activity may wait and in it the first thing we'll do is we'll resort resolve the intent and this will call the package manager and ask the package manager service hey please give me all the activities that match my criterias that match either my name or can perform a certain action and the resolved intent will go and be like okay I will give you the correct list or the correct single activity in this case we're going with a single a single activity so we will get get activity info with the one activity that we wanted to create a new activity record inside activity manager service will be created with all the information about this activity I put a generic example of how throughout this path activity managers service will talk to window manager service and do things such as create our window create a transaction between whatever was showing before to our new activity and now that our activity record is ready and our activity is ready to be actually shown and and created we will create it and resume it at the same time so now we are actually deciding if we already have a process to use and if not we'll also create a record for our process called process record if we do not have a process we would start a new one and we would call process start an end process start we will see that we are actually creating a new fork from zygote so this is the zygote call to hey please for yourself and let us know if it's if let us know when you're ready let us know when a new process was created and ready to run our run our activity run our app and again in activity manager service when the zygote calls back it will say hey I got this I'm ready what to do then we will bind the new application to it alright that's it thank you so much for coming [Applause] yeah I can take questions or you can also come over later if you want to I know there's like a lot of constraints with using a lot of legacy code is there a part of the architecture that stands out is something you would change if it were being rewritten today from scratch [Music] good question it's not just because of legacy code I think one thing I found out that there's a lot of edge cases that who we as developers aren't even aware of or don't even think about which make this whole tree very complicated um I don't know if that's really changeable because like really there's a lot of edge cases that a lot of things that can go wrong in the middle a lot of things that can be I don't know really exceptions to this very simple past that I showed I showed it in a very like this is like super simple but I skipped like maybe hundreds and hundreds of if cases and what exactly resides in the preloaded resources which is inside the zygote there was one section which is short say you have a zygote and inside zygote we have a dalvik and a pre-loaded resources so are the pre-loaded resources like when do they get pre-loaded when my app is installed or these are not your apps resources these are generic resources that are shared between all apps and shared between all processes that will be forth from zygote your apps specific resources will only be loaded once your particular app is running inside an instance that was forth from zygote okay thank you

Channel: droidcon NYC

Views: 4,269

Rating: 4.9534883 out of 5

Keywords:

Id: 0zJCyKp7-9s

Length: 41min 32sec (2492 seconds)

Published: Tue Oct 31 2017