Scaling Instagram Infrastructure

okay welcome to skating Instagram infrastructure Instagram is a community where people connect with each other through shared experiences in a visual way a bit of history of Instagram it was founded in 2010 less than two years later it was acquired by Facebook I joined Instagram three years ago and have been working on the infrastructure team since then fast forward to 2017 we got a new logo and in the just last six months a hundred million new users have joined the community since we're talking about saving today let's take a look at a typical day on Instagram four hundred million users visit Instagram every day four billion likes are registered more than 100 million media is uploaded and our top account has a hundred more than 110 million followers this is across the board four times increase in scaling than three years ago when I first joined Instagram back then it was a much smaller development team the app was waste and poor and we have a lot fewer users but it was fast growing and we were struggling to keep the service up all the time one of the common things you would see on a Friday afternoon right before the weekend peak time is our uncle engineers will be staring at the computer screen looking at the CPU load and decide whether we need to spend up a few actual web servers for the weekend well we have come a long way since that mode of operation and so today we're going to talk about in three dimensions of growth to non-native us to scale to today's level to scale out is to build an infrastructure that allows us to add more hardware when we need them and to scale up is to make each of these servers count and to scale the dev team is to enable a fast-growing engineering team to continue to move fast without breaking things so first scale out is the ability to use more servers to match the user growth from a few servers to quite a few in the data center to expanding to multiple data centers Instagram was running in AWS from the beginning and continued to do so on after almost two years with Facebook about three years ago it moved its service completely inside Facebook data center in order to take advantage of some of the scaling services like monitoring's search or spam but as soon as we moved into one of the data centers Facebook cut out the entire network access to that data center why Facebook takes disaster readiness very seriously so it would routinely conduct these drills called storms on a regular basis to make sure that all our services can sustain the loss of a region and continue to serve the users in a seamless manner Instagram a protector at the time was not able to operate the multiple data centers so we had to take play the gorilla tech lick and move up with our whole service to another data center unfortunately disasters are not just theoretical hypothesis so we needed to get ready for the next time the storm happens or the reality hits in addition thanks to our rapid growth we were quickly running out of capacity in that data center Facebook had capacity elsewhere but we couldn't take advantage of that power outages in the human errors also contribute to service unreliability just a few days ago a RM - RS type of you know operation on our container tier was wiping out quite a few of our production servers fortunately we had the ability to move our traffic away from the problem data center in case you think that this only matters to scalable services such as Instagram here is a headline from last week which reminds us even the worst most reliable infrastructure could still have outages at times and if you don't have another region to fall back on then this is a suggestion from one of the articles published I'd say that in addition to praying at the moment of this event let's put something on the roadmap for the next time you we would go business as usual how do we get there let's take a look at the back end stack Instagram hat at a time at the center is the web tier that runs Gengo with Python it receives user requests and accesses various back and storage or services for responding to the user a user request could also trigger some asynchronous tasks that are done in the in the backend for example sending notification to the person whose photos you liked these tasks are sent to rabbitmq and celery for processing so how do we distribute this back to different data centers we identified two categories of servers or services storage versus computing storage server store global data that needs to be consistent across multiple data centers with replication perhaps with some latency but eventual consistency computing servers are usually stateless they process requests by user traffic the data in these servers are temporary and they can typically be reconstructed from the global data using these two criteria we now rivet our stack to see what fits where we use post graph servers to store user media friendship type of data and a typical deployment in one region is a master with multiple replicas to support high to read RPS the webservers rights to the master where data gets replicated but the reads typically happens on a replica when going to multiple data centers it's pretty straightforward with cross region but multiple applications the jungle continues to write to the master possibly cross region but typically only need to read from a local region to address the increased latency for writing we did make modification to minimize the round-trip time for write by patching request wherever it's possible the increased latency between the master and replicas turns out to not have been a big problem for us next we use Cassandra to store user fees activities etc in Cassandra there's no master all replicas have the same copy of data with eventual consistency consistency can be configured based on applications tolerance for inconsistencies scalability service availability and latency so for example one application could choose to have right consistency of to and we consistency of one going to multiply the center again is pretty straightforward you can have the replicas living in different data center on the computing side we grouped Django and rabbits and a celery into one pod that goes in each region the global load balancer would balance us send the user requests to Django's but the asynchronous task would be produced and consumed in the same region that test has over here database replication across region and computing resources all in containing one region but we have left out memcache limp actually is a huge part of the scalability story it is a high-performance key value store that fund hence the data based year without which the databases would be crumbling under the read amend it provides millions of operations per second of each of the servers because of such high rate of services it is very sensitive to network conditions for latency is a big deal as such cross region read/write is prohibitive so our architecture decided we will not provide a global consistent memcache here the work that in the memcache in each region is determined by user traffic served out of that region let's see what problem it might cause as the user comes in making a comment on the media that gets into in word it gets into inserting the Postgres server the memcache also gets updated by the general server another reader comes in another user comes in and comes fetch his feed he's going to use the same memcached here where the commenter was so he's going to give them the latest comment no problem when both users are served out of the same region when we go to two different regions it could be a problem since we have the user requests based on a user ID two people sitting next to each other to be served out of two different data centers let's see what happens in that case again the users see makes a comment data gets replicates inserted into the post grant server memcache is updated locally in that region but it does not update the memcache in the data in the other data center so even though the comment is replicated in the Postgres server nobody is invalidating the memcache in the second data Center the user comes in getting his feet is going to get a stale comment of the media well it's not a great user experience and could cause a worse consequence than just social awkwardness so how do we fix the caching consistency problem as user comments comes in again we insert into the post graph database but we're now going to update the memcache from Jango instead we use the post class replication mechanism and we want to demon on each of the Postgres and replicas that tells the database updates and it dents in validates the memcache in its own local region now when the user R comes in and reads the feet it's going to be forced to go to many forced to go to Postgres because memcache does not have that entry anymore now the user gets the latest comments while it salts the stale memcache problem and creates a big load on the Postgres server let's see why this is a picture that has received 1.2 million likes how do we get this number to display to the user so we have a table that stores which user like to which media and we would do a select select town star on that table and this is going to take hundreds of milliseconds to execute it wasn't a big deal in the past because the cache cut the lights counter was caching memcache and every life would just do a memcache increment so there's rarely any need to go to the database to retrieve this counter but now because we invalidate this counter for every light that registers all these reads after the the life would have to be forced to go to the database so to fix this we created a demon life table that stores just a meteor and the number of likes instead of going through a select account star we'll just do an index look-up on this table that reduced the execution time of this query to tens of microseconds many orders of magnitude faster than the previous query however even in this case the load is still high you've heard a thundering heart problem in the morning on the Facebook live problem this is very similar when cash is invalidated there's no such counter in the memcache then all the jungles coming in would have to go to the database even with reduced query time it's still a lot of load on the database so we use memcache lease to solve this problem here's how it works the first angle that tries to read from the memcache instead of doing a normal get it does Alysse get and then cash returns instead of a Miss it says I don't have the data but you have my information to go to the database the second general comes in within a shorter period of time does a same lease get and instead of getting a Miss is told I don't have the data but don't go to the database I have a stale value for you you can use it or you can wait to you know we try to get the latest data in most about the application cases the still value would be completely valid so if you get a million likes versus the million plus 10 for user experience account are not going to be very different the first angle then goes to the database and updates the memcache subsequent we would be able to get the most updated value now with these improvements the database even though it's still getting higher load than before it's able to handle the load now and with the building the computing body building blocks and the replications for scratchin Cassandra we were able to build up an extend Instagram into as needed mainly data centers as we need to with that we now have capability wherever we need it and service is more reliable and we're regional failure ready but we're actually not everywhere we're mostly in North America right now because the latency from Europe and Asia is still very high so but we also see opportunities there with the latest features that we have we have four direct messaging and live streaming we have identified more and more users with more localized social networks whether it's celebrity whether it's you know a group of friends who are in a particular region in Asia it makes sense to move the data center closer to where users are to reduce this interaction latency so that would be something that we would like to tackle next well it is great that we have the ability to add more servers when we need them we realized we were leading them too fast while our user growth with healthy our server growth was far outpacing the user growth and we had grown at this space at this pace you know since two years ago we would be running three four times the servers that we do now so I really like this quote from last cucum so we talked about scale-up we don't mean to buy more expensive hardware's right with more CPUs memories and stuff like that rather what we mean is to use as few CPU instructions as possible to achieve something meaning writing good code to reduce the cpu demand on our infrastructure and to also use as few servers as possible to execute to carry out those CPU instructions let's just increase the supply each particular server has if actually each server will be able to serve more users that's what we call skill up so let's take a look at the men's side first we'll go over how we measure the CPU demand how we analyze where it comes from and how we optimize where it matters let's start with collecting some data so Linux provide perfect ABI for applications to retrieve CPU instructions for code segments so we sample user requests and collect these CPU instructions for specific requests along with tons of metadata such as what's the endpoint which data center they're running it from what kind of hardware they're running it on so that we can slice and dice on various dimensions so here is a time serious representation of CPU instruction usage by various endpoints we then implement a set of tools to monitor this metric and alert when the when it increases above certain threshold for example this way with this jump in the time series representation we can then match with the event log to see what are the possible causes is it a div that's rolled out is it some configuration changes or some back-end service roll out another very important piece of metadata that we log is whether a specific feature is on or off in a request this is very critical for us to detect increased CPU instruction demand by a newly added feature whose global in fact may not be very easily seen from the previous graph because it's still to a very small audience but its impact is amplified with this metadata and we'll have with having this data we we add this information right next to the UI where the feature is configured so when the even if the feature is rolled out to 1% or less this graph shows 1/8 more than 8% CPU increase if the feature was rolled out to a hundred percent so this gives the developer plenty time to address the regression because the report becomes a real problem this way would make the performance part of the development cycle rather than and afterthought but we still get you know innocent look some developers who say hey you know my div is only one log one line of log change you know can't possibly be causing 20% CPOE instruction for a particular end point so now it's time to develop some tools to deep dive into the code base for that we use a Python C profile C profile provides performance to fix-it function level with Co graph information very much like how Linux perf works the resulting data can be pre post processed and rendered with G prof 2 dot which is an open source tool using this you can look at the function and see its overall impact on your code path the color Collier relationship and is to help you focus on where you really want to spend your effort in optimizing in the past we already have this right this is all open source well some of our engineers have you know use these tools to to dive into specific performance problems but the cost of this C profiling is pretty high and the process of enabling disabling is manual and the prospect of messin with a production server to collect this data is pretty scary for most developers as our development team grows this is this did not scale so we needed to make the performance data readily available for more developers to access so we instrumented C profile collection in our production server continuously it is a conscious trade-off between cost of collecting this data and the visibility we would gain into our code base with this instrumentation at anytime our engineers can just run one command line and generate a call graph like what we saw before and it greatly improved their product productivity when debugging performance issues while snapshot was great when looking at the overall picture but it doesn't help a great deal when you look at regression so we also put this data in a time series format so if a caller is making multiple cost to other functions but you see matching regression in the CPU instruction you can be pretty sure that in fact that 20% of regression is caused by the one-line change of logging so it's time to spend some effort on that we see profile running everywhere we also integrated this data into code routing that shows how many servers would be needed just to run that function this is a great reminder for anybody who's looking at our code the optimization opportunities so now let's take a look at some of the optimization was done with all the visibility we have in the code base here's one example when you load up the Instagram feed we the server's return on number of URLs to the clients with each URL is the C CDN where the client needs to go retrieve the media this year our generation is based on the media as well as the user where the user is from so we can generate the most optimal CDN location for the user as we started to support a variety of mobile devices with varying capabilities there was need to generate multiple of these URLs so the devices can choose what's the best user experience they could serve depending on network condition the particular device the user has so we started to give multiple URLs to the mobile devices the only difference though is really the size of the media but as we were implementing this feature we were calling the generation of the URL function four times rather than once and with just different varying sizes with the help of C profile we understand this is what's happening so the first thing to do is less instead of generating calling this function four times we call it once and simply overwrite the site function this is one simple example but we have new more such examples with the visibility into the code base we know exactly what is happening and we were able to do a lot of these optimizations very quickly but you might also ask okay if the URL generation is so costly how about we make it less costly and you will be right so another major theme of optimal optimization opportunity was that we identify a number of functions that are excused extensively that are pretty stable there is not a whole lot of iteration on top of it and we cycle NIH's these functions or use C++ implementation to speed up the execution and we were able to gain significant CPU instruction to reduce the CPU instruction needs on that front so that's part of controlling demand well let's look at the second aspect of scaling up given certain CPU demand how do we make each server execute more ideally we want to all the multi cores on a server to be put to use and we do that by running many worker processes in parallel that process user requests but the number of processes is upper bounded by system memory a sticker how the memory layout looks like we run n parallel processes where m is greater than the number of CPU cores on the system each process has two parts of memory the shared part in the private part total system memory is made of this one shared memory and all the private memory added up together so Cal how can we reduce the amount of memory requirement so we can run more processes upon analyzing the memory configuration found that code itself is a big part of where our memory went so the first things we did was to reduce the amount of code in memory we started to run optimize which may be obvious but we were not running with - oh now docstrings is gone from from our memory the second part is removed dead code we developed new features very rapidly and there is a lot of legacy things that are not you know despite the best efforts from developers there are going to be old code laying around so with C profile data we were able to identify what are the code that's never gets executed in our code base and we were in the process of automating this code removal process the second part is if we could move some of the private memory into shared area where only one copy is needed then we would able to we will be able to reduce the total memory needs since configuration data is the same for all processes it makes sense to move them to shared memory so again this is a bit of a trade-off you can see between you know if you put in the shared memory then the access to the shared memory is going to be a bit more costly than if it's a private memory so you need some kind of metric to to measure the the trade-off whether it makes sense for you to do that the second part where we shared more is we actually disabled garbage collection for Python this helps to prevent some code being moved to private part for GC purposes it's not super intuitive but so we have a great blog post on this what we found and Python community is actually looking into how it can make GC more effective without moving memory into private part so with these memory changes we were able to actually get more than 20% capacity increase because we increase the CD instruction that we could execute on each server networks can also negatively impact scale-up of a server our django processing model is synchronous each process can only serve one request at a time so while it's wait for external services to respond it can cost servers duration and fewer CPU instructions that executed as our application become more sophisticated we are depending more and more on these external services and this week becoming a bigger problem one example is like this on our home feed we now have stories feed and sometimes given suggested users on the same screen this has greatly increased the latency for home feed retrieval however we also found that these services are typically independent of each other so instead of sequentially accessing these back-end services we explored using asynchronous i/o to access them simultaneously and greatly reducing the latency for whole for the for the this particular endpoint not only does it improve user experience it also helps to mitigate server starvation problem and increase the capacity of the user so conclude by optimizing memory and network access we were able to make each server serve more CPUs there still work to do as well although Python and C++ were great they're not as a debugging friendly as Python so we'd like to find a generic faster runtime solution for Python using JIT for example keeping the ease of programming and debugging as well as the runtime efficiency we're looking at asynchronous web framework so that we can remove more of our dependency and in external services and improve the resilience with newer Python version and better memory analysis tool we also hope to gain more visibility into the memory utilization as we do with the CPU utilization so we can further improve the memory of footprint and there are many other interesting opportunities that we're going to as well last but not least getting the dev team so instagram has an interesting combination of workforce we have about 250 engineers with 30 percent of them joined in last six month we have interns throughout the year who should numerous features with us in their three months working at Instagram we have hacker mentor from Facebook that works with us for 44 weeks on specific projects and we have good campers who sign up for Instagram tasks that gets done in two to two days to one week as you can see their familiarity with our system and the ramp up time would vary but not a lot but we're shipping features rapidly and each of these features will require some data to be stored somewhere right so when a product engineer starts to work on a feature your depressions that she needs to answer as you can see this is a pretty heavy process that each small feature has to go through and we don't give people a lot of rem-pod time like we said as the team grows and the MER features grow rapidly it is a big burden on the infrastructure engineers to babysit this process and it's a slowdown for product engineers to deliver their feature so what we really want is an architecture that would automatically handle caching and that allows the developers that define relations and not worry about detailed implementations it should be self-serve by product engineers and the infrastructure engineers will just worry about scaling this service turns out we have this infrastructure at Facebook Tao it's basically data based plus right through cache it still uses a relational database my sequel at the back and storage devices but it allows only very simplified data model basically the nodes with objects and the edges with relationships it may not be the most efficient as we used to have with in terms of you know how you store the data or you can't make direct sophisticated queries in the database itself but it does the most basic things at very large scale and this simplify data model allowed engineers to ship new features at a much faster speed without breaking things this is just one of the examples will continue to develop better data modeling and API to increase developer velocity so where are these features getting developed a typical source control at some companies with complex features will probably look like this like master will continue to contain incremental changes with smaller features perhaps but branches are created to develop larger features such as narcos live or direct messaging the problem with having the branches is that engineers need to be mindful of the codebase they're working with and do mental contact switching there's French management overhead surprises will arise when different teams work on different branches with overlapping codebase it makes it harder to do major upgrade or refactoring like the data model simplification that we we show before performance data would be discovered much later in the development cycle and will be much harder to fix so instead of branch management model for various features we adopted the webmaster approach with no branches every diff needs to be to keep master working so it's a continuous integration process engineers can collaborate much more easily they don't have to think which which repo or which branch of a particular feature is being developed on it's much easier to bisect problems and revert tips when necessary and we can continue to monitor performance for all features at the same time with pretty minimum overhead but if every feature is being worked on in master how do we shift the code so we use gates to control who will be exposed to new features typically at the beginning only a few collaborators in engineering would be seeing the feature the DA footers will test the future still under active development giving feedback so that the engineers can iterate on it and then employees are the next victims before we develop this to the world with this model we know the feature works but we don't know if it will still work when 400 million people start to use it the next day sometimes between employee and the world you know it takes one day to one week time so we don't know whether the database can support the increase week right RPS we don't know whether jungle has this much CPU capacity and you know whether the network has will become a bottleneck so what we also do is a load test we make up artificial load that are some are triggered by user requests we would make assumptions about you know how many users will be using this feature what type of load they would be generating and we would exercise the backend infrastructure in this process this helped us to prepare back and capacity to ensure a smooth launch of a new feature so we're ready to launch we need a release branch right let's see how often do we ship the code once we do what do people think once a day however once a day so we continuously roll out our master repo whenever a diff is checked in this result in about forty to forty roll-off a day a typical commit will go out within an hour of landing a master how do we make sure it doesn't all break loose right so code review unit test is very very important we're not perfect at unit testing but we do our best most in one of our major recent upgrades using tests was a you know a great help in capturing majority of our use cases and then we were able to bring up the service right away once code is committed it does another set of unit tests because you know you know the diff on top of diff may not work the next is production canary before the code is rolled up to the whole production tier we actually run it on a few of you production servers and compare the exception rates and five hundred two hundred raised with a the controls here and if this is a canary here is generating a higher than threshold failure then we would stop rolling this out to the world and finally we roll it but even despite this you know a lot of tooling and canary we still might have problems in production so extend is monitoring and alerting system helps us to discover problems fast and sometimes we do need to revert ok to wrap it up we have covered three dimensions of scaling instagram scaling out use more Hardware scaling up use less hardware and scale the dev team for velocity and productivity neuron uncle experience is a little bit different now but it's not all rosy scheming is continuous effort right there's no one endpoint with where we say we're done as you can see from my previous slides on the challenges and opportunities we continue to tackle interesting scaling problems at the next level scaling is multi-dimensional you want to look at different angles and makes your server so count and skating is everybody's responsibility not just infrastructure and efficiency team we're not the police or forwards and neighbors right I think in some I've got questions before you know how do you make the rest of the team be more efficiency driven I think you know the deep belief in helping each other is is really making a difference here so questions okay thank you very much so we have ten minutes for questions people have any sorry I'm just curious could you tell us a little bit about where you do your future load testing is it actually done in a production environment or non production thank you future developments does go to a production except it's gated and it's not exposed to the user is that your question load testing it's done in production hi thanks for the talk with this volume of distributing continuous deployment how did you convert the acceptance test you know the canary do you have a kind of really the ultimate ition so we measure the rate of 500 versus 200 errors and it's just one example we have a number of metric both in terms of the error rate exception rate in some cases even the CPU utilization the requests to the back end if you have if your dev is generating a ton more reply to the back end than your control system it could still be an indication of a bad gift and in this case the text is no longer too much and you know all this talk they cut our for every how long with the canary yes it's not long it's a couple minutes with each other over you know they do serve quite a bit of traffic in here we have we have the trade-off between the time we chased the canary the diff and and how we roll out the code fast the faster you roll out the code the more you can roll out to the the easier it is be able to identify which ative is actually causing the problem I think it excuse me are there any downsides to working on master and how do you effectively get the features as well I'm sorry coach sorry yeah there any downsides to all the developers working on master and how do they sort of gave their features within that single branch [Music] I'm not sure we have seen many downsides in terms of gating there is a risk where you know if you miss configure the future could be prematurely exposed to the world and it has happened unfortunately so but what it happens we develop more better tools we also canary the gates as well yeah thank you hi so you're deploying into your live environment forty to sixty times per day so how long does it take to roll out a new version to all our live servers do you mean between commits and roll out I know if you're just saying okay now deploy to all the servers I don't have a tape oh how long does it say ok 10 minutes yeah how many servers does it about over 20,000 ok thanks all right I was just wondering could you talk a little bit about your training culture and how you bring on your new young engineers you you mentioned that there was quite a few news characters in the last six months sorry I didn't quite get the question yeah I was just wondering what the training culture is like is it just learn on the job with the training I track training culture so Facebook engineers typically go through boot camp so there we we have training on basic Facebook infrastructure software pieces and deployments and things like that instagram has its own bootcamp forces so and then it's basically the park the point of contact for specific tasks those are boot campers for for hacker members you know who people who have already worked at Facebook Instagram for a while then it's a lot easier to understand the rest of the architecture interns you know obviously have longer time so they typically have their mentors working closely with them that helped a lot I had a cushion so the database wrote like a database in data center to you need to it has an issue the master you have to promote yes how quick is that we were able to promote a slave in so when we were still managing progress now we say from the last slides we don't anymore we we do not do automatic master slave promotion so once we detect a master going down and we promote a slave then it takes a lesson to manage to promote so during your data center like where they do the storms it's sort of fake right because do they alert yeah people that sell yes we don't do we don't do real cutoff we do promote servers before we come off yes and what do you do now is just pure Cassandra so we we use Tao which is also a relational database so again it's like you say it's not a real cut off without warning so you have masters with massive right failures and things like that that doesn't happen and yeah we do have it also Cassandra replicas and we need to prepare for the storm you know obviously we in a real disaster people would be more forgiving if there are more massive right failures but in a normal case we don't want to impact the user experience on a regular basis right so but other than the database is almost everything else is is automatic train I had one more question unless someone else are we have the computations can you just because you're doing like basically picture conversions right for that's what taking the CPU most of the time are yeah can you just delay it like let's see I upload a photo and your CPUs are bound can you just delay that computation so that my readers so the computation is not done upload time it's actually done by a tweet time any other questions okay oh hi I'm wondering in the very long term is Instagram considering biting the bullet and converting its backends to use the same way as Facebook to eat duplicate you know is infrastructure work and all that stuff Jimmy in Python versus PHP yes so you see HP hack yeah so right now we don't have much incentive to do that we have a lot of business logic built into the codebase most of the back-end access for example accessing Tao is already in C++ that is used by both PHP and Python so yeah so right now we don't have and using thrips you can use any access back-end services you can use Python or PHP doesn't really matter performance wise we are looking we have just converted to Python 3 which gained us quite a few percentage of performance gain as Python is being active developed 3 6 verses 3-5 also has performance gain we're also looking at as I was saying the runtime JIT optimisation that that hopefully could give us more CPU efficiency thank you I do actually also have a question a part of it you already answered you're using Python 3 the I was also curious about you used to just siphon as you use it do you implement because you mentioned you also use c and c++ do you actually implement stuff in C C++ or B you size into interface to existing libraries and if you if you write new C surface was covered in your experience be the performance difference between plain C C++ and like them we use both actually so it is especially for the code that is shared to access back-end by both the Facebook's content and our back our front-end we use C++ because that will go for both but in many libraries that we use in Python itself we use Python 2 to do it directly without using C++ in terms of performance I am Not sure I can now give you a number I don't think we have actually done like Apple to cap Apple comparison with the same module being implemented in C++ where to find them ok great things I in my experiences it's about 2% what I phone ok thank you ok thank you very much [Applause]

Channel: InfoQ

Views: 197,998

Keywords: Software Development, QCon, QCon London, Software Architecture, DevOps, Software Performance, Case Study Software, Scalability, Instagram

Id: hnpzNAPiC0E

Length: 51min 11sec (3071 seconds)

Published: Fri Jul 14 2017