How @twitter keeps its Search systems up and stable at scale

so search is one of the most important services for any product and at Twitter scale it becomes Ultra important but what does it take to maintain search at scale how can we ensure that a search service continues to function no matter what kind of load hits the system in this video we dive deep into how Twitter built its tooling around elasticsearch that helps them handle massive surges in search traffic do real time ingestion and ingest hundreds of terabytes of data in backfill but before we move forward I'd like to talk to you about a course on system design that I've been running for over a year and a half now the course is a code based course which means I won't be rambling a solution and it will not be a monologue at all instead a small focused group of 50 to 16 genus will be brainstorming the systems and designing it together this way we build a very solid system and learn from each other's experiences the course is enrolled by 800 plus ingenious spanning 12 codes and 12 countries ingenious from companies like Google Microsoft GitHub slack Facebook Tesla Yelp Flipkart dream11 and many many many more have taken this course and have some wonderful things to say the course is focused on Building Systems the way they are built in the real world they will be focusing heavily on building the right intuition so that you are ready to build any and every system out there we will be discussing the trade-offs of every single decision we make just like how you do in your team we cover topics ranging from Real Time text communication for slack to designing our own toilet balance live text commentary to doing impressions counting at scale in all we would be covering roughly 28 systems and the detailed curriculum split week by week can be found in the course page linked in the description down below so if you are looking to learn system design from the first principles you will love this course I have two offerings for you the first one is the live cohort based course and the second one is the recorded offering the Live code base course happens once every two months and will go on for eight weeks while the recorded course contains the recordings from one of the past cohorts as is if you are in a hurry and want to learn and want to binge learn system design I would recommend going you for the recorded one otherwise the Live code is where you can participate and discuss the systems and its design life with me and the entire cohort the decision is totally up to you the course details prerequisites testimonials can be found on the course page arpitbani dot me slash masterclass I repeat arpitbani dot me slash masterclass and I would highly recommend you to check that out I've also put the link of this course page in the description down below and I'm looking forward to see you in my next cohort so Twitter uses elasticsearch to power search of their tweets users and direct messages but why elasticsearch elasticsearch provides speed it provides scalability it is distributed in nature which means scaling out is just adding more nodes and more importantly it provides very simple rest interface to interface with it right because of the simple rest apis what happens is you just need a simple HTTP client to talk to elasticsearch servers and it makes it super easy to do so now before all the tooling that they built at Twitter what happened so every team was allowed to spin up their elasticsearch clusters to get their things done it's not just for search but maybe for other use cases they might use it for example real-time aggregation real-time analytics and whatnot but because there were many clusters like this the team owning that cluster had the responsibility to set things up correctly for example the because there are multiple clusters there are multiple teams owning it now who takes care of sanitization some team enabled logs on some clusters some team did not logs but in a particular format there was no throttling no rate limiting one team interfacing with other like one team abusing a cluster takes a cluster down affects end users and whatnot so there was a clear need that there had to be a standardized way to set up a cluster right but instead of just focusing on setting up it is also about how people are accessing the cluster so which is where the first tooling that Twitter built is an elasticsearch proxy now this proxy sits between the client and the elasticsearch cluster right so whoever wants to talk to this elasticsearch cluster cannot talk to this elasticsearch cluster directly but has to go through a proxy now this proxy is a very simple HTTP client and but because it is a simple HTTP client it can piggyback a lot of stuff for example this proxy because you know that all the requests will go through this proxy to any elasticsearch cluster now this proxy you put in all sorts of metrics all sorts of standardization that you would need to do for example thirdly right you don't want a lot of request coming from a particular client to to take down the elasticsearch cluster which is where this proxy comes in which has a rate limiter inbuilt right then routing you might have multiple clusters who like routing which request to wear take it's taken care by this proxy right security authentication right and more importantly monitoring when it comes to monitoring because this is a single point of entry for anyone to talk to elasticsearch now what happens you can dump key metrics out of this proxy for example what is the cluster Health what is the success rate what is a failure rate what is latencies and what not this proxy's job was really important this one layer that was added in within obviously it shot the response time by a fraction but this one centralized layer helps them ensure that the access to Cluster is standardized The Matrix that are published are standardized the logging is standardized authentication standardized right this makes their this increases the stability of the cluster because not people are not directly interfacing with elasticsearch right really important now second thing that can affect elasticsearch is injection because given at Twitter scale there would be huge amount of load coming onto this elasticsearch clustered when a new like for example a new tweet is posted where right rather when a tweet is posted it needs to be index in elastic search right now there are millions of tweets posted every minute now all of those gets indexed in elasticsearch so this means that there would be surge in traffic Whenever there is a spike in traffic at Twitter's usage right now elasticsearch is little tricky to handle such spikes and what happens is when there is a sudden Spike elastic surge does not get time to like like we do not get time to spin up more nodes very quickly because it takes time for the data to move from here to there right and elasticsearch is bad at handling such searches two things that happen typically is the indexing latency shoots up and the query latency shoots up so it like indexing the data into elasticsearch becomes slower and firing the search query on elasticsearch becomes slower so all the key operations take a hit now but at Twitter scale spikes or surges are natural for example if there is an event happening you would see a lot of people tweeting about it at that moment so there would be surge in traffic depending on how real world mimics but real world you cannot predict what's happening in what would happen in real world what would Trend at a particular time so that is where Twitter had to be very sure about handling spikes handling searches so what did you do instead of letting that thing directly get ingested into elastic surge they did it in an asynchronous way so any right coming to be put into elastic search now here it's more about that how it used to happen was people used to like directly call a pillar so the client used to directly call elasticsearch endpoint and used to put the data but now there is one more layer sitting before that which is like a normal single topic Kafka that you have like one topic for each cluster this is what they have configured and the rights are put into Kafka later to be consumed by workers and then indexed it to elasticsearch now what this ensures like there might be one more Kafka before that before it even comes here like maybe from API server one Kafka taken to this this is a proxy like so this is what is done by the proxiliant so proxy puts it in Kafka later to be consumed by workers right so there is no direct ingestion happening in elastic search so this one step one step deferred is what they did here it ensures that that we are putting in data in a correct streamlined way to elastic Source rather than relying on something else so few advantages over here first is request matching you can very clearly see how we have per topic uh one topic for each elasticsearch cluster when you put in a bulk of indexing request you can take few at a time and make bulk calls to elasticsearch reducing the lowest non-elastic search second back pressure given that if elastic search is slowing due to any reason the workers can slow things down very easily because elastic search is not responding they can just wait for some time before they start to readjust the data right then retrace in case there is a failure in case elastic searches are due to any reason the right request the injection is law is basically queued up in this Kafka and again later be consumed to be put into elastic search so you get resiliency just by adding this one more layer of Kafka in between it ensures that we are streamlining a consumption and ingestion into elastic search right the next service and really interesting one which is where we see the entire architecture together is a backfill service now it's a very common requirement across organizations that you will have to backfill a lot of data into elasticsearch this typically happens how so for example you have a job which is running which takes data from multiple data sources and it wants to index them into elasticsearch so typically what you do you write a simple mapreduce job this map reduce job would take data from multiple sources and make calls like each reducer will call elasticsearch endpoints to get it indexed now this is where the problem starts to creep in when you have large number of reducers running which is taking a lot of data mappers taking a lot of data passing it to reducers reduces ingesting into it into elasticsearch in synchronous call it would put load onto elasticsearch now imagine transferring hundreds of terabytes of data elasticsearch would cry it is not possible for elastic search to handle such high like such huge ingestion in a single chart so which is where you have to do it in a staggered way so how do you do it so they already had a service which was a map reduce job whose job was to read data from multiple sources when it was a backfill to read data from multiple sources and the reducers used to directly write to elasticsearch but now what they do is they read the data from multiple sources the reducer instead of writing it directly to elasticsearch they flush it into an hdfs hello distributed file system S3 any any compliant file system would work hdfs S3 GCS pick your favorite right so instead of their service the map reduce instead of writing it directly to elasticsearch what this job did is this job it put it into an hdfs and then there were consumers who were reading from it and then protected to elasticsearch these are massive ingestion happened right so let me walk you through this architecture now how how it beautifully solves the problem so what they have is they created a proxy for elasticsearch right so whenever someone wants to talk to elasticsearch it has to talk through this proxy so request comes to this proxy proxy sees if it's a read request read request directly goes to elasticsearch right there is no no points there you have to because you your you want answer to your query it will can only happen through elasticsears so read directly goes to elasticsearch now there is a write that is happening in real time for example a new tweet is created you want to ingest this tweet in real time into elasticsearch now some API server let's say this is API server API server is making a call to elasticsearch but what it's doing it is not directly making call to elasticsear because it's not allowed to do so it is making a call to the HTTP proxy a HTTP proxy instead of directly writing to elasticsearch it writes it it puts an event into Kafka from this Kafka workers consume and it puts it into elasticsearch right this way real time ingestion happens in a staggered way protecting an elastic search cluster right so like how users would a normally index directly to elasticsearch is now changed via this proxy which has very similar interfaces elasticsearch but now it happens in a staggered way and the more important part around backfill where they already had mapreduce job running which was directly putting it into elasticsearch now it takes like the HTTP request that we're making to index the data into elasticsearch now that same request with that exact same payload gets dumped into hdfs from these workers pickup and then they fire it into elasticsearch so there is no there is not a lot of change to be done from on the developer side of things it's all seamless like no need to change this map reduce job the only thing like one small change that needs to happen the map reducer where the reducer is used to directly put it into elasticsearch now they put it into hdfs and a bunch of workers are spun depending on the load or depending on the data that was created the orchestrator creates those very workers and they consume the data from the https and put it into elasticsearch right a very seamless way now here you see a pattern that the rights are always deferred the rights are always made asynchronous while the reads happen synchronously and this is one of the most common ways to build a scalable system the way you do it is let the Rights happen eventually but let the reads happen synchronously especially in a workload like this if it is a non-transactional use case you defer it all the time right so just to give you a gist read happens synchronously to elasticsearch Via this proxy the right through this proxy goes through Kafka consumed by workers put into elasticsearch while the backfill the job the map reduce job that used to directly write it to elasticsearch now flushes it into an hdfs which is consumed by workers fund direct spun dynamically using orchestrator and put into elasticsearch right this is such a beautiful piece of learning where we see how rights which are differed makes your system handle scale while reads happen synchronously right you basically there is no other way to do so right this is such a beautiful piece of thing like how Twitter at scale of Twitter how they are ensuring that their elasticsearch clusters don't go down while ensuring a very nice standardization by adding this proxiliary in front of it right great this is all what I wanted to cover I hope you found it interesting I hope you found it amazing that is it for this one I'll see you in the next one thanks [Music]

Channel: Arpit Bhayani

Views: 13,483

Keywords: Arpit Bhayani, Computer Science, Software Engineering, System Design, Interview Preparation, Handling Scale, Asli Engineering, Architecture, Real-world System Design, Engineering Blog, System Design for Beginners, How Systems Scale, System Design Simplified, System Design Interview, Twitter Engineering, Elasticsearch, Scaling Elasticsearch, Search Engines, Designing Search Systems, Scaling Search Systems, Search System Design, Elasticsearch Proxy, HDFS, Data Engineering

Id: dOyCq_mMtdI

Length: 14min 59sec (899 seconds)

Published: Sun Dec 25 2022