Deep Dive into Redis Replication

hi guys my name is Vichy Kassar and I work in Apple that provides infrastructure service for rest of apple first thing everybody needs to wear the headphone it has to be blue color for listening to me today I think I see most of it is blue so when I mentioned to my colleagues on hey I'm gonna be talking about Redis replication so they asked me why are you talking about Reddy's replication you know it just works it's been around for quite some time it's a very well-documented and it's not the newest shiny thing and if you're unable to hear me at any point can somebody just say this and I can get someone to help so the reason why I decided to talk on a redis replication is I've been fairly new to radisson register has been around for about 10 years and I've been working on it for about you know two plus years when I started with that is I thought I kind of knew how the rarest replication works and as I started supporting more and more that that is at in production then I realized I knew it's at the surface and I know there's this huge treasure trove when I dig deeper there's so much to know about a register application and today I'm gonna share some of that with you guys I cannot promise that each one of you will learn everything new here but if at least you know some of you take something new at the end of this talk I feel happy about the time I spent on it so here's a quick agenda you know I I realized that you know we have when you have audience like this you are at various levels of Radiesse maturity so I want to spend you know a bit of time in the beginning just talking about how replication works and then we won't move on to some of the readies configuration monitoring options then I want to dive deeper into two important topics which is data consistency with Redis application and then memory usage with ready application I'm going to share some of the common application issues that I see every day and we're going to top it off with a demo so replication basics you know we have lettuce has this Redis master and replicas so as a client you generally write to the Redis master some data and then that data as synchronously replicates to the replicas you can have more than one replicas and then from that point onwards client can actually read from the replicas or it can also read from the leader you know pretty straightforward so along with the replication you know generally you have a Redis failover service it could be Sentinel or something else it's a kind of monitoring your Redis masters and Redis replicas so if they master dies then it can actually promote an existing replicas to the new master so this is generally the idea so what are the some of the key facts you know replication is asynchronous in Redis first key facts to understand know as a client when you're writing data you do not generally do not get to control when the data has reached the replicas okay all that you know is your written to a Redis master and I think the data gets replicated as synchronously replication is non-blocking on the master side it's a very good thing right you don't want the replication that's happening on the master side to affect your client performance you know you want readies to response to your reads and writes as fast as possible irrespective of what happens so that's why you know it's been designed to be non-blocking on the master side and it's mostly non blocking on the follow on the replica side as well however there you know there are some windows there replicas when you make it read against the replicas you get the message saying hey I'm still thinking of it my master I'm unable to serve your reads right now master can have any number of replicas and generally the replicas read only but readies allows you to you know provide a config option where you can actually make the replica take right so but one key thing to remember is if you write to a replica that right is local it's not going anywhere else whereas if you write to the master it goes to all the replicas and also the rights that you do to a replica can get overwritten when the next time replication happens from the master if you write with the same key you know when the data comes from the master that gets you know your local rights gets overwritten but it may be applicable for some use cases but you know in general by default replicas are read-only and replicas can be master for other applica you know you can have a chain so this is super useful in the case where you have let's say three replicas in one DC and three in the other and you have a master in one of the DC's there's no point this master writing or the van to all the three replicas on the remote DC so instead what it can do is you know choose a local local master in the remote DC write to that and then that master can then replicate to rest of the replicas in that data center so what's the need for application you know first need of course is the high availability ready sees in memory database you know if if Redis were to go down you don't you want someone else to immediately take the data and solve it so you you can so this by having an in-memory replicas and you know there are instances where you need Redis replication for spreading the read load and this happens lessor you have some caching needs you can sell all your cashing needs with a single Redis instance and it can handle all the rights however the read transactions per second is so many that you cannot serve using a single Redis instance so for such cases again you replicate your data and send all your read traffic to these replicas so basically you're spreading the read load across the replicas that's the second need for replication and the third one is you know interesting where you do replication in lieu of persistence so you know that is as you probably know our first persistence but persistence comes with its own baggage I know you need a hard disk with your computer more moving parts or more chances for failure so there are you know some use cases where they don't want all these things but they still need a copy or multiple copies of your data and replications of those needs as well so what are the types of replication broadly there are two types of replication one is called full sync and second one is called partial sink and let me go or them full sync as the name indicates you know you're taking an entire dump from your master to your replicas all the data gets transferred all right how does ready smash to do this so and how does ready smash to do this in a way it doesn't affect the client performance right the way the Redis master does is it folks itself into a separate process and the full process is then responsible for sending the data over to the replicas it can do it in one of the two ways one way is the four process can actually write to the disk and then the disk file gets transferred over and this is how that is used to work couple of releases ago now there's also a disk last mode where the forked Redis master can actually write all the wire to the replicas so you don't really need the disk on the master side however on the replicas side you still this to collect this data so the replica can bootstrap and obviously no you use the full sink when a replica is bootstrapping it doesn't know anything so it says give me everything master I'll take all your data that's that's when this happens also then the partial sync which I'm gonna explain that whenever portion sync fails then you kind of go back to the full sync mode pretty straightforward partial sync basics the key to understanding partials saying is there's this concept of replication backlog okay in the in the master it keeps you know certain amount of memory reserved for the backlog and as and when new rights come in the new rights are also kept in the replication backlog with each byte that you write you keep an offset and that offset keeps increasing forever and this buffer is is not an endless buffer it's a circular buffer and it has a big in pointer of valid data and the end pointer like you know first byte offset and the current valid offset and in between that is your valid data right so the replication offset is you know where I am at currently in the buffer but you cannot take the offset by itself there's this concept called replication ID so when the master comes to life he's given a unique application ID if the master restarts he gets a new replication ID and the offset kind of goes hand in hand with that replication ID they're together you cannot just look at offset and say okay I want to i want that offset you should always say for this replica ID and this offset i want to sync right and there's also this replica client output buffer you know Redis is a in-memory database reading from memory super cheap but writing to the network is not cheap so there's an impedance mismatch right and plus ready single threaded so it wants to write as fast as possible to the client and then move on to the next work were you know you cannot write that faster the network and if you listen to Salvatore today he mentioned how that is the slowest part of writing so in order to break this gap Redis has this concept of client output buffer essentially whatever I just wanted to write back to the clients it tries to this output buffers and then merrily goes on and output buffer is then responsible for you know draining the data to the client and your replica is nothing but another client from a release point of view if you look at it you're writing data to it or the network right so replicas also guess a client output buffer so that also plays a role in the partial sync now let's put it all together so we have a client and we have Redis master and Redis replicas and if I double click on the Redis master we of course it's a you know it has a key value store that's the primary purpose of it and it has a backlog circularbuffer and it also has this replica client output buffer that I talked about now when the client writes some data so that data of course you have to first store it in the key value store that's the purpose of Redis but right after that the data is also written to this backlog circular buffer and to the repli'd client output buffer and whenever the network allows and maybe immediately that data now gets returned to Redis replica and the data keeps getting drained from the client output buffer but in the backlog the data is actually kept for any other replicas they'll come back bit later who is out to saying you know you you have still kept that circular buffer so you know quickly going over the replication backlog you can control the size through the replicas backlog size in your ready configuration and as I mentioned there are two key offsets here there is a first byte offset it starts at one and it stays at one until your circular but buffer becomes full and when you start over writing then this starts increasing and from that point onwards it's an ever-increasing buffer so then you have the master replica offset which starts at zero and with every bite that you write it keeps increasing let's take a look at this using a quick example so I have a replica backlog size with 50,000 and my first byte offset is 1 let's say I write some 21 K bytes so my master epoch offset most I write some more data now I'm near the end of my circular buffer so I write some more data my master replicas offset moves to 52 is an ever-increasing number and look at what happened to my first byte offset to 4 to 9 right it has more - what is the significance of the first byte offset at this point if a replica comes and ask me hey master give me 2,400 2,400 right then I'm going to respond saying 8 I don't have it in my backlog go away do a full sync okay so the valid data is always between the first byte offset and the replicas offset so now that we understood the back in a replica backlog I want to quickly go over some of the key configuration option so first how do you make yourself a replica of someone let's say we have two Reddy's instances one is running at seven zero zero one port on the localhost and a second one let's say I'm running at seven zero zero two if I want to be replicas after seven zero zero one I use a command such as replica of localhost seven zero zero one and replicas no one no one is a keyword where I say you know previously I used to be a replica but I no longer want to be a replica I want to be a master now so I can just issue the command replica of no one then I become a master okay so then the replicas backlog size that I mentioned the default value of that is one MB that you can change there's also this replica timeout this this has multiple meanings in Redis one key meaning is on the replicas side when you're getting this data from your master generally the data transfer has to finish with this within this time which is 60 seconds and if it doesn't finish within that time then that connection is cut off so this last one is the client output buffer limit this is the other output buffer that I talked about for the replicas and it has a hard limit by default 256 Meg and if the hard limit is breached connection is closed right away so what is the meaning of that you know under master you're collecting some data to be returned to replicas and that replica hasn't even come back so I do not have an endless buffer right so at some point I say eh you know I don't want to show the data anymore for you I'm just going to cut off connection so and then free this buffer and later you come back and they're full full sync if you wish so 256 Meg by default is the hard limit it also has a soft limit in this case it's the in by default is 64 Meg in 60 seconds what this means is even if the hard limit is not reached if you're at 64 Meg for about 60 seconds that means you're unable to drain the data fast enough also so even in that case I'm going to close the connection on you and ask you to come back so next I'm going to take you know make you go through some of the monitoring there are two key commands there is a Redis role and info replication so role is a very simple command when you invoke role on a master it simply says you know I'm a master I'm at this offset and I have one replicas in this case and my replica is running you know at local host and my replica is running at this port and here is my replica offset and as you can see in this example my master offset and replicas offset are not the same and replicas offset is a little bit behind what this tells you as a user is when you go against the master is that replica is a little bit behind and it needs to catch up okay and if I execute the same role command or the replica it says I'm a replica I'm at you know my master is at this IP this port and status is connected and words my offset so in for application is the other command that you can use to you know monitor rallies and will kind of in the demo I'm again going up or some of these things but let me quickly explain here in the info replication section the key thing to look at is how many replicas I have and for each replicas what IP what port I'm run the replica is running on and what is the state whether replicas is connected whether it's online or not and the replicas offset which is again you know same information that was given in the role and the last field there is the lag lag tells you when was the last time that this replica talked to me you know each replica talks to master have by default by every at every second so in a typical healthy production system you see this number going between 0 to 1 all the time but whenever a replica dies or the network connection gets broken you see the lag increasing okay and the next section I have is the the unique idea I talked about the replication ID for the master replicas offset first byte offset and the replica backlog size in this case I had set the replica bearclaw sketch to 50k and we also have this backlog hist LAN but that did confuse some of my colleagues so it starts at 0 generally increases up to the you know your buffer size and kind of stays there so I haven't found much use myself I'm sure there are some use that I'd not understand so in for application on the replica again as you can expect it gives you know the the replicas offset where my master is what's the replica ID key thing to remember is you are replica of someone else if you both share the same rappy an ID you can be a different offset and the job of the replica is to be as much in sync with the master get to the same offset but your application ID has to be the same that's just the that's the primary way you are a replica of some master and then you know for the replica itself it gives the offset first byte first byte offset and the replication backlog size it is not necessary for the replica to have the same backlog size as a master it can have a different one and hence it can also have a different first byte offset but your you know your replica offset should be the same or are trying to be the same there's also this master replica ID and a second replica offset that you know when you've seen for application you see what is this there's any of you know what a second replica ID is just by a show of hands okay then I can talk about this you know I mentioned that you know when you are a replica you actually faithfully copy a master by having their application idea of the master but there are instances when you are a replica you're promoted to be a master right now imagine there are ten replicas and you're one of those replicas faithfully copying a master and all of a sudden you are made the master by a failure process and immediately next what happens all these other nine replicas they come and hit you with hey I have I have this old replication ID I want to think with you the various used to deal with in the past was hey I I now have a new rep ID because my role change I don't know anything about your app ID come do the full sync with me but as you can imagine it's very wasteful because they were already pretty much in sync and an optimization in one of the recent release releases is to store your prior masters rep ID as a second rep ID and a the prior offset as the second rep offset so that way the replicas that were connected to the previous master when they come in to the new master you can still say yeah I got you you don't need to do the full thing you know you're already in sync and you know life goes on you know there's a pretty good optimization especially when you have large number of replicas so next I'm going to dive into data consistency on replication and let me start with an example so let's say you have a client it goes to Reddy's master and says hey I want to set my balance to 10,000 all right so now if the client leaves immediately from the racemaster give me that data of course it'll get 10,000 but with replication things are a bit more complex you know the client can actually go to a replica and say hey get me balanced but if the a synchronous replication if it hasn't happened yet then either you get you know this balance is not said I don't know about the key or you get the prior value of the balance both of which are bad right and you know I think replication can happen any time and as a client you do not really know when that has happened so now is relative 100% consistent database that's not one of the design goals of Freddy's but you know there are still some things that you can do to what I call as the best effort consistency okay I'm going to share some of the tips so that it will be useful to you to reduce the window of inconsistency okay the very first thing that you can do if it is possible is to read from the master you know you always write to the master your data is fresh current on the master if you read from master then it's very likely that you will get the right data okay however does it guarantee you hundred percent consistency no because you know your master can fail at that time a replica can get promoted and it's quite possible that the replicas that got promoted did not have the last bit of the right so boom there is inconsistency but you have reduced the window of inconsistency by reading from the master it may not be possible for all use cases you know what if you restore so many that you cannot achieve it through the right in which case you have to go to the replica but this is a choice for some use cases there's also this in a ready server config called min replicas - right then you see this name min replicas - right you think Oh boom you know there's a good option I can use I can let's say five two replicas I'll set this to two and as a client I will come to know when those are written to these many replicas but unfortunately that's not what it means this always goes to the second parameter here which is min replicas max lag what this means is if I'm a master if a right comes to me then I'm gonna look at all the connected replicas and look at their lag now if there are a minimum number of clip if I have minimum number of replicas that have a lag less than min replicas max lag then I'm going to accept that right what what it essentially means is you know I have so many replicas that are connected to me with you know not so much lag I'm gonna accept this right but I'm not going to guarantee that that they'd newly returned data is going to those replicas you know I'm just gonna you know give you some promises that I have this connected replicas but I'm not gonna promise you okay again not 100% consistent but you're reducing the window if there are some replicas that are like totally out of line then as a master you continue to say I cannot write your data because I do not have enough replicas with the sufficient sufficiently lowered lag so the next option that you have as a client is there's this great command called wait so you can say as a client you write some data and you you want to wait until that data reaches these many replicas but you know of course Redis is a distributor system in any distributed system things can go wrong you cannot wait forever so there is a timeout Milly's so as a client you can say I wrote this data and I want to wait this long or you know until these many replicas have got my data but but key thing to remember here is you have to do the call on the same connection and oh this confuses people some of the beginners the reason is you know when you're using a good kind library which does the connection pooling you do not know exactly on which connections things are going so if the weight goes on a different connection then you get the wrong answer okay so it's the key thing to remember here is you need to do this on the same connection that you actually did the set on or with your operation on to get the right answer this last one is interesting and we employed it a couple of times you know this is the case where you have a cooperative loader client so let's say I have a loaded client I want to load my master with some data you know generally I want to load it as fast as possible but not so fast that there is a inconsistency so what I what we did in this case is this loader client you know loaded some data then you know every few milliseconds it gets replication offset from master replication offset from my replicas and compare are they sufficient in sync yes then I continued my next batch of right this is this is again another tool that you can use when you have cooperative specially loaded clients so next I want to dive into a very key topic memory usage on replication you know the job of the raddest is to store your you know keys and values so you you need in large amount of readies memory is used for that purpose then there's this backlog buffer that I talked about so and Redis has this max memory setting you know what I want to explain here is the max memory actually is divided between the replicas backlog buffer and the memory used okay let's let me show that as an example let's say I set the max memory to 10 Meg and I set the backlog buffer to 5 Meg so what that does is the user space available is now only 5 again key thing to remember now again when you run that is in production you know you set max memory to some value let's say 10 gig and you know occasionally you see process is taking more than 10 gig so why is that so here's reason you know I talked about this replica client output buffer and that buffer size is not counted in max memory okay that is in addition all right and then is the replica client output buffer the only buffer that reddy's uses no actually it uses such buffers for every client so there are other clients others other clients and there are other buffers so your actual memory usage of the whole process is you know is like a total of this you know if you have to take one thing away from this talk I want it to be this I haven't seen this picture in you know that is documentation where it's documentation is great I love it by the way but I think this kind of puts together you know how the various buffers kind of play a role into your process size because when you operate read is in production one of the frequent things is why is the heck is ready you know or shooting my max memory and what can I do to tweak tweak it down or pick it up so some of the key points is you know replication backlog is a single circular buffer that's shared across all the replicas and you should remember that it reduces the memory available for user keys right so as I mentioned if you specify max memory of 10 gig and if you make this fight gig then your user memory is going down there and the replica client output buffer is one buffer per replicas if you are multiple replicas then you you know each one of them gets a buffer and by default each 256 Meg so you know if you have ten replicas it can easily add up alright so again key thing to remember when you're running Redis in production and this client output buffer has a hard and soft limit like I explained and it increases the memory used by the process so this is my last light before I dive into demo mode where I demonstrate some of the things that we learned today so I just wanted to share some of the common replication issues that you see in production one common thing is you know your full sink fails you know this is a common message fork was unable to allocate enough memory and this is very common when you you know host multiple various instances per host right so what is happening here as I mentioned you know when Redis does a full thing it has to fork itself when you fork you know it the 14 fork process needs some memory right how much memory does it need let's say I have a Redis master of 10 gig do you need another 10 gig when you fork right actually not you know with the modern operating system and copy-on-write semantics all the memory pages are shared between your master and replicas as soon as the replicas so not a replica with 4 process so only on modification the more and more memory gets used up so it is possible to actually fork your process with minimal amount of memory but then can you leave no memory you know that's kind of too adventurous and too risky right can you leave as much memory as needed by your master I think that's like too chickening out right so you need to find a middle ground and also you need to understand your data characteristics on how your data is getting accessed is the data getting accessed in such a way that may be in a single memory page is getting updated repeatedly or it's going all across all the memory pages in which case the memory needed for the full process is much higher right so those are some of the decisions that you have to make and even after you make those decisions there is this key thing called or commit memory and by default it is zero what that means is if I'm a master of 10-gig then I'm about to fork for full sink purposes unless I had the 10 gig OS does not let me which is kind of your not good for Redis so you should set this or commit memory to one what it says is yes I'm a 10 gig master but you know I understand everything about you know copy on rights and I know that I need lesser memory so I'm just giving in hint to the OS you know please launch me I know what I'm doing okay so I think for Redis even I think when you start up it says please set your or commit memory to one if we detects that it's been set to zero it's just you know it's it's really really essential so other common thing is replica gets a timeout receiving the full sync data from master this we have seen in production where you know your data set is so big and you know for whatever reason network connection was not that great so such cases you know it times out so obvious answer is you know increase your replica timeout or reduce your data size full full day to full sync data size this last one is partial sync fails with a message client scheduled to be closed ASAP and this happens when your client output buffer that I talked about is buffering the data for a replica it it gets overwhelmed so the master decides I'm not going to keep the data anymore I'm going to just you know shut down this buffer and close close your connection okay so these are some of the common problems and when that happens the last one so one one obvious option option is to increase your client output buffer size again none of the buffer is endless they all have a cost so you know many of them are trade-offs okay so next I'm going to dive into the demo mode so you guys can see everyone's can see the terminal right okay so for my demo so I'm gonna have two ladies instances my master running at seven zero zero one and my replica running at seven zero zero two and we're going to show a bunch of things okay so I'm going to first start the master master is starting in port seven zero zero one it's ready to accept connection we do not have any any replicas yet so I'm gonna I'm going to see show me the role of the master RC one is for my shortcut for readies CLI port some in zero zero one so it's going against poor zeros port seven zero zero one and asking for the role as expected just a master and it said offset zero because we are not yet written any data and the last line you see there is an empty list because I do not have any replicas so let's fix that let's start the replicas so replica started and I want to quickly show you what happened here replicas you know is trying to do some partial synchronization it was not possible then it did the full thing from the master the thing that you see with the for aid blah blah blah so that's the unique replication ID of the master and the zero that comes at the end is the offset and it received some standard bias in the beginning because there's no real user data so for everything is good so now I come back here and say hey master show me your roll again so master says now I'm still the master still offset is zero now I discovered a new replica and replica is running at seven zero zero two pretty obvious okay so let's now move on to the other monitoring command readies it ready see a lie-in for application this the command that I ran and this pretty much you know shows all the things that I described before I have one connected replica and the 4700 to its online offset is zero of the replica and lag is 0 because we are well connected okay so far everything is happening okay and the master replica ID starts with for a remember this because it'll come back replica I did - we don't bother right now replica offset is zero okay and my first byte offset as I was showing in the slide is one all right now let's start to do something interesting with this so I have this Lord loader client it Lords number of keys and with each key I'm going to write about you know 1000 bytes okay let's write one byte sorry one key with thousand bytes and let's see what happened here and this loaded client also does a in for application on the master as you can see now it wrote about thousand bytes but my replica did not have time to catch up it was still at offset zero that was because you know I did this in for application as soon as I wrote but has it caught up right now so if you now see master replica offset is one zero two three my replica is also at one zero two three and my lag is zero okay so now let's start writing some more data I want to bring it very close to the end of my replica backlog size my replica backlog size remember it was 50 K so I'm near the end of that circular buffer because I wrote about 49 K and my first byte offset is still 1 so how about if I just write one more key what happens to my replica offset so mass replica offset is ever increasing so it went beyond my circular buffer but look at what happened to my first byte offset that also moved now because the first 23 bytes have been overwritten right so at this point if some other replicas to come to my master and say hey give me a bite you know I offset 20 I'm going to say no I don't have the data for you all right now let's see what happens if i load 50,000 more i just want to show you guys what happens to first byte offset first byte offset is ever increasing from this point because I'm beyond my circular buffer size all right so you know so far we showed how things are working with offset but you know nothing bad is happening right because the sync is happening the connection between my master and replicas is going good though so let's start to introduce some instabilities so I have this stop replica command as you can see is going to send a kill stop so when you send this signal your replica is still running but it stops responding to anything okay so this say level of instability I'm going to introduce and at this point I'm going to write a key to my master so I wrote some data the first thing to notice here is lag is increasing it is 24 we never saw this before we are seeing this now because my replica is no longer connected okay so what is happening to our offset here I'm at 1 0 to 3 1 you know 1000 less but has the replica core top let me do an info replication lag is increasing and my replica of that is not catching up it used to catch up before now it is not catching up anymore because we you know basically you know the replication is not happening right now I have this continued replica so it's basically now says ok replica now you continue now if I were to do in for application on the master look at this log lag has gone back to 0 and the master and replicas are happily in sync again right so we did introduce some instability where the you know partial sync actually worked now right that data was actually buffered on the master and when that work came back it actually wrote that data now let's start to introduce bigger problems so I'm going to stop the replica and this time I'm going to load up so much data that I'm going to overwhelm its buffers so I'm gonna say Lord 10,000 keys and I want you to take a close look also at our server log so what did he expect to happen as I mentioned in my talk already our client output buffer was all run and master decided hey client you're too slow I'm not going to store any any more data for you I'm gonna basically close right close this connection so that's exactly what happened here now again pay attention to the server log on your top screen I'm gonna now continue the replica and what do you expect this time so the replica comes back and says ask the master hey give me the data but master rightfully responds I'm unable to partially resync because the backup that lack of backlog right as as expected and they go into this full sync mode and now they're in sync are they in sync really so let's check we're doing the info application and the offsets or is still catching up maybe yep six two one six two one and the lag has come back to one so what we demonstrated here is partial saying you know when it's possible it works when it breaks it goes back into the full sync alright so next I want to demonstrate the data inconsistency thing I mentioned so I'm gonna for doing this I'm going to start a Redis client session and then I'm gonna set a key and I'm gonna say has my key reached 1 replicas and I want to wait for 10 milliseconds ok and what do you expect answer is going to be because the full sync is now happening you should say yes one replica got your key all right now let's go here and stop replica well now stop replica and now right said let's say different key v2 and I'm going to execute the same wait command this time remember right now think is not happening what do you expect dance is going to be let's find out so it is zero no matter how many times I try it's going to be 0 because replication is not happening ok so this is a client you're controlling right you wrote a key and you can proceed only after that key has reached the number of replicas that you want and this is the way for you to control it and let's just finish the story here so I'm going to continue the replicas and if I come back here if I do the same weight command what do you expect this time the think is happening I should say yes wonder picker got mild right ok so what we demonstrated here is a way you can achieve best level consistency as as a client using the weight command and before I show the last memory part the thing I want to show it really quickly is switching the leadership switching the mastership and how it effects your second replica ID all right so I have this switch master what this command does is switches the master shape from seven zero zero one two seven zero zero two but before doing that it actually takes a dump of the info replication on the replicas side and it takes a dump of that after doing the master shape and at the end it switches the matched ship back so that the demo can continue so I need to mention I want to switch the mastership to seven zero zero two and let's go over here what happened in a minute so it does a before dump as I said it was the older up and the replica ID remember this it starts fit for a that by d2 was 0 the default value and offset was something that's ending with 7-5 3 and then it did the switch so I did I invoke replica of no one and made seven zero zero one the replica of seven zero zero two and let's look at after look you know what what happened when I switch to the switch the mastership so the new master got a different rep ID but then it saved the old one in this you know master rep ID - and the offset was still stored as is and the second replica offset was bumped by one and it's told the reason why is plus 1 is when replicas come they asked for the byte that they do not have it yet which is the next byte so it's gonna bump up that number from this point onwards if I write new data only master replicas offset keeps moving keeps increasing and second a pic offset is kind of frozen at this point okay so what we demonstrated now is you know the uses of the the second replica ID and offset so the last part of my demo let's see how you're doing on time should be okay the last part of the demo was the impact of memory right in order to do that I need to put Redis in a no eviction mode so I'm setting max memory policy has no eviction what happens with this is Rattus keeps taking the rise until it has enough memory in this case my max memory is set to 10 Meg and I want to see under these conditions how much data I'm able to write to the register I'm going to start with a clean slate I'm going to flush everything in my Redis master and I'm gonna attempting to load and thousand keys and I won't see how many I'm I'm able to write so obviously once that 10 Meg limit is reached I'll be unable to write though many keys so as expected I got a out of memory and I was able to write about you know 8300 keys I cannot write anymore because the Reddy's became four and I'll set the max memory policy to be no evict so that means it cannot take any more right okay so now let me flush it and this time I'm going to change the backlog size Oh a backlog size was a very small 16 K before for the first exercise I'm gonna now make the backlog size as 5 Meg and if you remembered my slide if my max memory is 10 if my backlog is 5 Meg that only leaves 5 Meg for the user keys so if I were to repeat this load of 10,000 what do you expect you expect less number of keys to be held in that is let's see if that is true boom you know I'm only able to store about half the keys because I consumed half of it by the backlog buffer ok let's take you to the extreme and this time I want to say I want to set my backlog buffer size to 10 Meg which is same as my max memory right let's do this let's do the flush and at this point I'm just writing this simple key 1 value 1 I'm no longer able to write even a single key 1 because even though I gave 10 Meg I consumed all of that with this replica backlog so I hope it sticks in your mind that you know how replica backlog and Max memory kind of work together so let me set this back to some sensible number which was same as before so the last thing I'm going to show you today is the impact of the output point output buffer on the process size again going back to the slide you know the output buffers they're not counted in max memory but they keep increasing your process so let's attempt if we can do that so my current output buffer limit you know this is the key number you know this number actually controls now this command controls client output buffer for different things but one that's starting with slave is the one for replicas currently at 16 16 K I'm going to change the output buffer limit to the default which is 256 Meg alright and I'm going to flush all starting with a clean slate and I have this command called churn what it does is I give some number of keys so this one basically keeps all writing the key with you know same value over and over and over again lettuce doesn't know that the value is same so if the impact of this is generating a large amount of backlog very quickly so your backlog gets full your replica output buffer becomes full so let's see when this is running what happens to our process and or on the top window I'm gonna do monitor my master monitor master is they come and basically you know I'm doing it top and grabbing the memory that's used by my master okay so we are at can Meg right now and this is just for master all right now I'm gonna start churning so I'm writing large amounts of data now and what what did you expect to happen for this you know memory right it's kind of staying around 10 Meg it increased a little bit the reason why it's staying around that is because your sink is happening correctly there is no nothing being kept all right so let's kind of you know let's be the bad boys and stop the replicas all right and watch that number moment I stopped it you know my client out buffer is you know buffering this data for this client which hasn't synced up yet it keeps increasing increasing it goes up to 256 Meg and boom it came down to 31 Meg can you guys make a guess on why it came back as reason is obvious because we closed that buffer say hey client go away I don't care about you and then it it releases and that can be evidenced by the by the master log where it says I'm going to close this connection alright so let me again continue the replica so it can do a full sync and you know that is is you know reasonably great but at giving back memory but I would have expected the memory to come back to 10 or 11 Meg here but it's not right I don't found answers of all you know all the answers here so there's still something that Redis escaping I think there's this Co for optimization here ideally in my mind if it was you know Levin back before I buffer something send the data now I'd no longer need the buffer it should come back but it hasn't so now I'm gonna attempt something very adventurous do not try it at home so output buffer limit and we're set to 0 what it means is I I set no limit let the buffer increase as much as you can right so I set this and I know nothing is happening to memory because the sink is in progress so again let's top the replica and this time it's a recipe for disaster it just keeps increasing there's no limit and then you know that's why I say never attempt this at home I'm just giving a demo here so I need to stop this pretty soon myself otherwise my left off will be no longer usable so let me continue the replica yeah and and the memory came back alright so so I hope I showed in a demo how'd you know the partial sink works how it kind of falls back to full saying how what's the use of this secondary replicas ID on a leadership change how you as a client you can drive the some of the data consistency parts and the relationship between the various memory and the buffers alright so let me switch back to my presentation and you know if you like my talk yeah I just want to plug here that we are hiring so please come and hit hit me up thank you and ten years of age [Applause]

Channel: Redis

Views: 4,020

Rating: 5 out of 5

Keywords: redis, redisconf, nosql, data, databases, replication, redislabs, ai

Id: esbRryo0Ty8

Length: 56min 54sec (3414 seconds)

Published: Thu May 02 2019