Automatic Redis Clustering

hey everyone I'm Luke hello so this is a automatic right is clustering and pitfalls this is about open source Redis cluster which we've used quite a bit at twitch and I'm gonna basically tell you guys what I've run into and how I've tried to build around it and the slide working there we go so twitch I work for twitch they've III honestly was a little too lazy to Google like stats but twitch is big we do video game live streaming we run by amazon where montgomery station actually we're just down the down the SF really close cool nearby and we have we have a lot of back-end infrastructure that uses Redis and in particular one of my services so you know you don't need cool stats it's enough before I start so at twitch I run a service that runs on bare metal we decide to run Redis on premise to minimize the latency but in hindsight we probably would just use a managed solution in the future especially based on this talk you'll see the end pitfalls part of the talk because somebody else can do this for you you don't actually necessarily need to implement all this yourself but this talk will give you at least some information how you could do it how you could use open source Redis for clustering mode and if you just want to use a managed solution and just be fine with that then you can just zone out like sleep whatever it's gonna be an easy talk for you then so operational goals so we have a lot of different hosts at twitch we run our own data centers at least four and this service runs on those hosts so we want to make it so operations life is easy so we want to run yeah exactly we want to run Redis nodes on hosts and just kind of run whatever like just a provisional host just start up some Redis nodes automatically create a cluster we don't necessarily want operations guys coming in and typing commands to try and create the cluster themselves we also want to just automatically create the cluster so you can pretty much just to think of operations is just hitting the on switch on a host and all of a sudden the host will join an existing cluster or create one if it doesn't exist of course redundancy so if a host fails we don't want to take down the website which has happened I mean it happened it happens to every big company that we forgot to have like a replica of the database and that went down and it's like oh no to that extent as well we want to avoid local replicas I say this because at least Redis cluster that's not necessarily built-in where you can have a case where you can have a master node and you can have a replica on the same host so if that has the host as a you know bad Ram or something you just you lose the the replicas as well as the actual master so not a good thing scalability so one of the really cool things with Redis cluster and what really drew me to it is the fact that you can run these single threaded nodes Redis is just single process it can't use more than a core and it's really optimized to do that very very well the idea is that you can just run multiple processes and you can run any number you want then the actual host to use as many cores as you potentially want to use so that's one of the the best things I would say about Redis is the fact that you can just run as many nodes as you want and they just they add add add add and then eventually you get a full cluster this is great for us because we can just if we add a host we'll just add the same number of nodes to the cluster and it scales that way so load balancing we're almost done with the goals but we want to make sure that again every host has the same number of nodes running that so if we have three nodes running on a host we want three and every other host we don't want to necessarily have one poor host doing too much work so that goes the same with the slots which is keys effectively and masters because master nodes do more work than replicas so effectively we're assuming that we have this big data center that has hosts of all the same type and we want to make it so we can basically turn on host it takes the same amount of traffic as every other node it just automatically you know Big O n scales up and onwards as you had hosted the cluster so operations loves me in theory if I do this correctly in practice they don't love me but so yes let's write a cluster management tool I want to go over what exists out there and why we decided to write our own and how you would go about doing it and etc so first off what's built into Redis so Redis does not have an automated management for clustering so Redis has a tool Retta cui that's lets you set up a cluster it's a pretty cool tool it has other commands like fix rebalance but it's very manual the idea is you're supposed to have somebody wake up in the middle of the night and realize oh I have to run this super secret command to fix the website if they don't oops the website goes down you also have to create it so let's say I provision a new host I have to go through I have to run this command to create a cluster or to like add to a cluster so the reddest existing tooling for for the open source clustering it's very very much manual so you have to list the actual some flags get the list of the IP port of every node you actually want to add to the cluster once it's set up it runs pretty well unless you like to make changes so Redis cluster as a whole is there's not too many tools out there there are some open source tools this is just the you know what I want to point out the the built-in also yeah you shouldn't use loopback and this is a this is it from the example from the clusters tutorial if you use loopback they'll and you don't specify that I bind address they'll all advertise loopback and then your cluster won't be accessible from outside of that single host so yeah some gotchas here is that to actually run the Redis see like commands all the times you have to figure out what you need some commands take IDs of nodes some take host port some of them certain configurations you need the special command so it's hard to automate this like you couldn't actually necessarily run just like let's say some script that will figure out what you read a CLI command to run it's very difficult to figure out what to do it takes a lot of I would say operational knowledge of Redis which is not something again to be friendly to the operations guys we have a lot of operations guys at twitch and almost none of them know how to use Redis I don't want to have to teach them you know how Redis cluster works what commands to run I want to write something you know so they don't have to also they have to wake up at 4 a.m. if they get paged or sometimes I have to get woken up if they can't do it and if you're running this big command like this is a pretty same command comparatively but let's say you typo something you just took down the website you know it's not safe so instead of alerting somebody instead of getting a human in the process we want to fix it you know I'm a software engineer I write software I automate things I try and get people out of jobs so you don't have that you know you don't need to pay somebody to wake up just and know about writers so there's a few different approaches so without exploring really the open source space which is trying to break them down into categories so if I wanted to have some management tool I could just have a single single script that just sits there and pulls the state of the Redis cluster and if it's in a bad state it will fix it this is kind of similar there's actually a fixed command in the Redis Eli which will fix clusters and do so you can also have multiple masters you could have multiple nodes that all try to you know some redundancy if that fixed script goes down what happens your writers cluster will keep running just fine but eventually it would get into a bad state so you want some redundancy and with redundancy comes coordination and synchronization so you need some external source to make sure that the management tools aren't going to fight with each other or you can be really clever like I tried to do and just have decentralized management just have it so you have this script that just tries to fix it and fights with everybody and tries these algorithms and the I failed it's really cool idea because the idea is I want to be like Redis cluster there's no owner of a Redis cluster every node kind of ends up gossiping with each other and in theory the management tool could work like that too but there's some gotchas hence the talk so I want to go over more of the details of how Redis cluster in works I know that some of you guys don't know about Redis cluster is and I know I'm glossing over a lot of important details but I want to cover a little bit so first off you have these nodes these Redis nodes a lot of these talks you assume everything is just a single Redis process so when you issue a command you send it just to that IP port the way that Redis cluster works is you effectively have a bunch of nodes that end up joining together and they all a little bit of the key space so you can create them you can make them join each other with this meet command IP port so the idea is if you know one node in the cluster you can say hey you meet this other node and then they'll start gossiping amongst themselves and no creo a cohesive like Redis application you can think of that it can then use all your favorite Redis commands with some exceptions there's a lot of there's a few commands that don't work because the data has to be sharted maybe that's the so for anybody this is database sharding is what Redis cluster is effectively so if I'm creating a new cluster I need to be able to run cluster meet on all the nodes I want to make it so I can turn up spin up my new host call cluster meet on the existing nodes and then I'm I've joined it so he's a little example where I have the new node on the left I can call meet and give an IP address and then it's effectively a ping command and you'll get back gossip about every other node in the in the system so the gossip is without going into much detail it's pretty cool just every X seconds nodes will send a random subset of other nodes they know about and that way you just by meeting one you eventually figure out everybody else in the cluster eventually consistent is the word for it so so yeah that's all you have to do to create a cluster right some gotchas they're so so Redis doesn't have any discovery mechanism which means that you need to figure out have some sort of discovery mechanism you need to how do you detect that there's a cluster running for example in this data center you how do you detect what node is running in the cluster so you can join it so you need some sort of third party orchestration there's no easy way you could use we'll go over the solutions but fully joining the cluster also takes time the idea being if you're at a management script and you say meet you can't assume that that node knows about every other node at least for a couple seconds which means whatever management tool you have to do has to realize that there is a gossip protocol there so nodes will have incomplete views of the entire cluster while they're joining that's a gotcha as well so you need to make sure either like sleeps or something you need to make sure that you realize that there's this gossip going on and then the random gossip to that point is it's not fully random but again nodes will have different views of the world this comes up a lot so different solutions like maybe you could just hard code IP port it's great for a startup or something or just I get a side project like yeah just sure this is my one master node just just connect to that that's the cluster owner you could also have somebody operations around a command you can have them figure out what the single hard-coded IP port is and then run the ready CLI command maybe land broadcast could do that or external source is probably a little bit saner like a zookeeper console etc there's a lot of different services out there you but you still need to have discovery be separate from Redis clustering Redis is great for discovery once you've joined because it's you can see which every other node and the cluster is but before joining you have no idea where they are there's no discovery in that regard cluster leave so you can leave a node can leave a cluster this is very dangerous but what you can do is you can take the d of a node and tell every other node to forget it so what it does is basically any internal state has about that node it just completely removes it and has a blacklist to make sure that that node doesn't try to join the cluster again because otherwise the node would be gossiped around right and it would rejoin naturally so hey you can just let's say you wanted to and I get in your in operations you want a deep revision a node you would have to run cluster forget on everybody on every node to try and tell them that no you shouldn't talk to this one anymore I'm about to shut it off gotcha is there so make sure you don't forget the node before transferring all the data away because Redis does let you do this you can forget nodes that actually have data on them or forget forget master nodes that have data the every other node will be like sure yeah okay I no longer know about this why I've just completely forgot about him and if you're in the process also that if a failover let's say you forget a replica while it's in the process of becoming a master you just forgetting is also from each nodes point of view so one node can forget about another cluster another node but not the other way around so a lot of got you there gosh is there read please can't forget masters so let's say you end up going through and try and make every node forget it like this one particular one you could totally get in the case where oops like you just this node will reject the forget command and you're getting this inconsistent state the inconsistent state is also particularly problematic mostly because Redis cluster requires consensus so in order to mark a node is failing you need to have a majority consider the notice failed which is really really cool and it's great for automatic failover to the point where you can just have you know a bunch of nodes they all independently have their own health checks and if a majority gossip that the node is failed it counts as failed the problem with these forgets is if you're the one that knows about a node you can't get consensus that that that node is failed so we've had the cases where we have nodes where they for whatever reason didn't get to forget command either because I couldn't forget the master or whatever and then they just had this entry in their nodes table that there's this node that's probably dead but nobody else seems to think that exists and it's not gossip because it's potentially dead so yeah forgets cause problems there's a and this is my little example here where you can have issue of forget command to the three nodes and one of the nodes can say actually no I'm not gonna forget that and then it will have this entry at the very bottom saying P fail means potentially fail in Redis terminology I think this node is probably failed because that my health checks aren't working against it but nobody else knows about it so I'm just gonna keep it in that perpetual probably fail but cake I can never fully upgrade it so this this has caused a few issues at least for me some solutions this is maybe too in-depth but yeah you want to make sure at least that you first make sure that the node doesn't have if you want to shut down a node you make sure that does have any data either manually do a failover or migrate all with slots then you have to do this like close to forget on every node and you just have to be very careful about making sure every node actually receives it if you ever have this case where again you're running a command that does a mess forget and it cancels or times out in the middle you're in this really scary state where your cluster is going to be it's not gonna reach consensus or do nothing this is actually what I ended up doing for the longest time where turns out if a nodes not alive it's gonna it's gonna be annoying because it's gonna show up as one of the members of the cluster but it's gonna be dead and for all effective purposes it basically isn't there it's just gonna be this like garbage that you of of X members in the cluster so yeah you can you can totally leaving a cluster is actually surprisingly difficult I would say it's a lot harder than actually joining cluster which is interesting uncovered slot assignment so here now we start to get into how Redis cluster actually works a little bit I've tried not to have a big slide trying to document I'm sorry confusing you yeah well but the way that Redis cluster actually works is you have these slots you have slots 0 through 16 K the idea is that you then hash a key and it will go to one of these slots now this is true for open source Redis this is true for some of the other managed offerings to they just use open source reddit so they just do this under the hood for you but the idea is that a node owns only a partial range of the slots and you want to make sure that those slots are covered so that somebody owns the slot and then clients will figure out who actually owns the slot so here's a little example first off is if I want to just call it around the command get X Redis has a hashing algorithm that's advertised it's both on the client and the server side so the client knows that X when hashed equals three nine nine nine and knows that based on the advertised slot ownership that the first node actually owns 0 through 5461 so that means it actually should own X and it immediately goes to the actual node that should own X so this is great because the instead of having redirects or proxies you end up the clients go straight to the actual source of the data I there's some some problems here though so first stuff let me go back to the slide so the command you actually run to assign slots in initial creation is the ad slots command you tell a node hey you're in charge of these slots and interesting enough this is also one of the reasons why it's not easy to automate this is you have to enumerate every slot manually so 0 1 2 3 4 5 up to 16,000 so you're gonna want it like some bash hackery if you want to do that by hand but usually you want a tool to do this running these raw Redis commands is is not quite not necessarily the easiest and there's also some an epoch but basically these the slot ownership is gossiped among every other node so if one node says i 0 able tell every other note I am slot 0 and here's my epoch which is used for tiebreakers epochs kind of like the time stamp more or less I went sited on 0 but in the cost in the gossip space cool so so in this case I've already have a set up cluster I just called ad slots on all these nodes to assign them ad slots so if you're thinking about i'm let's say I'm writing my cluster generation tool I want to create a new cluster how do I decide who's gonna own the slots initially if you have a manage like if you have somebody in charge you can say oh yeah this node owns these ad slots as ed slots just be careful if you have multiple nodes that decide to own everything so if you want to go to be the really smart decentralized solution you end up Epping nodes trying to own the entire key space because what happens is if a node doesn't own any slots if a slot is not owned it will just error so you want to make sure that at least one slot owns one node owns it but if you have all them try to own the entire cluster you'll just call as close tiebreakers you'll you you cause them to fight and in theory you can cause data loss as well in the case of slot 0 so it's already data in slot 0 and somebody else says hey I own slot 0 you'll it could just take over and cover any data that was there so first off I'm creating my cluster 1 solution you can do is just pick one a single node and tell it to own the entire key space the idea is that you'll migrate away the other option is to do it on every node so you decide to shard upfront and you don't use the rebalance so this is pretty easy though there's not many gotchas here this is actually kind of nice for just for initial assignment covered slot assignment or reassignment so to the example let's say we had the the cluster before but the node on the right dies and I didn't have any replicas you effectively have this this key space these slots that are not nola recovered so you're in this situation where oh no like I forgot to I I don't know who covers slot 6000 oh and this is really bad because if you're actually running in production in this mode you'll have a third of your requests effectively just error on this because nobody actually owns this data and Redis doesn't have a built-in way of recovering either usually the way that the Redis cluster works is that you have replicas and if a master dies the replica will take over but if there's no replicas like let's say they died as well or you just decided to be very bold and adventurous and just not use your redundancy you're gonna have this case where your your queries are gonna keep failing until somebody steps in now this is the case where we wanted to avoid especially we don't want to necessary this does a really good job of self-healing unless you're in this case where you you actually have a master that doesn't have any replicas so so yeah the cluster won't recover until the Masters turned back on that's how you're supposed to fix it you just turn turn the node back on it will realize oh hey I still own this data and it will if it was persist of the disk everything's fine but if you can't turn the node back on you're in a tricky situation so a lot of gotchas here so ad slots which your great command will actually not work if there's a master that it knows about so if we're back to this world that we we know that the node on the right owned those slots but it's failed we've it's failed to health checks my ad slots command won't work anymore it only works if nobody claims to own it which is kind of annoying but I also can't do a manual takeover because I need actually a replica in order to a manual takeover I I can't create a replica either because the node has failed so you're like crap okay so there's slots that are uncovered and I can't tell another node to cover them so you also don't necessarily know for example if there are replicas because it's perfectly valid for actually that'd be the master no that failed and replicas that haven't taken over yet because it's an election it takes time it's kind of cool how there's a there's a replica election but you don't know what the status of that is you don't know if that replica election is going to take place if it's still going how many how long do wait because they do need a majority consensus of note before they'll do the takeover and unfortunately there's a this may be a bug this is my little this talk is kind of a bug report I'm sorry everybody but the nodes replicas will report their previous master ID so let's say you have a replica that's replicating node a and then we decide to switch it over to node B but it can't do that for it or reason like maybe be turned off in the process or there was some some failure where it can't actually sync the data it can't become a replica properly it will still report itself as replicating node a so there may be situations where it looks like a master node has replicas but in reality those replicas don't know like they're actually in the process of connecting to a different node instead and that loops forever there's no timeout so you could be in this case where yeah I'm still just reporting the old node so you don't know when the cluster failover is going to take place and by the way I want to re-emphasize this has not happened with managed Redis a lot of people do this for you this whole talk is like if you want to do it yourself so because you're like this is these a lot of gotchas and this is something that I went into Redis cluster not knowing a lot of these and ended up figuring it out through a lot of production instance so there's some solutions here you need to timeout you to make sure that replicas eventually take over you also need to let's say you have this node the dead node that owns like slot six 6000 the only way to run ad slots and to fix the issue is to forget that node you have to tell every other node to forget it and then quickly do ad slots to say hey take over 6000 and do the tiebreaker so which is very dangerous because again the issues would forget I mentioned before and the hope is that the epoch will save the day so if you have one know that knows about the failed node and thanks to dawn 6000 and then another node that claims that one 6000 but but doesn't know about the previous one you're hoping that the epoch collision will save the day and swoop in and it will it'll now manage to cover the cluster although you will lose every all data like there's no way to avoid losing data and there's no way to bump the epoch either so you get this kind of convoluted case to try and fix what it really should be more of an edge case but it's trying to say I may be one takeaway from this is always have replicas because if you don't have replicas you'll get in this state where somebody's gonna wake up and have to figure out how to fix this situation and the answer is very difficult rebalancing so my my key space 0 through 16,000 let's say I had three nodes in the cluster and I want to add a fourth node so it used to have every node a third of the key space now I want them to own a fourth of the key space right you actually have to do that manually you have to do the migration and import in yourself again a tool like a Redis Eli will automate some of this for you they have a rebalance command but there's a there's some some issues there so the way that it works is every slot is in a state of either stable importing and migrating and you slowly go through every slot and you tell them to migrate to this one or import from this one and then move the data over gradually you choose the rate at which you can move data which is really really nice you choose the rate of the rebalance so you can do it either really quickly or really slowly but you have to know more about what you're actually use cases to decide what you should do and you just keep migrating data until there's none left so during the importing migration as well maybe there's a little example here if I want to move from two notes to from one node to another you would say that the slot is migrating and it's importing on the other end and you individually migrate key and keen key the way clients work this is just a little Redis cluster thing is they will they will figure out which node so let's say this X hashes to 399 they'll first ask the the owner if the data exists if it was an existing key and if it wasn't stinky it can just be served in the original note otherwise if it's a new key it should go to the new node the one that we're migrating to so there's actually a redirect with the Redis cluster during the importing and migrating State for clients so you don't want to stay in the state too long because it will effectively double your latency so and so you need to make sure that whatever again automation doesn't relatively quickly without overloading the nodes so going back to this slide again sorry so these are the actual commands again you could run this by hand if you'd like you can just connect to a Redis instance and run all these commands it'll take a while so you do need automation you want to set the first slot as importing you want to get every keys and slots cluster get keys and slots will just return an array of keys that exist and you want to individually migrate from a host host host and then say you're done I'm not gonna go into detail with these but this is effectively what it looks like again some gotchas so hotkeys so the way that the data is charted we shard based on slots so you have slot 0 through 16,000 but that makes in the assumption like let's say we evenly like each each node owns the 4th of the key space that means that you want to make sure that your data actually hashes pretty well if your data doesn't hash evenly across the key space like let's see if a hot key for example maybe there's just some some in allocation stream that's really popular then you have the case where a node will have more work than others a slot will have more work so if your data shards nicely it's kind of easy to write automation you just kind of say like you own a fourth of the cluster but if it doesn't like let's say you have a hot key problem you then have to write tools that try to resize based on workload which gets a lot more difficult you no longer doing this very naive like crust cluster create command like you do from reddit strip you or not sure above the CLI you now have to actually have some sort of automation goes through and looks at the CPU usage of nodes and figures out should I take slots away or give them you constantly keep rebalancing the cluster and error handling so I noticed all these these commands what happens if the node goes down or is overloaded while you're running these commands we've had cases where we do migrate we start migrating slots away when we want to shut off a node and somebody shuts it off shuts it off too early and now we're in this state where we were still in migrating a slot but the other side no longer exists so the source no longer exists and the clusters in this weird kind of bookkeeping state where it thinks it's still doing a migration so you need to have some sort of rollback mechanism or some sort of recovery mechanism and a lot of gotchas they're importing migrating is out gossiped so this is less of an issue but if you're really writing your own thing this just means that only the node the doing the migration or importing actually knows about it so if you're it's not gonna gossip to every other node saying this is a migration going on it's just a 2 B a and B know about the the migration the other nodes have no idea what's going on so again if there's a failure like let's say B Goes Down node C doesn't realize there's a migration happening and doesn't know how to step in especially if C's a replica that took over so migrations not being gossiped is mostly an issue if you just happen to have a failure while you're in the middle of a migration and one more gotcha I'm gonna say that word a lot so epoch collisions so what can happen is this epoch is more or less a time stamp that says that is used for collisions so when you migrate when you import a slot you will bump your epoch to say I now own this slot and I own it at this timestamp so and this time stamp is plus one of the highest known epoch every other cluster has every other node has in the cluster so the the thought is that if a node goes down it comes back up it have an epoch from much old like an older epoch so the other nodes and the gossip will say actually ignore that guy he's out of date this this person you know this node is the one that should we should trust because he's a higher epoch the main problem this is that if you actually since it's bumped when you're importing if you try to import and migrate at the same time you'll cause issues so what we can happen is there's a race condition where because the ownership is gossiped and because you bump epoch after importing you can actually cause nodes to think that you will in the wrong key space I'm not I don't have an example here I can make one but effectively you can create a situation where you read this cluster will be stuck thinking that a node owns it thinks the node owns a slot but it doesn't actually and the gossip will not recover it so you need to be very careful make sure that you count don't import and migrate at the same time this is especially bad for like a hot hot key or hot slot a problem because let's say I was had this node that was super overloaded I start migrating away from it and then the one that's over migrating to is overloaded I can't I can't then have that one off offload it he had to wait till they migrate like you can't import in migrate at the same time so it causes issues inconsistent view of the clusters a particularly bad state because you end up getting this that if the nodes don't agree who owns what you just clients will just break randomly it's never a good sign so sooner will slot solutions you can assume all slots are equal or do your own hotkey detection good luck don't import migrate on the same node so that means you have to wait for gossip to propagate that means you have to check every node to see if it's gossiped and then try to figure out a few migrations failed which is also very difficult how to have a fix that is another slide in itself but again I don't want to bore you guys too much more replica rebalancing I'm almost there we almost have a fully functional Redis cluster automation the idea is that you have replicas that will have one two one copy of a master node you tell them which ID to replicate and then they'll just get a stream of data they'll get a stream of pretty much every command such that they can then become the master if there's a failure right if does have a way of automatically moving replicas is its migration barrier to say that let's say if its value 2 if a node has three replicas and another one has one it'll move one of those replicas to the other node so it will try and rebalance so this is an example where we have the two master nodes and four replicas initially we have three replicas for the first one but the migration will decide actually let's take this one of these replicas and move them to the new master so instead of having n plus three redundancy and n plus one you have n plus two in both cases so they can handle the same other failures interesting gotcha here you may it's in this slide actually replica migration doesn't care about the actual IP or the host of the nodes so in this case I actually have to local host replicas for a local host master and I have to 101 replicas for you know that IP same ip address as well so if there was a failure let's say the host went down I would just lose the master and the replicas as well yes because replicas and masters can be on the same host with Redis cluster the CLI command will try to avoid doing that for you on creation but the automatic rebalancing won't so you don't want to host failure to cause data loss you don't want to be this it would cause the uncovered slot problem I mentioned where nobody seems to know who owns what slots and yeah it'll it's kind of a pain and I guess replicas also can get stuck failing to replicate this is less of an issue but you have to actually look at the log files to figure out when a replicas stuck which I've had the dijo at times so solutions I actually disabled the built in logic just said to value 999 or 99 so it will never happen and try to make it so host replicate other hosts with different IP address is the simple solution now but in theory if you get really when I get really complicated you'd start looking at the switches you start looking at the availability zones or whatever you start to trying to figure out like how can you decouple failures as much as you can roll with rebalancing so during a failover you'll have a replica will take over for a master and the old master when it gets turned back on will become a replica it will basically wake up realize like oh hey I'm part of this cluster right oh yeah I own these slots with this epoch and then it'll be told that no somebody else owns it now and be like oh okay and it becomes a replica unless the nodes ever turned back on we do have cases like for example your deep provisioning or there's just some catastrophic hardware failure you can't turn it back on so you kind of have this flow where a replica becomes a master and master could become a replica so he's a little example if I if I turned off the middle left one of the replicas takes over as a new master and we had this state where now we have two masters and three replicas where's before we had two masters in four replicas so what this means is that every node failure actually means minus one replicas because if a master fails the replicas becomes a master so that's always minus one replicas and zero net change in masters I never four replicas fails you just lose the replicas so that's just net negative one replicas so this is the problem when you have a big cluster because it means if you have let's say 30 hosts or 30 nodes every time you lose a node you lose a replica permanently so you have to either spin up new replicas to replace them or convert masters to replicas somehow so you could either you could create replicas so you can be in this case where I just start spinning up new Redis nodes as needed so instead of having a fixed like three nodes per host you could have three Plus notes per host and just create them as needed the main problem here is again the number of nodes in the cluster will never decrease because you always need replicas to replace any failed nodes so you do have a case where we started with like three three nodes per host and we in some data centers we found they're like six nodes per host eventually so we had to switch away from this approach you could actively demote masters as well to replicas which means you have to do the slot migration you have to move away any data you have to make sure that nobody tries to migrate data while you're doing it and you can basically take an empty master and make it a replica so it's not the most trivial thing but you can actively demote masters but it does require moving any data that was on the masters so a little conclusion here hopefully I didn't gloss over things too much but you should be scared you shouldn't understand most of the things I talked about because Redis cluster management is just hard the open source I really love the theory behind Redis cluster I think that it's the events eventually consistent design is amazing but there's probably too many gotchas there's not enough tooling around it and you should probably just use a managed solution I've say this as well twitch works for Amazon and Amazon which is part of Amazon I was a little disclaimer there but it's probably not worth the effort it wasn't worth it I think for my project because I spent a lot of time working on this and hopefully a good mental tool exists at some point so I can do other things and not give talks like this so if anybody's feeling inspired to tackle these problems you know go for it it would be great but there's a reason why many solutions exist and in this case there's actually a lot of hard technical decisions and problems behind it so that's my little conclusion I'm Luke L Curley at twitch if you wanna email me and any questions it was a question have you tried alternatives like an avoid proxy or just an application side sharding no this this tool we has twitch we've been working this for - for awhile before at least there was any existing I didn't do necessarily too much research either I've been trying to maintain this tool more than anything else than switch so no I haven't tried alternatives other questions is your tool available no we thought about it potentially open sourcing it but I'm actually just not happy with it it's something where I think there's a lot of cool ideas there a lot of cool approaches but it doesn't do enough to justify open source again thank you very much thank you so much everybody [Applause]

Channel: Redis

Views: 5,866

Keywords:

Id: RAPKJFAiBb4

Length: 43min 21sec (2601 seconds)

Published: Thu May 02 2019