AWS re:Invent 2017: How to Design a Multi-Region Active-Active Architecture (ARC319)

good afternoon everybody my name is Glen gore chief architect for AWS welcome bright lights so just want to go through what the session objectives are for you in this session what we're going to see is a little bit of context at the start there's a lot of interest around multi-region availability but well I talk to customers every week around this it's kind of the key point here is actually knowing when to use it and when not to use it because multi-region why it's really good from a increasing the availability of an application it does come with some significant costs complexity and we know let's go through some of those design decisions you make as you go through with this I need to talk a little bit about some of the services and how we make them available as well Multi a-z multi-region and go through some of the trade-offs so you wouldn't really understand how you think about availability when I talk to customers everybody says their application is absolutely mission-critical and I love hearing that because it means you use a lot of different services the reality is though you really need to understand availability modeling behind that and so one of the ways we like to think about this within Amazon is through the well architected framework there's five pillars and in this session we're really focusing on the reliability pillar here okay reliability availability durability they're all covered within this and there's some calculations you should be aware of when you're trying to model availability there's some key concepts used to use these way back decades ago when you're building on premise equipment you think about you know mean time between failure mtbf mean time to repair remember doing those calculations and so as you do this particularly in the modern cloud world where you starting to introduce things like microservices an organization suddenly have hundreds of individual components all working together to create a application that works and to end you're introducing more components and normally in an availability model the more components you introduce the availability actually goes down for example and so we've got three dependencies and they've been modeled on different availability ones at 99% the other ones at 95% and ones at 90% what's the overall availability of the system going to be 90% or less okay the chain is only is we as strong as its weakest link and so you can do a lot within cruising the availability of an application by putting into a high availability environment for example if I have two components exactly the same ones in one easy ones and the other each one designed to be 90% available what's the overall availability going to be 99% this is the important thing to know is why parallelization distributed architecture is so important when you're running the cloud okay and these understanding the mathematics behind what actually drives availability is critical as you design things what I see is a common mistake in the industries we take component one and we try to make it five nines on its own absolutely bulletproof and while a lot of the focus is put on the underlying infrastructure and the component that we can control we often forget about the quality of the software those they should be rid of what about the test use cases what about your juice things like chaos engineering where we're deliberately going to break things to ensure that that availability works under stress these are things you need to do if you really want to be true about achieving things like five nines of availability where you get five minutes of downtime and yeah you also get instantaneous failover this goes back to mean time between failure can be a low number if your mean time to repair it is really really quick this is an interesting concept as well so it goes through we know animations here as it fails so component redundancy increases your availability significantly so really be aware of what you're doing here's the good old chart of the nines of availability ninety-nine percent three eight days 15 hours all the way through to five nines five minutes the costs increased dramatically as you go up the number of nines that you're supporting you talked about the quality and the availability isn't just based on the AWS services for example what is the network connectivity between the end-users and the application is it distributed is it going to a single link devices the software quality control people I like organizations you've heard about chaos monkey and chaos gorilla go out working instances and some companies use that for people they literally have a lottery where they just send somebody home for the day you get a day off from work that's awesome but you're not allowed to check your email or contact your phone what they do is they see what breaks in the organization when someone disappears you'll be surprised what has happened if you don't get a free holiday in that you should go back and talk to your managers about it so as you think about these nines of availability as you start getting into you know 99.95% and above this is where you want to start considering a multi region deployment and there are different ways that you can do motor region which the team is going to go through as they go through this considering doesn't mean doing just means just have a conscious thought as to whether you're consciously wanting to go down this path or not now part of this is certainly need to understand how we make our services highly available we all know how a region works is made up of availability zones there's redundant transit in and out of those availability zones each AZ is made up of one or more data centers connected if you use the availability zones each availability zone is containment of the blast radius we try to do everything we can to architect and engineer that an availability zone is as resilient as we possibly can make it though there are some situations where an availability zone can still be a single point of failure control plane for example within an AZ therefore what do we do you have multiple a Z's you put the component in AZ ABC to protect against that we also have region-wide services that span those daisies often in a transparent way too you don't have to worry about it for example using s3 you don't think about what AZ is in it's spread across three or more OFS SQS Kinesis dynamodb there's a lot of them there where you'd go through some of these are default you don't have to worry about it some of their configurable there are different costs involved as you do this for example deploying elastic hash maybe that you just need one case server in one AZ it's perfectly okay if it's for tests and EV or something that's not critical to the organization but if it is critical you can choose how many replicas you need across as many a Z's as you require it's about giving you choice even within an availability zone there are techniques to help protect against single points of failure within the AZ for example with ec2 instance recovery instance ID if it fails just spin up another one where you go another way of increasing or actually decreasing that mean time to repair as you go through high availability auto-scaling or it's going groups across the available so these are all tools to allow you to hit pretty high levels availability within the same region still we don't think that you can hit four nines by the way within a region quite easily if you follow a lot of these techniques so you what are the reasons you may want to use a region active active architecture one of the most common reasons that I see that is defensible really independent of the availability modeling actually is if you have geographically distributed customer base you know if you've got customers in the US in Europe in Asia you want to have them using their closest region this helps with latency but when you're doing this you've got to thing about each service within each region so for example here you can see there's users in San Francisco uses New York using Western East and they're using a number of different AWS services there what happens if one of the services fails within a region well this is also kind of is how you should think about failure often customers talk to me about what happens if the region fails well that's not really actually what happens what happens is it's normally a component within the region for us it's a partial failure and those of you who work a lot in operations know that partial failures are the worst ones to do detect and actually work because it's not entirely broken it's just broken a little bit should I keep it running and hope it resolves itself or do I pull the trigger and fail the entire application over until the boss yeah I took out the other 70% that was working so I could get it were flipped over to the other site which is a hundred percent working doing that in a stressful situation it's not always as easy as it sounds so service has failed we're going to therefore fail the region from the application perspective and we're going to direct all the users to the other region the other region is still working for the majority of servers it's just at one service that's having a little bit of a hiccup this is how you get high levels of availability where our cost-effective do you know this is a another area where you see customers picker been doing lift and shift migrations legacy applications is I'm going to put the application in one region I'm going to stage it in another region but it's kind of cold or warm standby well we all know the problems of that they fall out of sync in that moment of need when you actually need it the most what happens it doesn't work meanwhile you're also paying a lot of money for it to sit there idle so where you ever you can get away with using active active you want it you're not wasting money it's forcing you to be in sync particularly you start doing things like more frequent code updates release cycles you're just forced to do good behavior and when you do need to switch from region to region you know it's going to work because I was already working for a subset of your customer base so I just wanted to set a little bit that context upfront for you around how to think about it there are use cases for region region active active deployments but for the vast majority of cases I don't see customers architecting all that well within the region using the availability zones and having the code quality and the testing around it and the methodologies to ensure it's going to work when it's most needed there so with that I'm going to hand over to Daren he's going to talk through some of the key technology requirements you want to think about as you start using some of these services thank you all right thanks Glen all right let's give so hi my name is Darren Brisbane I have been doing about gosh I'm old thirty-five years of this stuff dealing with developing multi-region architectures and deploying them all over the world so 30 years ago that made mainframes and global distributed parallel sysplex those of you I can't I can't see whether any of you are old enough to know what I'm talking about when I talk about that mainframe stuff but today we have lots of new ways to do it but having done this for three decades I've learned one important secret about doing active active multi reach in architectures ready here it is don't do it don't it's hard it has complexity complexity leads to failure half the time I've seen people do this they end up with lower availability because they created a more complex environment that they can't run you people do this for the stupidest reasons they just think well it sounds cooler to be active active so let's make our jobs ten times harder and set it up that way all right have I convinced anyone here not to do active active all right fine having said that let's talk about how to do it but keep this in mind if you remember only one thing today it's don't do this unless you really need it because multi-region active active put it in the same category like chemotherapy or great or brain surgery it's a really powerful tool but don't use it unless you really need it right don't do chemotherapy if you have a cold save it for the really deadly diseases that it needs to treat so that said what are the requirements to do this well first one of course is a network that you can trust right I need my network to be reliable and you might network to be secure I need to have a good backbone well of course if you're using AWS we have a fairly good backbone that is connecting together the AZ is and connecting the regions back and forth to one another if you're all in the cloud that's great and we've got a lot of great customers you know Netflix air B&B not all of them Internet companies places like a Penske hotels in Europe or Time Inc that are all cloud but most of you are not right you know we're in a world especially if you're with a large enterprise that you know you're not entire than the cloud so I'm going to have to have some way to talk back and forth to the cloud and then of course I'm gonna have clients that are not sitting in the cloud I need to communicate at some level so I need to have some sort of VPN or encrypted SSL or some way to communicate over IP and think about ways to make that reliable so point out my job is to give you all the questions the next week we'll talk about the answers data replication that can be synchronous or it can be a synchronous if I'm doing multi region data replication doing it synchronous I just introduced a huge lag in AWS the two regions that are closest to each other in millisecond terms or us East one and us each - all right that's Virginian Ohio that's still a 12 millisecond one-way trip so if I'm for example doing multi region synchronous on a database right what would have been a 1 millisecond insert if it was in one region just turned into a 25 millisecond insert because it has to go to one region right into the other region confirm that back to the first region now what if I'm doing say Virginia and Singapore it's a lot more than 12 milliseconds right it's usually somewhere around 200 to 300 depending on what's going on I actually had somebody asked why don't you just drill a cable straight through the earth in order to shorten to that engineering is working on that there's some challenges there so you can do synchronous but there's a latency cost right and you know you might think well 25 milliseconds isn't that long yeah but that's per insert right this adds up very quickly when you start doing these things or you can do asynchronous so part of that complexity and you'll be seeing this is whenever we're doing true multi-region stuff you've got to be able to deal with a synchronicity you've got to be able to deal with eventual consistency and learn to embrace it and design around it this is one of those the things the complexity that makes this hard because very few applications are written with this in mind so you end up having you can't just take an existing application set say I'm going to deploy this multi-region well you can but then you're gonna so you're gonna have to do some significant changes so asynchronous is useful there's lots of ways to do a synchronous replication Girish will walk through a good number of them but of course the issue have with asynchronous is if I'm writing to one part of the world and reading for another part of the world am I getting current data or stale data so one thing to think about is do I care does it matter if the data is a second or two seconds or ten seconds out of date for many applications that's fine right but for some applications especially things dealing with say actual say wire transfers or financial transactions or one that I saw somebody are playing with was autonomous driving and you really don't want to wait for you don't really want to wait for the car to communicate to the cloud to figure out whether to stop you know when it's on the road so you know we're gonna have to have some way to deal with a synchronicity and then deal with that is that is it nearly continuous am i shipping it with a delay how am i doing a batch and I sending things over once in a while then of course we have synchronizing the code so three basic ways to do this right when I stage deployment across regions because now I have the issue is one region has very one version of the code the other region has another version of the code so one way to do this is the rolling or canary deployment right canary deployment I stick a few servers or containers or land of functions or whatever on the new code see if it works if the canary dies roll it back if the canary lives roll out more can your applications survive that depends on the application if your application can't survive with some on one and some of the other then you're probably going to have to do blue green so blue green means I have the existing environment which is blue the new environment which is green and at some point I switch out all the IP addresses and DNS addresses and whatever and so now everything was running blue now it's running green and what do you do with the blue you leave it there for at least for a little while so that if something goes wrong you can switch it back some people like Netflix calls this red black instead of blue green I don't know why same concept though so if you ever see people talk about red-black de plan it's the same thing and then of course the idea of parameterize localization so if I'm running the code part of that I'm assuming you put in multi region for a reason one of those reasons usually is to do some sort of localization and to try to reduce the end-user latency by having to say the users in Asia I hit an Asian data center instead of having to go across the Pacific every time they do something so that gets the traffic segregation and management so there's a couple of options that we can do this you can explicitly do this by having different URLs or different approaches different end points so you know East out something or West out something you could do this with implicit route 53 does this really nicely so I you know have the same URL but it will get automatically routed and I can do some tricks in the traffic management infrastructure so I can throttle things I can do internal redirect I can do external redirect the issue you have the redirect is what do you do when somebody gets sent to the wrong region and then how do you define wrong region right and and then how one lives with that because sometimes the wrong region isn't a problem I'll give an example of a ride-sharing service that I use frequently and I live in Portland Oregon and they have a couple of distributes places that they have regions that they're running this in and I always end up logging through Portland now what if I happen to be in Miami that day still logs through Portland it said a little bit inefficient yeah but I'm using a mobile device am I kind of noticed as an end user in a mobile device a few hundred extra milliseconds probably not so for that it's the wrong reason but it's okay so again things to think about and then of course the whole idea of monitoring this becomes exponentially more difficult in multi region environments because I have more things I need to track and more I need to pay attention to you always need to pay attention application infrastructure health I'm hoping that all of you are doing some sort of log analytics on any critical application I'm regularly amazed by how many people have critical applications so they don't do log analytics or one that I ran into well yeah we we keep the logs in s3 and then when something breaks we spin up elasticsearch to see what broke okay yeah it's yeah that's putting on a seatbelt after the car crash right you want to be using a log analytics system to see what's going on as it's happening and then of course deal with the monitoring of things like replication lag and code sync and so on and making sure I'm understanding these globally for how they happen when you have multi-tenancy it's what is multi-tenancy well a tenant is a unit so sometimes it's a business unit sometimes it's a technical definition but it's the idea that I have multiple independent pieces so here's an example of doing it based on States so so one of the lessons that we're learning here is white on blue might not have been the best color scheme that we could choose so if you can't read that US West has a Texas in a California box us as in New York in a Pennsylvania box and of course the whole idea here is as I move things around over my backbone I'm I could have the I could move the Texas stuff into u.s. East if I need to for load purposes or other reasons so the whole idea here is to be able to have that unit as a tenant that I can decide where it goes and that gets down to the whole idea of static consistency when I'm doing multi region I really want to make sure that each region can handle my entire workload maybe not my entire peak workload but at least my entire average workload great thing about being on the cloud instead of using AWS changing that sizing is a matter of somewhere between zero and a few minutes depending on whether your auto scaling or whether you have to do it manually and then we have the whole idea of direction of replication right so I might have Texas with its core data in one side and then start replicating it in the other direction and we see that yeah fairly often because one of the things that you generally want to avoid in a database is having true active-active database where people are writing into multiple masters especially at the same time now as you'll hear today we've gotten as you may have heard from Andy Jesse this morning we're developing more services to make that easy but if you create yourself in a world where lots of people are writing in to different regions that could collide with each other you're gonna create complexity and failure so usually you want any given set of users to only be writing to one side with it replicating to the other side but it becomes active active because as we discussed back over here I've got some users that are active on one end some users that are active on the other end so they're active active at the system level that is a much safer and easier way to design this than try to make every database active active at every transaction and so if you have to do active active please don't try to make your databases active active active unless you really need to and then even if you really need to try to minimize it well you already pointed out you're not gonna listen to me so all right failover scripts and then I have to have some way to failover I need to reroute traffic for a tenant I need to change direction of replication right all of these are critical alright so what are the architectural considerations in here one I need to have tolerance for network partitioning I need to be able to have a way that if I fail in one region the application doesn't fail in another region I want to make sure that API calls never go cross regions that each region is independent from the other regions I particularly want to think about the worst possible failure mode which is a split brain which is not when a region fails but when the network connection fails so each region thinks the other side failed that works great how do you fix it afterwards you now got two different versions of the truth they're not easy things to reconcile so I don't have an easy answer for that that's one of the things to think about those kind of the complexity that I'm fearing when the backbone breaks now in the AWS world does that ever happen well it's incredibly rare but that's the whole point of this sort of disaster recovery thinking is think about those incredibly events one of our great customers is the Church of Jesus Christ of latter-day saints it's the only custom ever dealt with that that not only plans for catastrophic failure but cataclysmic failure which means the end of literally they put some of the family history records on to nickel sheets in salt caves to survive the end of civilization most of us are working in organizations that catastrophic failure is kind of the worst level that we want to pay attention to but think about catastrophic failure so to give my own example I am a proud citizen of Portland Oregon I love living important than I love watching the Sun Rise over the beautiful cascade volcanoes one day they're probably gonna kill me and in theory if we have one of those really big events like the Yellowstone and Cascades opening up it could wipe out not just the US west to region but most of the people living in the north western US that's a catastrophic failure is that likely to happen no but if you care enough about your applications you got a plan for it see what that will happen all right minimal data replication only replicate we need to replicate all right there's all data need to be replicated probably not it doesn't need to be replicated synchronously probably not right do I need to replicate my backups yes do I need to replicate my backups synchronously probably not there's all data need to be replicated continuously or can they be things that happen in batches so things to think about this kind of gets back to to a great saying that I've heard from many people because I've heard Winston Churchill and Eydie Grove quote of this which is don't worry about the hard decisions they're hard decisions because both both choices are probably pretty good you could flip a coin you'll be ok worried about the easy decisions we're about the ones that nobody is questioning go back in question should I be replicating this data do I need to replicate this data classify the data so I've got data that is transactions right this is stuff like purchase records I've got data that is catalog information this is stuff like product data I've got events and objects like clickstream I've got server logs and data like that so that first category is low volume but pretty critical right the other end is high volume but less critical we ever looked at how many logs your stuff generates in a day it's a big number those are the reason you need things like elasticsearch is part of that solution I don't know about you I'm not smart enough to look at 10 terabytes of JSON and figure out where the problem is so that's why you need to visualize these things but you do need to do some level of classification and have different rules for the different classes and then of course think about am i doing synchronous or asynchronous replication so when I move stuff across from A to B how is that going to how is that going to work so I can have I write my application to a database and have it replicates and then have it acknowledged and then acknowledge back that works pretty well right the problem there is as we said before where Network dependent but it does guarantee consistency or eventually it times out and the transaction rolls back or I can do the right and replicate an ACK but the danger here you might have noticed I did the acknowledge before the replicate is that they can go out of sync if there's a problem but that radically reduces that latency on the other end so basically here you have two bad choices you have to decide which bad choice is less bad for your application and your environment although as you'll hear today some of the things that we're doing with dynamodb and elsewhere help make this less of a problem and as you may have heard we are this morning was announced that we'll be dealing with both with multi master Aurora as well that minimizes these problems it means AWS takes care of some of it for you but as you probably know no matter how good the services are poorly designed applications can still fail no matter how good the services are underneath so we'd like to think of it as a replication Lane I got a sink lane and a nearly continuous Lane a batch lane and the one that off the bottom which is they don't bother Lane right like do you really need to replicate your logs probably not they're kind of huge plus if you're sticking them in something like s3 yeah they're only going to get lost they might get lost temporarily if we lose a network conductivity but the durability of s3 is ludicrously high right it's eleven nines synchronous is the hardest to manage the batch is easiest to manage but they all have to be managed and watched so transactions in that Lane catalog in that Lane events in that Lane logs falling off the bottom so it takes us to our ideal replication system so the basic minimum here is I have a source data here we say database that's how to be database right could be a data set all right but I've got a source chunko data that's going to go into some sort of replicator into its target so it should report the replication lag one way to know how far is that target behind the source so the it's like a good non database example of this is s3 replication and you can get the lag in there but you know s3 does not guarantee a delivery time so sometimes it's very fast but if you have like giant multimedia objects that could take a long time for the s3 it could take minutes for the s3 to replicate to a remote region but you need to report that lag it should record offset right how far off is the target from the source so have an idea of the record offset should be able to retry replication if the if it fails and it should keep trying until it's successful now many AWS services as Girish will walk through will do those steps for you but you need to make sure that every piece of data that you're doing active active has these steps in it that you are having a way to report on it and monitor it keep trying if it fails and keep trying again if it fails for the right amount of time so I'll finish with a few best practices and these are the best practices a little bit in computer science terms so if you're not familiar with idempotency idempotency means for things that are not changing you should get the result every time so for example for database that I'm not doing any inserts or updates on every select statement should give you the same answer right when I run the same select statement I should get this answer idempotency is the idea that no matter how many times I try the same piece if it's an idempotent operation it should remain consistent this needs to be item potent across regions if you're gonna be active active I want to make sure that when I run the same code in one region it runs the same way in the other agent eventual consistency this is the tough part learn to embrace and virtual consistency of writing code for eventual consistency is hard it's probably the biggest piece of complexity it's the only way to scale at Amazon we have spent most of the last ten years rewriting the monolithic applications we wrote in the 90s when we launched into micro services applications designer and eventual consistency it's been expensive it's been difficult it's been frustrating and it's lettuce grown from a ten billion dollar company to 180 billion dollar company right these are the kind of things you have to do in order to get to these large scales static stability it's another way of saying as I said before if I lose if I lose a region the other region should operate the same way so it should be stable from the point of view of the code which means they all have the same code they're all running the same way exponential back-off is part of the coding this is how you do try again right I'm presuming you're familiar with this pattern especially if you're a developer right you write it times out you wait 10 milliseconds try again wait a hundred milliseconds try again wait a thousand milliseconds try again how far does this go that depends on your environment and what that application can survive in most of the AWS services if I have a database primary failure and I'm set up for high availability my downtime will be somewhere between 30 seconds and 5 minutes depending on what I'm using so I need to make sure that my can my back off survive that long and if not will a timeout in a way that lets a user survive and deal with it throttling how do I deal with the cases that data doesn't always come in at the same rate this is when we use caching systems and and other things in order to be able to throttle and pull in the data and finally circuit breaking this is another thing that you need to write in code that you didn't need if you were before your active active which is when I'm doing things that go across the regions and something's not working how long do I let it hile up with other things depending on it waiting before I break that circuit and say this is going to fail so this is another place where you have to do some monitoring and some automation so TLDR don't do it but if you have to do it do it carefully and monitor everything now the good news is we have ways to make this a little bit easier Girish come tell us about nice so Darren laid out all the architectural principles I'm going to now discuss specifics of various services that help you implement this architecture and we are also going to see how we do how we keep you know those trade-offs that didn't discussed in mind when we are using those services and their features so I got this slide from last year's presentation by James Hamilton but there was Tuesday night live as well this year and you would have seen that all AWS regions across the globe are connected with each other using a very fat backbone right so network has been taken care for you so when a server in one AWS region tries to talk to server in another AWS region without you having to tell it the traffic is going to route over these high quality backbone so you get the benefit of this even without knowing that you are using it so this is pretty cool one less thing to worry about so backbone is fine right it's going to route your traffic what about the security of your data when it is flowing over the backbone okay so people use VPN right so you can use VPN there are variety of ways to configure VPN so either you can route your VPN data wire your corporate data center right that time you can benefit from a WSS hardware based VPN which is extremely high throughput or you can have a direct into an VPN using software based peon techniques right so there are a Mis from various partners like photogate which provide you VPN connectivity there is one downside of this though that you get limited by the instance bandwidth because now you are using instances to run your VPN software so there is a lot of complexity in doing this right it's quite doable there is a white paper that we have published as well as CloudFormation template also which can help you do the setup but wouldn't it be nice for AWS to take care of even encryption of your data when it is passing over the network so we heard customers the thing that we do all the time and we launched something new so people love VPC peering because it allows them to connect one VPC to another VPC in a very secure way and it always works but it used to work within a given region but now it works across the regions so this is a small diagram that shows two V pieces in two different regions so being connected so it takes care of a lot of your headache it's a free service ok so no VPN charges for you the only thing you pay for is the amount of data transfer at the rate of intelligent data transfer so that's pretty cool you have to keep in mind one thing though since it will form a single network by connecting them by VPN you should not use overlapping IP addresses they should be non overlapping IP address ranges and yeah as I mentioned all the traffic is by default encrypted so you're also once you do this so highly recommend it go for this whenever you want to connect two regions yeah these are some of the key benefits presented as bullet points it works just like intra region BPC data always stays on backbone it's always encrypted for you there is no need to use gateways third party VPN solutions saving you lot of cost right and no additional charges as I said for V PC peering all you pay for is data transfer at standard rates so now let us look into individual services so s3 is one of the first services that we launched and people put a lot of their important data in s3 because it is just so reliable right 11 lines of durability s3 now has this feature for quite some time which is cross region replication which very reliably replicates data at one end to the other end so whatever you are storing in s3 bit your files documents or your database backups all are automatically copied from one into the other end and it is done in a windy reliable way and since this is a AWS service all the security and things like that are taken care by AWS you don't have to worry about it it just works this is pretty cool now let us take a look at another place where you store your important data that's IDs and as Glen said in the morning not morning just half an hour back that before you think to do things cross region right or multi-region first learn to use a given region in the right way because you can derive a lot of benefits very easily there are a lot of low-hanging fruits and one of the low-hanging fruits is deploying a standby server for your master in another easy which is independent of first AZ which has got its own data centers power supplies network and everything and there is extremely low latency synchronous data copy that is happening from master to standby so if master is going to fail ever you always have a standby to rely on and you can switch over from master to standby within a minutes time the DNS rerouting happens automatically without you having to do anything so this is pretty cool you should always take advantage of this feature whenever you are going to store critical data in RDS now IDs has a neat feature which helps you with multi region architecture should you decide to go down that route and it makes sense in many many use cases so in such cases you benefit from a feature called as cross region read replicas by RDS so you can create a replica in a region of your choice and that replica is set up and managed by a W AWS management systems it works extremely reliably so currently the database engines that we support for cross region read replicas are my sequel Maria DB post gray and Aurora if you use sequel server or Oracle there are tools specific to those engines which you are free to use some of them are available in the marketplace but these are some of the most common database engines that we have seen people using so we support out-of-the-box Crossley replication you just have to turn it on yeah this is a very important thing okay so whenever you are setting up a cross region read replicas you need to be monitoring a replication lag and AWS provides you that information all the time to you through cloud watch so it's very easy to monitor you can set up Allah alerts and all that thing that you would want to do what is important here is to note that whenever you are creating your read replica in another region make sure that it has got enough capacity so that it doesn't fall behind the master it should be at least as big as the master that's what we recommend and if it ever falls behind there are some steps that you have to take to bring it in sync with the master but ideally you should tear away from that zone you should never have to do that so now once we decide to implement a cross region or you know an active-active multi-region system how do we use masters and read replicas of RDS so here is a very simple scheme so I have shown three regions up is Virginia then Ireland and then Tokyo okay master is only sitting in Virginia and read replicas are represent in the other two regions locally and servers in all the regions are writing to the Masters in Virginia it looks like a plan that is not going to work because the geographical distance is large but many a times all those read/write operations from remote regions finish in a second or so which is acceptable in many many use cases yeah it may not be acceptable in all the use cases but there are going to be many use cases where it is going to be acceptable and data is immediately replicated back to the read replicas so that it is available for real operations locally and in most systems you know you perform 100 reads for every write so this scheme of things however simple it may look it does work but yeah right traffic goes up read replicas traffic comes down and you can live happily but there are going to be some challenges there is going to be a little higher latency and there is going to be Network dependency what if the network breaks down so in such cases you can choose another design if you can't tolerate those situations okay so what you do is as Darien discussed you have different tenants in different regions right we have got region a B and C and they have got different denims being served from there each tenant has its own master to which it is writing locally okay and each master for reliability reasons it is going to have a standby which is synchronously replicated so that not a single transaction is lost and it also is going to have read replicas to which it is going to copy data or synchronously so that you have options ready should region a become unusable for whatever reason for you okay the thing that is going to happen though is that since the replicas are or sync in the other regions it they may lag one or two second behind the primary master right so some committed transactions may not have made to the other side when it is time for you to switch over because of whatever region but your transactions are always safe in the standby okay so you can use data reconciliation techniques which are very commonly known to you know reconcile the data if some star transactions were left behind you can always retrieve them later and make the system consistent again now multi master Aurora okay so we always recommend you having a master and a standby or read replicas we generally advise not going for multi master designs because it is very difficult for you to manage multi master replication both sides simultaneously replicating data to one another but we believe that there are some good use cases for multi master design one of them is extremely high throughput applications so we today announced a multi-master service it has right moves in all ages and you can write to any of the nodes in 2018 we will also be coming out with multi master Aurora cross regions so that's going to be pretty cool for you that's a lot of engineering work generally you will find it difficult to do it yourself but there are AWS engineers that are going to do it for you it will be available in 2018 so should you consider to do that architecture you know this is a service that will help you a lot Oh what happened okay yeah so right now I so far I was talking about the default replication of RDS but you also can use DMA service the difference being with DMS you can replicate selectively the default replication will replicate everything but sometimes that is what you don't want you want only certain tables to be replicated so DMS gives you that granular control and DMS is able to continuously replicate data because it works with change data capture technologies which allow you to just detect the changes and replicate them from one side to the other so this feature has been it's being supported for my sequel MariaDB Aurora and push for a sequel so yeah if you are using those technologies consider using DMS if you are doing you know granular replication now dynamically B is one of the most common services that people use to meet their new sequel requirements and so far you know dynamic Dubey's in two regions used to work independently so if you wanted to keep them in sync you had to do some engineering work yourself so random ODB is pretty popular because it is highly scalable it is fast it is fully managed and it provides you business-critical reliability so there are many many large organizations including amazon.com which deployed DynamoDB for their critical workloads so you would want to have die Namibia also replicated so today we announced one more feature dynamic global tables so now you can have DynamoDB tables spread across the globe and you can keep writing to any of the tables and they will be working in multi-master more from one end data will be copied replicated instantaneously to the other end and vice versa and all the tables across the globe will be in sync for you so these are some of the benefits but rather than me talking about it we have product manager for this service going who is going to show you the demo so what do you so if you're at the keynote this morning now you probably heard any of jazz you talk to you about global tables global tables is DynamoDB s fully managed multi-region multi master database solution just spent are quite some time sharing wisdom about why you should be highly wary of active active database setups I'm here to show you how easy it is to do active active replication in dynamo dB so why would you use mobile tables we really think it's about you know first if you have massively scaled applications that have users around the world you want to deliver you know the best possible and fastest performance to all of those users which means processing the requests in the most local region possible so essentially data locality and secondly it provides a new level of regional fault tolerance right because with global tables your dad has synced across multiple a devious regions and therefore it's able your application can withstand you know in a region outage or degradation of service and your app keeps on just chugging away now dynamodb global tables handles the difficult work of replication for you it propagates changes from any one region to all the other regions and it handles any conflicts using lash writer wins reconciliation between conflicting updates so let me go now to my laptop and I'll give you a brief demonstration of global tables great so we're sorry we're starting here in the the dynamodb console hopefully many of you are familiar with this page we'll go to our tables and let's first take a look at a global table that I created you know before this before this so now what you can see here is that you know this table has some items in it it's got auto scaling on and what you'll you'll see that it's new is this global table tab and in this case you can see that I've created a global table that spans several regions it spans Ireland Ohio as well as Oregon and that means that any rights to any one of those regions will be propagated within about a second to all of those other regions so now let's try doing this from scratch I'll go create a table I will call it global table example one we'll give it a primary key and you know rather than fiddle with all of the configuration details on throughput and capacity we'll just use the default settings for DynamoDB so this will take a moment it creates what was that oh yeah that happened to this morning so you can also back up and restore your DynamoDB tables so we created a global table example one it looks like it's ready so now let's go make this a global table so I go to the global tables tab it tells me that you know in order to use mobile tables I need to turn on streams that's because global tables works by using this DynamoDB stream to propagate all item level changes out to other regions so I'll click enable stream and that'll just take a moment to turn on and that's ready now let's start adding regions so I'm in Oregon but I want you know I let's say for example that my application serves customers in the u.s. west coast the US East Coast and Europe so let's go extend the global table to Ireland click continue dynamodb will work its magic in the background and create a replica for you in Ireland they'll apply some global table configuration settings we'll close out and then let's do this exercise one more time for for Ohio for the u.s. East Coast close out of there now let's just you know insert a little bit of data I'll create an item let's say this is item one and DynamoDB is you know offers a flexible schema so you can add whatever attributes you want and let's call this one hello from Oregon will click Save and now this item is written now if you just take a moment to do a brief refresh you'll notice that these two new attributes have appeared AWS rep updates it says update region as well as AWS ret update time now these are fields that the global table service writes on your behalf it tells you where the atom was written and it also tells you where it tells the system rather when this update occurred and these update timestamps are used in order to reconcile between conflicts to the same item so now if we go to another region we go to Ireland we can see that this table global table ax 1 was created for us and then we now see that oh look at that hello from Oregon and if we do the same thing now this is in Ireland and now we go to the Ohio version we see this replica table appear and hello from Oregon and you know if I get if I can say write another one let's say this will call this item 2 you know we'll call this one as well say message hello from Ohio you'll see it adds the fields and then it'll get replicated over to these other regions as soon as you refresh your your screen so this is all well and good if you're doing you know kind of like TOI writes using the management console but what if you actually wanted to do you know more rights than that let's say for example that you're doing several you know several hundred rights so in this case I've got a couple Python scripts one of them is going to insert items into the Dublin table and another one is going to insert items into the Oregon table hopefully this works and here we go you'll notice that the the inserts to the Dublin table are going faster than the ones going to Dublin that's just because you know we're going across the ocean here if you were to do this in you know real-life production you probably wouldn't be issuing the request from your laptop you'd be issuing them from an ec2 instance in Dublin so now we can start seeing these items we're starting to get inserted and we can go now to you know let's go to the Oregon table and we'll refresh and you can see now that there's all these transactions coming in and they're coming from multiple places some are being written you know in the u.s. to this table and others are being written from Ireland that are getting replicated over and if we go for example the Ohio table which is a passive participant in this it wasn't doing any of the writes we refresh here and we see that all of these rights have been replicated from Dublin and from Oregon into Ohio so that's Lobel tables for you oh yes one less thing and to clarify global tables is available today in five regions it's in u.s. West or US West or US West Oregon u.s. East Ohio us east Northern Virginia EU Ireland and EU Frankfort with more regions coming soon in 2018 thank you yeah so guys I'm aware that I'm the last guy standing between you and pub crawl so I'll make it quick I'll need another 10 minutes to finish all the slides and there is a lot of meaty stuff ahead so this dynamically replication it solves a lot of problems for you but if as of now there is a region where a dynamic global table is not available that feature is not available what you could do is you could use what is known as dynamic streams so streams is basically a place where a log of changes comes it is something similar to my sequel bin law so every change gets reported into stream so you can pick up the changes from the stream in one region you can have a lambda function picking it up and writing it to the other region so why we use lambda because it is just so easy you just define a lambda function and you hook it into a stream you can write it in so many different languages like Java dotnet node.js Python whatever you like and all the execution happens as many as time Records arrive in the stream and it is all managed by AWS all you have to do is you define your logic to handle that stream record whether you want to post it to another region or you want to do something else like write it into elasticsearch and all that so lambda is pretty cool lambda needs no more introduction it's a nice slide that shows how lambda works so basically it picks records one by one from stream it can also batch the records and it writes to the other end and that other end could be anything so there is one nice thing about lambda that I want to know that is highlighted in yellow okay whenever it is Raeleen from the streams say for example the target is not available what does it do okay in such cases the exception is treated as blocking and a WS lambda will not read any new records from the stream until the failed batch of records either expires or it has been processed successfully okay this is a very important thing right this is one of the features of the ideal replicators it shouldn't skip some records right it should wait until the target system is up and the moment the target system is up it should start processing from the place where it got halted right so it introduces a lot of reliability in your replication process now let us see how to replicate data across other popular products like elastic search and redshift all you have to do is your source post data to Kinesis firehose in both the regions and kinases fire rows gets this data efficiently into either elastic search or you know into redshift you have to take one care here though since you want to go same data to both the places your source needs to have tracking to make sure that the postings have happened successfully since you are writing the source you can always do that so this is how very easily you can get data replicated across elasticsearch clusters as well as redshift which are some of the most popular products that people use now this is probably one of the most important things that you will need to do when you are deploying a multi region architecture you will have to route traffic to appropriate region for a given user so new york user should consistently land on east coast you know excess users who are closer to you know west coast they should probably land on west coast so how do you do that right so route 53 is amazon's managed dns service which is fully configurable using a pas which allows you to do this in a very elegant way so let us take a quick look at the scenario you have got three different regions and then each region has got some users which are closer to it and would want to route each user to that region but sometimes that region is going to fail and at that time when I mean to say region is going to fail I mean to say from the applications perspective not AWS regions perspective whenever a particular region is going to be unavailable you would want to redirect users to other region so how do you manage that in a very efficient and elegant way so you use a technology of route 53 which is known as practic flow so what is the kind of endpoints it supports it supports routing to IP addresses load balancers s3 website CloudFront distributions and elastic beanstalk almost pretty much anything that you want to put behind your systems so what is that traffic flow does exactly for you okay so traffic flow is a system that implements dns base two rules for you okay so you can have a simple routing policy right you can have a failover routing policy what it means is go to a all the time but if a fails go to be okay then you could have Joel location based routing policy where you say that people from Florida should be routed to one region and people from you know Arkansas should be routed to another region then it could be geo proximity based policy you don't care which state people belong to just calculate the geographical distance and based on that do the routing the next one is latency based routing okay latency based routing allows you to pick a region for a given user based on latency so say for example if you are a streaming service you could always make sure that people land up in a region which is the closest to them with latency wise and last is multi-value answering policy which I will not speak much about for now so I have a question for you which of these policies you will use for multi region architecture which is the best policy is it Joe proximity is it geolocation is it latency based or you should have like static rules what is the best policy okay so one answer you can never be wrong with is it depends right so every customer is going to have a different scenario sometimes geolocation is going make sense sometimes the content is going to be jury stricted right so you want to make sure that people from a certain part can see the content and certain part can't see the content for whatever requirements so your scenarios are going to be extremely varied okay so people typically want to have a combination of all these rules a cocktail right can you do that okay so the nice thing is yeah the answer is going to be yes because I'm asking that question and it reflects positively on us so the answer is yes of course and you get this very elegant beautiful graphical design to design your policies if this do that if this do that okay and you can't just define very simple policies but you can also define much more complex policies let me jump to the next slide okay so here there is a much more complex policy so there is a failover rule under that failover rule there is a geolocation rule under that rule there is a again failover rule and like that you can have a hierarchy rules right first of all oh this policy if that policy is not appropriate for whatever reason then follow this policy and so on so route 53 in my you know research is probably the most indispensable tool that you have when you are going to implement multi-region architecture pretty neat I love it so now we saw how we like segregate the traffic right let us now take a look at some other operational things how do you do cross region code deployment so as Darian spoke in detail right you should either do blue green deployment right create another environment or make sure that it is working fine and switch all the traffic over to it or then you could base your approach on canary principles where you have a new version and rather than switching over all the traffic to that new version of your application you switch 1% user then 2 percent than 3 percent and so on so these techniques you are going to follow in multi-region world as well right but you will have to do it at two places right but it is not generally nice to do deployments at two region at one place because if you do something wrong both regions are going to be unavailable all at once and what is then benefit of having multi-region architecture right so never do simultaneous deployments to both the regions you should always do one region at a time deployments that is bad second is good okay now how do you do it in terms of AWS services so we have published this very nice solution which is ready to use you can spin it up as a CloudFormation template and it is very well explained on that link this deck is going to be shared so in this system we see that code is placed in an s3 bucket in region a there is a whole pipeline service from AWS that gets triggered it does the deployment testing and everything whether you are doing blue green deployment canary deployment whatever and once it is done it invokes a lambda function which copies the code to you know second region basically it places it in a certain place from there using cross region s3 copy the or gets copied to the other region then the moment it arrives in the other region you know s3 sends out notification when new files arrive if you configure it to the core pipeline get triggered in the second region it does canary deep lament Bluegreen deployment whatever you like in the second region and if all things look good in the second region it works in the third region it's pretty cool very easy to do it please do take a look at this blog as well as the CloudFormation template now this is very important parameter localization so this is a rolled brace practice that people always ignore the don't hardcore things in your code right so you should have a philosophy of storing parameters outside because you're owed ideally is going to be the same even in multi region world you want to deploy the same code in one region and the other your parameters are going to change based on the location so you should have some kind of a parameter DB where applications can query their parameters from and then you should have a need to ID to allow you to configure that parameter DB so either you can do this all by yourself or there are some customers who are kind enough to share their solutions like you can take a look at this open source product from Netflix called as Arceus which helps you do that right why reinvent the wheel whenever heal is available now cross region monitoring we routed the traffic we ensure the code deployments are happening properly but now we want to you know keep monitoring it how do we do that so first of all let us ponder how multi-region you know monitoring is going to be different from a single region so first of all you are going to be required to monitor few more things one of the most important thing is replication lag you need to know exactly how much you know data is yet to be replicated and you need to know the record offset basically how many records are yet to be copied so that you fully know the state of your system right you can't live without knowing these things ok but last but not least you want to have a single monitoring system which helps you see what is going on in both the regions so you can compare and contrast and take decisions yeah so let us take a deeper look into each of these aspects so replication log if you are using AWS services like RDS and all it is directly available in cloud watch so your life is easy if you are using something like s3 file replication which automatically replicates fire from one side to the other at this time you will have to develop simple custom solution so s3 whenever a file is added to it sends out the SNS notification right and when the file gets right written to the other side also it sends out notification so in this solution what you are basically going to do is you are going to observe all those SNS notification based on that you know you know things like ten files were returned to region a I received ten notifications but I received only eight notifications from the other region so probably two files are yet to be copied that simple map you can implement using this system it may look like it is having multiple blocks and difficult to implement implement but as always there is a cloud formation template to do this just follow this blog you will know how to do it it is very easy to setup and monitor it in a server list fashion okay so taking it a little further if you want to monitor you are now applications what are what is what are the tools and techniques that you have at your hand so we see these days that Amazon Elastic search has become one of the most popular ways to monitor your logs right so so many sources can forward of the logs to it right you can have a cloud watch agent running on the server which can forward the logs by a lambda then there are other services that generate logs like VPC which server is talking to each server which ports are being used in that communication so all these logs can also be forwarded to elasticsearch through this path okay then there are still more services like s3 ELB CloudFront that provide you logs to monitor so these are various paths that lead you to elastic search then there are still more services that make your life further easier by reporting you various things again you can get all this data into elastic search and if you are writing some applications and you want to monitor the custom matrices also like the number of purchases that happen and things like that you can have a REST API client that can directly post to elastic search so you can also monitor business transactions there and this is one of my most favorite right you can install a Kinesis agent on the server you keep on writing logs locally you don't worry about shipping the logs right this Canisius agent which is a very reliable component which works extremely efficiently will keep a watch on your log files pick up all those records and post them to firehose in a very reliable way and firewalls in a very efficient manner we load them to elasticsearch so this is all arsenal that you will need and you have at your disposal to build a very efficient monitoring system yeah and last but not least elasticsearch can be deployed across multiple Easy's so it works in a very reliable way because your monitoring systems by itself should be highly available so you have to again apply the principles of data classification even to your logs okay the principle that Darien spoke about all data is not created equal similarly all metrics are not created equal there are going to be some high level metrics like user experience metrics user count metrics transaction count how much business happen what is the replication status of your databases so these are very critical elements okay and then there are some low level matrices such that HTTP request count really versus write throughput cache hit vs cache miss rate so all metrics are not simple some matrix is higher impact or you want to monitor is much more closely right and some matrix is just too much so what we recommend you is that low level matrices be monitored using local systems and high level matrices also be monitored using local system in the other region also you do the same thing right but what you do in addition for high level metrics is you send them to the other region as well right so should region a become unavailable all those high level matrices which are much more important matrices they're also available to you in region B just ship those matrices with region tags and you know you'll be fine so this also reduces the cross region traffic because high level matrices are low in volume whereas low level metrics which are like HTTP logs they're high in volume you don't want to shipping you don't want to be shipping everything to everywhere okay this is the last part there are some open source projects which are contributed by other organizations that I personally feel you should take a look at one of them or rather two of them are jewel and if must I don't know what is the right way to pronounce it so what these products basically do is sometimes what happens is your users because of whatever region they end up in a wrong region so the user who was supposed to go to us west he lands up in u.s. East okay so these projects under such operations they allow you to route the users request internally to the other region where it should have gone in the first place and they also help you in the failover scenarios so it's beyond the scope of this article to this session to go in more details of those products I highly recommend you take articles by Netflix on that there one more product which I like from Netflix it's called as Eevee cache so if you want to keep your caches servers also in both the regions in sync Eevee cache can do it for you it can make sure that it writes to cache of both the regions so that you have the cache also in the you know almost the same state so being open source they're available for you to use please do take a look at them so now the conclusion you can leave in a couple of minutes avoid synchronous replication and simultaneous deployments as much as possible okay synchronous replication cannot be avoided in certain rare cases but simultaneous deployment should be avoided all the time okay design applications for idempotency and eventual consistency what that means is even if data is written twice right it doesn't corrupt your system or if there is some lag for data to arrive in the other region the application still is not bothered and it continues to function closely monitor your replication drags and synchronization delays how push buttons ready to switch the traffic so for whatever reason you believe that you can't handle Texas traffic in West Coast and you want to route it to the east coast you should have a push button or a script ready to do it because it might involve multiple operations like changing making some route 53 API calls changing the direction of replication data traffic and all that so you should have push buttons ready should you wish to do such failures the next one is make your high level metrics monitoring systems also multi-region otherwise if a region becomes inaccessible you completely lose history or track of what happened in that region okay so these are too many details because of which we went 15 minutes overtime so it is definitely an involved exercise even when you go to implement it so yeah it requires a lot of careful planning but we have services like VPC peering cross region dynamic global tables ideas replication s3 replication route 53 for you know traffic routing so these things help you achieve the goal of a multi region deployment which are unavoidable if you have a global streaming business or global ride-sharing business there are many new scenarios which requires this architecture to be implemented and we make your life as easy as possible last slide I promise okay so while it is very complex exercise and it requires a lot of planning and thought it can be performing ly beneficial if you have users which are spread across the globe and you cannot tolerate much downtime right so you should definitely consider that if your business case justifies it and if you think your business is of this type consider designing applications for multi region active active from day one okay but as we say in Amazon today's day one thank you [Applause]

Channel: Amazon Web Services

Views: 19,400

Keywords: AWS re:Invent 2017, Amazon, Architecture, ARC319, AI, DynamoDB, RDS, Route 53

Id: RMrfzR4zyM4

Length: 79min 39sec (4779 seconds)

Published: Thu Nov 30 2017