Distributed Sagas: A Protocol for Coordinating Microservices - Caitie McCaffrey - JOTB17

great thanks for coming to my talk I'm super excited to be here in Malaga it's been a really fun a couple of days but today we're going to talk about I'm going to talk to you about distributed sagas this is the protocol has been working on for coordinating micro services and making them more understandable so I'm Katie McCaffrey I am a distributed systems engineer at Twitter this is where you can find me on the Internet I'm at Katie it's pretty simple although if you have questions or want to reach out to me after the talk my D ends are open it's actually an easier way to get in touch with me than email so just use that so yeah let's considered ok so basically once upon a time we used to construct these monolithic applications and they ran at skills that could safely fit inside a single relational database and this is actually really nice from sort of an abstraction point of view because all of the complex logic about consistencies and different consistency models and like concurrency control was sort of all handled by the single database and it like was our nice abstraction that dealt with all the difficult stuff and you would just like horizontally scale out your application and that was really nice but then we are now in a time where we can no longer fit on a single database and the availability of a single database is no longer sufficient for our customers and for our applications and so you know in the mid-2000s we start seeing this rise of micro services and Nestle and basically what we what is the driving force behind these these patterns and the no sequel databases is basically we want availability and we want to be able to scale beyond a single machine and so these are really nice things because with micro service architectures we get a lot of benefits like you can have different teams independently to develop and deploy services at faster iteration speeds they own their own deployment and things like that so we can be more agile and then with these new sequel databases we basically have to denormalize all of our data and our different services but we get higher availability but we core consistency models and so basically what happens here is we've traded off of a lot of the strong consistency and a lot of that abstraction of like this one piece of our system mean like our single relational database owning all the consistency guarantees and sort of smeared it across this whole architecture and so there's kind of a term for this and Peter Bayliss wrote a paper about it called ferrule concurrency control and this is a really great paper I like it because it's academia looking at what industry is actually doing it in trying to offer some solutions or like get some insights and he looks at the Ruby on Rails community in that sort of workstation but I think it's broadly applicable to how we develop software today so what he defines Ferrel concurrency control basically is application level mechanisms for maintaining database integrity so this is basically saying hey we used to live in this nice relational database model that dealt with everything but now we have to actually write code to make sure the semantics and invariance of our applications hold throughout the lifetime of the system and I just think this is a really cool term like I laughed really hard when I first read this paper some people got mad I thought it accurately described what I do day to day as a developer so let's take an example of how we might start building an application with our micro services architecture so we're a small start-up and we want to help people book hotels on the internet and be a travel application but we'll just start with hotels and we want our application to hold the invariants that we will confirm a reservation with the hotel and then you'll like charge someone's credit card we will never like charge someone's credit card and then be like sorry we can't give you a room so that's the invariant that product has decided we need to hold with our application so we build a service we're like are your phone you might talk to some front-end that will route you to our hotel micro service which will deal with booking the hotel and all of the logic around that and it will probably have its own storage and then once it's successfully booked a hotel it will then forward the payment information to the credit cards API or whatever service you used to charge data and so what we're actually doing here is there as Pharaoh concurrency control mechanism built into the service and we're enforcing the causality of our invariant by chain request in this manner first I have to confirm that the hotel gets booked and then I will like make the payment happen and this happens causally because like they're happening sequentially inside of a process but what where does gets a little messy is that if you think through this whole scenario like what if I can't charge your credit card right that might just went out filtered back okay I have to free out the the hotel reservation and make sure that you don't hold a hotel reservation if you don't have a valid credit card so that's like additional ferrell concurrency control and failure handling that i have to build into my micro service okay fine that's not too hard we can do that so then our service so it's great we now decide we want to expand to help people book hotels and cars and airline tickets we get a lot more funding so we have a bunch of supplier bunch of people and we spin up a bunch of other teams and because we're micro services person can work on every up we have different people working on different services so we'll create a car service that will do the same sort of thing and it will like hold the same environment where we may want to make sure our car reservation exists and can be booked and then we'll charge the person's credit card and then we'll do the same thing for a flight right and this is like a super standard way that we develop code it's like very easy to see how this would evolve over time but then product comes back and says okay we're so successful with this we want to introduce a new feature called trips which will allow a user to specify their location and dates and will book everything for them it's like you know like a travel agent service as a service and so this should be super easy because we already have the functionality to book a hotel and we already have the functionality to book a car reservation we already have the functionality book an airline reservation so product is like yeah this should Rubies really fast like this should be easy so we spin up a new service micro service to like embody this trip feature and it'll have its own data store and it will talk to these other services to book trips for us but this thing has a ton of pharoah concurrency control mechanisms built into it because we have to handle a lot more edge cases and a lot more things can sail now if you look at this graph we have to handle of what if you can't book the car when you can't book the hotel and the airline reservation now you have to make sure that you go in like cancel those reservations that you've already made we have to handle the scenario of what if this service goes down that's handling the request and I've booked the hotel and the car and their line service like I have to make sure are either free them or like take that process back up and notify you as a user like we don't want to leak you can think of it as like leaking memory in a system where you leak hotel reservations and car reservations and things overtime that's obviously a horrible business and you can't do that so like we could do this and people do this all the time I've definitely built systems like this it's just tricky and difficult to get right and error-prone and it's usually like one of those things that product never understands where they're like you have all this stuff like why is it easy just to like combine them and you're like well distributed systems so this is our resulting architecture if we look at the system as a whole of all the different calls and paths this is super simple and there would be like a lot more things happening in here but I think it's pretty easy from this example to see how we end up with these desks our architectures this term is from aging and hot cross and stealing it from him but this is Netflix's a graph of Netflix's call diagram on the left right I'm backwards from you anyway not what is Netflix and this one is Twitter's and like you can look at a bunch of services and they look like this like uber also has one that some like when I'm a trainees talk teachers lose like really impressive video from now but you end up with these really really complex systems because we're having independent teams just like evolve a new architecture over time and no one's looking at the big picture and these things become super difficult to maintain over time they're really difficult to understand because you have all this painting of consistency models in micro-services and it's not really documented anywhere in like you don't really understand it then like people leave the company or they move on and work on a different thing and then someone has to understand this whole system and learn it new and then you have an intern excite like just looked at the micro service they were modifying and didn't know that like the order of those calls was important um so this gets a little no no this is coming back this time okay yeah back ok okay so this is kind of nothing and what I like to think about is can we do better can we make it easier to build these distributed applications like what are we missing because micro services are nice and I fundamentally believe that that's like a good way to build systems and scale teams and things like that but like can we do better than feral concurrency control how can we make it less difficult to build and maintain applications over time so what about spanner right spanner is Google's globally distributed database they actually just released it I think like totally GA yesterday I think I saw Herbert read about it but it's been in beta since earlier this year and they wrote a paper on it in 2012 but basically spanner is Google scalable multi-version globally distributed and synchronously replicated database they're basically providing access levels sequel server like transactions in a database that is globally distributed so you have replications all across the world the way they provide this and if you've heard of the cap theorem people sort of joke that it's like beats cap it doesn't it's just like it's still a CP system which means you give up availability but they have five nines of availability and the way they do that is via the true time API a maximum amount of hardware fiber between data centers and GPS and atomic and clocks installed of every individual machine so that's a lot but it is cool but and Google really swears by this and they say they've read the name deltas and invested so much money and this is because it's just easier to think about and have people program but then you have this other cake where Facebook's like nope like we're not going to do that because and they wrote a whole paper on it in 2015 about why they have an adopted strong consistency models at scale and maybe this is a 7 page paper it's interesting to read I think as a technical report but basically it comes down to the biggest barrier is that consistency mechanisms must integrate across many staple services if you think about our micro service architectures we do normalized state across everywhere right because like it's not in one single place and you know we might optimize one data layout so that this query is really fast and we might have an optimized as a different way so that a different core is really fast and that's like not a thing we can get around and so Facebook are using this paper and I actually take this belief is that really strong consistency curved systems are not not the end-all be-all you need it in some cases and if you saw like Chris permaculture on stock to talk about that a lot but you don't always need it and I think we over optimize for it and basically also Facebook goes and makes the argument that they think the users will tolerate inconsistencies over high latency I mean I think it depends on like what you're working on but like fine say that so it's also worth noting that this was protocol that you would have to have define between all the systems to do distributed transactions across many staple services exists in the world it's called two-phase commit this is an atomic commit protocol this was made a long long time ago actually trouble finding the original paper because it's really just that fundamentally part of distribute systems in computer science but basically what you do if you go back to our example it's Apple II and because it has two phases you can prepare a phase in a distributed system where the you will have a coordinator so in this example our trip service is our coordinator and it will say we're going to propose we're going to do something like book the hotel book the car book the flight as part of the trip and no one's going to do it yes they're just going to look at this and say can I do it and then like hold on to the resources to do it if they can and then they're going to say you know vote they're going to say yes I can do this or no I can't and then once everyone in the system responds which could be a long time because you're basically gated by the slowest node in your system and that kind of sucks for a latency then the coordinator is going to look at everything and say if everyone voted yes we'll send a commit message to all the parties involved if someone voted no we will say send an abort message to all the parties involved and so you'll either have it succeed or fail and then because it's a distributed system and because of the two generals problem in asynchronous networks everyone has to respond done to the coordinator before you can actually know that it has successfully committed and now everything is like atomically visible or it like never happened so this exists like this is the thing it's well known and we don't use it in industry very often the reason we don't is because it doesn't scale there are N squared messages in the worst case the coordinator is a single point of failure if that single machine goes down at any single time in that process the whole thing is sort of like goes haywire and then you have reduced essentially your gated by the slowest node in your cluster or involved in the transaction and then you're also sort of like keep their holding locks on things and that's like never really fast essentially oh I think it's also worth noting with two-phase commit like if this was scaleable like we would see the cloud vendors providing it as or actually has a blog post on Microsoft cloud has a blog post specifically stating that they don't for scale and liveness reasons and none of the major RPC frameworks like G RPC and finagle implement this because scale has problem okay so instead I want to tell you guys today about a protocol I've been working on it's called distributed sagas I started kicking around this idea in the Halo days I used to work on Halo the Xbox game and we built the halo statistic service and we were looking for something and we started playing with this idea I've been using it currently in building it out at Twitter to help us do distributed build infrastructure and then I've been chatting with uber as well I started talking with him in the middle of last year about distributed sagas and some of the problems they were having maintaining our my group of services and explained this protocol to them and Matt Raney one of their architects has really has a talk at yaa where he talks about how they're actually are implementing this so that they can make their micro services more reasonable and manageable and maintainable in the future so let's talk about this sagas this is Eckstrom your photos are inspired by the stapler in 1987 called sagas and so sagas are long-lived transactions in a single relational database this is not a distributed systems paper this is a straight-up database paper and so the problem they're trying to solve here is they would notice that there would be these bottlenecks when you would have a long-running transaction like something that was doing financial aggregation over a database for like a quarter or something I mean maybe you didn't need these locks to be held or a lot of times back in the day like you would have to have a user actually input things to make progress in the transaction so your gated by a human being which is pretty slow and so they were trying to find a protocol to say like can we provide some guarantees but not maybe acid guarantees that improve improves availability and latency and things like that so they come up with this pattern called a saga which is a long-lived transaction be written as a sequence of transactions that can be interleaved so essentially they're breaking a big transaction that touches many tables into a bunch of little independent ones and then they also define so basically at the end of this you basically have a sequence of all transactions in the sequence either complete successfully or compensating interactions are ran to amend a partial execution and they do the typical like thing it's at the time in the 80s when it was not fully understand how difficult some of the challenges with the tribute systems Ritter like clearly this could be distributed like that algorithm tend not so so I've been playing around with this and we're going to go through it today I actually find it incredibly useful so what is a distributed saga it's a collection of requests so like in our travel application it's either book hotel book car book flight or charge money and compensating requests it's sort of like the undo request cancel car cancel Hotel cancel flight refund money and these things all represent a single business level action and you're going to combine them into one so let's walk through the differences about distributed sagar request so they're sardar requests in a distributed saga can't abort so i can say hey book me a car and it is totally fine for that service am not doing it and say essentially abort right like in the in the database literature you would use abort you would just basically say like i'm not doing it or fail or whatever request must be i deficit so this basically means that i can make a request and i'll get a response and then if i make the same request again i should basically get the same response and it looks like the request has only happened once so basically you can do it many many times and it looks like it has always been affected once the reason you want this in a distributed saga and often in the distributed system is because our networks are asynchronous so if i make a request and then i like never hear back from it i don't know what happened i don't know if the message got there and got processed and then the response was lost i don't know if the message literally never got there because the requests literally never got off box there's a bunch of things that it could have failed and for liveness reasons an industry we say a timeout and say we think it failed but we're not really sure so what i want to be able to do is to replay the request and then get a response and if it happened to get there or not okay so we'll just replace until they work we also have the idea of compensating requests and so compensating requests have to semantically undo the effect of a request so I'm saying semantically because it doesn't have to exactly put the system back in the same state that it started in before the saga or that request executed so for instance like if in the example we're using if I charge your credit card and then refunded you money you actually might see two line items on your bill that month and that's okay because it's semantically in the same state another good example this is are they're just things you cannot undo because they have side-effects like sending an email so if one of my requests sent an email um you could just send send a follow-up email the compensating request to correct the information that was needed comments any requests also cannot abort they have to be able to execute to completion so when I say cancel the car it has to respond true it can't like say no I'm not doing it that's not valid compressing a request also must be a definite because I have to be able to run them until they succeed in order to provide the guarantee that we talked about or in order to provide the guarantee that we'll get to and then compensating requests also must be commutative and so this is another mathematical term but basically it means like when you book a car and cancel a car it's the same as if you cancelled the car set that message and then book the car and this is the same as like addition like one plus two equals three and two plus one also equals three and the reason we need this and this is also a property that I think we don't talk about a lot in industry and distributed systems but like it's probably kind of wrong and a lot of stuff you don't think about it but because of this reason where we use timeouts and we have asynchronous networks you might start making this request like book the car and then it waits a long time and we're like we think it fails we're not really sure so we're just going to like move on with our lives but we don't want to like just abort you know indefinitely we want to be able to complete this person's request at some point so we're just going to say book it again and then we get a response and then maybe later we need to cancel it so we'll say cancel car on the getter response and then this is a time is going down in this graph if you haven't seen these diagrams before so that's what's happening time's going down and then eventually you'll get this message this pesky like booked car message will decide to like actually come to the car service and we don't want there to be like a league where someone's holding a car reservation that they shouldn't be holding but because our cancel car request is commutative it doesn't matter and because the book car request is I definite it will look like no one will hold a car reservation and that's what we want okay so just to sort of like Risa mirai's requests must be identity' and can't abort and compensating requests must be I definite commutative and they cannot abort they have to always be able to be like you could fail and have a timeout but they have to be able to succeed if you can talk to the service so if we have all these properties that Sark requests we will get this guarantee that all requests either completed successfully once a distributed saga has finished running or a subset of requests and the corresponding compensating requests were executed so this is nice because it means our system is in a consistent state is semantically equivalent it's holding all the invariance that we sort of set out and defined for it it doesn't mean that like you might hold a car reservation when no one thinks you hold a car reservation or we accidentally charged your card but cancelled all your reservations because there was some failure in the system so this is super nice it does have some drawbacks like I said this is not an acid level transaction there is no atomicity or no isolation so what this means is that the the requests in the saga are visible for the entire Sagar completes so you might see that I hold a book hotel request and a book car request someone may come in and try to book the hotel book because I got the last room on that day they are rejected that request does not succeed and then later I might cancel that request or the saga may cancel that request and then the next person could then go and book that room this is okay I think this makes sense I think we're actually already doing all these mental gymnastics anyway in the current way we build systems we don't really live in an acid world anymore and for most things it's like like the world is kind of easy like I tell you information then you tell someone else and it's not like total knowledge transfer all the time like things don't just like pop into being fully formed so this is like important to know because like if you're talking about finances and things like that you have to understand that they don't like all commit at the exact same time and no one sees intermediary things but for a lot of the stuff we do day to day this is totally fine okay and so this guarantee may appear weak at first because we're not having like acid lemons actions I'm not going to give you a guarantee like spanner does but instead of having but what I think incredibly powerful but because instead of having to hand put a bunch of custom feral concurrency control logics and handle all the edge cases in every service that you're talking to we can simply focus on what we want our applications to do once we have this in place and so you're really sort of allowing people to focus on like the application they're building instead of all the edge cases and failure modes and faults of mains that could happen so how do we go and define a distributed saga a distributed saga it could be defined as an ace directed acyclic graph and so this is super nice and one of the things that I've been working on adding and extending and the reason you want to do this is because like sometimes you want things to happen like causally in a system like I want to make sure I book everything before I do the payment and so what this means is you have anything none of the nodes in the graph can be executed before all of their parents have completed so this is just sort of a way to give you a way to think about it so like in this graph you can book the car the hotel in the flight in parallel that could happen if you want to in your system and then the payment will be executed left each node in the distributed saga graph has a name you have to be unique name for the graph it has a requested enough information to make that request like book hotel and the same for the compensating request and then it has a status and so this is the only piece of state that will mutate as we're going through in this graph it also has a couple like special vertices that are the top and the bottom the graph I name them start and end just because it's sort of like I help helps me think about it but it's really just the top of the graph the bottom of the graph so we know where to start where to end and these are basically the same just they're compensating requests and their normal requests are no ops so they'll get processed the exact same way just like they have no effect on the system okay the other piece that we need is because we want this to be durable and persist across failure domains as a distributed saga log the alarm needs to be highly available this is the piece in the system that where you want to invest in something that does have fairly high availability since gotta be like a CP system but it needs to have high availability because if it is not available the sagas cannot make progress if they can't write to a log and then finally we need something to execute our code to and I call this piece of saga execution coordinator as the original saga paper this is literally just a single uh process of code that's running and execute the requests and the compensating requests according to how the protocol works it's not special unlike two-phase commit this process doesn't even have to do the same process throughout the lifetime of the saga it can literally be a different process for every single node in the graph and that is totally fine it's not special it's just like you know it's just turning it's stateless essentially all the state is in the the execution is in the log okay so how do we go through and execute this let's keep using our trip example if we started a saga we get a request that says book trip request I know that JSON is kind of like small I'm post all the files are already online I'll share a link but it's just to give you an idea of what happened I'd have all this information it would come in I start at the top of my start saga my trip booking distributed saga and I'm going to then log a start saga message to the log and this has to happen before anything else can happen and I'm going to store all of the data associated with the request that I might need to execute any future request typically when you implement this you destroy everything and you can just like read it back later and it's great once that successfully is durably stored it'll acknowledge it and then I can mark the start saga no to my graph as complete and we'll go on just for clarity I'm working all the stuff in pink on the slides that are like pertaining to that note and then the life change as we go on to help you follow along so now we can execute any node in the graph that parent is complete so that means the car note the hotel note or the flight node these could actually all run in parallel - that's totally fine so we'll log the start hotel message to the saga log this has to happen before we can do anything else and we need to get a axis that has been durably persisted we can then make the book hotel request I'm just using information that came from the original request to do so this is totally fine I will get a response this is yep I booked two requests and then I will log an end hotel message to the log and the information from that request that came back I'll do the same thing for the Karno this can now be processed I'll ready to start car message to the log once that completes I'll send the book car request message to the car service it will complete gives me a response I'll then log that in the end car message in the log and then mark that note is complete the flight canal B is the only node in the graph that can now be worked on so I'll say start slate and then I'll book the request once that is successfully stored once I get her a spot along the end flight message with the data and now we can mark that note as complete and finally we can do our payment node so I'll send the payment request it will respond will log all the data to the log that note is now complete and now our at the end saga node which is a no op so it just completes usually you don't even have to really write this message you can construction you have enough information in the log without it to construct it but it is usually for performance reasons that I log it because that you can just query and say like is n so I'll go there instead of reconstructing the whole graph and now we're done and our guarantee holds that all the requests in the graph executed successfully what's also really nice about this just from having use it in production scenarios is that you now have an entire log of everything that happened in your system and sort of like you can add additional messages too like the one I'm working on it sorry add messages to the start stuff as well just for debugging if we want to so then you can go and like easily see what happened with this which is nice for like anyone trying to figure out what went wrong or like eating customer service agents or like you could easily make a rollup email of everything you haven't it's there it's great okay so what happens when a system fails so like say we've started the trip saga in a different iteration we started the hotel and we've successfully booked the hotel and we're running things in parallel this time so the flight request is executing and then we are also executing the trying to book the car but the car service is like no I can't do that because I'm not going to I don't have a car available on that date so it responds and so now I need to log an abort car message to the saga log so now we're going to start a rollback basically because we can't guarantee that all the requests execute Excel successfully so once that is durably persisted to the log we're going to start rollback recovery and so basically what this is is I flip it you can think about this as I just flip all the edges in the in the dag I've renamed top and bottom just for clarity here and I'm going to mark all the nodes is not completed and then we're going to start rolling back in the opposite order working from the bottom in this case to the top so we're going to start calm saga this is a now op vertex we don't do anything we mark it as completed we then have to ask the saga log hey did anything are there any entries for the payment vertex and it says no so we can just you know we don't have to execute any complicating request that requests never ran nothing happened we're going to do the same thing for the car we're going to say hey do you have any entries do we do anything here it's going to say like yeah there was a start and abort but we also know because it aborted that that request can't complete successfully which means we also don't have to compensate for it so we just mark it as completed in move on for the hotel we'll ask for the log entries we'll get that there was a start and an end message logged this means that we currently hold a hotel reservation and in order to maintain the sardor guarantee we now have to compensate for that or cancel the hotel reservation so we'll send a canceled hotel request once that succeeds we can log the information to the log if we want or just the Cobb hotel message to say that we've successfully done the compensating request and then finally for the flight log entries we're in kind of a weird state here we saw that there was a start message and there is no end message we're not really sure what happened here we're not sure if we logged the start message in the sec crash and never was able to make the flight message I'm not sure if that message timed out or completed and we didn't get a response so because our message our requests our identity we can just force it to succeed so we'll book the flight request well look we'll get the response and then we'll immediately cancel it and this is the only safe way to do this because if you just think about if you just try to say like oh it didn't happen like you might still hold that flight reservation because it could have gotten there that message could have gotten there eventually and basically there is no safe amount of time that you could wait where that couldn't happen and because there are things are commutative this is totally fine so then now we have canceled the flight reservation and we can complete that node we now need to log the end Sarno message which is once again just an optimization but now we're done again and we've just walked backward and sort of undid everything we did and we still maintained our guarantee that no one holds a car a hotel or a flight reservation because the saga failed so I think this is pretty powerful because I spent a lot of time sometimes or had been funny a lot of times ad hoc coding this stuff one-off there's also another piece of this that I'm not going to talk about today just for time reasons it's recovering from saga execution coordinator failure or if you want to have a different process run it every time depending on the architecture your system depends on how you want to do that but this is like right it's not special a can't fail you can continue to make progress even if it does fail or the sec changes and this is possible because all of the data that like all that state any of that state that we changed in the dag whether it was done or not is maintained in distributed saga log and can be reconstructed from that so you can either rebuild it on literally every request or you can maintain it in stay for performance reasons and try to use the same process and then recover on failure so you can sort of if you walk through the examples backward I hope that's obvious that you can do that okay so let's go back and look at how we start running our travel application using distributed sagas it would look sort of something like this if we went through the same pattern where hey we just want to start booking hotels so instead of encoding our invariant that we want the credit card to get charged charged inside the hotel service via that chaining of micro service calls we're going to do it in a saga instead and so we'll have a hotel the book hotel and then we'll just say you know then once that completes booked the payment and then be done and so instead of like we'll have an extra service to start which is our sec or whatever you want to call it and that'll handle making the requests and then rolling them back if they fail when we add a car it's just a simple the car rental service it'll be just as simple will encode that dependency in the system in a saga so you can think of sagas like you're going to have to store them somewhere like it's like a stored procedure or you could like send it in the entire across if you want but usually you're doing the same thing over and over so just sort of like load it like code and then you can go from there and so then you just literally you have another team spin up a micro service car service or a Microsoft that does car rental and then and then you encode the information in the Fargo and now we don't have like we still can just reuse that payment API because we know it already works and the same thing for flights right we just have to add the new service and then we add a saga which should be pretty easy and then finally the real power comes now and we have our set of micro services to work with adding the trip feature is super easy we don't actually have to write any new code we simply define a new kind of distributed saga called winter trip saga and we add an extra API to our front-end and then this will execute because we already have all the services to do it and we're just encoding some causality and dependencies in a different way so I think the tribute starters are really powerful because you've now isolated complex code to a single logical area in your system and this thing is not a single point of center I think it's the other really important thing like a database or if it went down then like you can't do anything this thing you just spin up in another machine and then you just keep going this is really nice because unlike our micro-services thing the complex code lives in one place and it's hard to it's a little tricky to get right but not too time-consuming but then it's like in one service versus like this where it's like everywhere which kind of sucks I think this using distributed sagas gets us more to a modular service design and really goes back to what the promise of micro services or service-oriented architecture was where that service is only doing the thing it's actually responsible for which is like a bikini flight in the the you know sort of the normal microcircuit exer it has to know how to book a flight and has to have feral concurrency control mechanisms to make sure that we're not you know charging someone's card when we can't get the flight or cancelling cancelling the reservation when we can't charge the card and all that kind of stuff that goes on and then finally I think it's really powerful in the sense that you have you now have service composition if you think about this right like it's much easier to just compose services because they define API and then you just sort of define the relationships you want them to have between each other versus this you would probably have to add a new service and some new Farrow concurrency control mechanisms to get them to behave the way that you want them to so I hope I've convinced you that distributive sagas make building and modifying micro services easier I'd also argue they baked building and modifying like any actor based models easier you can use this easily send this use it there and then also I really fundamentally think that with servo LS this is a much better pattern to go with because there really is like you still have these cases where like writing to your database could fail whether it's like as your functions are like lambda I think in AWS and you still have to handle all of that in the code that you write in these serverless functions like you're still ring on a server they're still running inside of an asynchronous network if we just do this and instead it's much easier to like have building blocks and combine them and just have some generic protocol that deals with this for us so thank you this is me on the Internet and I'll take questions now [Applause] Thank You Casey any questions excellent presentation very thorough thank you thank you I just one question you have in this example that the flight booking that was unclear first needs to succeed before you can cancel it but since these two commute wouldn't it be enough to just cancel it no so the issue here is let me go back to this graph it's a little weird I totally agree it seems like super counterintuitive I have so many transitions so the reason you have to do this is because if we go back and I modify this one so the problem is is that if I don't have this say this book this middle book car one doesn't exist like I lose that one that doesn't execute book the car and that was a scenario wherein we don't try it again if I just run cancel car and that message never gets there like there is a property that you could define but it doesn't have like a really good math term then works on like the other iteration of this talk that I've done in the first version I basically just say oh you just run a compensating or a class and it's like like can't then the Assumption there is that cancelling the car you a can't use any of the information from the booking request to cancel you have to be able to somehow like magically just tell it to cancel something that you don't have a reservation for which is pretty difficult when you think about it logically and like how you would actually encode this and then also if you send cancel car and book car never gets this it also has to basically be the identity function on the server which is also kind of weird and I don't know many examples of that so I moved to this this is also the sort of like you know replay and then undo is also something that I believe is used in some indexing protocols for databases as well just because you want to make sure you know it happens then you don't have to get into this wonky world of like did it happen or did it not you just you know right it's a stronger guarantee any others yes do you know if there are any implement like ready generic implementations of this protocol so there is not one that I would recommend using that don't like skew be fair if you go look at my github there is one that is like a prototype and then we built off of that at the one that's being run into Twitter I would not like you want to look at it and like play with it it also doesn't encode it's just the saga messaging part because the sec is very the sec is the hardest thing to generalize I think because I've done this in a couple different places it's really dependent on how you execute your your infrastructure uber is working on this and they may want to open burst it at some point but I think it's also tied just the way I actually works is tied to some stuff that I don't hook they have open source yet I am also playing around with writing one like an example this was maybe in Orleans just cuz I think it would be fond and I used to work on Orleans so TBD I guess it's actually not super hard to write if you want to go look at my github at ADM 20 you can see sort of what we're doing there but then it's also like there's a bunch of different trade-offs in how you actually implement writing to that login what storage you're going to use because it depends on like if you request a really big like with our distributed build graph or our distributed build infrastructure we actually are running like it doesn't have to be super fast but it has to like run like you know thousands and tens of thousands of test cases and that's like that many different tasks or whatever requests in the system and so that gets a little you optimize for different stuff so there's a there's a toy one I want to say a toy very strongly like please don't pick that up and put it in production I think one more the back No how do we handle when the service is not available you mean like the saga execution coordinator recover me recovering yeah how do you recover that's also very sort of like depends on your architecture if you use like one of the major cloud vendors and like I'm pretty sure it'll just spin you up a new one if you're using any of the past solutions like it'll just restart the process and then it can just relook at the log so that's totally fine you can totally there's a bunch of different ways you can use timers you can use callbacks it's sort of based on however you what works with your infrastructure and where you are I'm a big advocate of like use what you already have that work and just building something I'm custom so the simplest we can assume that it is not available as a Justin and it or so the simplest way is to just have another service like heartbeat it like and you probably already have this if you're using any kind of monitoring or logging right so like the really simple like I have no automation in my system is like you might have Nagios like literally ping it every five seconds and then if it dies you can have someone manually go restart it most people have something that like does something more sophisticated than that and will like automatically reboot the machine so like for instance we use Aurora and Mesa set Twitter and so the way we have our sec recover is that Aurora does heart does excuse me does failure detection for us and we'll notice that one of our nodes died or if it kills one of them it'll just bring up another one somewhere else in the cluster and a bunch of other I think there's a bunch of other cloud solutions that do this as well okay thank you yeah so related to the same the same question what happens if the SEC fails you mentioned that you actually store some state there the data like dibs on some of their steps or some of the notes complete or not so you have to restart that somehow what do you do like you and do the whole transaction just in case or no but so you don't have done do the transaction with SEC fails so because the SAR like this graph you may use a graph and your in your code as like an optimization like to persist that state but you don't need to all of the information to like reconstruct this graph on the fly and to think about the algorithm is actually in the log messages and so you can basically like you can also just literally have the SEC like read from the log and then and then reconstruct all the states so that's how you recover did that answer your question okay and what if we don't want to allow a user to see intermediate results does it mean that this protocol is not applicable for this use case yes so this is very much you I don't provide any atomic guarantees at all and most things don't honestly the only really thing at a distributed level out there is manner you can use two-phase commit at low scale like some people do it it works okay but I would not recommend using that in any critical or hot path I would also argue that like our desire to use acid is sort of just a desire to make our lives easier to think about when we don't we're a we don't have it today like and so people have figured out how to get around it and build really complex applications without using it a ton because that's how a micro services were coming to know what has acid and so you might need it for things like in Twitter we use strong consistency for like giving you a username because that would be really bad if we give you a user names and we're like sorry like you can't have this username so there are things this doesn't work for but for a lot of stuff where you're just literally trying to think about something to happen and then this thing to happen and then this thing to happen it actually works very well and gets some of the crazy logic out of the code thank you cool thank you are we out of time we've got five more than one more question I know if you're giving me to like we're done look okay start so follow thanks for this presentation was very nice and the weakest point is still discord coordinator so how to provide cover failover or something like this yeah so it is it is still I guess what I should really draw this as is that it's not really a single point of failure because literally any process in the system can do it it is stateless you may be preserving some state as it is running as a performance optimization but you don't have to and so like really you are only gated by the time it takes to spin up another machine or like hot failover to another machine and so this is basically like this works today this is how like we provide high availability and no sequel databases and distributed larges we just like have multiple of them running or we allow them to recover right so okay thanks I think that's it Thank You Daisy thank you you

Channel: J On The Beach

Views: 35,284

Rating: 4.8981614 out of 5

Keywords: Distributed Systems, Microservices, J On The Beach

Id: 0UTOLRTwOX0

Length: 44min 34sec (2674 seconds)

Published: Fri Jun 09 2017