Do you know Distributed transactions?

every distributed system at some point of time needs to agree upon some decision or some value and this process is called as consensus and here it is the meaning of consensus and there are so many algorithms are methodologies which actually helps us to implement consistence some of them are wrapped saga axes chubbie two-phase commit and our newly added one more methodology called as Nara in LRA so in this video we are going to talk more about distributed transactions because you guys learned in a couple of earlier videos about more about database transactions how it is implemented and how different kind of logs like pessimistic and optimistic logs will actually help you to implement transactions in database and non database kind of environment in this specific video we are going to talk about more about disability action because we learned that in database the concept of transaction is all or nothing basically when you have so many queries and updates we treat all of them as automate block and then we always expect all of these to pass or none of them to be executed but think about it if we don't have one database and we have so many queries and updates where we want to treat all of them as atomic then how do we do about it I hope you guys know different kind of software architectures one is micro service architecture one is service-oriented architecture and the other one is monolithic architecture moralistic is cold and it is gone and service-oriented architecture is being used somewhere and the most popular one is micro service architecture monolithic architecture is basically is like kind of like a single bundle of all different services and it's usually in a single deployment and you basically do a vertical scale whereas in service-oriented architecture it is more of you know similar kind of services are bundled into one specific service and that itself can be scaled independently whereas in micro service architecture is a different way of service-oriented architecture in which you basically the application is made of small you know independent or loosely coupled services to understand transactions in micro services or transaction in distributed systems we need to take couple of examples and then see how it used to behaving monolithic architecture and then how it behaves in micro service architecture to understand it better so I have an example here so this is the architecture for monolithic so so so this example is for monolithic architecture in which the user is trying to place an order at Amazon website by paying some money which he has in his wallet so we are interested in transactions right because the video is all about distributed transaction in case of monitor I think it is not really distributed at all because we have one service which is basically the Amazon website itself so all of the service is connected to one database this particular user is trying to place an order so what happens the sequence diagram will explain explain that so when the user may be for example take ID with seven places an order at amazon.com some order so so you will make a call to Amazon service and this is whole of monolithic service and then this service is going to start a transaction and there's two important actions to successfully place an order he has to first check the balance and update the balance by deducting the price of the product and then he has to place the order there are so these are the two important you know queries which is actually happening so before doing that as you learn from previous you know videos that need to have a transaction because we have to treat these two queries like it could be select or it could be update or it could be insert has one single unit of work to make all of this process atomic otherwise there are a lot of problems which happens please refer to the other videos which have made so so we have put all of this into a transaction so you can see have referred it as DB transaction so first what we do is update balance in the wallet and then second place and order or update an order into the database so this works smoothly because it is a modern with the architecture nothing is going to go wrong because we have isolation it's completely atomic so everything works fine suppose what happens if the user doesn't have sufficient balance in his Wyatt so this is the step where we are basically updating the balance of the wallet that said we actually check whether he has the balance in his wallet or not in that case this query will definitely fail so we basically does nothing so because of the transaction every previous action will basically will be rolled out and be basically doesn't commit so nothing is going to happen basically the user will see an error showing that you don't have sufficient balance so what if he has a sufficient balance but we don't have enough products in our inventory so in that case we will update the balance remember we haven't committed yet we will definitely update the balance by deducting the amount here suppose to paid and then we will try to place an order and we see that we don't have enough product in the inventory so this particular query will fail and we don't commit and basically if rollback so all the deducted amount will be put it back to the in this wallet and then again a user will see an error showing that we don't have the product into the inventory please check back later so that's how it is protected even if two or three you know parallel transactions are going with the same user ID but still since we have started the transaction that particular row will be kind of blocked so other transaction will be waiting until this transaction rule is a sell arc so the isolation is provided so what happens in case of microservice how do we do the same transaction and microservice and now we have the microservice architecture for the same example and as I told you this is loosely all services basically this man services which are which makes a complete service or complete application so in this case the same application has Orchestrator basically that's the one which basically this is the call from the user and then files so many calls and then call it collects the results and give it back to the users or the API cards so in this case whenever user plays our order basically the orchestrator calls to different API calls of micro services and now we have two micro services for this purpose one is customer wallet micro services and the other one is order micro services and they both have their own databases they are not sharing the database ok so now whenever the user I want to place a order he makes a call to ARCA straight on an Orchestrator file it was to customer wallet and order and then the process would take care and let's see the sequence now how exactly it happens now this is same customer with ID 7 places an order and the orchestrator receives that call and then the our constructor basically makes two calls one for customer worried because he needs to deduct e amount and then he also needs to make a call to place an order or create an order to the order micro service so now what he does he does either fire those coins one after one or in parallel so let's consider one after one so first it makes a call to get up the balance and then he makes a call to create an order so do you think this would work this will definitely won't work because the transactions are limited to their own local or individual cause of that particular micro services the transaction is not kind of global usually in if you see the monolithic the earlier sequence ride-around we're in monolithic we had the transaction for both of these kind of quits since the services are split into small small services individually now we can't have this one transaction for both of the calls because these two services are not sharing that there is the they have their own databases because that's how microservices are big so now how the isolation is not definitely not there because when you whether when this first call happens he will deduct the amount and then the create order will quad will happen by the time what if one more might call for the same user gets into here and he sees the deducted balance basically still we haven't placed the order but other DB uses the user is seeing the deducted balance and the order is still not placed so these kind of problems are there what if this fails what if this fails and who will take care of the failure scenarios what if the balance was deducted and then the order is failed because we don't have and if you know products in our imagery so who will take it off you know rolling back the amount whatever we deducted so this is all little getting messed up so how do we solve this particular problem of transactions in micro services the concept to solve this is called as distributed transactions and there are so many algorithms are you know the process to make sure the transactions work in distributed systems so before solving that here is a little crazy idea so instead of doing anything can we do something to this micro service and solve this particular problem so yes actually we can write because see instead of having two different databases for these two services because we know that these two service is a kind of dependent and we need a transaction why don't we just connect order and customer wallet to the same database so with this work definitely it will work because these two are services and they both are connected to database and they should work but this is not the ideal way of building micro services because this is anti-pattern where two different services shouldn't be connected to the same days instead a couple of guys have you know you know a couple of days of while it and still go ahead and do it buts file but it's not really scalable and this is not a reliable way of doing it I mean course you can do whatever you'd like end of the day the solution should be neat and it should be level and should be scalable and all that so the other techniques what people recommend is instead of having you know two services practical same day we have a replication strategy of you know kind of like you have the two different instance of TV and they both are replicating the data between each other and this is still fine but the only problem you see here is because of the replication the consistency goes for a toss in this system the consistency will not be there if your system doesn't need consistency then you can do this way otherwise you will have to stick with having individual databases and stuff you should still implement the distributed transactions for micro services so how do we really saw the distributed transactions problem the answer is two-phase commits our saga so let's learn the two-phase commit first and then we'll concentrate more on saga so there are so many people who has implemented I know two-phase commits in a lot of different project and everyone has their own different opinion some say is two-phase commit is not really good for micro services some say is it all depends on the use cases personally I haven't implemented on my projects this is all the information I was able to collect from different blogs and sites so I'm not going to recommend what to do but this is how we can actually solve the problems if it really fits to your use case definitely go ahead and implement it if it doesn't fit sit use a different way of implementation maybe Sado will definitely help you if you think two-phase commit helps you then please go ahead and implement it now let us understand what is two-phase commit so as in the name it has to face that means that we have two phases to do it so the first one is prepare and the second win is come basically who tells you to prepare or commit so if you remember in the earlier diagram we this much right we had this much so we her Orchestrator we had customer wallet micro service and other micro services we had two different databases which are connected to individual services now we have introduced one new server or component cordis coordinator now for the sake of simplicity a Brittany tasks operate one but in real word this could be present in any micro service or it could be a independent services so let's keep that as independent services so what this coordinator does is this is the one who actually takes care of two-phase commit so in case of any other commits like a layer transaction committed they are all kind of like or exactly one phase commit they are like it's just one entity right so any come it happens it happens at one shot but in this case the Komets won't happen at once basically it has two steps to do commit when it's preparer and vice coming so how this exactly works so let's see what happens when the user places an order to the system in the form of sequential dagger so the user places an order the first thing will happen sees the coordinator should first create the transaction ID because that's the ID which will be referenced to or given as a reference to all of the different micro services which the coordinator is going to talk the first thing the coordinator does is the preparer face and then the second commit phase so first he has to do the preparer face in this case that's the first stage right so in this stage the coordinator will ask the customer wallet and also the order micro-services to prepare their states what that means is basically when coordinator has to customer Wallet microservices to prepare what he does is he basically checks for the balance does the customer has the balance or not if he has it he basically locks that particular row using the transaction okay and then the same thing happens with the coordinator when when the coordinator talks the art of my son's husband he will basically check if that particular order exists in the inventory or not if it is it will basically block that particular order for himself and then both of the services basically reply to the coordinator that yeah okay we have prepared so this is the card which basically says that we have prepared by the customer wallet and also by the honor that ok have prepared the order so now the coordinator will be basically waiting for two different calls because it knows that I have made a request to two micro-services one is customer and one is art of micro-services so he will be waiting until two callbacks or two responses if for example if the order response took some time to reply then the coordinator will not do anything but wait as soon as he gets to calls he will basically continue with the second phase of the two-phase commit the second phase of the two-phase commit is commit so in this case one successfully prepared the coordinator basically asks the customer wallet and also the order microservices to start coming what that means it basically giving the confirmation that yeah you guys can go ahead and commit whatever you guys have prepared a lot then the two calls will be basically placed one is commit wallet and then commit order both you basically committed at this point of the time and then they both have to respond it back to the coordinator saying that did we successfully committed or not so in this case both sent back the reply that yeah okay commit and also this guy replied back to the coordinator that okay we ordered so this complete cycle finishers of the distributed transaction using two facedowns well then I have to look into the failure scenarios so so the first case will be for example what if the user doesn't have sufficient balance in that case the customer wallet should basically throw an error so what happens is the coordinator billing makes a call prepare basically this one the customer one at microservices basically tried to check whether the customer has enough balance in his wallet if it doesn't have basically in this face itself he starts to throw and how basically we see the response but it's not a bouquet preparation ok prepared but instead it will be throwing an error and as soon as the coordinator sees another from there you know customer wallet micro-services the whole process will be aborted and that's how the destroyer transactions basically handle the failure in case of prepare wallet and the same goes with prepare order as well if the order is not there Elementary basically in the first step itself the transaction will be about you for some reason after prepare statement if there are any years then also there are possibilities that the whole transcription process might have Mike about say in case because some are have the sufficient balance and image should be had the order so in that case the prepare face will succeed with okay preparation and okay preparation by the order so now basically the coordinator tells customer wallet service and order services to commit it when both guys try to comment if there is some inconsistent value for some reason just for for example the definitely the error will be thrown at this stage and again the transactions will get aborted so how exactly the isolation is provided here because as I mentioned in the stage itself basically both will lock their respective rows using their local transactions so no other you know requests we basically can modify the same row which is locked saying yes of here if we are modifying ID 7 so in that case in the prepare stage itself the customer wallet 487 is completely locked and order for ID 7 is also locked so no other you know concurrent requests will not modify the wallet they have to wait until this transaction you know returns the lock so that's how the isolation is also provided and the whole thing is kind of behaving as a proper transactions in distributed environment so the only important thing we need to remember is we need to definitely have our time out because say for example the the coordinator should definitely have a time out why because he said I may say to prepare wallet prepare wallet to both of the services what if these guys never returned the poky appropriate message then it doesn't mean that coordinator should keep on waiting so he has to abort the whole transaction so we also need to set some time out badges for you know basically waiting time out to make this thing work properly what are the advantages and disadvantages so the advantage is it is definitely a strong consistent model of distributed transaction okay but the problem here is if you see it is the coordinator who is basically handling all of these micro services and that's kind of bad and also if you see in the from the prepare statement the resource is kind of held up or locked since all of these dialogues are are the talking is happening over HTTP that that's kind of slow as compared to the other database transaction so these locks are kind of held up for so long time that it kind of causes a lot of latency in the system and that is the one important reason why people doesn't suggest to use two-phase commit in case of micro services so the two-phase commit has strong consistency but there are some serious problems with the way it works and that actually makes the development developing this particular protocol little more tedious job say for example let's go one by one what happens if the coordinator trains at this stage we have just created the transaction ID and the coordinator free what happens to the a procedure of distributed transaction we don't know right so what happens if for some reason the North didn't even respond back when the quad rangers sent a prepared message the coordinator would be keep on waiting hoping that the order micro-services will respond to me but what if this particular web service is down and he never responds back so the coordinator neither can't abort or you know commit he has to just keep on waiting that is the reason why I mentioned he had to have time out to not happen this way so that some of the other resources will be held until this day Britten's the okay proportional message but since this services and this is the extension of the two-phase commit called as three-phase commit in which one extra step is intuitive called as pre-commit stir and how this pre commits step basically helped us to recover in the case of coordinator failure or participant failure or both the participators can be one or more those are the different micro services or different nodes which are actually participating in case of for three-phase distributed comment and the coordinator in the three phase commit the advantage is coordinator can be any one as I have earlier explained by leader election any node or any participated can become a coordinator and also the advantage is because of SAP recommit any coordinator which takes over the failed coordinator can start from the middle by understanding the state of recommit so entry phase the first stage is called as can commit in this case the coordinator basically understands how many participants are there and so it basically since the request chorus can come in and all of the participate or will reply as S or no or whatever so that's the first phase and then in the pre commit phase this is where the coordinator basically tends to lock or you know to prepare the data to start the comic procedure and then once the acknowledgement is received from all of the participators which the count we got from the first step so the coordinator waits for all of the participate to stove to play today by acknowledgement and then it basically calls every participated to do commit and then waits for all of the committed acknowledgments so this is how the three phase commit works it is somewhat similar to two phase it's just that one pre-commit stages added and i'm going to tell you how exactly this stage helps you to take over when some your coordinate or participative phases so let's see how three phase commit helps us to recover from the state where the earlier coordinator died and how new coordinator will actually take over state the new coordinator definitely needs to talk to all of the participate is to understand the state say for example if the coordinator when the coordinator talked to participated if at least one participated says that I have B do commit you know message received that means that for sure the new coordinator can guess that the old coordinator was definitely ready to issue or do commit and hence the coordinator can take hold from there by ashame do commit to all of the participants and committed for some reason if none of the participant says that I haven't received pre-commit that means that for sure the new quadrature can guess that none of the participants participate as would have committed the data so that means that we can either start from pre commit stage or we can start over from the can commit to understand how many participants are there and then the pre-commit stage would continue and all of the data will stay consistent we learnt two-phase commit three phase commit and also dynamic way of choosing coordinator and all of that they are good these procedures works well and they're con strong consistent but the only drawback is they are synchronous and also they take take a lot of I doe time that means that increasing the latency this is also a widely used pattern called as saga and this was a synchronously that means that we can implement this kind of pattern when we have WebSockets or long polling or even in case of polling as well it's up to us to choose what kind of technology but it works well with the asynchronous way of sending the messages between client and the service but nevertheless the how this pattern works is if the series of my two services basically talks to each other using messages over even burst or queuing systems let's take a let's take the same example of placing an order in Amazon or in a ecommerce site how does it work when the user places an order the request actually lands up to order MS or it could be vice versa let's take it came to the first went to service called as order micro service in this case the auto microcells basically checks the inventory if it has the product which the user is requesting for he basically adds a message for us order created into this queue or it's even worse let's let's take this as a simple queue it adds that message over here and then when the customer wallet multi services is free he basically picks that message and then processes by deducting the respective amount from the wallet so if all goes well everything is succeeded so the whole cycle is kind of finished what if for some reason if the customer does didn't had enough money in his wallet then this guy will add a message to our different you basically basically we need to draw back the audit which is space right so in that case we'll have one more cube maybe maybe somewhere here so we have one more cube so so basically this customer management and a message called as roll back order or something like that so if basically for any actions we need to add a message to the queue and there is a spective micro service which basically takes care of that so when this custom of wallet micro services pays it adds a message called as roll back after this message will come and land up into this queue and then there is a micro service which basically takes care of customer wallet roll back micro services and this is basically does the rollback of order and everything so they need to be a lot of micro services to take care of different kind of actions and all of these actions are basically transferred through the queue or even buses and hence there is an ends it is a synchronous way and it is little faster compared to the two phase and three pace because we aren't using a lot of locks over here and all of the transaction is local transaction with respect to those micro services so when it plays the order there will be a transaction inside this micro services not in the transaction on all of the metro services and like the two-phase and three-phase that's why it is much faster than the synchronous and also how isolation is provided since any order requests are going through the event buses all of the messages are sequential in here that's the reason why the micro services will not get the multiple requests at a given point of time you can scale these micro services into horizontal horizontally and these micro services basically receive the sequential messages and hence there will not be a you know isolation problem at all so the complete Rudrama city will be still provided using this way so I think that's it this hand pattern works well as well I think have covered a different ways of handling transaction in distributed systems basically micro services is one gurai sum for the distributed systems I hope I have covered all of the information so if you like this video please do subscribe to my channel and if you guys have a new topic please do write that as well in the comment I will make a new video on that I think yeah that's it do like this video and share with your friends thanks loved

Channel: Tech Dummies Narendra L

Views: 128,661

Rating: 4.857801 out of 5

Keywords: Amazon interview question, interview questions, interview preparations, algo and ds interview question, software interview preparation, developer interview questions, Facebook interview question, google interview question, Technical interview question, software architecture, system design, learn System design, 2 phase commit, 3 phase commit, microservices transactions, transactions distributed, distributed systems transactions, learn distributed systems

Id: S4FnmSeRpAY

Length: 31min 9sec (1869 seconds)

Published: Tue Jun 18 2019