What is the Publisher Subscriber Model?

hi everyone we are talking about event-driven services now and as you can see we have a micro service architecture so s1 s0 s2 s3 and s4 other services we have we have a client connected to s1 it sends a request after s1 has processed this request and made some changes if required it needs to send two messages to s 0 and s 2 the important thing to notice here is that the order of these two messages don't matter s2 can get the message first or s0 can get it first similarly s3 and s4 need to get messages from s2 after it's done it's processing and the order of these two messages don't matter either if you are using a request response architecture you be sending a request to s2 and the request to s0 the smart thing to do is to send these requests asynchronously and wait for the responses once they have sent our successful response you just do the same thing with s2 the main drawback with this architecture if you're using requests and responses is that s2 might be waiting for both of these services to complete even if it is asynchronous it might wait for s3 and s4 s response to come once it gets that response it's going to send it to s1 and then send to the client imagine if s4 fails if s4 fails and s2 is going to be waiting for it after a timeout it's going to send a timeout to s1 which in turn is going to send a timeout to this client and it took a long time for this request to fail not just that s1 actually processed it correctly so this data now might be still right so when the request is sent again s1 is going to make a change in the database again and then s2 is going to get the same request it might make a change in this database there are two places where there might be multiple changes for the same request the better way to handle this is something called a publisher subscriber model with the publisher subscriber model what we can do is we can remove this kind of dependency on the request and response and make it pass a message right messages are fire-and-forget so if I send a message here to a message broker like Kafka like RabbitMQ and depend on it to correctly send the message to s too so it's probably going to persist some messages and abstract out a lot of things now the s-1 can actually send a success to the cell phone right to the client kafka the message broker is going to be waiting till s2 is back up and then send the message similarly instead of s2 sending the message to both its clients it's going to pass it to a message broker in between and this message broker is going to take up the responsibility once it's completed to send these two messages to s3 and s4 one thing I forgot here is that the message broker is going to be sending this message also so this request response is taken out of s1 there are some advantages and disadvantages with this architecture let's start with the advantages the first one is that this is going to D cup allowed of the responsibilities that you had so s1 is no longer dependent on s 2 and s 0 instead it just publishes to the message broker and then it's relieved so its responsibilities are over it sends a message to the client saying that I'm successful client says ok till both the messages are sent to s 0 and s s to this message broker is going to be persisting those messages let's say s2 is down and once it comes back up that time the message broker is going to replay those messages to s2 so that it can then send it to s3 and s4 using this message broker the best thing about this is that it makes the system easier to understand instead of having multiple points of failure it's got a single point of failure which is far more easier to deal with one good thing about that again is that if you are having a single point where you're sending messages you just need to have one interface so s1 this is a developer thing if s1 is going to be sending a message to the message broker it sends a generic message with a lot of data the message broker then sends generic messages to s 2 and s 0 which in turn consume that information and then according to their requirement take data so that's a that's a good thing for the developers there's only to worry about what is the interface interactions that they have with these two messages this also provides you some sort of transaction guarantees right so if you are sending a message to s1 and it's able to persist to the message broker that means that the message will somehow reach s3 at some point of time in the future because this message broker is not going to lose messages it has some persistence in it it's going to ensure that s 2 gets a message similarly this message broker is not going to lose its messages so s3 and s4 are insured to get the messages so if you send this request once and you get a success it means that this this process will terminate at some point of time in the future so that's a loose transaction guarantee of at least once and it's also more scalable because if you have a new service s6 which is interested in s ones messages all you need to do is register this service s6 with the message broker and then it sends the messages being produced by s 1 to s 6 also write s 1 did not need to know the the subscribers for its messages so there's a lot of things which are standard publisher subscriber advantages which we have just found over here talking about the disadvantages of this service let's get to a financial system which requires a lot of consistency let's say you have an invoice generator and you have fun transfer service and s1 is a gateway so the Gateway does nothing it just processes the message and then sends it to s2 and s1 I mean s 0 and s 2 so you get a request from the client saying that please transfer my funds from account a to account B and the amount that I want to transfer is 950 the initial amount in account a is 1,000 so the bank is going to charge 50 rupees for this transaction as a commission so after this transaction a should have 0 rupees in its account and B should have 950 in it that's expectation but let's say the Gateway gets the message forwards at s 2 and s 0 a0 says remove 50 rupees from the account because that's my commission and sends an email to the client saying that you know what we have 950 rupees in your account or rather it just sends an email saying that this is the transaction amount that you asked for so if you piripi deduction that makes 1,000 minus 50 950 right this is the current amount in the in the account and the fund transfer service is down which means the message broker is not able to send those messages to s2 nothing has happened yet it's still at 950 the client then sends another request saying that you know what transfer 800 rupees so 800 rupees means a a commission of 50 rupees so in total 8 50 rupees needs to be in your account the message comes to s1 sends it to s 2 and s 0 s0 picks up the message sees that the account currently has 950 you need 850 for this transaction to be successful and so no problem deducting 50 rupees from the total that you have right now which is 950 - 50 which is 900 is your current balance and the fund transfer got two requests of 950 and 800 when the funn transfer runs what's going to happen is that it got a request for 950 transaction that is going to fail okay that's going to fail because the current funds you have is 900 950 is beyond that the first transaction fails the second transaction of 800 succeeds leaving 900 minus 800 100 rupees in the account okay with 100 rupees in the account the initial expectation was there's going to be 0 rupees left that should have happened the second transaction should have failed instead you have 100 remaining you have 800 transferred and 100 rupees as commission to the bank so there's a lot of confusion because of having a transaction across services right all this could have been avoided had you not hired a transaction across services and there are ways that you can avoid this which we'll be discussing in the next session on consistency but this is a major drawback of this kind of architecture you can't use it for mission critical systems where either there's a success or a failure like there's no otama city in this the other drawback about this is that it doesn't guarantee anything about idempotency so you send the message from here you give it to s2 s2 makes a change in its internal state and if s2 is doing something which is not I'd important let's say withdrawing 50 rupees from it from an account s2 is going to do that and when publishing over here to this broker it might fail so it sends the failure response over here after the message is replayed it might debit 50 more rupees from here and then send the message again so you're seeing a similar kind of issue where 50 rupees are removed multiple times because this message that the message book is sending is not important it's asking you to debit 50 rupees instead it should be also sending a request ID along with it so if the same request ID comes along and you have made a database change according to that request ID you can ignore it right but this is something that the developers need to do so that is application logic and this architecture in general is not helping you do that and this point we understand how publisher subscriber models work and that's the basis for event-driven services so because of publisher subscriber models you can architect a lot of complex systems but be aware of its advantages and disadvantages it can be used in the finance service but it can definitely be used for let's say gaming services where you have a lot of analytics being done or you are going to be sending events which have occurred so you're going to be publishing events and then you have subscribers for those events one of the major organizations which uses this architecture is Twitter so for that business requirement of posting a tweet and many people actually consuming that to it the publisher subscriber model is perfect right you have events published and subscribers for that and this is the basis for event driven services I'll be taking event event services in the next video because this is pretty lengthy anyway if you have any discussion you want to do on this architecture we can have them in the comments below I'll see you next time then

Channel: Gaurav Sen

Views: 162,018

Rating: 4.8561487 out of 5

Keywords: publish subscribe, pub sub, message passing, system design, message queue, event driven systems, software interview question, system design question, request response, scalability, fault tolerance, gaurav sen, gkcs

Id: FMhbR_kQeHw

Length: 11min 24sec (684 seconds)

Published: Sun May 27 2018