Publish-Subscribe Architecture (Explained by Example)

what if you guys ask me a question which is better rest or message queue I struggled to answer for a while but after thinking about it I discovered that both rest and message queue does boil down to the basic elements which are basically rest being HTTP which is request response and message queue being published subscribed in this video I want to discuss both the request response and the publish/subscribe architecture and talk about the pros and cons of each so if you're interested stay tuned you new here welcome my name is Hasan and this channel we discuss all sorts of software engineering by example so if you want to become a better software engineer consider subscribing hey that'd be like and so you get notified every time I upload on your video would that say let's just jump into this video pub/sub let's do this alright here's their agenda what you're gonna do is like we're gonna talk about a request response a little bit and then we're gonna talk about where the request response actually breaks right because nothing is perfect in life apparently right and as I start talking about that you guys are gonna see time code where you can jump to the interesting part of the video request response pros and cuz that's another thing we're gonna discuss right what's the good what's the bad about this technology publish/subscribe architecture we're gonna talk about that how it actually works what is it right we'll example and i'm finally we're gonna talk about hey observe pros and cons if you're interested stay tuned let's start with the request response model it's a very elegant and simple design that i've been designed in the 90s and the whole internet is running on this thing it's very very simple design that's why it's popular so your client your web server and the client makes a request let's say this is a get request over HTTP the client waits the client is blocked now you can argue with that whether it's blocked and cannot do anything else right which is no longer true because we asynchronous requests right you can make a request and you can do other stuff in the background that's okay right because this is just you make a request you wait and you don't actively wait you don't spend any processing power waiting right he's just make a request and forget about it right and then once the request comes back you get back a content and then you do something with this content right so that's the request response model it's always the client initiating the request there is no other way coming from the web server okay and I'm gonna reference a video here for the synchronous versus asynchronous there are lots of playlists that you can go and watch that if you're interested to know more all right and where does this break is this perfect obviously nothing is perfect let's assume you have this system where you want to upload a video let's say this is YouTube and obviously when you upload a video to YouTube you don't just upload a video there are a lot of stuff happening in the background so you wanna upload their own video you wanna YouTube want to compress the video because usually it's that raw and before file is very huge right you want to compress it and after compresses is done you want to pick that up by the format service you want a format and what does that mean you want to produce different video ties for the appropriate devices so you want to provide a video for mobile phones so that's up maybe 480 or 720 or 1080 or 4k I want to produce different kind of content right based on the viewing platform and provide that and you want to also once this is done you want to notify subscribers so how do you do that with just request responsible it's so simple but it really breaks down do you think about it so client makes a request uploads that beautiful raw mp4 video right and it waits right they can do other stuff in the background but still waiting right and then a plus server is processing that stuff and then once it's done it now it is waiting and it's making a request to the compress service to compress the uploaded video all right to that maker cause now the compressed video is processing and what is it is done it is waiting and make a request to the format service and waits right so a lot of people are waiting there's a chain of people waiting for this thing to get done format service is processing producing all this for 18 720 and 1080 and 4k and 8k okay right it's bad I don't know faster okay format service is just processing that stuff and then once it's done it will make the request to the notification service like okay I'm done let's make a request to the notification service which will notify all the subscribers obviously that hey we've done this videos uploaded go and and that and then the for my service is waiting for the notification service and you guys can't argue with this like oh go no really we can notify people once it's uploaded right no Bailey you want to notify people when it's ready to be consumed when it's here it's not ready to be consumed right so and that even get more complicated as we talk about it alright notification servers dead okay I'm done the first response format service say I'm done don't press edit I'm done so people start unblocking services start unblocking requests as they come in and finally the client would say who done upload it alright obviously guys put anything that breaks in the middle the whole thing is broken I'm sorry about that okay so the whole thing is essentially broken right once you put any obstacle and network error the whole chain is broken and you don't know if this thing is finished or not that's the problem we're facing with the requested response if you're chaining multiple service especially in a microservices architecture that breaks down let's say we I want to add another copyrighted service and we want this to the copyright service like to check the contents for content IDs and check if there is like a copyright infringement so you want to consume the compress service need to send the compressed file to both format and copyrighted oh my god right this topology gets really complicated real quick son right all right what's good what's bad about lays pros it's very elegant and simple yes if you have only two pieces of software talking to each other that's beautiful and I still love that but once you get into complicates it on you're not really right it's only got some point I love that it's stateless specially when you HTTP that's not really true for like requests most like a database like database you make a sequel that's still request response you make update table where blah equal blah that's a request and then when you get a better result decade 70 are all updated that's a response right that's still requests respond but that's that's nowhere stateless this estate was gets all right HTTP another is stateless but yes you can argue with that but but it is if it's stateless it's good because it's scalable right you can scale it horizontally and that's a very overloaded word I hate putting it there but I have to put something in the pro section scalable here means that it is scalable at the receiver end where you can duplicate the receiver if it's the same content right if it's the same has the same functionality you can duplicate it easily and you can scale it easily right because it's just making the same request you can put it behind a load balancer a beautiful idea right lord answer can just route requests to any service and just scales very nicely right but it's not really scalable in other terms right so scale bar is a very overloaded Ward and I'm really mean about horizontal scalability of the same duplicated service oh my god that's getting complicated okay what was that cons what's bad about this obviously as we said it's very bad for multiple receivers the moment you start seeing a lot of receivers a lot of consumers you start really wearing yourself down and and and yeah things can go really long moments you insert any thing in the middle right you your architecture falls apart okay and that's why you have to start hacking things around how do you hack things around right you basically introduce high coupling so people a lot of services start talking to each other which produces a lot of high coupled services so services start knowing each other and that's bad we want software to have social anxiety we do not want software to talk to each other that's bad and software high coupling is bad we want software to be as oblivious as possible about the whole system high coupling yeah we talked about that client and server have to be running you want say Hussein that's just weird of course I'm gonna send out a quest of the servers down yeah of course the client have and server has to be running to communicate are you telling me there's something better than that where the client can go offline held the server can go fly and still they can communicate oh we're gonna see about that all right yeah so that's kind of a disadvantage up sub has this advantage still have my concerns about are we gonna talk about that all right so chaining circuit breaking retries all that stuff that we introduced just to solve the problem of of how can we guarantee that these topology of systems connect together right the highly coupled system can correctly talk to each other right we need to timeout correctly we need to retry if that didn't work and that just put a lot of pressure on the service and complicate things even worse right and that is a problem by itself it's not an easy problem to solve yes service missions take care of that stuff a clients like finagle on Twitter that open source to an angle can take care of that stuff for you right if you make a request but to me still sounds like complex systems right yeah all right pops up to the rescue it's the best thing ever not really all right so let's talk about pad cell publish/subscribe model all right we're taking the same example the YouTube uploading service and we're gonna upload a video and we're gonna compress the video we're gonna format the video we're gonna notify people how about that so there's as you can see here that you can see still the services here the client here the notification services all that jazz but there is some box here this box is the middle middle layer that everybody's communicated to and this is called multiple names people call it broker people call it message queue people call it streaming processing a lot of names but it's a middleware layer where you can push and publish content to it and it will take care of delivering that content to someone else okay based on the subscription mode so these guys subscribe these guys publish everybody can publish and subscribe at the same time and you publish and you subscribe and publish to what is called a queue or sometimes called topic like Kafka called a topic rabbitmq call it Q channels is called I think it right is called channels right so whatever it is and tough car and rabbit in Q 0 and Q all that stuff orgies this message queue that supports pops up right all rights go through an example see if this thing is good or not all right so here's what I'm gonna do I'm gonna I'm making a client I'm gonna upload the service still this is to add to an extent this is still a request response right you can still mission mash what does that mean mission matter I don't think it's a word but yeah you can do a hybrid between request responds and pops up so that's okay right so you make a request and upload service stop processing waiting for all your stuff so the client is blocked now right and again guys when I say a block doesn't mean he cannot do anything or she cannot do anything why am I earning to computers as gender okay I don't know all right so you make a request and it waits right it still can't do other stuff in the background obviously but it is asynchronous not nevertheless so upload service processing once it's done the upload service would say you know what just give me a second give me a second give me a second I'm gonna I'll brush this really quick to this topic and once I get the result and which is usually this quick because these guys are in the same network interface I assuming this is another network interface and this is another network interface so they communicate with this I have popping this is in the same hopefully a land area and they will communicate what they're like in a 10 Gigabit Ethernet network or whatever and I will become fast right so you make a request and then get response real quick upload that five gig video once it's done that's it your job is done as applause service so your clone you can notify the client that it's done let's let's repeat that's right alright so upload once it's done publish it to a topic or a channel and get back a result and then shoot upload it done that's it the client now done job you can disconnect the client you can just move on with your life this video is in the system and leave obviously that means this guy has to be up all the time that's what she said but what we have here is essentially a topic and that topic or the channel has the content it's sometimes called a queue a topic a channel and that has the wrong before videos now how do people consume it these guys would have been subscribed already to a topic that already exists I just hide it I did some animation but compress service in this case is subscribing to this topic now we're gonna talk in details like how this is actually done is it what does it mean to subscribe right it's very weird abstract word right is it are you pushing the result preserves the moment you have a raw mp4 video art is the compressor is actually polling information or is it like pinging what's happening here right so you gotta tell me more right so the compressed service there are a multiple implementation I'm gonna talk about them but Porsche and long pulling and pulling just polish the shield's list I don't know why would you do that it's just same as a request response but alright so compressed service will receive now the raw before for you let's imagine that it just got it like immediately pushed to it right that has this the limitation obviously we're gonna talk about that a compressed service receive this raw mp4 video start processing it and guess what it will publish its own compressed video to another topic that says hey don't press video here is that all right it's a topic so it doesn't really know who's gonna consume it the applause service didn't know that the compress service gonna consume it and that's the decoupling that we talked about they're just decoupled that everybody from each other the format services now subscribe to the compressed video topic and it will get it immediately Alice this is a little bit okay yeah you get it whether it's for short long polling whatever you get it format service it's just like publishing content look at that 480p 1080p this is Gary Vee man for my services Gary Vee just just producing content like there is no tomorrow for my service publishes all our stuff so that it's done now the notification service let's stay on it speaker sure you want to notify people when the 4k video is ready I know this is a little bit harsh you want to notify people the more the 480p videos is ready so it's up to you you want for example to notify people when the highest quality videos available that's gonna take long time to notify people obviously but nevertheless right you got it alright and then you can easily fit the copyright service right we talked about him just slam it there and subscribe to them I don't know the compressed video and I will just immediately find any Content ID and then right so it's a beautiful design is it perfect though I don't know let's find out pros what's good about this let's find out obviously scales with multiple subscribers right multiple receivers multiple consumers is great for multiple receivers because they can you can add as many receivers and unique receivers you're talking about unique different distinct receivers right they are different from each other they have different needs they have different wants and they want to be scaled you can scale them beautifully right unlike the request/response well they have to be aware of each other on that pad alright so this thing is great for microservices all right it's amazing right because the moment you have multiple services in order to avoid the spaghetti mesh topology of everything is connected to everything you can you can just have this one place and everything's connected to this place now you have a sense almost like a center single point of failure that can be dealt with differently right we're going to talk about the pros and cons loose coupling you just decoupled things now the services are not aware of each other that's good we like that stuff right the moment that the less things are couple teacher the more they can essentially scale and be added and the the system can be modified easily right because you cannot you can smoothly change one thing without breaking the entire system because low coupling or bad because the moment you have a service that depends on 700 services any small change can break any of these 700 clients right so if you have you feel sure that your only client is the pub/sub system you're golden right works well clients not running well yeah but right so if if we go back to the example there and we select the notification service if if any of these services are offline we don't really care the moment they come back online they'll will subscribe and they will say hey we have a new message we have a new topic let's consume it right so that's okay because their messages are already stored in the queue and the topic in the channel right so that's how but that also mean there's problems let's talk about the problems cons all right there are obviously message to the video shoot all right so the message delivery issues about when we have a subscriber and we have a miss message queue or we have a publisher and we have this message queue you're publishing something how do you know that the the queue the message has been published its first thing yeah well you can say that well the same message queue will will will send me a notification I can only mean hey you're done good stuff we're good alright and that you don't care about anything else for the publisher that's good okay if you don't receive that connotes mean you might try again and that's a problem okay but how do you know that if if I actually if I published again how do I know I didn't publish that same content twice right we have ways of solving this with idempotency but still can it still complicate complication going on there right and who's taking care of this complication should it be me as a publisher or should it be the message queue I don't know right the other problem is the subscriber now the subscriber that's not that's the challenging part how do you know as a subscriber that that's up this subscriber this consumer this compressed service actually got the content first how do you know that it actually processed that content because it has to tell the service hey by the way I read the I read this message just the fact that this acknowledgment that hey I read the message people are taking phd's in this because it's so hard right it is so hard to know that a message has been consumed or not how do you know right you can send an economist but about what if the economic didn't you didn't get it all right so we've talked about the message delivery issues obviously and now let's talk about the complexity it is a very complex system because of the message delivery issue we try to find ways around it okay and to find ways around it we add complexity okay so this first very simple let's talk about the Porsche and the pull and the long polling model how do you deliver a message from the topic right how do the proctor deliver a message from the topic to the subscriber to the consumer okay how does that happen right you can you can imagine like rub it into your Redis and we made a rabbit in the queue video here I'm gonna reference it here so you get you guys check it out but it's very interesting and what you want to do essentially is first thing is you wanted to establish a two-way communication like a TCP channel right and then rabbitmq has its own protocol I think it's called it's not really its own or it's a standard say advanced message queue protocol and they're using that it's a two-way communication it's a two-way binary protocol and they talk about it Redis have its own rest protocol I don't what score stands for what but I forgot so they again it said by WIC two-way communication where it's like you communicate on both directions and then now you can start sending push messages as a result because it's a two-way of communication right and push messages push notification or push models very complicated you might my sound nice hey the moment I get a topic immediately pushed to the subscriber that's so beautiful because it's almost real-time but what happened if that client is offline whatever if the consumer is offline you okay you can say they let's hold the message on until they are online and then when they are online push it okay that seems reasonable all right but how do you know that they are offline than that that that status right it's you're almost keeping track of the subscriber in your message given that might be okay if you think about it but that adds additional complexity another complexity is the back pressure where you the publisher is so fast in producing content let's say a publisher is garyvee it's just like grinding content daily right not daily every second just Bob Lorsch and stuff all time and then the poor consumer can barely consume these messages there right is just the flood of push messages them do tiny device for example that consumes that stuff cannot handle that load it just cannot it cannot handle something that shoved down there at your throat obviously it's gonna wait right and this what do you do with these awaited messages do you just put them back and do you do you timeout do you put them do you and do you now keep track oh this client is slow so I'm gonna slow down its kind of fast I'm gonna speed up that's a very complex problem to solve by itself right that's myself that's a problem right and then another problem is okay so might say that's the rabbitmq unreleased uses that okay that has its own limitation so the other approaches to do like the polling method right where do I have a message do I have a message do I have a message as a clients you say do I have a message do I have a message do I have a message do I have a message do I must do have message well if you don't then you just kind of saturated the network with empty requests which kind of busyness right all right that's a problem okay that's it that's that's a lot of processing wasted cycles for requesting empty responses right there's nothing for you so why you keep requesting so the so the solution the parcels use long polymethyl Bush Kafka uses write which is like hey you make a request and you block yourself right we can block you essentially as if you made a request and we're taking a long time processing you but there's really nothing to process nobody's busy and because it's a beautiful of asynchronous II in this age it's fine you can wait but you can do other stuff in the meantime you the only thing I've spent is memory here that's the only thing I've spent maybe a little bit of an event main loop just checking if there's something came back or not at but that you're doing that anyway right so it's it's not much so the long poem that's what Kafka uses right and then that has it's also limitation obviously it goes like leave my not did best real time but you solve the problem of essentially the back pressure of shoving a lot of messages for clients who cannot be ready to consume messages right obviously it's a very complex system and network saturation in case of push you pushing a lot of messages you're shoving the network with a huge amount of notifications and content right that that's a lot of content going at the network sometimes unnecessarily because the client doesn't really necessary can process these messages which which might lead to failure which made which may lead to you retrying the broker can have to retry this request which is also bad right and then network saturation in the other hand is like the other way under utilization which I didn't write here is just okay I make a request but I don't see anything I make a quiz I don't see anything so summary what did we discuss in this video we talked about whether a quest response model breaks right it's a beautiful design but it has its limitation right we talked about the pros and cons of this request response we talked about the publish/subscribe pattern or architecture I don't know what you call it right publish/subscribe architecture or pattern whatever rocks your boat and we talked about the pros and cons of that thing okay all of this is Kafka RabbitMQ and radiused we made a video about Redis we made a video about RabbitMQ I am making a video about Kafka still in the process I'm gonna let you guys know if you want to know if you all really see the video about the Kafka write down in the comment section right now right Kafka so we can actually I see if there's a lot of interest in that you guys give me a lot of great suggestion I love those videos suggestion I'm making them I'm enjoying making all this content it's a long formula content I know it's a lecture this is not a three-minute video they watch right and then you move on it's a fee this is videos that you watch if you're serious to learn obviously and know like no 10 minute video can give you any value really or five minutes video you get a watch and learn about a lot of stuff okay that's the kind of content we making this channel so if you're interested consider subscribe and like this video like it share it with your friend I'm gonna see you on the next one you guys stay awesome

Channel: Hussein Nasser

Views: 49,291

Rating: 4.7942691 out of 5

Keywords: pubsub, pub sub pattern, publish subscribe architecture, kafka pubsub, rabbitmq pubsub, pubsub explained

Id: O1PgqUqZKTA

Length: 30min 2sec (1802 seconds)

Published: Fri Nov 08 2019