2 Hours RabbitMQ Course with NodeJS, Pros & Cons, Cloud RMQ, RMQ vs Kafka, RMQ in Wireshark & MORE!

what's going on guys this is a collection of my popular rabbit mq content here on youtube so i'm going to discuss rabbitmq as a technology i'm going to show you a little bit of my deep discussions of rabbitmq as a technology my pros and cons i think what what i think is good what i think is bad about rabbitmq then we're going to dive deep into some sort of philosophical discussions of what rabbitmq can evolve into like a can can you implement like quick technology and rapid mq or or something like http 2 multiplexing and instead of having their own custom channel idea and their tcp right some a lot more discussion there i'm gonna dive a little bit deep into how do you how rapid mq works today under the hood by doing a little bit of a wire sharking uh uh that protocol the advanced message queue protocol and see how it looks like how chatty it is how how consumer uh and consumes and how publisher actually publish pop sub cues all that jazz can i talk about all that stuff so if you're interested check out this content and i'm gonna see you in the next one check out the other content of of this channel i discuss anything back in engineering that's what i love to discuss i'm gonna see on the next one enjoy what is going on guys my name is hussain and this video i want to discuss cues and when to use the cues message queues to be specific like there are other type of queues but i don't i'm not sure what are they so i'm talking about rabbitmq 0mq kafka when do you want to use this in your architecture and do you really need it right and that's the question here you always have a question that hey do i really need to implement this in my system design or not and i'm just trying to kind of assess that and help you with that if possible so how about we jump into it and if you're new here guys i discuss all sorts of backend engineering in this channel so if you're interested subscribe and like this video and share with your friends that's it let's just jump into it all right so what is a queue and when do we need to use it and uh guys if you if you already uh subscribe to my channel you would see me repeat this over and over again any technology out there any back-end technology out there it exists for a reason and it exists to solve a problem so i know that might sound cliche and makes just perfect sense right just yeah of course it exists for reason and that also means that there's no technology just exists for a fun of it or because it's cool right you need to use it if that problem exists for you you cannot just use grpc because it's hip and cool right no you should use it when you absolutely the problems that grpc addresses solves your problem right addresses your problem same thing with a queue so how about we talk about the actual problems with the queue solved back to the request response architecture when i make a request to a backend and regardless of the communication protocol that i use whether it's tcp stateful tcp raw or whether i'm using grpc again stateful or with mm whether i'm using a stateless wrist architecture that request requires some resources at the back end to be served right to be consumed and executed that request what does that mean it means that request might be less to get all the employees right or an update to do a booking system right so hey i'm on a book this seat that's that's a that's same thing right and this requires a finite amount of time of your server to actually process this and we talked about the ways you can serve your request and one way to solve this problem is asynchronous execution with a single thread like your server has one thread and that thread just keeps working the problems that it have right it's so serving request this is now it's listening to tcp connection this is now doing that that's how node.js does it right other other web servers uh does it differently multi-threading multi-processing right regardless right so apache does it multi-threading node.js does a single thread but it's asynchronous and we talked about that i'm going to reference the video here i think it's here go check it out but sometimes a single thread in a node.js or multi-processing or multi-threading in a web server could not cut it because you will quickly overwhelm that single server to execute all these requests right it really depends if that request is taking a long time to process and if it does that if that equals taking a huge amount of time and unpredictable amount of time to process then there are flood of other requests that is coming i'm not talking about queues yet guys right just normal request response there are a lot of requests coming and they are waiting and when i say waiting they the client is actually just blocked because that access to the tcp connection didn't even get a response back okay and that could be harmful for the user experience right the user will feel it so what is going on i click and nothing happened and users hate that when they click and nothing happens you show me something that happens or tell me that something is happening but don't tell me that i'm doing something and i did something and i don't see any results they hate that you're a user you probably seen that so how do we trick that a requ a normal request response architecture doesn't cut it in this case if your response time is unpredictable right because you have a lot of requests coming and you might say hey who's saying i'm going to scale horizontally and that's absolutely fine you can do that you can put a reverse proxy have it configured to be a load balancer and swizzle the request to all the other services and if you have if you start waiting if you've started seeing requests taking a long time to process right then you start spinning up more services or containers if you're in a microservices architecture and then start serving that and people do this to this day without a queue without the idea of a queue right and as i said this doesn't really scale well if your processing at the back end is is very hungry processing hungry or cpu hungry or even ram hungry because you cannot spend a lot of time just uh having this process take time so if you're predicting that responses will always take a long time probably spinning up multiple services will not help you right because the request will be the same whether it's going uh it's sending to another server which are for which is free or a service that is server doing other things as well yeah you're gonna see some mindless cool minus school is that the right word minuscule difference but still it's going to take a long time so here's where a cue is useful if you really think that request will always exponentially go large yeah maybe if your database is uh doesn't have any rows but as you grow large that request will go slower and slower slower would such exponentially not necessary exponentially just uh polynomially with your number of rows so here's where q really beneficial so what you would do in this case is what is what i'm gonna do i'm gonna employ a queue in my system a message queue and that means if i am receiving a request a server i will do a very quick operation that is constant that is a big o of one it's a very fast operation and i'm gonna respond to the user with a with some sort of an identifier right and here's that's that's how a queue works so if i send me a request i'm gonna put it in a queue that's a big off one because writing is always fast especially if you're in a lsm3 kind of a database right and most databases now especially write only just right to the end lsm right log structure multi you write it and then you respond back to the user hey i committed to you user that i have received your request and it's now processing or it's now it's in the queue it i can't promise anything else but hey i received it better than having a request that is not served right that is not just waiting so check user experience better right okay i'm willing to wait as a user yeah at least i still they received it and now really up to you as an architect you can have a client come back and ask and paul p-o-l-l this task id that we're given he's like hey how's how's this job down we're doing how's this job going how's just just job done and once that response actually complete the response will come back so hey that job is done okay you can now do whatever you want to do that's one way of solving the problem rabbit in the queue doesn't do it this way uh rabbit mq does it the push way right where it's just like a stateful connection i forgot with the the protocol that robert mcqueen is but it's a it's a it's a very elegant way of using channels it's awesome i love it and i'm going to make another video about this compared to http 2 the idea of rabbitmq using channels it's very similar to streams and i don't know who came up with this idea before regardless get back to the point if i response back if that job is dequeued right or executed that could push results back to the client immediately as they are received right so this way you eliminated the latency of waiting clients still technically didn't receive the result right because you don't receive the result but i can unblock the user experience i can show some sort of a progress bar i can i can give a better user experience and i elevated the flood of request on my server now i'm going to have a nice queue yes it's a centralized system still but it's an ice cube and people people services can listen to the skew and start pulling jobs pulling tasks and execute and write it back to the queue right very very similar to a pub sub system except the only difference between a queue and a pub sub a queue is whenever you remove an item from the queue it is gone right that service owns it it is dq'd versus the pub sub system you have a topic or very similarly right the brokers have these topics and the service can as infinitely consume the same item many services can consume the same item right but now each service have some sort of a position that remembers oh i consumed this yes i consumed this i consumed this and the service optionally can have a way to go back and forth in the queue and then the pop sub system so that's a very cool very quick way of knowing how do you actually when do you want to use a queue versus just a normal request process circuit system and load balancing and all that stuff right so very quick if your request is indeterministic you don't know how long it's going to take a queue is probably a good idea for you if your process is by nature long running a queue is good for you right just cue it and let other process pick up the work and write it back to the queue or if it's a resource hungry if you're by default your process back in processing is a resource hungry it's a bad idea to have the web server itself do the work for you the web server should do one job and one job only shouldn't process your stinking request it should just response back to web traffic it serves web it is a web server it serves web traffic and that's it don't let it process your prime numbers or do a very complex operations in the web servers stuff try to separate concerns as much as possible all right guys that's a quick video just to let you know the difference between when to use a queue when do not use a queue hope you enjoyed this video subscribe if you like this content like this video if you like it i'm gonna see you in the next one you say guys stay awesome one of you guys asked me a question which is better wrist or message queue i struggled to answer for a while but after thinking about it i discovered that both rest and message queue does boil down to the basic elements which are basically rest being http which is request response and message queue being publish subscribe in this video i want to discuss both the request response and the publish subscribe architecture and talk about the pros and cons of each so if you're interested stay tuned you know here welcome my name is hussain and this channel we discuss all sorts of software engineering by example so if you want to become a better software engineer consider subscribing hit that bell icon so you get notified every time i upload a new video what that say let's just jump into this video pub sub let's do this all right here's the agenda what you're gonna do is like we're gonna talk about request response a little bit and then we're gonna talk about where the request response actually breaks right because nothing is perfect in life apparently right and as i start talking about that you guys are going to see time codes where you can jump to the interesting part of the video request response pros and cons that's another thing we're going to discuss right what's the good what's the bad about this technology publish subscribe architecture we're going to talk about that how it actually works what is it right with example and then finally we're going to talk about pep seb pros and cones if you're interested stay tuned let's start with request response model it's a very elegant and simple design that have been designed in the 90s and the whole internet is running on this thing it's very very simple design that's why it's popular so you're a client you're a web server and the client makes a request let's say this is a get request over http the client waits the client is blocked now you can argue with that whether it's blocked and cannot do anything else right which is no longer true because we have asynchronous requests right you can make a request and you can do other stuff in the background that's okay right because it's just you make a request you wait and you don't actively wait you don't spend any processing power waiting right just break a request and forget about it right and then once the request comes back you get back a content and then you do something with this content right so that's the request response model it's always the client initiating the request there is no other way coming from the web server okay and i'm gonna reference a video here for the asynchronous versus asynchronous so there are a list of playlists that you can go and watch that if you're interested to know more all right and where does this break is this perfect obviously nothing is perfect let's assume you have this system where you want to upload a video let's say this is youtube and obviously when you upload a video to youtube you don't just upload a video there are a lot of stuff happening in the background so you want to upload the whole video you wanna youtube want to compress the video because usually it's that raw mp4 file is very huge right you want to compress it and after compress is done you want to pick that up by the format service you want to format and what does that mean you want to produce different video ties for the appropriate devices so you want to provide a video for mobile phones so that's up maybe 480 or 720 or 1080 or 4k i want to produce different kind of content right based on the viewing platform and provide that and you want to also once this is done you want to notify subscribers so how do you do that with just request response well it's still simple but it really breaks down if you think about it so client makes a request uploads that beautiful raw mp4 video right and it waits right it can do other stuff in the background but still waiting right and then upload service processing that stuff and then once it's done it now it is waiting and it's making a request to the compressed service to compress the uploaded video all right okay that maker code now the compressed video is processing and once it is done it is waiting and make a request to the format service and waits right so a lot of people are waiting there's a chain of people waiting for this thing to get done format service is processing producing all these 418 720 and 1080 and 4k and 8k there's no 8k right it's a fad all right i don't know if that's true okay format services just processing that stuff and then once it's done it will makes a request to the notification so it's like okay i'm done let's make a request to the notification service which will notify all the subscribers obviously that hey we're done this video is uploaded go and and that and then the for myself is waiting for the notification service and you guys can't argue with this like oh go no really we can notify people once it's uploaded right no really you want to notify people when it's ready to be consumed when it's here it's not ready to be consumed right so and that even get more complicated as we talk about it all right notification server said okay i'm done the first response format server says i'm done don't press it i'm done so people start unblocking services start unblocking requests as they come in and finally the client would say done upload it all right obviously guys put anything that breaks in the middle the whole thing is broken i'm sorry about that okay so the whole thing is essentially broken right once you put any obstacle and network error the whole chain is broken and you don't know if this thing is finished or not that's the problem we're facing with request response if you're training multiple services especially in a microservices architecture that breaks down let's say we i want to add another copyrighted service and we want this uh to the copyright service like to check the contents for content ids and check if there is like a copyright infringement right so you want to consume the compressed service need to send the compressed file to both format and copyrighted oh my god right this topology gets really complicated real quick son right all right what's good what's bad about this pros it's very elegant and simple yes if you have only two pieces of software talking to each other that's beautiful and i still love that but once you get into complicated scenarios not really right it's elegant and simple i love that it's stateless especially when you use http that's not really true for like uh request response like a database like database you make a sequel that's still a request response you make a update table where blah blah blah that's a request and then when you get a better result like hey the 70 year old updated that's a response right that's still a request response but that's that's nowhere stateless this is the stateflies gets right http another is stateless but yes you can argue with that but but it is if it's stateless it's good because it's scalable right you can scale it horizontally and that's a very overloaded word i hate putting it there but i have to put something in the pros section scalable here means that it is scalable at that receiver end where you can duplicate the receiver if it's the same content right if it's the same it has the same functionality you can duplicate it easily and you can scale it easily right because it's just making the same request right you can put it behind load balance and beautiful idea right load balancer can just route request to any service and just scales very nicely right but not really scalable in other terms right so scalable here is a very overloaded word and i'm really mean about horizontal scalability of the same duplicated service oh my god that's getting complicated okay what was that cons what's bad about this obviously as we said it's very bad for multiple receivers the moment you start seeing a lot of raseivas a lot of consumers you start really wearing yourself down and and and yeah things can go really long at the moment you insert any thing in the middle right you your architecture falls apart okay that's why you have to start hacking things around how do you hack things around right you basically introduce high coupling so people a lot of services start talking to each other which produces a lot of high coupled services so services start knowing each other and that's bad we want software to have social anxiety we do not want software to talk to each other that's bad in software high coupling is bad we want software to be as oblivious as possible about the whole system hi coupling yeah we talked about that client and server have to be running you won't say hussain that's just weird of course i'm gonna send a request of the servers down yeah of course the client have and server has to be running to communicate are you telling me there's something better than that where the client can go offline and or the server can go fly and still they can communicate oh we're gonna see about that all right yeah so that's kind of a disadvantage pop sub has this advantage still have my concerns about that but we're going to talk about that all right so chaining circuit breaking retries all that stuff that we introduced just to solve the problem of of how can we guarantee that these topology of systems connect together right the highly coupled system can correctly talk to each other right we need to time out correctly we need to retry if that didn't work and that just put a lot of pressure on the service and complicate things even worse right and that is a problem by itself it's not an easy problem to solve yes service meshes take care of that stuff clients like finagle on twitter that open source to finagle can take care of that stuff for you right if you make a request but to me still sounds like we have a very complex system so i don't know about that right yeah all right pops up to the rescue it's the best thing ever not really all right so let's talk about pabst up publish subscribe model all right we'll take the same example the youtube uploading service and we're gonna upload a video and we're gonna compress the video we're gonna format the video we're gonna notify people how about that so there is as you can see here that you can see still the services here the client here the notification services all that jazz but there's some box here this box is the middle middle layer that everybody's communicating to and this is called multiple names people call it broker people call it message queue people call it streaming processing a lot of names but it's a middleware layer where you can push and publish content to it and it will take care of delivering that content to someone else okay based on the subscription model so these guys subscribe these guys publish everybody can publish and subscribe at the same time and you publish and you subscribe and publish to what is called a queue or sometimes called topic like kafka call it topic uh rabbit mq call it q channels it's called i think red is called channels right so by the way this and kafka and rabbit mq 0mq all of that stuff are just this message queue that supports pops up right all right let's go through an example and see if this thing is good or not right so here's the thing here's what we're gonna do i'm gonna i'm making a client i'm gonna upload the service still this is to add to an extent this is still a request response right you can still mission mash what does that mean mission match i don't think it's a word but yeah you can do a hybrid between request response and pops up so that's okay right so you make a request and upload server is still processing waiting for all your stuff so the client is blocked now right and again guys when i say block doesn't mean he cannot do anything or she cannot do anything why am i referring to computers as gender okay i don't know all right so you make a request and it waits right it still can't do other stuff in the background obviously but it is asynchronous nevertheless so upload service processing once it's done the upload service will say you know what just give me a second give me a second give me a second i'm gonna publish this really quick to this topic and once i get the result and which is usually this quick because these guys are in the same network interface i'm assuming this is another network interface and this is another network interface so they communicate with this i am hoping this is in the same hopefully a lan area and they will communicate whether like in a 10 gigabit uh ethernet network or whatever and that will become fast right so we make a request and then we'll get a response real quick upload that five gig video once it's done that's it your job is done as upload service so your client you can notify the client that it's done let's repeat that right all right so upload once it's done publish it to a topic or a channel and get back a result and then sure upload it done that's it the client now done its job you can disconnect the client you can just move on with your life the video is in the system and leave right obviously that means this guy has to be up all the time that's what she said but what we have here is essentially a topic and that topic or the channel has the content it's sometimes called a cue a topic a channel and that has the raw mp4 videos now how do people consume it these guys would have been subscribed already to a topic that already exists i just hide it i did some animation but compressed service in this case is subscribing to this topic now we're gonna talk in details like how this is actually done is it what does it mean to subscribe right it's very weird abstract word right is it are you pushing the result to compress service the moment you have a raw mp4 video or is the compressed service actually pulling information or is is it like pinging what's happening here right so you got to tell me more right so the compressed service there are multiple implementations we're going to talk about them but posh and long polling and then polling just polling is just useless i don't know why would you do that it's just same as a request response but all right so the compressed service will receive now the raw mp4 value let's imagine that it just got it like immediately pushed to it right that has this limitation obviously but we're going to talk about that the compressed service receive this raw mp4 video start processing it and guess what it will publish its own compressed video to another topic that says hey compass video here is it all right it's a topic so it doesn't really know who's going to consume it the upload service didn't know that the compressed server is going to consume it and that's the decoupling that we talked about just decouple that everybody from each other the format services now subscribe to the compressed video topic and it will get it immediately unless this is a little bit okay yeah you get it whether it's push or long polling whatever you get it format service it's just like publishing content look at that 480p 1080p this is garyvee man format services garyvee just just producing content like there is no tomorrow format service publishes all our stuff so that it's done now the notification service let's say you're notification that you want to identify people when the 4k video is ready i know this is a little bit harsh you want to notify people the moment the 480p videos is ready so it's up to you you want for example to notify people when the highest quality videos available that's gonna take long time to notify people obviously but nevertheless right you got it all right and then you can easily fit the copyright service right we talked about them just slam it there and subscribe to the i don't know the compressed video and i will just immediately find any content id and then there right so it's a beautiful design is it perfect though i don't know let's find out pros what's good about this let's find out obviously scales with multiple subscribers right multiple receivers multiple consumers is great for multiple receivers because they can you can add as many receivers and unique receivers here we're talking about unique different distinct receivers right they are different from each other they have different needs they have different wants and they want to be scaled you can scale them beautifully right unlike very cost response where they have to be aware of each other and that's bad all right so this thing is great for micro services all right it's amazing right because the moment you have multiple services in order to avoid the spaghetti uh mesh topology of everything is connected to everything you can you can just have this one place and everything is connected to this place now you have a center almost like a center single point of failure but that can be dealt with differently right we're going to talk about the pros and cons loose coupling you just decoupled things now the services are not aware of each other that's good we like that stuff right the moment the the the less things are coupled to each other the more they can essentially scale and be added and the the system can be modified easily right because you cannot you can smartly to change one thing without breaking the entire system because low coupling are bad because the moment you have a service that depends on 700 services any small change can break any of these 700 clients right so if you have if you're sure that your only client is the pub sub system you're golden right works while clients not running well yeah but right so if if we go back to the example there and we like the notification service if if any of these services are offline we don't really care the moment they come back online they will subscribe and they will say hey we have a new message we have a new topic let's consume it right so that's okay because the messages are already stored in the queue and the topic in the channel right so that's how but that also mean there's problems let's talk about the problems cones all right there are obviously message delivery issues all right so the message delivery issues about when we have a subscriber and we have a message queue or we have a publisher and we have this message queue and you're publishing something how do you know that the the queue the message has been published it's first thing okay well you can say that well just saying message queue will will will send me a notification i can only mean hey you're done good stuff we're good all right and that you don't care about anything else for the publisher that's good okay if you don't receive that acknowledgement you might try again and that's a problem okay like how do you know that if if i actually if i published again how do i know i didn't publish that same content twice right we have ways of solving this with item potency but still can it still complicate complication going on there right and who's taking care of this complication should it be me as a publisher or should it be the message queue i don't know right the other problem is the subscriber now the subscriber that's that's that's the challenging part how do you know as a subscriber that that's this subscriber this consumer this compressed service actually got the content first how do you know that it actually processed the content because it has to tell the service hey by the way i read that i read this message just the fact that this acknowledgement that hey i read the message people are taking phds in this because it's so hard right it is so hard to know that a message has been consumed or not how do you know right you can send an economy but what if the economy didn't you didn't get it all right so we talked about the message delivery issues obviously and now let's talk about the complexity it is very complex system because of the message delivery issue we try to find ways around it okay and to find ways around it we add complexity okay so it's very simple here let's talk about the posh and the pull and the long pulling model how do you deliver a message from the topic right how do the broker deliver a message from the topic to the subscriber or to the consumer okay how does that happen right you can you can imagine like rabbit mq already and we made a rabbit mq video here i'm gonna reference it here so you can you guys check it out but it's very interesting and what you want to do essentially is first thing is you wanted to establish a two-way communication like a tcp channel right and then rapid mq has its own protocol i think it's called it's not really its own but it's a standard advanced message queue protocol and they're using that it's a two-way communication it's a two-way binary protocol and they talk about it redis have its own rest protocol i don't know what's stands for what but i forgot so they again it's a byway two-way communication where it's like you communicate on both directions and then now you can start sending push messages as a result because it's a two-way communication right and push messages push notification or push models very complicated you might it might sound nice hey the moment i get a topic immediately pushed to the subscriber that sounds beautiful because it's almost real time but what happened if the client is offline whatever if the consumer's offline you okay you can say they let's hold the message until they are online and then when they are online push it okay that seems reasonable all right but how do you know that they are offline and then that that that status right you're almost keeping track of the subscriber in your message queue and that might be okay if you think about it but that adds additional complexity another complexity is the back pressure where you the publisher is so fast in producing content let's say a publisher is garyvee it's just like grinding content daily right not daily every second just publishing stuff all the time and then the poor consumer can barely consume these messages they're right this is just the flood of push messages them little tiny device for example that consumes that stuff cannot handle that load it just cannot it cannot handle something that shoves down their throat obviously it's gonna wait right and this what do you do with these awaited messages do you just put them back and do you do you time out do you put them do you and do you now keep track oh this client is is slow so i'm gonna slow down this kind of fast i'm gonna speed up that's a very complex problem to solve by itself right that's itself that's a problem right and then another problem is okay so you might say that's the rabbitmq and redis uses that okay that has its own limitation so the other approach is to do like the polling method right where do i have a message do i have a message do i have a message as a client you say do i have a message do i have a message do i have a message do i have a message do i have a message do i have a message well if you don't then you just kind of saturate the network with empty requests which kind of business right right that's a problem okay that's it and that's that's a lot of processing wasted cycles for requesting empty responses right there's nothing for you so why you keep requesting so the so the solution the part solution is use long polling with which kafka uses right which is like hey you make a request and you block yourself right we're going to block you essentially as if you made a request and we're taking a long time processing it is really nothing to process nobody is busy and because it's a beautiful of asynchronousy in this age it's fine you can wait but you can do other stuff in the meantime you the only thing that i spent is memory here that's the only thing i spent maybe a little bit of an event main loop just checking if there's something came back or not but that you're doing that anyway right so it's it's not much so the long pulling that's what kafka uses right and then that has it's also limitation obviously because like you might not get best real time but you solve the problem of essentially the back pressure of shoving a lot of messages for clients who cannot be ready to consume messages right obviously it's a very complex system and network saturation in case of push you're pushing a lot of messages you're shoving the network with the huge amount of notifications and content right that that's a lot of content going at the network sometimes unnecessarily because the client doesn't really necessarily can process these messages which which might lead to failure which may which may lead to you retrying the broker can have to retry these requests which is also bad right and then network saturation on the other hand is like the other way under utilization which i didn't write here it's just like hey i make a request but i don't see anything i make a request i don't see anything so summary what did we discuss in this video we talked about where the request response model breaks right it's a beautiful design but it has its limitation right we talked about the pros and cons of this request pose we talked about the publish subscribe pattern or architecture i don't know what you call it right publish subscribe architecture or pattern whatever rocks your boat and we talked about the pros and cons of that thing okay the example of this is kafka rabbit mq and radius we made a video about redis we made a video about rabbit mcu i am making a video about kafka still in the process i'm gonna let you guys know if you wanna know if you wanna really see the video about the kafka write down in the comment section right now write kafka so we can actually i see if there's a lot of interest in that you guys give me a lot of great suggestion i love those videos suggestion i'm making them i'm enjoying making all these content it's a long form content i know it's a lecture this is not a three minute video they will watch right and then you move on it's a this is videos that you watch for your series to learn obviously and no like no 10 minute video can give you any value really or five minutes video you're gonna watch and learn about a lot of stuff okay that's the kind of content we make in this channel so if you're interested consider subscribing like this video like it and share it with your friend i'm gonna see you on the next one you guys stay awesome rabbitmq is an open source distributed message queue written in airlang and supports many communication protocols it was trying to solve a problem with what we call the spaghetti mesh architecture you know guys remember where every client is trying to talk to another client in the system remember guys the days of the enterprise message buzz and all these god awful days you know so probably q was introduced to solve this problem right by introducing this intermediate layer in this video we will explain the main basic components of rabbitmq right and you got to start seeing jump codes or time codes gear guys where you want to jump to the interesting topics and and you don't have to watch the whole videos just whatever you're interested in you can jump to that right so we're going to talk about the rabbit in queue components i said such as the advanced message queue protocol amqp what is a channel what is a queue what is a publisher what is a consumer what is agreement exchange what is all that stuff right and then we're going to talk about the meat and potatoes of all this stuff where we will actually spin up our own rabbit mq server with docker okay then obviously that's not enough we're gonna write some code we're gonna write our own publisher that uses node.js okay that publishes to the rabbitmq server okay and then we're gonna write our own consumer piece with javascript node.js that will consume that queue that will consume these messages okay finally guys i want to talk about my thoughts my personal thoughts about this technology finally obviously we're going to summarize the whole thing all right guys if you're new here welcome my name is hussein and in this channel we discuss all sorts of software engineering by example so if you want to become a better software engineer consider subscribing hit that bell icon so you get notified every time i upload a new video would i say let's just jump into this video guys all right so rabbit mq so here's the architecture of rabbitmq rabbitmq uses this middle layer and solves as we talked about so the problem where clients want to talk another to other clients in the same system or external systems and instead of having each client having knowledge of other clients we kind of grouped everything into this layer okay we call this rabbitmq server rabbitmq server listens to 57672 port by default because it's a server it has to listen right because it's using tcp okay so that's the first abstraction the rabbit mq server there's a lot of pieces inside of that i'm not going to go through that yet but i want to explain that first piece which is the server this could be multiple servers could be distributed replicated and all that's jazz right but we have this first piece robert mcq server the second piece third piece is the publisher where hey i am a client and i want to publish a message for certain consumers that are interested in this message and we're going to talk about an actual example guys where we write the code but essentially what the publisher does is establishes a stateful tcp connection between itself and the rabbit mcq server and this is very critical to understand it's a two-way communication okay so the underlining transport protocol is tcp okay it's not http okay it's using tcp raw tcp and there's a nice protocol on top of it okay there are multiple protocols but we're interested in is is the advanced message queue protocol there's like certain headers certain messages right think of it's just like an application layer http where it uses tcp right it has its own headers and body and git and methods and posts right the advanced message queue protocol has its own format as well it has its own protocol okay so it uses that okay and then it uses that and and publisher can send messages to the server the server kind of sends messages to the publisher so it's a two-way communication very critical to understand okay so there is a connection so that's the second abstraction a connection third approach exactly so a publisher a connection to the server there's a consumer who want to consume messages so they will connect again using a stateful two-way bi-directional tcp connection to the server using the advanced message protocol there are other protocols but i'm not gonna talk about this in this video right so let's focus on just one protocol here so the am amqp protocols man that's hard to pronounce all right so the consumer establishes a two-way communication with the server and says hey what's up server sup give me some messages right so and the server will start pushing messages emphasis on the pushing guys very strong word here the pushing the server pushes messages to the consumer okay very interesting okay so the server pushes messages to the consumer when they are ready okay when they have them messages and there is the publisher sent messages to the server another abstraction for rabbitmq is what they call the channel okay and the channel is a logical connection in your connections just like a mini connection think of it right and the reason they did that they want to separate the consumer connection from multiple consumers inside that consumer an example would be you would create this consumer have three channels using the same tcp connection and the reason they're using that is this instead of if you're trying to like instead of having three consumers having three tcp connections let's have one consumers or three channels with one tcp connection this is called multiplexing where you bring a lot of stuff into one pipe and that's a good thing i like this about it i like it a lot right you can use the same tcp connection but you can send certain bits with the channel id say hey this is by the way this belongs to this channel this belongs to this channel this belongs to this channel so they can segregate and do certain things based on that so that's an a nice abstraction right a channel okay same thing with the publisher so you can send through a certain channel okay if you don't have channels then the publisher let's say you write a publisher that does more than one thing okay you have you will be forced to use multiple publishers instead of kind of putting all your code into one publisher or one consumer right another abstraction is is what the q stands for is the queue right all of this stuff goes into the queues okay so the queue is like you send an information goes to the queue and then other the consumer will per basically pull information from the queue right and that that's that's where the queue is right but however that's very important the publisher and consumer are not aware of cues okay technically they are aware of what they call exchanges and that's the seventh abstraction on this rubbish mq there is an exchange where the whole thing is is an exchange there is a default exchange and you send all the stuff to an exchange and that will take care of propagating into a queue right you don't have to know about this we're not going to talk about exchanges as well we're going to use the default exchange with takes at all i'll care about a lot of stuff right an exchange can be used to use different algorithms to to kind of fan out and do a round drop and into uh different cues but that's that's out of the scope of the video as well because we have exchanges we have queues but let's let's just think about the concept of a key where you put a messages publisher publish a message goes to the queue consumer consumes a message from the queue and the exchange is that default exchange thing right you can have multiple exchanges as well all right guys how about we spin up our own rabbit mq and we play with this thing okay all right guys so to spin up our own rabbit mq server i'm not gonna tell you to download and install that thing right you don't have to pollute your machine with with your with these installers so what you have to do is just install docker make sure you can say docker and you get some sort of a feedback here and you make sure to you can do docker run hello dash word if this gives you back hello docker that means you have docker installed once you have docker installed we're gonna spin up our own rabbitmq message container if you will and that message container message server that container will have a rabbitmq server from which we will start writing code okay and we're going to talk about what we're going to write but let's first spin up that that thing how do we do it very simple okay so first thing we want to do is write docker run okay and by always make it a habit to give your container a name let's give it a name rabbit mq okay and then once we give it a name we will expose the port of that container to my machine so i can communicate with it right so that's essentially what's the default part of rabbitmq you have to memorize that 5672 right and what is that's what i want to expose to my machine this could be anything right the second part five six seven two is what is inside your container right so that has to stay that has to stay five six seven two this could be anything you want okay and then let's let's just use this default thing okay and then finally we do rabbit mq and that's the image from which we're gonna pull the rabbit mq docker image from which we're gonna spin a container let's go ahead and spin that and just like that we're gonna start downloading all that stuff right and then yeah just like that this terminal is now occupied all right so this you can start seeing the messages that goes into this docker uh container which is the rabbit mq okay now we have a container that is running and half rabbit mq what we're going to build with this technology here's what we're going to build we will build an asynchronous job execution okay engine and how this works is essentially we'll have a publisher that will publish jobs and say hey consumers i want you to execute this job and you need to execute this job and then there was would be adequate consumer which is essentially a process that will take that job and start executing it right this could be like think of it like you you're publishing tasks and these tasks are executed by actual processes right and this could be heavy processes like i don't know calculating the prime number of certain number or finding or going to the database and doing a batch job or doing a hadoop job right a map reduce right so all kind of job you're going to publish it to the queue i'm going to publish it to the rabbitmq message and then the consumers will start pulling the don't stop pulling they will get that message that get that job and start executing it all right so now we know what we're gonna build let's go ahead and build it all right guys so what are you gonna need here is you're gonna install visual studio code or any favorite editor you have i'm gonna use node.js as my execution engine here to build my publisher and consumer and here's what we're going to build right so let's go ahead and build a publisher okay my publisher and consumer will live in the same project that's okay because we're going to do have a different kind of input to things right so let's go ahead and build this project so you go go ahead and go file open and then we'll go to javascript here playground let's get rid of project called rabbit mq just a blank folder we're going to go ahead and create a publisher.js file and here's the thing guys let's go ahead and initialize npm so we create a project in pm init dash y right that means shut up i know what i'm doing just give me a package.json we're going to use that later and now here's what i want to do i'm going to create a const and then what we want to do is we talked about this a little bit and when we explained that right and that we talked about rabbitmq uses the advanced message queue protocol amqp and this protocol has a lot of clients and node.js has that client so i'm going to create an advanced message queue protocol instance and then we're going to acquire that library which is called a advanced message queue protocol library okay that's what it's called and that's a promise based library which is amazing because we love permafsside stuff it's using something a blue bird to promote from permusify a cut i can't pronounce anything okay promisify this things okay so i have that so that what is the first abstraction we do right i'm gonna create a connection to my server do you remember where my server is running do you remember the port guys i hope you do yeah because now we're going to use that but first of all since we're going to use a promise based thing let's create a function obviously it's going to be an async function i'm going to call it connect right you can call it publish anything you want and we're going to do a try catch just to catch any bad things right and then console.error ex so just just build a skeleton here and here's the first thing we're gonna do we're gonna do a connection create a connection how do we do that amqp dot connect that's not hard what do you expect you want a string from me what's the url here's the url guys since we're using the advanced message queue protocol that's the protocol we're using and since my rabbit mq is used running on my mac it's gonna be localhost or the machine so what is the port guys it's five six seven two i can memorize that thing okay and here's the thing guys since this is a returns a promise right it's a blue bird that means it's a promise you gotta await that thing await that thing and here's my tcp connection remember there's another abstraction with rabbitmq which is called the channel which is just one channel that goes in your connection so you can create i don't know how what's the maximum number of channels per connection but you you have to create a channel because that that's the channel of communication so that's not enough so let's create a channel how do you create a channel again you await a connection.create channel that sounds not hard at all okay that creates also that's a promise based so that means it will be awaited there's a rejected promise we go go here you can use then dot then up to you guys but here's the thing now we have a channel what do we do right the idea here is we're gonna publish to a queue okay rabbitmq documentation says the client doesn't know off a queue i kind of disagree because i write a cue job i literally specify the queue name in my publisher that means so don't tell me that the client is not aware of that queue it just doesn't make any sense to me right so that coin is aware of a cue right but internally you kind of publish to an exchange but i don't know it's just a whole just mumbo jumbo abstraction to me right i don't i don't believe in any of that so a channel can do something called assert queue and that assert queue will make sure that your queue exists on the server and if it doesn't it's going to create it for you so what do we do we're going to create a queue that's called jobs right and let's just use use the default there's like whether you want durable that means the persisted disk or now let's just use the default here okay and then constant result equal await since this is also going through the tcp you get away because it's going to take some time await everything and to know basically if you see bluebird as a result that means that it's a promise based okay we got a result now we got a queue what do we do we asserted that we have a queue if you don't do this you might get an error the second statement so here's the thing what do we do now i want to send something send to a queue what do we send which queue which queue or sending here i'm going to send to the jobs queue what do you want to sense on i'm going to send an array of buffer essentially which is just like a buffer and the buffer is uh you have to send a buffer and to send a buffer you can do buffer from and then pass a string so let's just send a json object here and then let's just do a message what's our message here what's the message what are we going to do const message equal let's say i'm going to send a number here okay and this is the json object with the number 2 right or just 19. and then i'm going to send that json object so you cannot send just the json object as as it is you have to stringify it so you're going to do a string json.stringify message and that string will become a buffer so you obviously need to send a buffer you can also if you want just to send text say hi you can do that right but we want to be fancier guys we want to be fancy okay we send the job what else guys is that is that all i think that's it so we can if you want to you can close the connection here okay and just be happy but let's just just say console.log let's say job sent successfully all right just and what's the job that's the job what's the job the job is the number i don't know you're sending a number and that the process will take that number and process that input right think of it as an input you're sending a message to the queue and then one of the your processes will pick that and start processing that okay whether it's gonna do whatever right okay how about we run this thing guys before we run we have to call this function we never called it so let's go ahead and run it and hopefully that works let's do uh i might it might fail because i didn't install yep cannot find module amqblib well how do we install a module very simple sir we do npm install amqp lib just for you to see it guys we have to install the library that we use it doesn't come default by no with node.js okay now we have it let's run i'm gonna by the way guys give you the code so don't have to worry about pausing the video and doing all that stuff right today there get a channel after we get the channel we got the channel successfully what is the result what is that so it gives you back the queue essentially so we can print that queue if you want okay and then that sends to the job and then we say job send successfully and then obviously my debugger still running because we didn't kill the connection right so it might be a good idea just to kill the connection afterwards okay so we submitted something to the cube how about we consume it guys so let's create another another one right another one consumer.js who is a consumer it's the same code almost the same code so here's what i'm going to do same mq so it's just it's just steel code guys because it's almost the same but i don't want to do all this mumbo jumbo right i don't want to send anything to the queue i want to do something else so just delete that all right okay so that's i don't think i need a message because i'm consuming messages here so we connect to the server we did the channel because you have to do a channel by the way this channel is different than the channel of the publisher they are completely different channels right they are completely different tcp connections to begin with right so what i want to do here is we have to keep the consumer alive unlike the publisher publisher you can actually kill the connection if you want after you're done but you might want to keep the client the consumer happy and dandy so what do you want to do is the console log listen uh waiting for messages something like that right we're just waiting for messages but how how do i do how do i get messages well you do this channel dot consume and guess what what do you consume guys you consume a queue and what's the queue name it's called jobs and what do you want to do when you consume a job i want you sir to call this function when a message comes in okay and here's what i want to do i just want to do console.log all right and then just print their message okay and then i'll go pay attention to what will happen here i'm gonna run this thing now i'm gonna run the consumer and let's see what will happen this is a debug i put a debug here so after if the after the connection if the server which is rabbitmq sent to me as a consumer a message i am going to get triggered here and here's what we're going to do what we're gonna do i just wanna print what we got oh we went to the publisher for some reason i don't want that all right so let's run my consumer ooh nice look at that so we got a message what does this message have it has a content that's the buffer we got guys remember we sent a buffer we get a buffer back how do we get from the buffer we get a string back which is what we're interested in we're going to talk about that so it's it's unreadable so let's make it into but we didn't we didn't get the message which is pretty cool-ish guys all right so here's what i want to do if i just started again what i thought a cue when you just read the message the message is dequeued right but every time i restart the consumer i always get the same message again let's print it so we can we can see content the two string just print it let's see what we get as a as a consumer and look at that we got the number 19 okay it's a little bit boring so let's make it into actual json right input equal json dot parse right how this is how you convert the string back to jason maybe we get the basin and then we do console.log received job with input input dot number right that's all that's the function and then let's do the fancy text here template lateral literals right guys let's do that consume again and we got it we get it then we get receive job with input 19. now you can take that number query the database do your thing and here's the interesting part guys the reason we keep receiving the same message over and over and over and over and over again right is because we did not tell the server that we have received it as a consumer that's what's called acknowledgement and i didn't talk about that but we're gonna we're gonna talk about a little bit now okay so why do we do what do we do all right so let me first clean up some some of stuff guys right i want to do is there's two certain things i want to create an a script here an npm so that if i do npm run publish right i will run node publisher dot js okay that's what i want to do and if someone says consume i'm gonna do go to node consumer is that what it called the js okay this way if i do if i go to a terminal how do i clear this thing right if i do a camera i do npm run publish i'm going to publish the message okay and let's kill it if i do npm run consume i'm gonna consume that message and you can see i received two jobs with the same id because i just published another one another thing i want to fix here let's kill this job in the publish i want to get i want to publish a number that the user sends me how do i do that right as i wouldn't do npm run publish and then i put a number and that number will be published this way we are always hard-coding the number 19. we don't like that right so here's what i want to do to do that kind of hacky but why not so do process dot arg v and i believe it's the third element is of the input will be the input right because the i think zero is the node js.xe or whatever in your mac what's it called i forgot the second thing is the path to the javascript the third thing is the input right so if i do that that will be the input so let's try it out so now if i do npm run let's zoom in here so you can see npr run publish eight job send successful e8 right that's pretty cool if i do npm run consume yay we got 19 19 and eight okay so we keep keep getting these jobs here's the thing now okay so we now have that let's get out of this and show you that we can actually run it out of the terminal guys it doesn't have to be in the visual studio code so let's go ahead and do something fancier here i'm going to create a new terminal right and let's go ahead and create another terminal here all right so i have two terminals here we're going to do here go to the same folder which is i think called javascript javascript playground and then it's called rabbitmq right and then what i'm going to do is nvm run publish 7 and just publish 7 successfully and i'm going to do here is go to the same folder javascript playground wrap it in queue and then npm con run consume and then we can see that we got all of them now okay so that's just another way of consuming stuff here's the thing guys the last thing before we end this video with the exercises i want to tell the server server please dq this job because i have processed it how do we do that here's what we do go back to the code the consumer job is to dequeue or to acknowledge that they receive that job in order for the server to pull it from the queue okay and there's a lot of stuff here this is a complicated topic kafkas have their own implementation of how like guarantees of delivery like uh at least once or at most once or exactly one right so there is like a lot of guarantees without which i'm not gonna go through this because it's just gonna get longer and longer but that would be another topic to discuss but the rabbit mq guarantees i think at least once delivery or at most once it cannot guarantee that exactly you received it once right it's just a little bit tricky stuff that people take uh phd in right this is really complicated because even above my head really it's just really complicated stuff all right so here's what i think if i received a message i want to acknowledge that message certain thing let's say if the input dot number is equal equal to seven just like that i just want to process number seven for some reason right i don't want to process all the cues go ahead and do channel dot ack and you can acknowledge everything you can ignore just a certain message so i'm gonna acknowledge this particular message which is called massage right when a message is received if that message is satisfied certain criteria let's say you went to the database and you process and you get a successful message you processed it you run your hadoop job you got all all of the stuff you run your uh everything and you got back the results you want the way you want it then you can tell the server it's safe to remove it from your queue and that's what acknowledging me okay let's go ahead and acknowledge this thing only okay seven if i go back here and i say npm run consume you can see that i received job with all of them again but this one was acknowledged but it doesn't show here so now if i kill that consumer and i run another consumer you can see that it's still there this is a problem with it's it's it's my problem now i'm going to show tell you exactly why it's just a javascript and something with javascript right let's go ahead and show all right the prob the the reason it was not acknowledged is because i think this is this is a very javascripty thing right where i told it hey if the number is exactly equal seven and the type is also matching right but i bet that this number is actually a string right it's not actually a seven so if i do just double equal this will work so let's go ahead and do that all right so what what happens here is equal equal the reference quickly really quick triple equal that means the numbers the the values do match without implicit conversion right that means if this is a number the second side has to be a number if you say just equal equal that means this is true right if you do this is yeah let's just let right now this is true but if you do this this is actually false right that's the difference okay so now that we actually coursed it let's go ahead back to the terminal and clear npm run consume and you can see that we have acknowledged it hopefully this time run it again and then it's no longer there all right guys long video but hopefully you uh have you got some benefit from it so let's jump back to the slides continue this give you my take into this technology then we can summarize this video let's go back all right guys so we finished the consumer we built the consumer visa we built an asynchronous job system with rabbitmq we have a publisher we have a consumer we send a job and we execute and we have all this fancy stuff and we talked about all the beautiful stuff in rabbitmq and here's my thoughts about this technology it has too many abstractions and here's the thing my old boss told me says if you building a system with a lot of abstractions and you want to introduce a new abstraction you gotta remove an abstraction you have to remove something you cannot have this many abstraction what do i mean by abstractions well let me explain to you right when i started researching this produ this this technology seriously last week i was so confused because there are so many terms for this technology it's a message queue what the heck are you guys doing what the heck is a publisher consumer channel exchange what else there was like so many things and channel and then connection and then two duplex and then push and pull so many stuff and then they are supporting three or four protocols so there's an advanced message queue protocol and there is the stump protocol there's the http protocol there's too many abstractions you know what you get when you have too many obstructions you get a complex system and complex systems do not scale with a human we are not designed to work with complex systems we are not designed and this is my opinion take it with a grain of salt guys you might be smarter than i am but if you're introducing this much complexity right i i even i don't remember all the abstractions this is just few of the abstractions i'm pretty sure there are more and if you're introducing all these abstractions your system is going to get complex and if this is good complex adoptions will slow down okay and that's why i believe rabbitmq is getting less and less popular with these years and that that's one of the reasons in my opinion it's just too complex does it do the job of course it does but if you figure it out right there's like and i didn't even go deep into this like you know what i'm i'm just gonna scratch the surface i'm not gonna go deep because it's so confusing guys right so to me it's a very complex system right another thing i want to talk about here is the posh model the the the decision that rabbitmq team did says as a consumer if you connect to a rabbitmq server which is called an exchange which is as talented and ultimately goes to a queue through a channel oh my god look at all this stuff right that's what i'm talking about look at how complex that to talk about things is so much look at the http how simple it is request response system over tcp that's it right that's a whole internet is running on this thing guys difference between this to see how complex this thing is and i think the rabbit mq like community made the same problem with with the enterprise servers bus is a little bit better enterprise service audio asb was even worse it was so complex and it's just an equalizer between whether you want to have a completely open system extendable extensible versus a simple system it's that is kind of closed and i don't mean closed source here closed in in the limited functionality that it gives right reddit redis did this very well i think redis zaid hey it's an in-memory database yeah we can give durability but i'm going to give you functionality very simple publish subscribe and that's it right it's a very limited set of functionality and memcache d i didn't research memcached but it has the same concept which is a very simple model simplicity rules guys so let's talk about push model so when a consumer consumes a certain queue rabbitmq pushes messages as you as you saw guys i didn't ask for it i didn't actually hey say hey do i have a message do i have a message do i have a message i didn't do that i just said hey by the way if you got a message push it to this function and the function got called and triggered and then we got the results right the problem with this is it does not scale how do you guarantee that the consumer will receive that message right and how do you not overwhelm a consumer because guys remember i am a just poor consumer if you have like thousand messages and you're pushing this this down my throat there's this what is called a backfill problem i think where where as you start pushing more messages i cannot process them fast enough for you so now you're kind of putting more work on the server right to to start pushing start slowing down the pushing process because now your thing that can see you're telling the server says hey server please slow down that's the option that the rabbit mq have i think it's called the maximum something right so it's like hey by the way i can only process five messages at a in a second right so don't send me more than five messages now you're pushing this complexity to the server that's why your server became more complicated and that's an a kind of an unneeded complexity on the server side because the server now will start pushing more and more messages to you and let's start oh i have oh this consumer is slow this consumer is is fast so i'm gonna pushing more message to this consumer versus this one it's very complicated kafka i think i agree with what kafka did with this and they said you know what we're not going to do a push model we're going to do a long polling model pull it back push this complexity on the consumer and let the consumer consume whenever they want right whenever they are ready to consume they run into another problem which is the acknowledgement problem which way which they solved in a very smart way i didn't research kafka in detail but when i do and leave a comment in the comment section below if you want to see the kafka message right a kafka video and um i'm really interested to make that video but i think ra kafka did a better job at this by shoving that responsibility to the consumer to the client instead of having the server push push push messages right so that's what we're doing with with the notification system as well like why youtube have problem with notification because of the push system right there is always pros and cons with this redis picked the push model but i would understand what the red is it has limited clients consumers because it's all a back end technology unlike this this is very front endy the message key because consumers will immediately consume the red uh the rabbit mq message queue versus redis it's always a caching technology so it is always going to be in the back in deep deep deep back end so the number of clients will be limited compared to the number of clients to rabbitmq i might be wrong i might be mistaken but that's my take on it hopefully you guys uh give me your second what do you think right all right guys summary what did we discuss in this video we just got the components right i'm not even i don't even remember the components there are a lot of components guys exchange and and queues and publisher and consumer and channels and connections there is other stuff that i didn't discuss by the way guys by that we just got therapy in queue we discussed how to spin a rabbit mq server with docker i think this is if you want to try anything you use docker all the time don't install garbage on your machine guys don't install things and just take memory whenever you install something piece of technology do docker start rabbit mq when you're done docker stop rub it mq so you just even if you're studying machine the containers will not stop by default so you kind of save on memory right and if you're done with it docker rm rapid mq okay we wrote a publisher right we also wrote a consumer and we did essentially we we built an asynchronous job system where a publisher publishes a job and the consumer just starts and pick a job and start working on it and then do this thing and then produces the output okay and i i finally gave you my thoughts about this technology guys hope you enjoyed this video i'm gonna see you in the next one you guys stay awesome what is going on guys uh my name is hussein and then today's video i'm gonna spin up a cloud version of rabbitmq using the cloud advanced message q protocol so the advanced message queue protocol uh protocol applies for many messaging queues right and one software that uses this protocol as this one protocol is rabbit mq so in this uh video i'm going to spin up one rabbitmq instance on this service and then i'm it's it's very simple and i'm gonna uh create a publisher and create a new queue on this uh instance and then start consuming the same instance from um from a consumer that i'm gonna build i'm gonna use node.js to build these applications and this is just to tell you that you don't have to basically spin up these uh rabbitmq instances on locally on your machine so you can just use one of these services and it's really nice and easy and how about we jump into it guys all right so i'm gonna go ahead and log in to go to cloudamqp.com and hit enter log in and this is my email feel free to email me uh anything any questions i really appreciate all your stuff guys and we go ahead and create a new instance and what we're going to do is going to call it i don't know test and you can choose any uh any plan you want free is more than enough right i'm just going to go ahead and select region east really where whatever you wanted this instance to be and the closest thing is the west i'm going to pick it like i don't know north carol california sure this is the closest to me because i'm in california i'll go ahead and review and just like that zero dollar a month by the way this this is not a sponsored or anything like that it's just i just enjoy their services and i i thought i'd just give them some love and trying their service so go ahead and create instance and just like that we have a new server ruby in queue manager you can go to the rabbitmq manager take a look at that obviously i don't have any queues i don't have any exchanges i don't have any channels nothing right so all i'm going to do is go to edit and let's take take a look at the services that's the host that's the host bubba and this is the amqp url so i'm going to go ahead and copy this puppy and let's go to the code guys how about that go to the code this is the code guys i'm going to share with you i have done a rabbitmq crash course go check that video off you want details about what rabbit mq is what is all that stuff what is a channel what is the connection talking about all that stuff where we actually built this application from scratch but in this video i'm not going to show you how uh how i'm going to build it from sky i'm going to consume this stuff all right so publisher the publisher will publish will connect to the rabbitmq server which is currently set to my localhost i'm going to change that and i'm literally going to put this huge url that i got from the server right and can i connect to the server i'm going to create a channel because this is like the stream level just like http 2 create a channel and then we can do stuff in this channel we can have many many channels by default i think that servers give you like around 200 channels by default that's pretty much uh for a free version is enough right we're gonna create a new queue called jobs and we're going to send content to the queue based on whatever the parameter that we send to this application right in this case i'm going to send a number and then once december we're going to make it into a json and then send it back to the server and then the moment we send it we close the connection close the channel and close that connection at the other end i'm going to consume that new content right on the consumer side right how about we connect and test this thing node publisher.js and i'm going to send the number 107 boom just like that we have connected we have sent the job we have created the job cube because it didn't exist right this is what essertz does and then we send the queue and then we close we send that job an entry to the queue and then we close the chat and we close the connection let's look at the at the dashboard how do i go to the dashboard still figuring this thing out probably in key manager all right look at that we got a brand new queue it's called jobs how about channels there's no channels weird because yeah because we just closed the channel right and uh yeah that's the jobs there's some content in it probably right this is the queue message there's one ready hasn't been there's no connections and yeah there's all this stuff there's one queue seven exchanges i didn't create any exchanges these are the default exchanges i guess right i never use these exchanges in rabbidm here so i'm not really familiar with it but yeah let's go ahead and play with the consumer side of this start consuming this we'll take this thing consumer and uh yeah we can just uh do the same thing here huh let's connect that's and what do we do we're going to consume same exact thing right i'm going to connect create a channel assert that the queue exists right we don't really need the results i'm not going to do anything with the results and then we start consuming and the moment we consume anything we receive the job with the number and we print that number and that's just this is just wasn't some example with the acts and stuff so i want to acknowledge that stuff but i'm not going to acknowledge anything i'm just going to consume and print let's go ahead and do that node consumer.js i'm not going to pass anything else when i do that so look at that receive job with input 107 and the connection is still there i did not close the consumer for some reason right i i want to show you that if i go back to the dashboard and go to channels look at that we have a channel that is opened it tells you this is my public ip address it's dynamic so don't worry about it the moment i restart my router is going to give you another uh stuff and just generate an android username that's the that's my local port apparently cues that's the cues a high availability queue and all that stuff so yeah and if i kill my q my consumer and i did this i'm going to receive the same thing you know why guys right because we have not told rabbitmq to actually dequeue that job however while i'm working on 107 that job 107 if someone else connected to the queue at the same time what they will do is let's do it how about that actually just know it consumer.js it's great two connections look at that this this puppy had did not get the queue did god did not get 107 because to robert mq right the other puppy is using the 107 right but the consumer didn't did not acknowledge it right so for all we know if i kill this connection i go back let's see what will happen right look at that because the first consumer died i go to the second consumer immediately got the job because they say hey looks like this consumer didn't acknowledge me right that's why acknowledgements are very very tricky to get right right so that's why uh get that thing right guys and i talked about this stuff in rabbitmeq all right guys so a very short video to talk about ramroom q and how to spin up a rabbitmq in a cloud version that's i think that's the first video i make on a cloud and start using the cloud more because it's just very convenient all right guys that's it for me today uh hope you enjoyed this video give it a like if you like it share it with your friends i'm gonna see you on the next one you guys stay awesome what is going on guys my name is hussain in this video i want to discuss rabbit mq and http 2. uh rapid mq i made a video about ramen q check it out guys here it's a very interesting message queue that i have my doubt about but i a little bit changing my my my mind about it because it's a really really great tech and it had a feature that i fall in love the moment i read about it and it's called channels and let's talk about that a little bit and then compare it to http 2 streams that also we're going to talk about it right so when you have um a rabbit mq server or a broker right here right and we have a client which is usually called a consumer the client establishes a tcp connection between itself and the server and we know how tcp works guys check it out there is in the client library for rabbitmq they add some abstraction called the channel and the reason they did that is like every request you want to send to the server you have to specify the channel id and they have headers for every request they send tagged with the channel id and you might say why do you want to do this this is so awesome technology the reason is instead of having multiple clients to establish multiple tcp connections to the server you now have one tcp connection and all of these requests are funneled in the team's tcp connection and now you can send a request and expect a response back even if that request is already you're sending another request in the same tcp channel because they are very uniquely tagged with the channel id so now you can simultaneously send requests on the same tcp connection and that's exactly how http 2 works guys the idea of streams browsers before http 2 what did they do remember every request you send you can only send one request at a time and you better send one request at a time because you cannot we don't know the response that comes back which request was it for because it's just garbage data coming out and end right so they made their decision way back to so oh it's syndrome request and wait for the response to come back and that request must be for that that response must be for that request and so on that's that's called essentially head offline blocking so it's like so that the rest of the requests are blocked until that response is actually the first response received and that's a problem right and that same problem exists with tcp and any request response system so what rabbit mcq did is they added this abstraction of the idea of channels in that tcp connection so every request is almost uniquely identified in the channel itself that is very similar to the idea of http 2 streams which we talked about it right here go check it out so http 2 streams serves a very similar idea you have one tcp connection instead of six like what the browsers used to do in http1 and then if you make a git request they are tagged with a stream id if you make a post request simultaneously they are tagged with a stream id if you make another get request to fetch an image if you make another request to fetch in css or or an html file or javascript file they are tagged uniquely with a stream id very similar to the channel id and they are shoved into the tcp connection right and now when you come back the the response that comes back is tagged with a stream id so you know oh so you know that oh this this response is actually for this stream all right so you actually can assemble things in the client so it doesn't matter what the order comes back if this comes back or this comes back first and it doesn't really matter so even if there are some delays right so why i made this video was like i'm not sure who actually invented this idea of of multi-concurrent tagging of request so rabbitmq or http2 or speedy google was invented this idea right so i was just like i'm just were very curious about this so both obviously come back to that final my final point is both http 2 and rabbitmq suffer from a very severe problem which is the tcp it's a problem with the tcp itself right and the problem with tcp is like if you sing a packet and the packet is damaged or haven't received by the server and there's no acknowledgement and that's a packet right it's just a packet the client always retransmits that packet back right and that causes any other requests packets that you are about to send to have wait for an acknowledgement to be received from the server this is at the packet level guys this is not at the stream level so if you're that means if stream one technically didn't receive an acknowledgement stream two cannot even be fired to the server which is very bad right that's a problem with tcp that's a very large and and that problem carries with rabbit in the queue it's a very similar problem and that's why guys quick was invented to say so you know what we cannot rely on on the idea of free transmission to put it at the at that lower level of our architecture you cannot implement that at the lower level so what they said is that i'm not gonna we're not gonna use tcp for re-transmission we're gonna use the bare bone udp which doesn't have any of this feature it just sends a request and relies on the higher level of application two actually does that re-transmission and verification of the packets and that's what quake did quick implemented re-implemented tcp by the higher level and they built in stream in it so now you can use one logical connection there are no connections in udp so they create a kind of a similar logical connection and now you can send as many requests as you want and they are tagged with a stream id and if one stream had a bug in it i don't know the packet didn't receive correctly that the quick client will only retransmit that it will not stop the rest of the streams right all the all the channels if you want to call them right the rest of the stuff will just go normally right because they have nothing to do with that first one because if the stream 2 is good or stream 3 is good and stream one is bad why do you have them to suffer right like what http does today right htb2 if stream one is bad or has a i don't know it was transferred in the ocean and the fiber was caught right the stream one will will suffer and stream two as stream three or stream five all the streams will suffer so you will feel the slowness with http two that's the problem with this right so quick solves this problem very elegantly at that level and now http 3 is built as a very thin layer on top of quick that uses all these particles that brings me back to the farm my final point is like i think most message queues right and databases and protocols should start using cues to be honest right because that solves a huge problem with tcp tcp had this problem with re-transmission and the packets right and now concurrent sending concurrent requests at the same time and that just magnificently quick solves this problem for you right today i just wanted to talk about rabbit mq and the idea of channels and then uh http 2 and then how both of these are having this problem now solves with quick and quick have their own problem i'm not gonna discuss on this channel but definitely i'm gonna make a video dedicated for quick but i was very interested to know that the similarities between rabbit mq as its own from scratch protocol building of the channels very similar to the http 2 protocol all right guys i'm going to see you in the next one leave a like if you like this video subscribe and what do you guys think what is the future protocol do you think quick will hold to its glory and it's going to become really well known pearl that is used everywhere because i'm i have a good feeling in the next 10 years quick was going to become a big thing there's one problem that i'm going to i'm not going to mention here is essentially the udp and how the internet treats udp but that's in for another video like this video if you like it i'm going to see in the next one you guys stay awesome what's going on guys my name is hussain and welcome to another episode of wireshark them all and today we're gonna wire shark rabbit mq so we made a video about rabbit mq guys check out that course discussing this tech discussing this beautiful message queue that uses the advanced message queue protocol and as we did with the previous episode with wireshark we're gonna establish a connection between a client which is this node.js application that connects to an advanced message queue protocol server which is a rapid mq server in this instance that i spun up in a cloud-based architecture and i'm going to make another video showing you how today i did that it was literally two clicks i love this stuff and the reason i needed to do that because i cannot run a rabbitmq locally on my mac and run wireshark the same machine because i won't capture those packets right and uh yeah so how about we jump into a guy so here's what i have i have a rabbitmq server running on the cloud and uh i am this piece of application literally connects creates a channel and i described in that video what a what is the difference between a connection and a channel right it's like in http 2 you have the connection and streams very very similar right and then uh i'm not sure who came up with this idea first so yeah similar like even ssh same thing right their idea of connection and channels so we can allow multiplexing essentially all right so the channel and then what we do here is we create a channel and then immediately in this channel we create a queue and then we send a job to the queue and this is basically whatever we pass in in the in the in the buffer uh i'm gonna say hi for example and that's it and just we print something and then immediately we close the channel we close the connection and i'm gonna capture the whole thing right how about we jump into it so create a brand new terminal do node publisher.js that's the file and the source code will be available guys for you i'm going to say hi boom so he's going to it just prints it and immediately quits so let's go to wireshark and see what we captured here's our shot guys and i did a filter just so i can only filter between my server my ip address client and the server which is uh that's the who wherever this thing is i just pinged the host and then got the ip address and literally filtered it so let's go let's go through all this garbage let's see if we can explain all that stuff guys how about we do that first three things guys we know what it is is right this is the client is start with 10 and this server is 72 just for simplicity we know this what these three things are right sen senec ack that's the three-way handshake to establish a tcp connection that's just to prove that uh rabbitmq uses tcp and that's not bad that's just something we need to be aware of so that's the three-way handshake to establish the tcp connection right then we know that uh sen senec enac and here's the first contents that the client sends protocol header says hey since the robert mq uses the advanced message queue protocol the first thing it says is hey i'm using the advanced message pro q protocol right and this is my minimum version this is my maximum version version 1.9 whatever right 0.9 actually right that's the version of the amqp 091 right and the server acknowledges that packet and then once we acknowledge that packet says okay i got your request and now i think we're good let's establish a logical connection and that's look who's who's initiating this it's very interesting guys right that was i believe a result of the connection when we created the connection right the first thing but the server responds back okay i'm ready to establish the connection for you right so now it establishes it's just to us at that level it's just a bunch of data well to the to the tcp stack but we're sending connection.start and look at how beautiful this thing is wireshark actually shows us this stuff it actually decor doesn't decrypt it's just it's already plain text i'm not using any encryption here so it knows says hey this is the version this is the capabilities and that's all the things i support and here let's start the connection and the client acknowledges that request that packet and then this the client now sends its version of connection start okay i'm happy with everything you did server here's the things i support uh here's my configuration and let's go ahead and establish a connection right so technically if you think about it we have a tcp connection but this is the logical advanced message queue protocol without rabbit mqs is to establish a physiological connection on top of the physical tcp connection the server says okay let's tune things a little bit right and you might say why do we need to tune if we can we do the tuning in this that's kind of a optimization that the advanced message queue can do i guess but for some reason we're doing it in another step but sure so i'm doing a tuning and here's the thing channel maximum right the server says i only support 200 channels and that's a good idea for uh for a cloud application because they don't want to make it in like a thousand channels otherwise they they're gonna deplete a free i'm using the free tier right and this is the maximum frame size every each frame that you send that's the maximum i can support and here's the heartbeats like we're gonna ping each other every 120 seconds to see if we are alive or not right so that's the thing so the third the client acknowledges that packet that connection tune and then the server the client sends back connection tune okay and here's the thing the client agreed on the 200 the client agreed on the heartbeat but he didn't agree on the frame max it says okay dude your frame max is so big son i'm gonna only send you for 496. that's the only thing i um support also awesome so that services acknowledges that and now we're actually opening the connection see we didn't even start this thing yet so we're opening that connection and this thing is like look at this this is now the client actually opening that connection that's was just starting the connection now we're actually opening the connection for to send to send data we acknowledge the server acknowledges that connection open and then the server sends back a connection open okay so if we get any errors we will see these errors and um yeah it even sends back known hosts that's if it's null i guess that's the only host that supports in case if you want to do like a load balancing or stuff like that uh client acknowledge that now channel open let's look at the code we're right here guys now all of that stuff was here right now we're here okay line 13. look at that stuff now the client opens a channel and and the channel what is a channel this is is like a strip in a connection that we allow uh basically multiplexing right so we can have multiple threads in the same process mult ascending in parallel multiple packets on the same tcp connection right so this is a great idea and people have been doing it for years re-transmission something happened here apparently the client restaurant retransmitted some packet that have been lost or because either we didn't get an acknowledgement or something bad happened right so let's send back and the server replies with share channel channel open let's open let's go ahead and open that connection and we we don't have a channel id for some reason right and uh yeah looks like i have not specified a channel id i have an option if i want to to i believe to specify a channel id huh maybe maybe not i remember there was like an idea to specify channel id but looks like uh there isn't never mind acknowledgement from the client that we receive the channel open and here's the thing guys we're declaring a queue now the client is about to click create a new queue that's the queue right this is it we're asserting the queue so we can create a new queue on the server and since i created this before that queue already exists so a bunch of free transmission we're going to ignore this stuff right because that's what happened when you connect things to the internet dupe acts stuff like that and then we come back and then he's like queue declare okay that's the from the server says okay i got you this queue already exists and uh go ahead and this is we have nine messages in the queue currently because i i used this cue before to do stuff in it and the client acknowledges the queue and here's the thing we're gonna publish now we are aware we're sending content we're sending the hi which is this is two letters right two letters two characters right let's see if this is actually there and the payload where's the high there we go i actually send the jason it says number i why is it number high let's see what did i do i forgot what i did a code oh look at that i'm sending a json of message object what is this oh there you go look at this i'm sending i forgot i wrote this code a long time ago so it's it's a json object with the number i'm supposed to be numbers but for some reason i said hi sure right so we're sending adjacent number and high doesn't matter guys you get the idea all right so we're sending that and now we're acknowledging the server acknowledges that we got the publish and now here's the cool thing guys the client closes the channel right right here we're closing the channel but the server didn't respond with anything and that's a very very interesting thing guys right if because when you publish something and that's one powerful feature in advanced message queue protocol when we publish the only thing we need to know that our content has been published to the queue is this puppy the acknowledgement and that's it that's a lower level thing that we don't have control as developers as programs back and engineers we don't have any control over this thing right so it's just the moment we get acknowledgement we know it's it's it's in the server and i don't really care about this reply because we're publishing a queue it's going to be at the other end of the consumer to consume that stuff it's not our job anymore we're closing the connection the server acknowledges the close that the server closes the channel from its side right and then acknowledge the client acknowledges the close and then what is it so the client now physically let's say logically closes the connection right because you can open and close as many channels as you want while keeping the connection alive right you can have many channels in the stem tcp connection and no i shouldn't say tcp action you can have many channels in this logical connection that is actually just one physical tcp connection does that make sense guys and then yeah we closes this puppy right the sir the client says okay i'm done even with this uh connection just like let's go ahead and close the connection right and the server says okay here's the interesting thing this is the delayed beautiful acknowledgement part of this thing what happened here is when we closed it if the server can it start instead of acknowledging the close data packet a wait since i'm about to close it anyway let's wait let's not acknowledge that stuff and let's send you the data with the acknowledgement so if we look at the acknowledgement we say okay we're acknowledging this the same content right here right so if you look at this it should be the next number is 5569 and if we look at technologies is also 569 so yeah we're acknowledging and sending the data that's the best thing that we can do if possible and that's where a good well-written server shines really right and that's the back-end is written very well we're acknowledging and sending that data at the same time if we can do that low level thing we'll see great performance despite some dupes and stuff like that happen in the internet we cannot control that obviously but what happened here where the client acknowledges the close okay which is this response from the server and here's the thing who's initiating the close hmm the client is initiating the close awesome so if the client is generating the close right and uh look at that some error happened it got reset right so so far from test to multiple tests to rabbit mq i never seen a clean close of the tcp connection and i'm not sure why and we can dig deeper in into this thing and i i i'm probably going to need some help from the network engineers because i don't know why these resets happen but whenever you see reset that means something wacky happened in the internet and and uh the packets went out of order and then uh we couldn't close the connection in a clean manner so we reset so to the client and the server this is an error we close the connection in error despite me just closing in neatly to be honest right i just immediately opened and closed it right and if i look at the code i also awaited i made the mistake of the i i ha i didn't have these puppies right i didn't have a weight i'm i'm mistakenly didn't that and when you do don't do that since this is asynchronous code we're gonna do that and since this is asynchronous we're gonna send the code and immediately go to the next thing and close the connection we don't wait for a result from that right and then when when you do that bad things happen man right because you start closing the connection and and before actually receiving content from the server it's going to be uh awful all right so that's uh that's essentially robert mcq wireframing i hope you enjoyed this video guys give it a like share it with your friends i'm going to see you on the next one you guys say awesome goodbye what should our wire shark next let me know in the comment section below goodbye what is going on guys my name is hussein and this is a very interesting article from doordash uh they're basically uh at a food delivering service here in the us and uh they have a very interesting piece of article here they're moving from rabbit mq to apache kafka back in technologies to back in technology that we discussed in this channel guys so check out the rabbitmaq video right here also you can check out the apache kafka video right here if you want to learn deep dive around an hour video detailing both of those tech so guys that let's be clear about this guys that does not mean rabid mq is bad right let's be clear about that and even after i read the article i still think that is very specific use case to do door dash that just didn't work with rabbit mq right so the outages being caused with rabbit mq i think it has to do with something that i'm going to discuss in this article right i'm going to reference the article here you guys go ahead and read it i'm going to summarize some of the stuff here so i'm going to summarize the article what what the doordash engineering team did to overcome these problems and then that why did they choose kafka they did a great job guys detailing the problem detailing the the scalability and availability problem they have detailing even the solution they they even came up with a proposed different solutions and pros and cons i like that the only uh if i'm gonna critique this article is that i'd like to see a little bit more details technical details that is yeah even that yeah i i would like to see i had a lot of questions that i just couldn't find answers for in the article itself right and i'm going to get into these in this video so guys they rabbit uh no rabbit door dash have a back in task processing specifically an asynchronous task processing uh system eh exactly that's it an asynchronous task processing so we talked about that a little bit guys we built one actually we built one very very simple one with rabbit in qfr if i if i remember correctly and and what that what does that mean right what is what is an asynchronous task processing really mean right so that means synchronous versus asynchronous we talked about these two guys what is the difference between these two techniques check out this video if you want to learn more about but synchronous is a i'm gonna make a request and i'm gonna sit down and wait right for the result and this waiting doesn't have to be blocking necessarily right because i could wait in an event loop right like i can have an event loop that okay i'm waiting for a result a callback to give me the result but i'm doing my my other stuff as well that's that's how java's gonna node.js works right even that that's the synchronicity or asynchronous means i'm gonna make a request and i'm gonna give back a response immediately from the service from the server from the backend but that service is queued that job that task is actually not executed immediately but it is essentially queued to get executed at a later time right that's asynchronous processing here so let's be very clear about that asynchronous in the front and it's a little bit different from asynchronous and the back end right we're talking about the back end here it's all back end we're back in here right guys so if i if i if i make an asynchronous job that's it means i'm gonna execute a request and i'm gonna immediately immediately get a response back saying here's your job id come back and check with me later right so that's that's in a nutshell doesn't mean this is exactly what's happening here but that's what an asynchronous system is and then once the job actually finishes we write the result back into some sort of uh place it could be the same message broker could be it could be a desk it could be another service who cares right and that that basically picks up the result and you can do stuff with it right so that's that's what asynchronous task process and you they have tons of this stuff they have a lot how much did they say there there was a number 900 different asynchronous tasks ordering merchant order check out you can imagine right all of this stuff you want to get a response back to the server to the user as much as fast as possible right let's execute a request get the result that okay i have submitted your order i am processing it go do something else right that's how the back end works yeah the back end is executing it right versus synchronicity where you the back end is actually serving you and executing you and you're waiting for it so here's a technology that i never heard about before celery guys and you might you might laugh now because i i have never heard of celery before so celery is exactly does the job that i talked about here it is it's an it's an asynchronous and synchronous task processing software it has nothing to do with brokers has nothing to do with messaging it is it is it is is a software written in python that you submit tasks for it it spins up the thread executes the task for you and then gets you back the result so that it runs you can either run it in a single node or multiple node and that's it right celery and that's that's one critique that i'm gonna say that maybe it's well known that's why they didn't mention this in the article i wish they talked about more about this their existing architecture i felt that they didn't clarify that to be honest right they jumped into the problem but they didn't draw uh diagrams of their existing architecture so uh that if i'm gonna critique it that will be one critique in this article right i'd love to see their existing architecture instead of the problems right make sense so they have salary as this asynchronous job executing but they pull the jobs and and and all these messages from a broker because salary can be actually uh connected uh actually let's let's pull it where is salary let's go to wiki yeah so salary can be optionally connected to a message broker and the recommender is robertson q and orredis right because writers have this pop sub system right rabbit mq is also there they are not supporting kafka as far as we learned from this article beautiful article guys beautiful good job saba and ashwan good job guys so so this is the current architecture salary as this task execution system or written in python and then rabbit mq as the back end that that basically submits acts like a queue basically the message broker that submits to the exchange and then writes to the queue and then a salary agent a salary worker will pick up a new job will pick up a new task and it's from rabbitmq and start processing this stuff so they're using celery as the vehicle right for all for the longest time i didn't think and i still believe we don't really need that extra layer right i would have built this myself if i would write and i i built a task manager before right in my my carrier so i would i would basically use a queue i still looking for a simple queue that does only a q right i couldn't find one yet rabbit mq's i had my little bit of of critique let's say for rabbit mq if you if you know this channel i talked about a little rabbit mcu i like this architecture but i think they went too far right they started with being a cube but then moved to a pop sub system right kafka started as as a streaming pop sub system and and and that's that's that's the good thing right rabbit mq is starting to do so much they're trying to expand when you try to expand you don't have focus anymore right and that your software becomes a little bit complex and it suffers i feel mongodb is hitted the same route in my opinion i know i know some of you like mongodb and nosql i'm have nothing against those right but i just think if you're starting with nosql and you are nosql and you're schema-less then trying to expand to become relational is a little bit weird right you're trying to solve different problems now you're trying just to steal to eat from the market share from the relational right so that's why mongodb is i know i'm getting off the road but i'm i'm just talking about the rabbit mqs the same size says the same problem that i have with these systems right mongodb now started supporting transaction full asset transactions why right just because people were complaining that oh i would not move to because it's not relational right if it's not transactional so they built in transactions on this and they have some other problems like limiting the transactions and stuff because of that right i think they they're heading into the wrong direction if you ask me mongodb but still it's a great database the engineers behind are so smart solving extremely hard problems they're solving transaction across chart which is which is nuts right this is nuts it's a very difficult problem to solve but my point with rabbit and q is the same right that rabbit and q is from the name it's a message queue but it does so much other stuff well there's these exchanges and stuff right awkward but regardless it doesn't mean arbitrary is bad i just my opinion and you can have your own opinion so celery and rabbit mq are two systems and what happened if you include too too many systems guys right complexity right this is not a system written by you it's an open source yeah but you have dependency on this you have dependency this and this is what happened this is what causes the problems they start to see a spike and you can read this article spike on rabbit mq robot mq will just go down because it will hit a limitation in rabbit mq and it's called the high surge turn churn excessive connection turn they will hit this excessive connection churn guys let's read this this is what they hit the door dash team that what what what happened is like they will get a lot of orders a lot of these execution and salary jobs will just skyrocket and it will start shoving lots of jobs to rabbit mq and execute not just publishing consuming and publishing both right and rabbiting q hit this high connection churn let's read this a little bit let me zoom so we can read clearly a system is safe to have high connection churn when it's rate of newly opened connection is consistently high and and its rate of close connection is also consistently high what do we know in the back end guys why did we build http 1 1 because http 100 had the same problem http 10 we're trying to to be so precious saving our precious memory so we said every request opens a connection and and and sends the request and then closes the connection we regretted this decision immediately because it was a bad idea opening and closing connection is bad it looks like from what i thought and it's not clear salary is the cause of this thing celery is every time you want to all connect to the rabbit in the queue they open a new tcp connection which we talked about and then they send the request and then they close it resulting in a churn of closing and opening closing and opening you shouldn't be opening and closing connections in my opinion keep them alive why do we have hdb11 hdb2 keep those dang connections alive keep them running and that's the problem of using existing software and not building your own and i'm not saying don't use existing software of course but and again i don't know if this salary is the problem i never used it but it seems like it from the description here from what i'm reading and guys if you if anyone from doordash here and i said anything that is dumb or incorrect correct me guys i would love to learn more details i'm starving for details here i'm starving i'm starving i need to learn more is salary can we i need to know what is the architecture between celery and rabbit mq here and why are we so chatty when it comes to that because rabbit and mcq connections are not cheap they are very similar to http 2. very similar and we know how expensive http 2 when it comes to cpu usage right we know that because all these streams which ruby mq have it's called channels right all these channels are in the same tcp connection so robin mcq team i loved that in rabid mq i love their design and i don't know who came up with this idea first rabbitmq as the advanced message queue protocol or http 2 from speedy google i don't know who came up with this stream or channel i prefer the word channel over stream i hate the word stream and hdb2 i don't like it one bit i think it doesn't have anything to do with stream it's it's a channel i love the word channels more clear if you tell me his channel it's called channel in ssh it's called channel rabbitmq speedy decided to call his stream okay wherever all right so i think that that's that's what caused basically the problem and you have a huge churn in in rabbit mq causes what it causes downtime it causes basically you cannot accept more because th the the the the single master we can't say master anymore uh the leader node in rabid mq is busy doing closing and opening connection which is extremely expensive and i and i bet it's even more expensive in case of ramen with you because they have the the logical connection idea of channel so i bet that's even made the problem even worse so yeah availability scalability so they they they clarify these problems in details they talk about all that stuff they're using python which is which is not a bad idea i think it's just just any language will run into the same problem if you ask me and uh here's another thing guys here's another thing i forgot to mention so celery talking about that southern burst of traffic would leave rabbitmq in a downgraded state where task consumption was significantly lower than exception in our experience this could only be resolved with a rabbit in queue bounds wow that's just unacceptable rubber mq has a concept of flow control where it will reduce the speed of connections which are published too quickly so that that cues can keep up floor control guys it's a it's a very similar concept in tcp tcp have flow control quick that's quick they built i believe they built the flow control constant in quick yes so rabbit rabbitmq also implement an application level flow control you know which makes sense you don't want to shove thousands and thousands of packets all at once right you need to taste the network see if it just like tcp slow start which we talked about right here this is very similar i believe right which i i never knew that robin q had a flow control kudos i love it so yeah so that's why it is it is a how what does what's the english word for it double-edged sword yes it is a double edged sword so could it could hurt and it could its beneficial network latencies yeah but they will kill the system otherwise so they talk about all this stuff here consumers the salary consumers to kafka no kaka robert mq we don't talk about kafka here so all this stuff they they propose so many other solutions uh and then they they say okay let's switch up since celery supports redis let's switch up radius instead of rabbit in the queue i don't know if that will help maybe it will because what does radius use yeah rydis uses the resp protocol right there their own custom one i have no idea if they're using channels or not though so maybe if seller is doing the same thing closing an opening connection with every single request and consumption and i think you're going to have the same problem right that's the what they call the harakiri that's the closing and reopening harakiri connection churn does not significantly degrade redis performance okay all right so they actually tested this thing i like it so they said that the closing and connection churn is not a problem in redis probably because it's not it's not the chat it's not it's not the advanced message q protocol in rabbit mq is different than redis right it's a completely different protocol so then maybe it's a little bit more lenient but to me i would solve the root problem why we do we have this churn of connections you open a connection open-end connection is so expensive open-end connection is so so expensive especially if it's tcp what other connections do we have i mean quick we don't have that yet but yeah opening tcp is just so expensive guys three-way handshake and then they didn't mention if it's tls or not but yeah if you have tls on top of that if you have ts 1.2 sheesh i bet it's secured they didn't mention anything about tls did they nope see this is what i crave i crave informacion guys guys i crave information i need more information there is not enough details here so yeah like is the architecture between salary and rabbitmq is this secure doesn't mention because if it's secure this will compound the problem even worse and if it's ts 1.2 it's even even worse than that because we have like what two round trips plus the handshake and then you close it that's why we need to understand how backhand works guys i i i i keep saying this in this channel i keep saying it fundamentals guys fundamentals not tools do not use tools and obviously i'm talking to to to to my community here i know not your dash engineering per se but yeah is so important forget about this all these tools once you understand how stuff basically work you you you are unstoppable you just understand everything you just need to understand the basic fundamentals you build this stuff on top you you know what you're using so they switch it to kafka and i believe first of all they built a a smart python interface on top of their celery uh workers so that they they kind of canary deploy the thing between rabbit mq and kafka right wait a second i think they they removed celery altogether did they yes because you can you cannot connect celery directly to kafka as far as i know so they built their own custom system from scratch right that's the mvp they built it from scratch it's a custom solution right is that what i a pretty custom custom i read the word custom here custom solution there you go they build their own custom switch again guys if correct me if i said anything wrong with that right so even that custom solution they still had problem with kafka guys right so before we go to the problems of kafka they started just canary deployed between kafka and robin say this was shifted all the load from kafk robert mc2 kafka i'd like to see how did the system behave without kafka and the custom solution celery out of the way how did that behave did they try that because i have a strong feeling that celery is the problem here i don't know why again guys i don't i don't know i don't know i might i'm i don't want to be that guy but it feels like celery is the problem here custom solution very thin layer job processing and just use rabbitmq as a broker i still don't understand why do we need the salary as a software when was it built i'm just interested when was salary built right it's like was it before the idea of broker was it was designed to solve certain problems i think i don't know i know instagram uses salary i read this somewhere here but i still don't know again guys i never used salary before so i don't know let me know if any of you guys use celery that would be interesting to know right if you guys have similar problem but yeah just because uh doordash moved to kafka does not mean that kafka is better than rabbitmq i mean kafka's is way feature feature-wise is better than rabbitmq that's for for sure just because of the design decision we talked about that i'm not gonna dive deep into that again but the design choices of seamless adoption guys so they started shifting everything back to kafka all right and they they had some problems they hit some problems which is called the head of line blocking problem guys and i was surprised when i saw this like wow how can i never thought about that because we talked about consumer grouping kafka check out the kafka video and and you can use the youtube chapters to jump into consumer group to read exactly you're going to understand whoa exactly you'll understand why this happened so consumer group guys is the ability to is kafka's solution for parallelism so that you can spin up in number of consumers in a given consumer group and what will happen here is every consumer will get one partition and one partition only right and and and yeah so that's that's this that's the thing that's that's the idea of uh kafka topic partitioning right so so that means also means if you have a consumer that is just a slow consumer then and this is a slow message this consumer is just taking a long time processing this message for any any reason right then you have to come it's like okay first of all if this is slow then the whole thing is just blocked right this is called head of line blocking we have the same problem with with http uh in pipelining http one one right where you send a request you cannot execute any of the other unless that that the interquest is executed right it's just ordered so that's the same problem here right see the fast messages are okay slow message is the problem and and and now we start questioning why do we have a slow message versus a fast message message right why would an order one order take more to execute than others they should be identical right it's not that again i'm i'm not 100 sure but if an order is an order if i submit a thousand if i want thousand burgers from in and out versus one it's an order who cares if yeah i'm not it's not like me i'm gonna make the burgers just in and out so yeah i'd like to know more about why what is a slow message per se i'd like to know more about that so yeah solving these kind of problems will be interesting i'm just very interested so yeah parallelism and then they're going through all that stuff so they they start to break down the the consumer and trying to find other solutions they're still still still working into these problems guys so i'm gonna i'm gonna leave i'm gonna leave this uh to you guys it's a it's an interesting read definitely i loved i'd like to see more and more articles like this written in a very detailed manner uh it could be just me i'd love to see more details in my opinion because like as i read i have more questions but yeah it's worldwide good job good job guys i love it saba and ashwan excellent job guys excellent i love it this is a great art article uh we need more visibility like this we need to know real peop real people real businesses uses using these back in technologies and how this technologies are performing right again guys again i'm going to repeat it for the for the millions time does not mean that apache kafka is always the right solution to you does that mean rabbit mq is a bad solution to you although most of your comments guys on my rabbit mq video was like yeah you're right robin mq we we moved from rabbiting q because it was so complex again guys that robin mq i used it at my work and i built a course i made a video talking about rameki and i made a video talking about kafka and kafka if you start with kafka it's it's it's way more how do i say what i cannot say than complex it's more intertwined compared to rabbit q right but i came with rabbit mq a little bit confused i saw there are so many abstractions that kind of uh the barrier to entry becomes real high and and learning rubbing mq becomes really difficult to the amount of features that i have and that is that was my problem with it again i might change my mind anything that i say in this channel i could tomorrow then i could definitely learn something new and say okay i was wrong i would i'm i'm very happy to be proven wrong in anything i say obviously that's how we learn right we keep learning to learn everything who who knows everything nobody right we keep learning guys all right what do you think of this article guys have you used celery have you used rabbitmq have you used apache kafka did you build your own asynchronous task processing at all let me know in the comment section below and then see you on the next one you guys stay awesome goodbye

Channel: Hussein Nasser

Views: 53,545

Keywords: hussein nasser, backend engineering, rabbitmq, rmq, rabbitmq vs kafka, rabbitmq message queue, rabbitmq course, rabbitmq tutorial, rabbitmq vs zeromq, rabbitmq publish subscribe, pub/sub, message queues, distributed message queue, stream processing

Id: e03c3CIGtYU

Length: 153min 21sec (9201 seconds)

Published: Wed Oct 21 2020