RabbitMQ : Message Queues for beginners

what is up youtube and welcome to another video in this video we're going to be taking a look at how to run rabbitmq the basic fundamentals of a message queue why you even need one and we're also going to be taking a look at how to write a basic application that produces messages as well as consumes messages from the queue and if you guys enjoy this video be sure to like and subscribe because in the future i'll be showing you how to run a rabbitmq cluster for a highly available production system as well as running rabbit and q on a service like kubernetes so without further ado let's go so what are message queues now message queues is a form of service to service asynchronous communication this is common and very popular in microservices and serverless architectures now why would we need rabbitmq or a message queue to understand why message queueing is important we have to understand the concept of asynchronous messaging now let's take a look at a basic microservice example of service to service communication in this example service a makes a call to service b this is usually done over a tcp connection now this call here is synchronous meaning service a has to wait for service b to respond if service b takes a long time to respond this connection is potentially dragged out then we get added latency if service b crashes or dies or fails to respond service a may have to retry potentially multiple times in order to get a successful response now as you can see service a is somewhat coupled to service b now with asynchronous messaging service a sends a request and continue with its life it does not care about an inline response it does not care who processes the message how long the message takes and it's not directly coupled with service b now in order to understand how asynchronous messaging is achieved we have to understand message queueing now asynchronous messaging is usually achieved using a message queue now what is a message queue well this is where rabbit nq comes in so instead of service a directly calling service b service a places a message into the queue now service a can go on with its life and not worry about who processes that message service b now picks up that message processes the message and we have an outcome in message queuing service a is called the producer since it's producing messages service b is called the consumer since it's consuming messages asynchronous messaging is achieved with message queues and this is where rabbitmq comes in now there's a couple of advantages here message queues can help so that one service is not directly dependent on another service when you're writing code to build up service a to create a network packet and a request for service b you usually start writing api contracts and this means that service a is somewhat coupled with service b now these queues can usually help decouple these two services that would normally depend on each other cues can also help in cases where we can generate messages way faster than we can consume the messages so this allows us to scale up the backend consumers to cater for the need of processing a ton of messages it can also help with spreading the messages out to multiple consumers this is called batching so rabbitmq is the most widely used open source message broker and rabbitmq also has all of these features now we've spoke about asynchronous messaging also he's spoken about message queuing but it also provides delivery acknowledgement now delivery acknowledgements is a special feature of rabbitmq let's say service a puts a message into the queue and let's say service b picks up that message from the queue for some reason service b crashes if you configure rabbitmq with delivery acknowledgement it will expect service b to put an acknowledgement message back to the queue if service b dies and the queue does not receive an acknowledgement rabbitmq will automatically put that message back to the queue for another consumer to pick up and this adds resilience to message queues rabbitmq also has a great developer experience with many types of programming language compatibility so we'll take a look at that in a second now very importantly for devops folks rabbitmq also has a distributed deployment mechanism so you can set up instances in a highly available manner like a cluster in the next video we're going to be taking a look at how to start up rabbitmq in a cluster mode and how to join multiple instances to make it highly available ebnq is also enterprise and cloud ready so in a future video we'll also take a look at how to run rabbitmq in a highly available cluster on top of something like kubernetes in the cloud we'll also take a look at some tools and management plugins that rabbitmq supports as well as management and monitoring tools so rabbitmq also supports tools like prometheus integration so you can monitor it from your prometheus instances so let's take a look at getting started and how to install rabbitmq now the best way to run rabbitmq is using the docker image so you can run it in production on something like a kubernetes cluster if you head over to docker hub you search for rabbit and q you'll find the official images with all the different versions they also have documents around showing what is rabbiting queue how to run it using a docker run command there's some features like memory limits we're also going to take a look at authentication using an erlang cookie there's also the management plugin that we're going to be enabling to get the ui dashboard we're also going to take a look at how to enable plugins so for those of you who are new to this channel everything i do is on github i have the docker development youtube series github repo in here i have a messaging folder and i have rabbitmq everything i've done is documented in a readme file over here that you can find in the rabbidmq folder so remember to check out the links down below to this github repo so you can follow along now especially if you're in devops it's very important to understand the networking of running rabbitmq especially when it comes to running in a cluster so what we're going to do locally the first thing is we're going to run a local docker network called rabbit this is so that every instance that we run can talk to each other we're also going to run an application that's going to produce messages consume messages and they all have to run on the same network now the easiest way to run rabbit mq as i said is in docker so we're going to say docker run in background mode we're going to remove the container if it gets deleted and we're going to run this container on the rabbit's network and then we're also going to specify a hostname of our rabbit instance and name for the container and then we're going to run rabbitmq the official image 3.8 now it's very important to understand the hostname rabbitmq uses an identifier and then an at symbol and then the host name in order to talk to each other so if you're running rabbitmq on something like azure vms or ec2 instances you need to make sure that you know the host name of those instances so that these rabbitmq instances can find each other if you're running on something like kubernetes it's important to run it as a stateful set and not a deployment because deployment pods are ran have random host names so now that i've started the container we can see we have one rabbitmq instance up and running and you can see these are all the ports that it's exposing now it's very important to know if you're running it in a cluster we'll be using this port and this is the communication portal for applications to consume the queue we can then say docker logs rabbit one which is the container name and we should see that our rabbitmq instance is up and running and healthy we can also go inside of our docker container using the docker exec command and we can run bash and now that we're inside we can use the rabbitmq ctl this is the cli tool we can use to manage rabbitmq so the rabbit cli is very useful for controlling your rabbit node so it has a couple of commands regarding node so you can stop and start your your application which will leave the container image running you can also reset it so that's basically a command for the redis node to leave a cluster it also has a bunch of clustering commands we'll be taking a look at this in a future video of how to create a rabbitmq highly available cluster there's also some replication commands managing of users access control monitoring as well as policies virtual hosts node configurations the list goes on [Music] so rabbitmq also has a separate cli called rabbitmq plugins and if you run that there's a bunch of things you can do you can disable enable and list different types of plugin integrations for rabbitmq now these are very useful since rabbitmq provides a wide variety of plugins one plugin is for example is the management interface and another plugin that's quite useful is the prometheus monitoring plugin so if we run the rabbitmq plugins list command we can see a list of plugins that are that are available in our instance and you can see none of them are enabled by default but you can see here we have like different authentication backends we have peer discovery plugins we have tracing plugins and we also have the prometheus plugin now rabbitmq also provides a management plugin as i mentioned earlier so to enable that plugin let's exit out of the container quickly and let's quickly delete the container and let's recreate the container but this time i'm going to expose port 8080 which internally in the container i'm going to expose port 15672 this is the management plug-in ui port that we can use to browse to the management instance over the browser so if i go ahead and run that and then go back into the pod i can then enable the plugin using the rabbitmq plugin cli so i can say plugins enable rabbit and queue management that'll go ahead and enable the management plugin so now we see it's gone ahead and enabled a bunch of plugins to make the management capability to work we can then say plugins list and we can see it's gone ahead and enabled web dispatch management agent and the overall management plugin and if i go over to the browser on localhost 8080 we can see we've arrived at the rabbitmq dashboard and you can just log in with the default credentials which is guest and guest and this is the rabbitmq overview page so you can see i have a one node in my cluster we can also see a listening port so this is the port that we're going to be using to develop an application that can put messages into the queue you can see the connections that are established to this instance and then rabbitmq also has the concept of channels now channels of virtual connections to a specific cue so this is important because when we write a program we're going to be creating a connection to our rabbitmq instance we're then going to be creating a channel which is a virtual connection directly to a queue so we have to define a queue and then we can start putting messages into that queue so here you can see we have channels and then we also have views that will be established shortly and then we also have an admin interface where we can add users and service accounts that can interact with rabbitmq so let's take a look at what it takes to write an application that pushes messages into the queue so we can see this thing in action so to do that we're going to head over to the docker development youtube series github repo i have a message in queue folder grab it in queue and in here i'm going to create a new folder called applications now the first thing i'm going to do is create an application that publishes messages into the queue so we're going to do one application called the publisher and for our application i'm going to create a new docker file so you don't have to install anything on your machine i'm going to write a very small go application so i'm going to say from golang 1.14 i'm going to run the alpine image and i'm going to run that as a build container and the next line i'm going to go ahead and install git since it gives us the dependency manager for go i'm then going to set my working directory to slash source and then i'm going to pull down three dependencies i'm going to say go again i'm going to pull my web application so this is going to be an api where we can push messages to and the api will put the messages into a queue using the rabbitmq sdk and then we also have a logging driver just if we want to log stuff now our source code is going to go into a new file called publisher.go and after we've written that source code what we're going to want to do is we're going to want to copy publisher.go into our source directory and then to build it we're going to say go build build the publisher.go and now we have our docker container for our application but what we're going to do now is we want to make a lightweight one so we're going to save from alpine as our runtime we're going to then tell docker to copy out from the build layer which is this layer up top here we're going to copy out the publisher binary that we've built using the go build command and we're going to copy it into an app folder and then to start up our application we're just going to start it up with a command called app slash app slash publisher now we need to go and write some code so let's head over to the publisher.go and we're going to start with package main we're then going to import all our dependencies so this is just to log that's to log as well we import net http so that we can run http functionality we're going to run an http router which is going to be our web server for our api and we're going to pull in the rabbitmq sdk as well then to make this simple i need to obviously pass in a hostname and a port and also a username and password to interact with rabbitmq so for that i'm just going to use some environment variables it makes it really easy to configure and then i'm going to go ahead and and create a main method now first thing we want to do in our main method is define our web server we then want to go ahead and define a route so we want to say router.post and whenever messages comes over the slash publish endpoint we're going to take whatever's behind that as a message and we're going to put that into our queue so for that we're gonna define a basic submit function just to keep the main method a little bit lean and we're just gonna write something to console just our application is running and then we're gonna start up our application with http listen and serve method on port 80. so now that we have a web server up and running and go let's see what it takes to actually put messages into the queue so for that we're going to create and define our submit function which is going to be taking a bunch of parameters from that web request the next thing we're going to do here is grab our message this is the message that comes over this post request as a parameter for interest sake we're just going to write our message to console we're then going to go ahead and establish our connection with the queue so we're going to use the rabbitmq sdk and we're going to pass in a user password host and port now we also want to go ahead and handle that connection failure in case it fails and comes back with an error we're also going to defer this connection and we're going to close it whenever this function exits and the next bit is we're going to need to create a channel now remember a channel is basically that virtual connection that we establish with the queue so in order to talk to a queue we need to create a channel so to do that we use the connection and we create a channel similarly we also have to handle the error just in case we fail to open a channel and then we also have to defer that channel in case this function closes or errors out we want to make sure we close the connection to that channel and now that we have a channel we can go ahead and declare the queue so we say channel dot q declare and we can give it a name so i'm just going to call this q publisher and there's also a bunch of settings and configuration you can look at about how the queue should behave then again this is golang so we have to handle any errors that would come back from that in case we couldn't declare a queue and now we can go ahead and publish a message on that queue so we say channel dot publish we pass in the queue and we also pass in our message in here as a byte array then we also have to handle that again in case that comes back with an error that we failed to publish the message and last but not least i'm just gonna write to consoles saying that we've published successfully now to build and run that it's very simple i've created a bunch of commands here you can follow along we're to change directory to the location where that docker file is and we're going to say docker build.t and i'm going to pass in my image name that's going to go ahead as we can see it's going to go ahead and compile that application and build up a container image now remember you've got to check whether your rabbitmq instance is up and running also it has to be running on that rabbit's network we created and then the next step we can do is we can run our publisher application so we say docker run minus it we run it on the same network we also give it a host name we also give it a bunch of environment variables so we say um the host to connect to we're going to connect to rabbit1 we're going to connect over that port and we pass in a username i'm just going to use the guest username and password we're going to expose port 8080 so we can interact with this api and this is the image name so you can see our application is now up and running and then what i'm going to do is i'm going to open up postman i'm just going to create a post request to localhost port 80 to that publish endpoint and i can basically pass in any message here so i can say hello and go ahead and send that we can see that's gone ahead and published we can see that our publisher has put the has received the message hello and it's published it successfully to the queue and interestingly if we go over to the dashboard we can see that there is now some interaction and activity happening on our queue and if we head over to queues we can see that our queue has now been created here by our application we can go ahead and access that queue we can say get messages let's get the message and we can see that we have a message in here with the payload of hello so now we have an application that can push messages to a queue but more importantly how do we consume messages from that queue so what we need to do now is write a consumer application that targets that same queue and that's very simple to do in my applications folder here i'm going to create a new folder called consumer and very similarly i'm also going to create a new file in here i'm going to start with another docker file now this docker file is identical to my producer but i've just gone ahead and renamed the application file so i'm still using go i'm still using similar dependencies but this time i'm not using an http web server i'm gonna instead of running publisher.go i'm gonna write a consumer.go i'm gonna pass it into sourcefolder i'm gonna say go build and i'm gonna produce a runtime container image and i'm just gonna start up this time a binary called app slash consumer so to create the application we're going to create a new file called consumer.go and similarly we're going to start with package main i'm going to pass in my dependencies i'm also going to pass in the similar environment variables because we want the consumer to connect to the same instance of rabbitmq and i'm just going to write a main method and in this method i'm going to create another one called consume so let's go ahead and write that up now the code looks very similar to a publisher it also needs a connection so we need to establish a connection passing in the same connection string we also need to handle the error in case the connectivity fails we need to go and create a channel the virtual connection in order to connect we have to handle that in case that fails we also go after going to clear our queue so we're going to declare the queue with the same name that the publisher is using and in case the cue declare fails we have to go ahead and handle that i can just write a message to console to say that the queue and the channel have been established and then i'm going to go ahead and defer those connections so if this function exits or cancels or errors out the connection and the channels will automatically be closed and now i'm also going to call the channel.consume function and pass in the name of the queue and then here you can also pass in a bunch of settings so this is where you can enable things like delivery acknowledgement and things as well so you can see here we've we've disabled auto acknowledgement so i want to manually acknowledge when i know my transaction has been completed and then we obviously have to handle in case that um consume function fails we have to handle that error and then i'm gonna go ahead and create a go channel this is a channel that's gonna run forever and it's gonna be basically listening to messages coming onto the queue so then i'm going to pass in a function to that go channel you can see here i'm just creating a go function and what i'm doing is i'm just looping through the messages that are coming in over this consume function i'm grabbing the message out of the body so i'm also writing that to console to say that we've received the message and then here i'm passing in acknowledgment back to the queue and then i'm also writing to the console that we are now running so to build this is very simple as well i've got it documented into the readme if you want to build it you just basically change directory to the consumer folder and we're going to say docker build and we're going to tag it this time as a consumer image that'll go ahead and compile our app and build a container image out of it and very similarly we're going to run this by saying docker run it on the rabbit's network we're going to pass in the same connection details to connect to our existing rabbitmq instance and watch what happens when i run it we can see we've established connection to the queue and we've automatically received a message that's because our previous message is still sitting in that queue and has been picked up by this consumer and if i go back to the rabbitmq dashboard and i say get messages we can see the queue is now empty since the message has been picked up by our consumer automatically so now we can go ahead and interact with our api and publish several messages so i can say what is up youtube and if we take a look at the at the consumer below it's received all the messages and processed them so hopefully this video helped you guys understand the basics and fundamentals practically and theoretically of rabbitmq and how to write an application that produces and consumes messages out of that you hopefully this video also helped you get your first rabbitmq instance up and running and understand the basics of a queue now be sure to like and subscribe and tune in and follow along since i'll be showing you guys how to run rabbitmq in a highly available manner for production systems so we're going to take a look at how to set up a rabbitmq cluster and then in a future video we'll also be taking a look at how to run rabbitmq on kubernetes so hope you guys enjoyed the video and as always like and subscribe and until next time peace [Music]

Channel: That DevOps Guy

Views: 54,864

Rating: 4.9587379 out of 5

Keywords: devops, infrastructure, as, code, azure, aks, kubernetes, k8s, cloud, training, course, cloudnative, az, github, development, deployment, containers, docker, rabbitmq, messagequeues, messagebroker, messge, broker, queues, servicebus, aws, amazon, web, services, google, gcp

Id: hfUIWe1tK8E

Length: 22min 6sec (1326 seconds)

Published: Wed Aug 05 2020