Kafka Tutorial - Spring Boot Microservices

what's going on guys assalamu alaikum welcome to amigos code in this video i'm going to teach you everything you need to know in order to get up and running with kafka so this course is part of the microservices course which is on my website and currently is the most popular one and the one that you have been requesting so go ahead don't waste time the course is over 12 hours long packed with awesome content enroll right now because there is a discount code and soon the price will increase if you're new to my channel go ahead and give me a thumbs up also take one second literally one second and smash that like button so i can keep on recording videos like this also don't forget to comment down below and let me know what other topics you want to see next or or any other content or any language or framework please just give me a shout because i do take all the suggestions and that's how i come up with these videos if you're not part of the discord community as well as private facebook group go ahead and join because the community is growing and yeah so it's really awesome because people usually ask a bunch of questions and we can exchange ideas on and so forth so do join because i would love to see you there without further ado let's go ahead and learn about kafka so for this course we're going to build everything from the ground up i'm going to show you how to set up a kafka broker locally and then you'll see how we're going to configure a topic so that producers and consumers so the producers can send streams of records to the topic and then consumers can consume those records and do whatever they want to do with it so we're going to configure the whole lot and you'll see how everything is put together and as i said if you're not part of the community go ahead and join because i would love to see you there and if you have any questions or you are stuck you can basically just ask questions there the source code will also be available on github so if you are stuck you can just pretty much clone and just have a look how do things and compare with yours and just get things up and running locally as well without further ado let's kick off this course so in order for you to understand what kafka is i would highly recommend you to go and enroll to this course right here where i teach you everything about microservices in terms of setting up everything from scratch microservice communication connecting to databases logging and a bunch more so basically everything in here that we are doing and previously we actually did connect to a rabbitmq in here so if you've missed that video as well i would highly recommend you to go and check it out now because we are learning about kafka in here we just have to replace this part right here because kafka can be also used as a message queue so let's dive into kafka to understand what it is kafka is a distributed streaming platform that allows producers to send events of streams right here to the broker and then consumers can actually receive those events and the way it works is through topics and i'm going to explain topics in a second but kafka is a distributed streaming platform now you might be wondering why distributed well you can imagine that you want your application to be resilient i.e you don't want just one node to act as your kafka broker and instead you can have two three four or thousands of nodes so these are brokers that basically run your kafka process so these could be cross region different data centers so on and so forth and with kafka you can build real time event driven applications such as for example location tracking so maybe if you order food from uber it's then when you see the driver updating its location on the app that is then through kafka kafka is fault tolerant so you run a bunch of these nodes across region it can scale so if you basically need more brokers or if traffic increases kafka is highly scalable and as i said kafka should always run as a cluster now one difference here that you might be saying okay so what's the difference between kafka and rabbi mq for example well with kafka messages when they are sent to the topic they don't disappear so they can stay there for a couple of minutes couple hours couple of days or even forever whereas in a traditional message queue the messages are gone as soon as they are consumed so we have producers and we have consumers on this side producers these could be micro services or web application or anything really that wants to produce messages or events of streams and then you have consumers so again this could be microservice to microservice communication or maybe you want to send logs as well or you want to build monitoring systems including dashboards or even do work with hadoop also right here on the side you can see that kafka has something called kafka connect so these are mainly for data integration you've got kafka connect source so this pulls data from your existing data sources into kafka and then you have kafka sync which takes data of kafka topics and then puts it into your data source so the connect right here so these are java applications that run independently from the main process there's also something called kafka streams so this is a java api and it's mainly used for data processing and transformation you can enrich data you can transform data you can perform filtering grouping aggregation and a bunch more now here you can see that the arrows go both ways so as data comes to kafka basically streams performs all of these data processing and then once you're done you can actually put it back to a topic so that any other consumer that wants to consume that enriched data they can do so now let's look where these events of streams are put inside of kafka so with kafka we have this concept of topic and if you've seen my mq you've seen topics and different types of topics but in kafka it's a little bit different so a topic is a collection of events which is replicated and partitioned so if you can imagine so if you have messages going to your topic then if you have a bunch of nodes these are replicated and also partitioned now partition here means that so if you don't have a node which is big enough to hold this topic right here the topic itself is actually partitioned across multiple nodes as i said it's durable so you can sorry for hours days years or even forever and there is nothing that says how big or small these copies can be so there's no limit so you have to decide what you want to store and you can basically have really really big topics or tiny topics now in contrast to a message queue such as rabbitmq as i've mentioned a topic in here you can think of it as a log so if you've done any java using log4j storing logs to a log file this is the exact same concept right so data going into the log can be kept for hours days years or even forever and once data is read of a topic it's still there it's still there whereas with a message queue if for example this is red then it's gone but with kafka it's not the case meaning that you can then have a bunch of other consumers reading off the same topic so we have producers that stream data of records so these are the records in here and then we have consumers on the other side so in our example we are working with microservices so we'll have a microservice that produces and another one that consumes these records now let me quickly touch on partitioning so as i said there is nothing that says whether a topic can be small or big so there's no size limit for them really now you could have one node you could have one node that is big enough to hold one topic right so if you have a massive topic you can have one giant node to hold that or instead what you could do is so these days as you know we can scale up or down because of the cloud and resources are cheap and we can have a node which is kind of a medium size right so in this case here let's say that from a giant node we go to three or four small nodes now because we have the topic so we have a topic this because it doesn't fit on these smaller notes we have to partition it we have to partition it so here we have petition one partition two so on and so forth this is the main idea of kafka next let's go ahead and build a project with kafka so you see the integration and we're going to create the broker which will run locally we'll also configure a topic producer and consumer and then we'll have everything working together by sending and receiving events of streams so far in our microservice architecture you've seen that we've used rabbitmq right here so that if notification is done for example our application doesn't crash also we can send asynchronous notifications and we get a lot of benefits by introducing a message queue so we've used rabbitmq but what i want to show you is how we're going to use kafka and because you understand how rabbitmq works then kafka should be really straightforward so if you look at the use cases for kafka on their website they say messaging so kafka works well as a replacement for a more traditional message broker i.e rabbitmq or others right so message brokers are used for a variety of reasons to decouple processing from data producers to buffer and processed message etc in comparison to most messaging systems kafka has a better throughput built-in partitioning replication and fault tolerance which makes a good solution for large-scale message processing applications and really they're referring to activemq or rabimq so this is exactly what we want right so we want a messaging system so let's dive deep into kafka so by now you should know how all of this works but let's first set up our broker in here so i said kafka runs as a cluster that's why we have n brokers now a broker is just a vm a machine or a vm which runs the kafka process you can have two or more but there's no limits to how many brokers you can have and usually some companies they have thousands of these up and running so a broker is just a machine or vm running somewhere in the cloud maybe or in premises and it just runs the kafka process it manages all the partitions for us the read and write requests and a bunch more let's go ahead and set up a kafka broker locally in here i'm within the official page for kafka which is kafka.apache.org and you can see that they give fifth description of what kafka is and some of the industries that use kafka and if i scroll down you can see that it's basically high throughput scalable permanent storage and high availability so it stretch clusters officially over availability zones or connect separate clusters across geographic regions which is really nice and you can see the ecosystem in here so built in stream processing connect almost anything a bunch of client libraries as well as large ecosystem open source tools so let's go ahead and in here navigate to get started and quick start within this quick start page you can find this video that explains what kafka is but if i scroll down have a look we've got a couple of steps in order for us to get up and running with kafka locally so in here i want to download the latest release so let's just download and within this website have a look at this http link let's just click on it and then i want to save this into my downloads folder and if i go back to the previous website and now you can see that it says cd into that folder and then we need to basically start the kafka environment so here we need to basically start a zookeeper so this is where kafka stores all of its configuration and then we can start the kafka broker so let's just take this command in here command c go back and here i'm going to cd into kafka and now i'm going to paste this command so this will basically start a zookeeper there we go and you can see that it's up and running now i'm going to open up a new shell so shift command d cd into downloads and next what we need to do is to start the broker so here take this command and basically just invoking the sh file so back in the terminal i need to cd into kafka and paste that command enter and there we go so you can see that from now on use broker on localhost 1992 so we have kafka up and running which is really awesome next let's go ahead and build our springboot application that will send messages to a given topic so we have the broker up and running now let's build a springboard application and this could be for example a microservice that will basically act as a producer to send stream of events to our broker which is running on 1992. so let me open up intellij and in here i'm using the ultimate edition so i'm going to say new project and i can choose from various templates so here spring initializer kafka dash example and i'm going to leave everything as default next and right here just pick whatever version you have in here and for now i'm going to select in here i'm going to say web spring web so we can build a restful api and also search for kafka so kath so here spring for apache kafka so let's just select the first one we're not doing streams for now so just pick the first one finish and there we go so now we have the project right here so if i open up the application folder and then go to palm.xml and in here so if i collapse this so in here we've got the standard xml file for maven applications and in here have a look we've got java 17 and two dependencies which i've chosen so springboard starts a web as well as kafka and we should have one for test as well as kafka test and also the springbook maven plugin to be honest this is it so now that we have the project all bootstrapped let's go ahead next and start building the necessary code within src so that we can start publishing messages to our broker the first thing i want to do within src is to have the necessary configuration so that we can have the connection details to our broker and set a topic that we can send events to so open up main resources and then application.properties and let's have one single property in here and this will be bootstrap servers so spring kafka bootstrap service and currently we just have one so here i'm going to say localhost column and then 1992 and within this package right here which i'm going to name to just com.egoscode just like that and i can get rid of this in here and there we go so within this so this contains the code for our main application now if i run this we should have no errors hopefully and if i close this there we go so you can see that in here tomcat started on port 8080 which is fine now within the main package let's have another package and i'm going to name this as config and let's have a class called kafka and then topic config and let's annotate this with add configuration and in here so this class will be responsible for creating topics now i'm going to say public and then new topic and this comes from the kafka clients admin package there we go and i'm going to say that this will be amigos code and then topic now inside let's just say return topic builder dot name and this will be amigos code and finally on a new line dot and then build and as always and then bean so that this gets instantiated and we get a new topic so let's just restart the application and there we go if i scroll up you can see that bootstrap service localhost 1992 and there's a bunch of other configuration that we didn't set but this is just the defaults and if i scroll down you can see that tomcat started on port 8080 which means that we are up and running now if i show the cluster you can see that our cluster also is up and running and currently zookeeper is also up and running so things still intact and we can proceed to the next step which is to configure our producer configuration catch me on the next one so we've got the configuration for a topic right here now let's focus on the producer so that we can produce stream of events so back to intellij and still within config let's create a new class this will be named as kafka producer and then config let's annotate with add configuration here and we're going to have a few things so one is if i put this full screen i want to have the variable that will hold the bootstrap server url so private and then string and then bootstrap service and this will come from value and remember so within our configuration file we have spring dot and then kafka and if i show you so here basically i just want this take that go back and let me just replace here job done now we need to have the configuration that we can pass to a producer factory now this has to be in form of a map data structure so let's just define that public and then map and we want to basically have the key to be a string and then this will be object because it can be anything really so maybe you want to have an integer or a string or a class or anything right so here i'm going to say producer config and inside let's define the map so new hash map extract this to a variable and i'm going to say props for property now let's say props dot put and i need to put this configuration here called producer and then config and this comes from kafka clients producer dot and then have a look we can basically specify a bunch of configuration in here but the one that i'm interested is the bootstrap oops so bootstrap service comma and then the value will be bootstrap and then service i also want to define the serializer so this is key and then serializer class config and this in here will be so string serializer from kaffir common serialization dot class so the key will be a string and also the value so let's just duplicate this and here i want to say dot and then the value serializer and for now let's just say that we will send strings and finally let's just say return and then props there we go and this has to be a map just like that and also i just realized that the extraction didn't work as i wanted so this has to be a string and job done now let's define the producer factory which is responsible for creating producer instances so here we need to have a public method and i'm going to say producer and then factory the key for this will be a string and the value will be a string and if you were to for example send a java object right so your own class or record this would be for example customer or maybe notification but for now let's just keep things simple and have a string in there and i'm going to name this as producer and then factory and within this method all i want to do is to say return new and then default kafka producer factory and one thing that i need to pass inside is the config so producer and then config and finally now that we have the producer config which allows us to create kafka producers we need a way for us to send messages now this is possible with the kafka template so kafka and then template so the template also will be of type string and then string so this is using generics and again if you were to send for example a custom object this will be that custom object data type so let's just say here kafka template and inside of this kafka template i want to return new and then kafka template and kafka template if i press command p in here takes the producer factory now this producer factory let's just instantiate it as a bean so at and then bean and let's take this in here command c and we're going to paste it inside here and so this is using dependence injection now and then pass the factory inside and job done and last but not least so that this works we need to say at and then bin and there we go now that we have the producer configuration next let's go ahead and learn how we're going to send a message to our topic our broker is running on 1992 we also have configured the producer now let's see how we're going to use our producer which is kind of a microservice really and send a message to our topic that we defined so let's open up intellij and within project open up the kafka example application in here and this really should be named maybe kafka application so let me just rename that to kafka application i think it sounds better okay and what i want to do is i want to have a command line runner and this command line runner will receive the kafka template that we've just defined this will be string and string and i'm going to name it as template and inside here i need to return so args so this is using dependence injection by the way so this right here will be this guy that we've just configured if i go back and here i'm going to end this with semicolon there and now i can say kafka template dot and then have a look the available methods so we have send and pass a message send with a producer record with a string which is a topic and then the data and we can also pass a key so this will guarantee that the messages are delivered to the same partition if you are using partitioning so for now let's just say topic in here and i want to send to the amigos code topic and the message will be hello and then kafka and then basically end this with semicolon now this guy here so amigos code if you forgot so this is within kafka topic config is this guy right here so this is the topic that we've just defined in here and to be honest this is it so obviously i'm going to change this in a second so we can have a restful end point and from there we can basically put it inside of the topic but for now here let's just say bin and with this in place this method here will run as soon as we start the application so let's just restart the application and hopefully things should just work and have a look we have this log line in here which says add and then producer one and in here it says basically producer client producer 1 and then the cluster id so at this point we have sent a message or a stream and if i scroll up i just want to see that basically you can see that things are working so tomcat started on 8080 and we have no errors in here so at this point our application is working successfully next let's go ahead and set up a consumer so that we can see this message being consumed within the quick start page for kafka if i scroll down you can see that we can create topic so they have a script for it and in here so we can basically read events so have a look it says open and at the terminal session and run this command so here what i'm going to do is i'm going to take this command so let's just take this command command c and open up my terminal and at this point i need to open up a new shell so shift command d and cd into downloads and then kafka there we go now i'm going to paste this command and before i run it so let's just have a look so here we just run the script passing some arguments but the topic here have a look the topic is quick start events now our topic is called amigos code so let's just change this to amigos code and i want to read from the beginning and also i'm passing the bootstrap server which is localhost 1992. now if i press enter we should get have a look so before i did have a few things inside but we have hello and then kafka so this is really nice so if i open up intellij and collapse this and let's just say for example a smiley face in here and restart let me go back as soon as possible and there we go you can see that we are reading from the amigos code topic which is really nice now obviously kafka handles lots amounts of data so if i was to go back to intellij and let me just cancel here for a second or hide my bad and what i want to do is in here so let's just say for example 4 i and i want to loop let's say 10 000 times so this is nothing but you could loop more than this if you want but i'm going to paste this inside and here i'm going to say plus and then i if i reload and then go back we should see oh have a look so this was really quick right so basically with no time it just read everything from the topic right so we produced 10 000 events and you saw that it took no time now let's try a million for example so here let me just hide this and from 10 000 i'm going to say one million so i want one million here so this is 10 million oh actually let's try 10 million and see what happens so let's just reload go back and have a look how it's processing everything so you can see it's basically handling all of this with no trouble and this is basically the power of kafka that you are seeing in here so eventually it will finish just give it a second and boom have a look so 1 million records processed without no issues so hopefully now you can start to see the power of kafka next let's go ahead and build the consumer code within our java application all right now let's build the microservice logic to act as a consumer so that we can pull the events from our topic let's open up intellij and in here before you saw that we basically we pushed 10 million events into our topic and this was nothing for kafka so let me just basically say 100 here because of time and i also don't want you guys to wait around so here let's just open a project and within config let's have a new class and here i'm going to say kafka consumer and then config this will be at and then configuration and within here so this will basically follow the exact same thing that the producer had so let's just basically copy some code so let's just take all of this command c go back paste that in and to be fair we could have made this right here a bean so that we can reuse and make it more generic but at this point let's just copy and paste so consumer and then config and let me put this full screen and in here let me just scroll down and the only thing that changes really is that instead of the producer factory so before have a look we have producer factory now we want the consumer factory so this will be consumer and then factory just like that and this will be consumer config and if i take this command c and let's just space it here and also this is consumer and you can see that it works so let me just import and job done now the next thing that i need to do is in here i need to specify the listener container factory so this will be public and then kafka listener container factory and this listener here receives all messages from all topics or partitions on a single thread so let's just say that we want this to be a concurrent message listener container and the key for this will be string as well as string so remember this is what we are working with and if i put this let's just see if i can put this on a new line even though this is ugly and that's because my font is quite big and have a look at the method so i'm just going to say factory in here and put it on the same line here and inside now i'm going to say concurrent kafka listener container and then factory and you can see i'm using keyboard shortcut there and don't have to remember this long class so here i'm going to say string and then string and i'm going to say that this is factory and this will be equal to new concurrent kafka listen container factory just like that i want to set the consumer factory and this will be consumer factory and in fact because this is a bean in here we could just use dependency injection in here so again this is really ugly code so excuse me and this is only because my font is quite big so let me just paste that in and just like that so this is the argument and now i can just pass it like that and there we go now the last thing that i need to do is return and then the factory and job done now let me just scroll up and see if everything is looking good and i've actually noticed that this should be consumer config instead of producer config so let's just change this consumer and then config and let's just import that and also instead of serializer so on this side we want to deserialize so let me just select here ctrl g and then d serialize just like that and i think this should be it if i press cape and to recap the producer uses a string to serialize for the key as well as for the value and the consumer basically has to deserialize using the same data types later we'll change this for a custom object but for now this should be good to go next let's go ahead and write the code that allows us to consume events from our topic with this configuration in mind for the listeners let's create a listener and within the main package i'm going to have a class and i'm going to call this as kafka and then listeners so obviously you can basically structure this the way you want in terms of packages but i'm going to leave it within the root so press enter and now this i'm going to annotate with add component and in order for us to have a listener that listens on our topic let's have a method here i'm going to say void and then listener so currently we only have one listener so listener and within this listener i'm going to receive the data and because we're using spring boot in here i can just say at and then kafka listener and here inside i can pass the topic or topics so i'm gonna say topics and currently i'm going to listen from america's code and also we have to pass a group id so this is so that if we scale ie if we have more instances of the same application they can basically read from the same partition or topic so here i'm going to say group id so group id and for now this is a bad name i know but i'm just going to say full for example right or you could say group id but as you have more listeners then you want to change this so that it's unique now in here i can then just do anything i want with this data so for now i'm going to say south and full screen received or listener received and then data and let's add an emoji there so i want to have an emoji here and i'm going to choose the tele yeah there we go tada there we go and we are good to go so i've got typo here and job done now let's try and restart and see whether this works so let me restart the application and currently let's have a look so you can see that we have group id which was assigned and i think at this point so i think i need to restart let's just see i think i need to restart once more but yeah there we go have a look so now we're actually reading from the topic right so we've published a bunch of messages so here have a look we did publish before kafka application we published all of these messages in here and then we consume them inside of this listener so if i go to run you can see that we have tada so also remember if i go back to my terminal in here so this was open have a look right so i've got another consumer so what we have is we have two consumers reading from the same topic next all we need to do is to build a restful api that allows us to publish messages to a queue let's go ahead and do that next so to build an api let's within the main package let's just have a message controller class and here let's annotate this with at rest controller and at request mapping and this will be api v1 and then messages and let's have one single method here so public and then void and this will be publish and we're going to receive and we're going to receive a message request so in our case let's just have a record so class and this will be message and then request and this will be a record and inside i'm going to say message so now at and then request and then body and from it i want to get the message request and all i'm going to do is so obviously i could structure this into services so on and so forth but for now all i'm going to do is i'm going to inject the kafka template so here private and then kafka template this will be string and then string and kafka template and inside i'm going to say kafka template dot and then send i'm going to put into amigos code topic and in here just say request dot and then message and also we need to have a constructor here so constructor and things are now auto wired and because we want to expose this method to our clients we need to say at and then post and then mapping so that we can issue post requests against it so let's just reload in here and for this you could use pretty much any res client but within intellij i can just say generate requests in http client and this will be 8080 messages enter and i think so let's just have a look examples post request and i need this guy here so let me just take this and then go back and paste that in and the content so here so this will be message and i can pass anything so here i'm going to say api with kafka now if i send this request by just pressing display button there we go 200 status code and open up run and if i scroll up you can see the headers as well as the url if i open up run have a look so this is good stuff so listener received api with kafka so let's just say here hooray amigos code send the request there we go go back to run and hooray amigos code so you can see that our listener i.e the consumer is receiving streams of event of a topic so we've got a restful api we publish to a topic and the consumer reads it out so we pretty much have built a restful api with kafka integration also if i show you the other consumer which is running on my terminal you can see that here it says hooray and then amigos code api with kafka and we are good if you have any questions please do let me know otherwise catch me on the next one if you want to learn more about spring kafka navigate to spring.io then click on projects and then spring boot and right here if i scroll down you should see that there is a link for spring for apache kafka if i click on that and you can see that it gives you the overview and we went through all of this but basically in here so if i click on learn and you can see the reference docs let me just click on that and in this page there is everything you need to know about using kafka and springbook so if i scroll down it's quite an extensive documentation which i recommended to read so you can basically if i click on the reference have a look configuring topics it gives an example in here sending messages so this is what we went through right here so this is it to be honest if you have any questions please do let me know otherwise catch me on the next one okie dokie you've made it this far hopefully you saw the power of kafka we build producers consumers and they both exchange messages or events via a topic now the next step for us really is to basically change the value data type that we produce to our topics so currently we're just publishing strings but we can also publish custom objects now if you want to change that it's super easy and i again encourage you to enroll to my full course to have a full understanding of microservices so the course is over 12 hours long right now and i'm still adding a couple things to the course and right now there's a discount which you can find the link and description of this video and yeah this is all for now i had fun teaching you kafka and if you have any questions please do let me know otherwise i'll catch you on the next one you

Channel: Amigoscode

Views: 230,514

Keywords: amigoscode, kafka, kafka tutorial, kafka spring boot, kafka tutorial for beginners, microservices tutorial, microservices architecture, microservices interview questions

Id: SqVfCyfCJqw

Length: 51min 29sec (3089 seconds)

Published: Thu Feb 03 2022