Spring Boot with Spring Kafka Producer Example | Tech Primers

have you worked with Kafka and spring boot in this video we are going to create a spring boot application which can publish messages onto our Kafka topic I am going to create a Kafka topic locally so I'm going to run my zookeeper node along with the zookeeper server and we are going to create a Kafka topic and we will create a spring mood application and publish a message on to that Kafka topic and we will see from the console by consuming that message and seeing whether we are able to publish it or not [Music] press the bell icon on the YouTube app and never miss any update from that premise as usual we are in the start at spring dot IO I'm using Java spring boot 1.5 to 13 let's take the group as calm tech primers I will make it as Kafka because we might do more videos on Kafka because this is going to be a cough cough producer example where I'm going to create a spring boot with kafka producer as an example so I'll just make the same as the artifact I'll just say spring good cough cough producer example what do we need we obviously need cough car dependency so there is some module called spring Kafka which spring has created which can be used inside the spring boot application so that it is easier for us to publish messages like how we used to query rest endpoints using the rest template we can do it something with the Kafka template additionally we are going to use spring MVC so we will expose a web server web service and we are going to hit that web service and that will publish some message let's create this project let me open it in IntelliJ so the project has got imported here let me change the version to Java 8 and let's go and check the poem to like someone so the palm dot XML has spring boot start over which is the spring MVC additional to that we have something called spring Kafka so this is the module which we are going to use to publish messages onto the Kafka topic let's see how easy is that with spring I'm going to use the server port 8081 because 8080 is occupied in my system I'll use that by default form for bringing up this particular spring boot application additional to that let's create our usual resource I am just going to create a resource here so let's call it user resource as usual right so this is going to use ping MVC I'll just say stress controller and I'll map it as request mapping only thing is I will just make the controller as Kafka so that way we know that this is going to produce some Kafka message now I'll just create a get mapping so that I don't have to use post or postman I just do a get mapping and let's do a post message right and I will say publish message I'll create an published message or I will just make it as publish and this could be a message right whatever the publishing and just say final string message and I also say published successfully now we need to publish the message which you are going to push so how do we publish onto Kafka it is pretty simple if we have the Kafka in the same machine I'm going to use Kafka template like how we have a rest template we do have something called Kafka template by default let's publish a string as a message so what I'll do is I'll just say Kafka template string string and I'll just author why this so spring will by default take care of all the configurations which we need we don't have to provide now now we need to publish this on to a topic and we need to publish this message so message is a string so let's create this topic and just create it as a field let's provide it as a name right so let's provide a name to this particular topic and just public stuti final it's going to be a constant one so I'll just say Kafka underscore example you can provide any named to it I'm just going to say Kafka underscore example right so that's it I don't have to do anything else I just said Kafka template since I'm going to use the Kafka also in the same machine as my laptop so I'm not going to provide any configuration details but in the later part of the video I will show you how to provide that so right now that's all I don't have to do anything this will automatically publish a message on to the Kafka topic so let's see how it happens right before this we need to start the Kafka instance right now I don't have it running so in order to bring up the Kafka server we need to connect we need to create a zookeeper instance because Kafka is a zookeeper so I already have the zookeeper command here we need to say zookeeper server status search this is already there in the Kafka folder so if you see here this is the Kafka folder which I had already downloaded from the Kafka website and I'm just going to start the zookeeper node now so this will start a zookeeper instance here in my laptop once the zookeeper node has started now we have to start the Kafka server so in order to start the Kafka server we just need to say Kafka - server start a search which is already there inside the bin and we need to provide these several properties which is also already present inside the config folder so once we start the Kafka server we should be able to start our consumer as well so we need to it sorry we need to first create the topic so once the Kafka server is up now we can create the Kafka topic see I already have the command for creating the Kafka topic we just need to say Kafka - topic toward a search we need to provide what we need to do - - create and we are going to provide the zookeeper instance which this particular cough-cough car topic needs to be created on and it is inside our local machine and the port by default is - 1 8 1 so that is what we are providing here if you have multiple zookeeper nodes you can provide additionally along with this right but we have only one right now and the replication factor is 1 you can provide the different replication factor based on your cluster and needs and SE mueve the number of partitions so right now I am going to create only one partition but if you need multiple partitions you can mention that and finally we need to provide the topic name so here I will say the topic name is Kafka half an example underscore example this is the topic name which we saw all right so this will create a topic for us so first we need to create it so the topic got created so once the topic is created we should be able to publish it before that we need to consume from this particular topic so that we can see the messages are getting posted or not so what I will do is I will consume I'll just create a consumer so this is the consumer which will just connect to this particular topic whatever we mentioned here and it will show what messages are coming in so right now there won't be any message see right now there is nothing coming in because right now we are not published anything let's try publishing messages right let me start this spring good application so the topic the port number is 8 0 8 1 let me go to the browser 8 0 8 1 so the spring buta plication is coming up still once it is up we can go to the browser and hit the get URL so we have it ready 8 0 8 1 I should see yeah the server is there so we need to go to Kafka slash publish I mean to provide the message right let's say hello so this should publish the message and if you notice here we got a published successfully and let's try yeah see that we already got a hello so let me try showing this in a single page right let me push this here so I have just now made both these windows appear in the same screen so that you guys can see how the messages are getting published say hello YouTube I'll just read rigor it and you can see in the right side here this is the console where the message is getting consumed from the same Kafka topic where we are publishing so if I just hit on this you can immediately see that because the hello YouTube option so this is how we can publish a message on to the Kafka topic but what we did here is we just published a string right so what if I want to publish a JSON message so this is just a string but however I want to publish a JSON message so in that case what will I do so I can create a model here what I will do is I will just create adjacent model and I will show how we need to modify the Kafka configurations in order to publish adjacent so this is a normal user class which I have created I will just create a named user's name will be named Department and let's say and just do a salary so we have created the model now instead of this string here we need to use the user model that's it but you need to provide need to create a new user with the name what is the name of the user let's say in a name was always Peter I'll just hard code it for now so that we are just saying this is the name let's provide a name and the Department is technology and let's say the salary is 20,000 long that is so this is created now this part is the same the publisher part is going to be the same just that we have used the user model and we are going to publish that directly however we need to serialize this particular object in the type we want to publish so by default Kafka uses the string so it said that's why we do not do anything for the string message publishing part however we are going to publish JSON here now we need to tell Kafka that I'm going to publish a JSON message so we need to do some configuration so let's call it Kafka configuration and this needs to be loaded during the spring boot initialization so I'll just add the ad configuration annotation now we need to create a beam of type producer config road user factory sorry so we need to create a producer factory and whatever am I going to do inside the producer factory I'm going to create a configuration and I'm going to provide that configuration for creating a new default Kafka factory when I am creating this new default coffe cup producer factory I need to provide some configuration so let's see what is that configuration right this configure just a type hashmap and I'm just going to put valleys on to it so what type of values I'm going to put I need to get the producer configuration right so I will just say producer config this has some keywords these are the different configurations which we need to provide right so the basic one is the bootstrap server configuration where I need to mention what is the server name right in our case it is the localhost I'm just going to say 127 dot 0 dot 0 dot 1 : 9 0 9 2 that is the default port which we are going to use for consuming messages right same way for the publisher party it's the same so if you see the consuming part where we are using the port number is 9 0 9 2 I'm just giving the same thing here so that is the Kafka pot the next one is what type of key are we going to put so producer dot key serialization class conflict I am just going to mention the serialization class which we need to use as a key so I'm going to use the key as a string so I'm just saying string serialization serialize a class this should be coming in from this spring one so I'll just say serialization dot class that's it same way for the value I need to provide the similar configuration however in the value we are going to use the JSON serialization so we will say jason serializer dot close so this will tell Kafka that ok I'm going to use the JSON configuration for the values and when I created the producer Factory I'm just providing these values here now who will tell spring that ok this is the thing we will have to now create the Kafka template we created the we did not create the Kafka template we were just Auto wearing it now here is the chance for us to create the Kafka template with that string user whatever we created we'll just say a return new Kafka template and I need to provide the factory so I'll just say provide the producer factory so that is what I am doing here and this should be of type string comma user here I need to do the same thing here I don't need this that's it so what we have done here is we have created a producer configuration factory where we provided the configuration details the three configuration details which we provided where the bootstrap server retains the key serialization class and the value serialization class we are going to publish a JSON message so we are using the JSON serializer here if you have a different type of message for example if you have a JSON message or if you have a row format you can just do that change here and we are injecting this particular producer conn factory on to the Kafka template so that this Kafka template can be used inside our resource directly in fact we can rub this Kafka template publisher part on to a service but I'm not doing that here I'm just making it simple so that's it so what I'm going to do is I'm going to hit the publish option which in with a name and the name will always go with the default technology and salary so I don't want to use postman here to show that so that is why I'm making it simple there is no other reason for me to do this but if you are using a post mapping you can add it in under the post you can you can provided Jason and you can push the Jason directly on to the topic but I am NOT doing that right now I'm just making it simple by using the get mapping here so the server is up now let's go to the browser and I'll keep this in the side right so that we can see what's happening so what I as I have said that do a publish and we need to provide a name so I will just provide a name as Peter so this should publish Peter along with the Department and the salary see that right we we got adjacent message in right side so let me reduce this a bit so wherever we are consuming we are getting Peter so let's try some now let's try publishing Sam so Sam yeah we got the name Sam and the Department is technology and salaries same so that is what we did right so this is how we can publish a Jason message so this is nothing but a custom message so this is how we can publish a custom message on to the Kafka topic by providing the configuration we provided this utilization configuration here to the calf car template and we used that Kafka template in the resource and we published that so this is how you can use spring boot to publish messages onto the calf car topic I'll just summarize what we did we used the calf car template like how we use the rest template for calling res messages now we are using Kafka template to communicate with Kafka servers if you have a custom Kafka server which is outside your machine you can use that here you can just say bootstrap server configuration you can provide the Kafka server here in my case it is the local machine the same machine so I don't even how to provide this configuration until or unless I need to change the Jason civilization so the key and the value can be serialized separately you can provide a different serialization right now I am using the Jason previously for string I didn't have to provide anything because by default it uses string serializer I'm using the Jason here so I'm just using it I'm just overriding it now and I'm providing that particular producer factory to the Kafka templates that spring knows that okay we need to publish a custom Jason message so that's how Springwood knows what we need to do in this particular project and directly I am just doing a send on to the topic so this is the topic which I created I just showed you few minutes back I have just created this topic in the Kafka server and I'm using the publish option and I'm using the send option to send this particular Jason message on to the topic this is how you can use spring boot to publish messages on to the Kafka topic as always this particular project will be available in github for you to use also I will mention the Kafka zookeeper commands whatever I had used what you can do is you can go to the Kafka website and download the calf server or the cluster basically and you can just try it out in your machine there are batch files for the windows as well so you can try it out in Windows as well so right now I'm using the Mac version of the Linux version so you can use the Windows version and you should be able to do the same so I will give the commands for all these things do check the trail DV dot MD file which is available in the root in the github link which is available in the description I hope you like this particular video if you like the video go ahead and like it if you haven't subscribed to the channel go ahead and subscribe to it meet you again in the next video thank you very much [Music]

Channel: Tech Primers

Views: 202,811

Rating: 4.9173937 out of 5

Keywords: techprimers, tech primers, spring boot kafka, spring boot with kafka example, spring boot kafka tutorial, spring kafka tutorial, spring kafka example, spring boot with spring kafka, spring boot kafka integration, kafka example java, spring kafka java, kafka topic creation, kafka spring boot, kafka spring tutorial, distributed messaging system, distributed messaging, distributed message queues, distributed topics, kafka topics, spring boot kafka explained, kafka producer eg

Id: NjHYWEV_E_o

Length: 21min 4sec (1264 seconds)

Published: Fri May 18 2018