Run Kafka Consumer in GoLang - From Scratch

so basically in this video I will cover how to using golang language and you know connect with the Kafka and read messages so let's get it started so it's pretty straightforward on the ceiling goal and as an IDE just create a sample project and clearing a go project is really really really simply that it's just create with may not go as an empty file all right because it's my main file I'm gonna rename the package 2 min and just get a main function and okay that's pretty it that's it it's working at least the project that it is done correctly so alright this is that easy okay now we have created our main project let's see how we can write a Kafka consumer and its messages so let's create a directory Kafka and let's create a profile that's they need to talk convict all right so this is a convict all right function Kafka alright now I'm going to define some kind of a configuration for Kafka so actually cook up library comes very handy so let's import that fast a cough cough yeah ok it goes alright let's organize that and that's for the format also because we want to print some statements okay so Kafka dot reader config and let's define some configurations so the first one is like the brokers so I'll be planning to run Kafka in my local so let's put some broke a string so this will be like a strain and basically most and ninety-ninth is a default over to the idle listen alright the topic topic name let's give some topic name that's a topic and let's give some group ID okay not even should be okay okay this one imax white is really don't want to wait for a longer time so just give it a very small back Speights so which means once this is over this will read that message so let's assign it to a variable this is of type okay so this is a reader config has been created all gates come complaining because it's not been you so we will use that as well now again Kafka wrote the reader provides contact pilot call it a reader this is a tie okay so this is where you have created a reader now you can actually start an infinite for loop to basically read from dot read message and then provide the context object within port act to read that message so this is this will give you a message or error in case of error so let's hold them into these variables and if in case of error if it is not know no equals to nil just print something for now that should be good so format dot error okay let's print the error as well and you continue because you don't break Conte you want to break consumer just because one message has caused some error all right now here you can print some message so let's say fmt dot println message is and then you print the color in a string because by default you just spin a message should be some bytes all right so your start Kafka is all set which will read to this topic on this group and immediately but very few bytes it will like you know it will be ready to read that message and this infinity for loop will make sure that you know this broker this consumer is always up and running and as and when you get these messages you just just print that message okay now the only thing left is basically trigger it from the main point so let's call this or that simple that fun fast k-dog slash Kafka and I'm gonna call it as a Kafka okay Kafka yep start Kafka all right FM t dot println of karl has being restarted okay now I don't want to exit my program so I'm gonna sleep for some time so let's say ten minutes more than sufficient time don't mean it I like you all said and there's a little problem when the main thread executes this will get it stuck here so because it is going in a for loop so we need to call some data in a separate thread so I'm gonna write it go which will start as a separate routine the the go routine it will be called so this will make sure that you know it is running separately without affecting the execution of this code so this code is ready the only thing left is basically to start Kafka and that's I'm gonna the main focus of this video is basically to focus on the go Kafka consumer so I'm not gonna cover much on how to start a CAFTA but I have few scripts already ready which can do its job so cough game is starting is a talker container this is an alias for me so it's like CNS like docker PS okay I'm gonna start a Kafka which will actually start my Kafka broker's okay alright so I'm gonna watch my containers so you see just these containers have been started they really take a little while to basically get up and running okay let's wait for a few seconds mm-hmm okay all right it looks good so these I'm finding the producer and consumer scripts they come quite handy for me because I can just copy and paste and play around with them okay so this is my first message so this on the left side is starting a Kafka producer and the right side sack of a consumer and whatever I'm gonna print here from the console producer should be available in May Kafka conservation shows my brokers Kafka brokers are running perfectly fine all right let's go back to the code all right let's run that okay now you see guys this has printed this okay this is also started Kafka and this is also printing okay no not this one I need to start actually this one Kafka has been started oh yeah sorry they're spending here so basically this is coming here started coffe kinda Shepard said I'm gonna clear this console so clear buffer and what am I gonna do is here this is my second message and they should get painted here when I press Enter yeah so here you go okay and then you keep sending message message one message to play message four and so on okay you see all these are getting printed so that's pretty straightforward guys go lying in itself is very cool the integration of go with Kafka consumer and producer is like very very straightforward thank you

Channel: Sachin Shukla

Views: 6,388

Rating: 4.9215684 out of 5

Keywords:

Id: Ur4ZtuMpj14

Length: 9min 0sec (540 seconds)

Published: Sun May 03 2020