Getting Started: Apache Kafka + .NET

are you interested in playing around with kafka i am so i'm going to walk through setting up zookeeper and kafka using docker then i'm going to create a simple console app using the official kafka nougat package to create a consumer and producer hey everybody it's derek martin from codeopinion.com if you're new to my channel i post videos about software architecture and design and dot net so if you're interested in those topics make sure to subscribe all right the first thing i need to do is actually get zookeeper set up and zookeepers needed to run kafka because it is ultimately what is used for kafka to keep track of consumers and brokers so i'm going to be using docker so i'm on the zookeeper page here on docker hub and i'm just going to jump over a terminal and i'm just going to docker pull this latest all right now that's done i also need to pull the latest image for kafka so let's do that so here's the docker up page i'm just going to copy the docker poll all right so now that i have both uh images downloaded what i'm going to do is actually create a network that i'm going to use for both containers of zookeeper and kafka so let's do that all right so let's create a new zoopy keeper container pretty much the entire docker command that docker run command i'm using here is from their getting started guide so it looks like this so the network is gonna be the kafka one that i just created we'll just give it a name of zookeeper so this is kind of standard port 2181 and then i'm just going to bind 2181 to my docker host from that container and then what i can do is run a docker logs on zookeeper to see that's running and we can see that it's binding to 2181 all right so now we can create the kafka container gonna again use the same network and then i'm gonna bind 9092 because so what i'm doing here is i'm providing the environment variable to say that zookeeper was running on 2181 that we defined earlier and like i said for uh 1992 because there's another environment variable that we're going to tell our listeners for kafka here so so there's localhost 9092 all right now that we have it running do the same thing we can see do a docker logs and we are basically running and connecting to zookeeper all right so now that we have zookeeper and kafka running first thing i'm going to do is i created a blank.net new console app so i'm going to take the confluent kafka nuget package and i'll jump over to writer and i'm just going to add this to the project now what i'm going to do is i'm going to create i'm going to use kind of the generic host to basically configure both a consumer and a producer so the first thing i'm going to do is actually create our producer that's going to be basically publishing messages so before i even get there i'm going to use the generic host which is going to require me to also need another package this is what i'm going to use all right so now let's create our producer generic host i should say our hosted service that's going to live in the generic coast all right so we're going to implement this so basically i'm taking pretty much kind of a starter sample modified a little bit here so i'm actually going to create this create a constructor and i'm going to take the logger so we can log what we're doing here and now i'm actually going to create the producer itself so to do that we need to create our producer config and this is ultimately where we're just specifying where our bootstrap servers are and that is going to be on localhost 9092 which is what we configured when we were setting up our containers so our producer builder takes two type parameters the first one is going to be null from their particular type here the second one is going to be a string and i'm going to pass this config all right so there's our producer then what i'm going to do for kind of demo here is i'm going to just let's just iterate let's say it'll do 100 times and then i'm gonna log kind of a hello world message hence the string as the second parameter now i'm actually going to use the producer to actually um send that message so that's going to look like this and this is going to be the first thing we're going to specify is the topic i've already created a topic called demo i'll show that in a minute and here's where we're actually going to create our new message of type null string so here i'm actually just going to specify the value which is going to be from our type parameter to string and i'm just going to use our value that we're also logging and since i have a cancellation token i can also pass that along the way all right and flush make sure everything it gets delivered and then in our stop async just because this is our hosted service i can dispose the producer all right so now we have our hosted service where i've just kind of created the basics of a producer and i'm producing a message that's going to have hello world and then our incrementing number all right so lastly like i mentioned i'm going to use a generic host i need to wire that up so let's create a so it's gonna be our host builder so configure services where we're going to add our hosted service and then from here now that we're creating our host let's just run it all right there we go all right so let's give this a run and we'll take out the messages and look at the messages it's producing so there we go from 0 to 99 now what i can do is i've downloaded an app called conductor this is where it's just a visual way of looking at a ui of kafka so if i go under topics this is where i've actually created that demo topic beforehand so if i look at the topic and i look at the stream here you can see this is right now hello world 99 and everything we've basically been producing all right so for our consumer what i'm going to do is show a completely different library than the official because i uh last week noticed this tweet from ian cooper and scott hunter replied asking what package he was using for kafka and because they have a customer that is using this particular package this kafka sharp um so here's the github and what i'm going to use this package as the actual consumer all right so let's add the package reference to kafka sharp and then i'm going to jump back over to the cs proj and i'm going to add a new hosted service but this is going to be actually our consumer all right so pretty much gonna do the exact same type of thing is i will create the client and then in the start i'll actually just consume uh messages and in the stop i'll just dispose of the client so let's first create the client so we create a cluster client this requires a configuration just like we were doing with our producer to wear our 1992 instances and then we got to provide a logger so i'll just use the console logger all right so there's our cluster before i forget i will dispose of it and then in our start async is where we're actually going to start consuming messages so let's just do consume from the latest and we gotta provide the topic which when i was producing i was using demo so i'll just use the same one and then here i can specify a message received and then record is going to be the actual kafka record so at this point i'll just log what's actually coming back um this one from what i'm aware of doesn't actually do any of the deserialization for you so it's just going to come in as a bite array versus the the other package i was using does the serialization for you so this is one that's a little bit more lightweight hence why it's more performant and like i said the record there's gonna be record dot value and it says object but it's actually a byte array so what i'll do here is i'll just cast this as bite all right so let's give this a run what i'm actually going to do before that is we will add our consumer all right so we'll run this we'll actually see the consumer pick up so there's publishing everything and then we just received all the messages so we initially published uh let's go up even farther so these are publishing and then subsequently here's our message for receiving i can also take um our conductor ui again and let's go to the demo topic and i'll just produce one from here actually make this a little bit smaller and we'll see in the logs it's actually so let's add and when i send this off there was our received hello world as i sent that from conductor so if you're using kafka let me know in the comments i'd be curious to know if you're using it alongside some kind of command dispatcher or event dispatcher like brighter or the project i reviewed a few months ago called cap let me know in the comments if you're using a dispatcher with it or if you're just using the libraries directly like i was using right now thanks if you enjoyed this video give it a thumbs up and please subscribe for more software architecture related videos you

Channel: CodeOpinion

Views: 49,380

Keywords: software architecture, software design, cqrs, event sourcing, design patterns, software architect, programming, .net, .net core, asp.net, asp.net core, soa, microservices, message queues, kafka, event bus, event driven architecture, azure service bus, rabbitmq, distributed transactions, service bus, mass transit, nservicebus, message queue, message queuing, messaging patterns, service oriented architecture, microservice architecture, domain-driven design, enterprise service bus

Id: n_IQq3pze0s

Length: 17min 9sec (1029 seconds)

Published: Thu Aug 06 2020