How to Connect to Apache Kafka running in Docker(multiple scenarios)

welcome everyone it's otto here today's topic we will be discussing how we can connect to apache kafka broker that is running inside a docker container so lots of people have troubles with connecting to the broker especially if it's running in the container after watching this video you will know all the details to connect to kafka right because we will go with different scenarios which you will come across so before i jump into the docker and starting the broker i opened this server.properties that comes with kafka without docker without anything i just downloaded kafka right i have kafka here and if i go to bin not bin but config and if i go to this server.properties it will show this file and this is basically file where you can configure your kafka broker now when it comes to connecting with coco broker there are three sorts of main parts one is the listeners that is listed here and commented one the second one is advertised listeners and the third one is listener security protocol map right so we will go through these properties because these are the key in order to understand how we can connect to kafka broker now you can also read the description here basically the listener is the socket configuration for the broker that the broker will use to you know get the traffic and it's it has the url format the first one is the listener name or listener protocol name the second one is hostname and the third one is port right so for example let me actually uncomment it what would be one typical listener we can give any name in the listener name i can call it for example my listener now obviously will give more meaningful name but this is just for example we can also give the host name but if we want to like bind it to the specific network interface but typically people omit it so that it is bind to all available interfaces but we can also bind it to specific one for example localhost if we wanted to in this case i will just omit and the port let's put it 99 1992 right so this would be a typical example of a listener now what is the advertised listener right advertise listener basically in this property we tell the clients which is producers and the consumers how they can reach kafka broker right and if it's not configured it will use this list which in many cases creates the problem and let me actually open this picture here uh by the way this picture is from the book called effective kafka which i highly recommend to check it out it's really great book and it explains difference uh it explains how clients connect to kafka broker in this picture and let's assume there is some sort of private network and we have three brokers and we have one client right and those are running on different machines so first step is to connect to random broker right so this internal client should have some sort of ip address in the bootstrap list bootstrap server list to connect to at least one broker right in this case it connects to broker zero and after this connection is successful it downloads this metadata now what is in this metadata there are a lot of things but one major thing is the list of advertised listeners for this broker and for other brokers so it actually returns the list that we will configure here if we omit it it will return this list which again in many cases it creates the problems and i will show you why it creates the problems so let me tell you that each broker in the cluster has a complete view of other brokers so even though we connected to broker zero sprocket zero knows what is the advertised listener of broker one and broker two same for example broker one knows what is adore test listeners of broker zero and broker two and whatever we get in advertised listeners that address will be used for subsequent calls right so any address we specify in bootstrap server list right it's only for the initial connection for subsequent connections the clients use the address from advertised listeners that's why it's extremely important to configure this this advertised listener properly let's actually configure it now this is obviously just an example to show you a theoretical example we will get our hands dirty and i will show you in practice so what would be a typical advertised listener in this case first of all we need to advertise the listeners right so the listener name comes here again we are advertising this listener which is called my listener so obviously we have that name exact name here also and in this case let's assume um let's assume uh our client is on the same host as our broker so we'll just say localhost here and the port stays the same right let's let's assume we have this kind of setup in this case right let me let me make it smaller in this case so then this client connects to the broker and gets this property localhost 1992 now in this picture the client is on different host machine will it create a problem or not will it can it use this address for subsequent connections well it cannot use this address uh for subsequent connections because there it's running on a different host machine so localhost for this client is a different than localhost on this broker right but if they were running on the same host machine in this case this would work right and in this case even if we did not define the advertised listeners it would still work because the broker and the client is on the same host machine and we are not using any docker or any any containers in this case right so sometimes you can get get away without without with not defining advertised listeners but in many cases it creates a problem if we don't define it in a correct way and again it has this url format the name of the listener that we want to advertise the host name and the port and there is another property called listener securities protocol map where we map our custom names and this is our custom name that we can name anything we can map it to valid predefined security protocol for example i just met my listener to plain text plain text is a valid security protocol predefined i cannot change its name and it basically means that clients in this case producers and consumers they don't need to authenticate with the broker and the data between them will uh will be unencrypted right there's also other protocols like ssl for encryption sassl etc we will be using the plain text which is like uh the most basic one to simplify things okay so first glance we know that the listeners is just the socket configuration for the broker advertised listeners are how clients producers consumers will reach our broker and if we don't define the list of listeners will be returned which creates the problems in many cases and in listener security protocol map we just map our custom listener names for example my listener here to valid predefined security protocol names okay so let's get started with uh running our uh our broker in docker by the way we can configure this file server.properties if we are running kafka without docker but in our case we're running it in docker so we don't need this file so let me close this don't save and i have a docker compose file here on my desktop which we will use to run our coco broker in docker so if you don't know how to run kafka using docker compose i actually have a video about it and you can watch it watch it we basically define two main services the zookeeper and the kafka and we are also creating our custom network which we give a name of kafka docker example net and we also obviously specify this here that so our uh services so keeper insert and kafka will have we will be associated with this network okay and pay attention uh that uh in uh docker we configure the listeners with this property kafka uh underscore listeners and we configure their task listeners with this property and security protocol with this property now as you can see i have three listeners basically separated with commas you can have many listeners you separate just commas same same goes with advertised listeners and same goes with security protocol maps right so i have an internal listener here which i gave it the name of internal because in this case internal means that we want to connect to kafka broker within the same docker network right that's why i called it internal you could have named it differently listener names are are absolutely up to you to name right and i omit this host host ip address uh in all listeners which means that it will bind to all available interfaces in the docker container because this broker will be running in docker container right so let me actually run it so i have my terminal here i mean on the disk desktop folder so i will say docker compose app to start our services hit enter and it will it will start our services it will create zookeeper and kafka containers let me actually open another terminal here to check the status and i can say docker ps for short and at least running containers and we have our kafka and zookeeper running so what i want to do now is to log in in this container and show you what ports are exposed for the broker so we'll say docker exec it for interactive terminal we will specify that we want to login as root and we should specify the name of the container which is kafka and the program that we want to run which is posh okay i misspelled talker and we are in the container right so to check the ports in sockets we need some net utilities we need to install them first so we'll say net tools yum install net tools and it will download those tools from confluent repository it generally takes a couple of seconds and it's done and i can say net stat dash a n and it lists all the sockets active sockets and the ports and we can see here that the port that we defined so let me actually put this together if i may yeah so pay attention to this listener which is 1992 and we omitted the host so it bind pound to all available interfaces so these four zeros is meta address which means that it is bind to all available interfaces if we put localhost here for example something like that localhost it will only bind to localhost right and you might be asking okay what are some available interfaces in this container let's check with if config command and we see the loopback interface which is the localhost and it also has the another interface with this ip address that docker basically used this container right so it's bind to the port 1992 basically listens on both interfaces in this case okay what do you think what should we advertise for internal listener so i put this here so we're advertising internal listener and i put the address how others in the same docker network can reach this broker and if you know docker if the containers are on the same docker network they can reach each other with just container and names that's why i put it kafka here because the container name is kafka and obviously the port that socket listens on right and so if we want to start a container in the same uh network uh this address should be used so let me actually uh let me actually exit this because we don't need uh to be inside the container clear and let's start kafka console producer which will be running in a container in the same docker network as we had this broker which is this network okay so i actually have a command here that i don't want to type so i'll just copy here so we're basically running a container from this image that we use for kafka broker and we are specifying that to put this container in the same docker network as these services are right and we are executing this kafka console producer script we specify the bootstrap server list basically how we can reach kafka for initial connection again bootstrap server list is for initial connection only and we use the same address here as we had in advertised listeners and we're specifying the topic which is just any topic name the topic will be created automatically if it doesn't exist it doesn't exist in our case so it will be created so let me just hit enter and it should try and give us this greater sign and let's produce in the topic now it gives us this warning but don't pay attention to this uh it this means that the connection was successful the initial connection again let's i will bring this file again a picture so the initial connection and then we get the metadata which contains the advertise listener list in this case we connect it uh here on internal listener and we get back the advertised listeners of internal listener which is basically this address and for subsequent connections this kafka console producer as any other kafka client will use this list not the not the address that we specify in bootstrap server then it will use the address that we specify in advertised listener for that listener okay uh by the way just a side note it doesn't return all advertised listeners because we just connected to internal listener which means that it will return the address for internal listener for it will not return this address for example because we don't need it right and if we had other brokers as it is in the picture it also would return the advertised listeners for other brokers as well right also side note the listener names should be identical for all brokers in the cluster right so when i have internal listener here i also should have the same user name for other brokers just a side note okay how can we really really verify that the connection was successful and everything is good so what i will do here i will create another terminal and i will run the consumer so this is just a producer it produces to our broker and let's run the consumer and verify that we can consume the message so instead of producer let me specify the consumer here hit enter it doesn't do anything because it waits for me to produce so let's produce something and we should see it below and we see it below here let me actually produce some messages hello this is same docker network and we see that the message is consumed here now this means that our setup our advertised listener setup works as expected okay so let me actually uh close these uh consumers and producers let me actually remove this if i may but for some reason it's stuck uh okay not this yeah let's remove it now i want to do some one experiment here also i want to change this address and let's put it [Music] kafka1993 will we experience the problems or not i think you should also already know the answer that we will experience some problems so let me stop this docker compose and clean everything so we start fresh again i will start our services again because i changed the compose file and i need to restart clear check the containers and for some reason our consumer container is not dead it's okay i will just uh remove it okay container remove dash f oh dash f is wrong here no problem so we removed our console container which should be removed um and we have our kafka zookeeper and we changed uh these from 1992 to 1993 so let me start our uh producer once again and i specify the bootstrap list as 1992 because the socket for the broker listens on 1992 and the name of the container is kafka so initial connection should be fine initial connection should be filed hit enter it shows this uh sign creator sign and if i try to for example produce something it gives me warnings it doesn't stop why is that because after the initial connection is successful and we didn't see any errors right until i started to produce for subsequent calls it uses this address and since there is nothing on port 1993 right we have 1992 here it cannot connect to this broker it could connect initially with bootstrap list but after after this initial connection after this initial connection we got back this address kafka 1993 which our console producer will use and there is nothing on 1993. therefore it just spams the warnings that it cannot connect so we know that this address that we specify here really matters all right we saw how to connect to the broker from the same docker network we basically expose our broker we reach our broker with just a container name here which in this case is kafka and by the way in security protocol map i map this internal protocol to plain text as well as other listeners i map all of them to plain text for simplicity right now how can we connect this broker from just host uh host machine right without any docker let's say our client we have console producer that is not running in the docker it's just running on our host machine how do we connect for that i created another listener i named it external same host and i omitted the interface so it will bind to all available interfaces and i use different ports 29092 typically for different listeners you use different ports and there is also advertised listeners obviously for that we are advertising our listener which is external same host and for hostname we put lockhost basically we are saying to our client that will be running on the host machine to use the local host and use this port obviously this port which is 29092 should be accessible outside docker so that's why i i published this port here now this is not something kafka specific this is docker that if you want to access the port in the container you need to publish it so i want to access this port inside the container because the broker's socket one of the sockets will be listening on this port and i want to expose it outside docker and that's why i list these ports here especially when i go to localhost29092 it will redirect traffic to our our broker in the container and for this listener i also map it to plain text so let's actually run uh not run but to try to connect i i obviously have this running our kafka i didn't stop it so i have kafka and zookeeper now i need the console producer uh on my host machine on my laptop right so for that i actually go to the kafka that i downloaded and i will go to bin and i will have my kafka console producer here so i need to cd here cd here great and we need to specify kafka console producer so kafka console producer dot ch i think we can omit it but let's put it here bootstrap list bootstrap server which is basically the same as we specify here topic let's say test topic two you can name it anything it will be created automatically and let's try oh it's not recognized okay let's let's omit sh yeah and i think that we can uh we can produce something so let's say hello and use these warnings but it's not it doesn't mean that we couldn't connect um and we can also start as we did before we can also start the consumer so i will start the consumer here let's do this and i will say i will basically copy this command uh which is kafka console producer but instead of producer i will use consumer consumer hit enter and let's produce something for example hello from the host and we see that it's printed here which means that the initial connection and subsequent connections where this address is used uh was successful meaning that we have advertised listeners configured correctly right so let me stop this consumer and producer and this also now there is also another scenario that will be useful i think when kafka broker is running on external external uh server in this case we will be running this broker on digitalocean so i actually have a digital ocean server here with this ip address and i have kafka running there now for that i created again another listener so for these different situations you typically create different listeners and i i called it external different host right and for port i choose 29 0 93 and obviously i advertise it here and what do i need to use here i cannot use localhost for the host name because it is running on a different machine in its it's running in cloud and localhost for cloud machine is different than localhost for my laptop right so i need to specify the ip address here which is this ip address basically and the port and don't forget that we also publish that ports this is purely for docker that this port is reachable outside docker docker container this is just docker not not related to kafka so let's actually try to produce something here uh for that i will start the producer but instead of localhost i need to specify this address right this address and the port is three let's call it topic test three so i will uh start and i also start the consumer right away basically i will copy this and let me paste it here and topic three started consumer producer hello from remote server and we see the same message here which means that uh connection initial connection with bootstrap server list and subsequent connections where this address is returned was successful meaning that our advertised listeners are configured correctly uh just a side note in typical scenarios you do not hard code the ip address because ip addresses might change all the time people typically use the domain names like you can for example have something uh broker xero my company my company.com something like that because it's very easy to update like dns names to point to valid uh ip address uh then hard coding ip address because ip addresses especially in the cloud and especially if you're using docker and kubernetes they change all the time so it's better to use the dns names instead of hard-coding ip addresses okay let's let's quickly summarize what we learned and what we did we understood that uh in order to properly connect to kafka we need to understand what is a listener what is advertised listener and what is a security protocol map we saw that the listener is just the socket configuration for the broker for incoming tcp connections we also learned that advertised listeners are basically a way to tell the clients producers or consumers how they can reach kafka brokers and the security protocol map in secured protocol map we just map our custom listener names to valid predefined security names right the listeners and advertised listeners have url format where we specify the listener name first then we specify the hostname and then we specify the port we also discussed three cases of connectivity with kafka broker firstly we saw how we could connect to kafka broker from the same docker network and we basically said that we need to use the uh the containers coco containers name because this is how docker network works that you can reach other containers by the their container names we also saw how we could connect from the same host when client is running on the same host without any docker we just use localhost in this case and different port and finally we saw uh how we could connect to external kafka broker which is running in the cloud for example and we used uh the ip address of the server and typically people use the dns name because it's more it's easier to update dns names and we used different port so we had three listeners we advertised the same three listeners and each listener has different ports this was all for now thanks for watching and i'll see you in the next

Channel: Otto Codes

Views: 15,801

Keywords: Apache Kafka, Kafka, Kafka Docker, Connect to Kafka, Kafka Connection, Broker Connect

Id: L--VuzFiYrM

Length: 33min 9sec (1989 seconds)

Published: Sun Sep 12 2021