Mastering Kafka E2E Integration Testing in Spring Boot with TestContainers | JavaTechie

hi everyone welcome to Java techy if you remember in the last Kafka session we have designed our Kafka producer and consumer application right so in this tutorial I will guide you how to write Kafka integration test with test containers okay all right being a developer always writing a Kafka integration test can be frustrating because that's mainly due to the complex test configuration that involves for registering the consumer and producer or to read and write messages isn't it and also without proper integration testing you cannot be confident about the stability of your production environment so to leverage that we can use the test containers which can reduce test configuration to almost zero and not worry about how to read and write messages from one to Kafka when writing a test case instead a developer can entirely focus on only testing needed functionality okay so if you are not aware about what is test containers and how to use it then I would strongly suggest you to check out this particular video in my YouTube channel spring boot 3 integration testing with test container where I have explained in detail by taking the example of MySQL using the test containers I will also share the link in the video description for your reference okay so without any further delay let's Circle back to the demo so let's get started so before we start writing the test case first let's verify whether our producer and consumer example application is working or not so I have started my producer application and also I have started my consumer application here and if You observe here in the terminal I have started my zookeeper and Kafka server okay that's enough now if we'll go to the producer app in the producer application we have defined the control load from this control load method we are sending the customer object to the topic fine so the endpoint is slash publish let's go to the postman and I have this particular endpoint with this particular payload now just trigger the request status code is 200 now let's verify in the consumer whether that particular object is received or not yeah can you see here consumer consume the event this this is what we are sending from our producer 101 name is basant email ID and contact number so all good now let's start writing the test case for that I don't want to use my Kafka which I installed on my machine so I will stop everything I will stop my consumer and producer as well and from the terminal I will close my zookeeper and Kafka server because I am not going to use this particular Kafka instance I want to take the help from Kafka test containers to spin up a new cup instance for me so I'll show you that how we can do that so let me close everything now the local cup guys are done for me so what I will do very simple just go to the palm.xml in palm.xml we just need to add the test container specific dependency if I will open it so I have added the test container dependency then test container for Kafka and I have added the Jupiter and also I have added this utility to just pause for some time I mean whenever I am running the Kafka producer and consumer I want my thread will pass for some specific second or for some time interval that is the reason I have just added this utility okay I will tell you in the code whenever we will implement this that's fine then just update your project then what I want to do I want to disable this config class because I want to load it from the application.yml file so I will just uncomment this that's it just go to the test class and start writing your test case now here okay let me close everything so first step we can just run this particular test case in random Port okay so I'll Define web environment there is something called random port then I just let me Zoom this for you then I just need to annotate here at the red test containers by adding this annotation I am telling to the spring boot test that this particular class is using some container fine now which container I want to use I want to use the Kafka container so for Kafka container from where you will get this Kafka container just go to this documentation you can copy this and then then simply just paste it here we are using the Kafka container by saying to the Kafka container hey container can you please bring this specific Kafka version for me 6.2.1 if you don't want to specify any Kafka version just tell that hey container can you give me the latest Kafka version which you have with you that's fine you can Define the version whatever you want here okay now annotate this at the red container next since we have created the Kafka container we need to configure the bootstrap server so for that I will simply write a method public let's say init Kafka Properties or something like that then you need to just pass your Dynamic Property registry and also here you need to annotate Dynamic Property source this annotation came from the this specific test containers okay so you no need to remember this annotation name but whenever you are using the test container you need to play with this annotation fine now registry dot add key name is spring.com I just need to copy the same name what I have defined here so this is right so you can Define then bootstrap servers and what is the bootstrap server you want to specify I don't know exactly the bootstrap server name because this particular container will get the server for me every time you will run you will get different server okay so what we can do we can tell here hey get it from the Kafka container itself this is how we can specify fine so we are done with the infrastructure setup now let's start writing our actual test case so let's go to the project we are going to write the test case for this guy okay this Kafka message publisher we want to send the custom object which is customer this particular method so let me copy this method name then I will simply write the test method public test fine so to call this specific method from the Kafka message publisher first I need to inject this so just go here and I just need to automate that private then just do the auto add that's it now I just need to call Publisher Dot send events to the topic there are two method we have defined in that publisher one will send the string message one will send the custom object where we have just defined the customer as a detu and we are sending it so I am using the second method I mean wherever I am sending the custom object so I will create a new customer new customer but all field I need to pass ID name email and contact number so let's say ID something some random number name let's say test user email I guess email is test at the rate gmail.com then what else contact number some random number okay next just annotate this at the rate test make sure to input the test from Jupiter not from the junit once you have written this this sent even to the topic will publish the message to the topic and after that I want to pause for some time interval so that is the reason we have used this utility class so let me add the input statement this should be import from org dot this particular class okay and I want to wait the pool interval is 3 second and minimum I want to wait for 10 seconds and next to that if you have any asset statement you can add it inside this particular block so just import this so we are specifying the pool interval of 3 second and at most I mean the minimum amount I want to wait for 10 second once this particular message will push to the topic so after that if you have any asset statement you can write it here since my method is but I don't have any asset statement I cannot write anything here even you can give a try by persisting that particular customer object to the DV and you can just write the asset statement here by fetching it from the DB so this is a simple example guys okay nothing to worry now I believe all good let's run our code so let me run the test case okay it seems there is error this particular method must be static okay let's change it to the static so if you'll make the static better let's change this particular field also static now looks good let me run this we'll find out some error then I'll tell you how we can resolve that so if You observe here when the particular test will execute first it will get that particular Kappa image can you see here starting to pull the image what is the version Let test now it will take few second to pull the docker images so you can see here it successfully pulled the image of this specific version I mean the latest version then it's trying to start that particular container this is what right now to verify whether it pull the specific Docker image or not what you can do go to the terminal just type docker images If You observe your Chemistry here CP Cup and the tag is latest now if you want to verify whether that is the container itself is running or not just run Docker PS you can see here right the Kafka container is running here let me Zoom this for you the container is running here and this is the docker image it pulled fine so go back here let's see what is the result so if You observe here it's giving us the error because this particular test case will look into your application.2 ml file which you have defined in your Source folder I mean in the main folder not in the test folder there you have specified the bootstrap server as a 9092 localhost but our bootstrap server is running dynamically fine so for that reason you need to override this application.yml in your test folder so just create a directory resource just paste this and remove the bootstrap server so that it will pick from the container and will run it Dynamic bootstrap I mean the bootstrap server will be dynamic here so this is not required for test you can remove this fine now go to the test class and run it again now second time when I am running it it don't pull the image again because it is already pulled that specific image at first time fine so let's wait it to complete see it directly pull the image here and restarting the container can you see here we got the message here sent message this is what the object we are sending right with offset zero now if you go and check in the publisher class after successfully send the messages we are just printing the statement system dot out dot println sent message this is what right if there is a exception then it will print this particular statement since in our case we are able to successfully send the messages we are able to get the first statement that sent message to the particular message unto which upset that is what we can see here right if you go here to the test case get the result go to this specific test case now if we'll scroll down you can see the log messages here right this is what we have written for producer now since I am a producer I might have 100 of consumers right so now consumers people need to write their own test case for Consumer class okay so let's start begin writing the test case for Consumer we are good with the producer now if You observe in the code I have not worried more about the infrastructure I have just defined these two line everything will be taken by this Kafka container itself what I focused I only focused on writing my functionality test whether this particular method where I am exactly publishing the event is working or not okay now let's start writing the test case for Consumer application so I need this information so first of all first let me copy all the dependents what I have added here because same dependence you need to add in our consumer as well copy this go to the consumer application go to the palm.xml then just scroll down and paste it here just update it fine now go to the test case here this is where the test case now let me copy all the field from the producer I mean we are going to reuse the same annotation and Kafka container so I will just copy paste just go to the consumer paste it here go to the producer get the container and Dynamic Property source copy this then paste it here I mean whoever have the consumer for your application they need to test their application that whether they are able to successfully face the events from that specific topic or not okay so that is the reason they need to write the test case so for that how they can validate that whether the message is coming to the topic or not so forcefully you need to send some event or messages to the topic then in consumer itself you need to write the logic to the or you need to validate that whether it is consuming or not okay that is simple statement guys so let me write the test case public first let me go to The Listener class right this is the class where my consumer is written here I mean this is the place I am consuming the event from this specific topic so that is the reason we have defined the other capitalism so I'll just take the method name go to the test class public board test so to validate whether this particular method is being consumed or not so what I'll do forcefully I need to send some message to this topic so for that what I need I need a cup template so I just need to inject here Kafka template of type string as a key and value as a object just inject using Auto add so here what I will do I will just send Kafka template dot send object to which topic the same topic what you have defined here because I only forcefully sends a message to this topic to validate okay this consumer is working correctly just specify the topic and just Define the customer object so I will better copy the customer object from here itself just paste it here or I'll just Define above okay then just pass this customer object to this Kafka template so that it will publish the message to this topic and this particular listener need to execute and it will print this log statement that is what we are expecting fine so what I will do go to the test first I'll do one thing I will just add some log statement here sl4j then I'll just add log dot info let's say test consume events method execution started something like that and also I will just add execution ended so execution will be start then we are building the customer object and through Kafka template we are sending it to the the specific topic fine now once it will publish to the to this specific topic this listener will read that messages and it will print the log statement once everything done we are just printing that okay the execution is ended and you can also annotate your other test and make sure to input it from the Jupiter okay this is as much as simple fine now here also what I want to do I want to enable the application.yml I mean I just want to uncomment I want to use this application.yml and I want to comment this config class again you can try in both the way but let's go with the application.aml so that if any test configuration required I can add directly the yml file here rather than Define the config class now here also I need to define the resource folder new directory resources and just add the application.aml file fine so make sure you need to define the same group in your test case so let's go here I mean in not in the test case this particular Kafka template will publish the message to this topic now if you go and check in The Listener the topic name is correct and the group name need to be this okay I mean it is same only that is how I have configured in yml file so go to the test case all good now let me run this so it started the container here can you see here now it will start executing our test case so this test is executed successfully now if you filter this we have not received the I mean the request is not goes to my consumer when if you will see here not here to go to the main one we'll find some error right what is common producer is closed forcefully so for that what you can do you need to wait for some time I mean the way we have done in the producer similarly you need to wait for some second because it might take some time to establish the connection right or it might take some time to connect to the container so it's good to keep some buffer uh time interval for sleep fine now let's run it just run from here restarting the container now so it started our application now can you see here let's put it to complete yeah within a second it will be complete now if we'll scroll down The Jetty group partition assigned to this particular topic partition I mean the partition count is zero and this is a topic name and this is the group name and can you see here the messages let me Zoom this consumer consume the event this is what the email and contact number this is the object we are publishing through the Kafka template and we have defined the consumer who listened to that particular topic and my consumer is able to consume it so if You observe here go to the consumer the statement we have written is same right consumer consume the event then what is the event it is getting we are just converting it to the string the same statement we are able to see it I mean we are able to successfully consume the event from our test case and using the test containers so if You observe here again I didn't take much headache to Define my infrastructure I just Define these two statement and rest everything will be Take Care by this Kafka container itself I mean the test container itself okay so the another Advantage is here in this particular approach the Kafka Docker container can be kept alive which can help with the test performance and results in a testing environment closer to the production one so this particular tutorial will give you the fully context how you can write the integration test for Kafka using the test containers whenever you will try to apply this specific use case to your real life situation you can take it further by adding more test cases you could also start refactoring your app or adding new features more reliable with the test that you have written okay so do let me know in a comment section if you guys have any doubts that's all about this particular video guys thanks for watching this video meet you soon with A New Concept

Channel: Java Techie

Views: 16,695

Keywords: Spring Boot Kafka, Kafka Integration Testing, TestContainers, Kafka Testing, Spring Boot Testing, Test Containers Tutorial, Spring Boot Integration Testing, Testing Kafka in Spring Boot, Kafka Docker Container, Integration Test Setup, javatechie, kafka

Id: UvTXArpgrtY

Length: 22min 55sec (1375 seconds)

Published: Sat Sep 09 2023