Mastering Testcontainers for Better Integration Tests

foreign testing is all about testing how well the different parts of your code work together I don't know testing these scenarios a bit is said than done because it is difficult it is not easy to simulate the environment for integration testing setting up databases with data as real as possible simple dependencies such as message views servers or even uid so do it when things are difficult we will drop that part of himself I'll barely do them so if you are either of these categories I'd recommend you try out text containers which help you simulate your autistic climate and take the food without all the pain is this High chances that you have more questions that you will see answers to like how to debug your contain including complexes if you're looking through solutions to simpler questions more this session is for you hi everyone thanks for joining in for another idea live stream I'm your host malagupta and now let me add our amazing speaker to the Stream hi unique uh it's a pleasure to have you present with us today hi it's a pleasure to be here honored to be invited uh now let me introduce also works as a developer Advocate with creator of the skin and he also colleagues the virtual Doug before we start the session and I let you know what is your favorite feature on test containers favorite feature of test containers that that's a good question uh let me answer it with the with the question as well what is your favorite programming language model because we have libraries for test containers in different ones like.net golang um um let me guess dotnet uh no Java because we are in a travel issue yes yes absolutely we love all the languages to be honest um and both in the test containers community and at Atomic jar but I'm very happy that you said Java because my background is mostly from the Java ecosystem so I would be way less confident in talking about task Engineers for go besides like the the sort of the high level basics and and examples I think for uh for the Java version I have to the absolutely key thing is the ecosystem of modules which have different Technologies sort of pre-packaged for you to run with test containers uh and if we're talking about a particular technology I think that would be the toxic proxy module and we're going to show a little example but in in short you can build complex topologies of the containers that you want to run uh with your tests and then you can use toxic proxy to kind of sort of proxy and introduce various Network effects to to do like case engineering things so like cutting connection to the database middle in the middle of the test and like checking whether your application survives that a lot or introducing latency like that's that's a very fun uh type of testing and and I think that is my absolute favorite uh that makes a lot of sense and I'm glad you mentioned Java because what else could we expect from a Java Champion right I don't know I don't know Java is my bread and butter so this is good me too right and um I know that we are talking about the advanced features of test containers in this session so anyone who is watching this session and they want to get started with test containers is there any session of yours or any other resource that you would like them to refer to uh well we in the session we're going to go through a little bit of Basics as well very quickly okay we're gonna so I think even if you are not particularly experienced with test containers you can get up to speed here very very very much and very well um and it's just a matter of I know uh trying it out on your projects that's that's how you learn the best uh yeah but I'll also we can we can see other other conferences live streams and maybe resources uh I can think about it and then give you the links that makes sense I just wanted to address this question because uh a lot I would assume that a lot of viewers now might not have any experience working with test containers and it could be kind of a showstop for them to kind of jump on to the uh Advanced features without knowing the basics so thank you so much for answering that part and before I let you take uh the the floor let me share some quick housekeeping details everyone who's watching please use the YouTube chat to post your questions Olek will take breaks uh while presenting so that he can answer the questions and we'll also try to answer your questions from the jet planes team uh the session is being recorded and will be hosted on intelligent ideas YouTube channel so if you haven't already subscribed to our Channel now is the time to do so and if you like today's session which I'm sure you would please like the video and subscribe to the channel did I repeat that okay um so um or like the floor is all yours and I'll be backstage in um where is your stream yes it's already here so okay have fun thank you very much thank you very much uh we gonna try to have fun uh just if the stream disconnects and it happens that I'm just seeing here in my room yelling at the computer for no reason then please tell me interrupt me because I I might miss that moment uh but that will not happen it's all well tested we all know that if it works in the test it will work in production and so on right we're going to talk about test containers um and I work as the developer Advocate at the company called Atomic jar which was created by the maintainers of the task containers Java last year and uh we we try to support well we not try we do support the task containers Community to the best of our ability and if you have questions and you're watching this in the recording and you don't get to ask them live here in the chat just reach out to me online uh I feel like I can I'm old and wise now uh recently became old and wise and I can I can give advice I can give advice but not just about test containers but internal testing software development uh I I'd love to chat with you about uh what you think so let's talk about tests though tests are essentially in software engineering in our day-to-day life tests are sort of the answer to everything and you might not believe this uh or you might be not fully appreciate this but it truly is right all the normal software development question am I making progress towards creating a feature do you have tests are they test passing more than they were passing last hour good like when you develop something like are you breaking somebody else's code right can you merge things well run the test like do tests pass then you're good if tests fail then probably you shouldn't shouldn't be merging things right um so it's just literally the the only question that matters and of course that's not uh a hundred percent true but a lot a lot of a lot of questions actually are answered by having tests and running them uh because sort of if you even if you don't do the devops methodology right if you just uh sort of work in the like smaller teams that where they have the operations separate from development uh where you're working on things like libraries where there's not so much monitoring uh and operating but like you just throw the jar into the maven Central something like this uh still tests are on the critical path from typing in the code to actually giving that code to your users and you cannot avoid that so if any of the any of the phases becomes a bottleneck then you are struggling and your whole cycle is becoming slower and that is very very often not desirable so tests are very important uh hopefully everyone agrees so in the past what we did we imagined that there are different types of tests and we would like sort of cover most of our code with unit tests and then we'll be the integration tests on top of that and unit test will run everything in isolation as small units of code as possible and then they will be quick they will be fast we're going to cover all our logic with them and then integration tests will will be slower and harder to maintain and will have less of them and then end-to-end tests which run in the environment similar to like production they will be like even fewer of those and for a long long time this like Paradigm sort of worked but in the recent years specifically with the migration towards small applications microservices and so on we started and we as a community here different teams started to rethink how they do testing and uh the there are a number of Articles how people kind of organize their testing but the main concept sort of is that integration tests are rising in popularity in importance because the smaller your application is the more it depends on its environment and how it runs against the the certain Technologies like maybe Cloud Technologies or databases or um it always needs like a message broker for its for working so integration tests that can run in the in the environment similar to production uh become more and more important because maybe your application just passes the the data back and forth which a lot of microservices do and then there is not so much to test with linear tests so those higher level uh more sort of functional tests uh become more and more important and you still have the unit test implementation detail test and you still have the integrated tests which run in the environment similar to production sort of like staging environment maybe or or like a canary deployment if you if you actually test in production but the bulk of the decision whether the code that you have is good to be rolled out or not lies on the integration test sheet and and that's where the test containers comes in test containers is the collection of libraries in different languages to uh that integrate with Docker environments and allow you to run various Technologies like databases message Brokers uh headless browsers anything that runs in Docker it allows it you to run in a Docker container and on the other end it you can use that it has connectors gives you the programmatic API to to manage the life cycle and configuration of those services so what you can do you can create you can make your tests create the environment for the integration testing that the tests require and because you have the flexible programmatic API you as a developer hold the key to the success of those tests right you don't need to ask your platform team to kind of provision your instances of the services you don't need to install it locally on your machine uh you kinda you can have your tasks consistently across all the environments where you run tests which means all your colleagues anyone who clones a project your CI servers test containers has been and it has seen an incredible growth lately uh earlier this year uh the technology Raider uh by thoughtworks put test containers into the adopt category saying that oh this is a great default option and uh implying that you should have sort of a you should understand what you're doing and have a good reason to not use test containers uh for your project which is which is very very good uh how does it work test containers requires a Docker environment to this is where we actually run the containers and that could be any Docker environment compatible API environment so it could be Docker desktop that you run locally it could be a Docker machine which is how we previously ran Docker on uh Mac OS and windows which is a deprecated project now but it still works it could be your favorite kubernetes cluster exposing Docker API or it could be any uh compatible Docker API environment so at Atomic jar we are building the the cloud backend for running containers during task and their tests which is the called test containers and you can just come and join uh come and join the test containers.cloud you can look at the benefits and so on I want to go very deeply into the details uh but yeah you can lean on the cloud to run the containers and so on uh Docker is Docker is great why do we do Docker why don't we for example do something else is because we rely on it to isolate the processes and run all the software that we want in in any local system right or in any environment where we need to run tests so you can do the processes at isolation you can specify the configuration for the file system or which I don't know if your container runs a database you can specify what schema you want you can put that memory limits and and everything and it's a a very good platform which is just like Java it is sort of available everywhere it is available with the consistent user experience on all necessary platforms um and what it's more important almost all software that you run can be run in Docker right like you have the images to run all the databases all the um message Brokers your Kafka your hive mq your kubernetes clusters your uh I know everything mongodb or any redis or anything you can also run your services right if you have your internal services that your test depend on you can also run them in Docker because you can package them into images and run the containers out of those but Docker API using the command line directly or using the configuration options that Docker provides out of the box is a little bit clunky specifically for the integration test because during the test we want to do today we want to put our application in the environment into the sort of H cases right that's that's what they tests are for right we want to check what will happen when the database schema is this in this this way or when your Kafka topic has those messages like do we process that the information correctly or not and for that sometimes we need more flexibility in configuration that uh the kind of sort of out of the box none API Docker Integrations can provide right so test containers does a few things it gives you the API to do the container lifecycle and cleanup to container both container and service configuration and it gives the integration with Frameworks or your test libraries and if we talk about specifically task containers for Java then it's it's a mature project it's very very old it's not like that old in Java years right so what is Java 2027 27 coming 27th which is ridiculously old I feel I feel I feel like it's almost as old as me but it's in the in the darker years task containers Java is very very uh mature Docker is barely older than the project itself and of course does continuous libraries are open source projects uh they sit on GitHub under the test containers organization uh and they they use Docker API so we don't they don't rely on the particular common line uh tooling uh and then they integrate with the Frameworks in the respective ecosystems and languages so for Java that is integration with J unit or Spock integration with spring and quarkus and Micronaut and so on uh so you can use it you can use it without those two tools you can use the API directly uh but there is also Integrations right and there is the whole library of uh which I mentioned in the beginning right the real Library there is ecosystem of libraries where the configuration for the particular Technologies is already created for you so you don't have to fully understand how to run like subtle details of how to run Kafka in a Docker container or how to run uh your kubernetes in the docker container or whether that is even possible or what exactly needs to be done you can just use your uh programmatic API add a dependency into your project and uh click uh click the the the dot and the Alt Enter in in your IDE a uh and then go go right it will write the code for you haven't tried maybe if I like maybe you can even ask the the AI to write things for you and it will respond meaningful though personally I would not recommend that if I would choose oh this is an interesting question and it's completely off topic but I feel it's very very modern and very interesting so 20 seconds of attention if personally if I would choose which part of the code to generate whether to generate the code and write the test myself or write the code and generate the tests I would choose which is to write the test myself because they verify I need to be certain that uh that the things that the things we're writing uh or generating they actually do the the thing or that they want to do right writing the code and generating the test sounds like a counterproductive uh measure but this is I don't know I haven't thought about this for enough time to for this to be like an important takeaway this is just uh yeah the Alt Enter comment uh made me think about that right um so sorry about that the module of the ecosystem of modules which is great and the other thing is as uh I mentioned earlier test containers is not just the Java Library there are other implementations in the popular uh programming languages uh so like things like jvm languages like kotlin for example can easily use test containers Java because there is a native uh interoperability between those um but uh for the other languages like golang or net languages or rust there is test considers python so you can use the same approaches of integrating with the docker environments and creating environment for your tests FML environments for your tests on the Fly they are available in your in your favorite languages and they're all open source projects so if you if you use particular language just go or you have friends I know this is Java oriented stream so we have friends or colleagues who are using another language there's continues.net for your uh JavaScript and typescript developers so you can go to them and say like oh look this is a great library in Java we're using it all the time uh and I just learned from Oleg that there is Task containers go maybe you should go check it out and it will be like oh my God this is the best early winter's holiday present ever uh and they will be grateful and they will bring you uh Beverages and things right so this is sort of the introduction very quick introduction I'm gonna go now into my ID and we're gonna try and see what the tests are and how they work and what are the what are the common patterns and what are some features that you might not have heard of uh and how to do how to use that effectively right uh do I have any questions in the chat I have a bunch of comments uh but okay but if if we have questions right now I think this is a good idea to ask them mother um I think we do have a couple of questions but they are more as you said they're more like comments uh just one question um okay so there's a question by Yuri you can see that on your screen now taking to account a Docker license doesn't allow to use it in any companies is there any choice to use test containers without Dockers so the question is probably about the alternates would you like to address this question now or yeah yes yes of course I can so test containers requires to you to have a a Docker compatibility like API available somewhere so that means that um you can you can use any Docker implementation and I think the question refers to the changes to the docker desktop licensing that came earlier this year I think in January which feels like an eternity go unfortunately but um it like it doesn't have to be your Docker desktop client for example it could be task containers cloud or it could be your any Docker implementation available where if you're running it on Linux or in UCI then you can run the same test and using the docker uh demon directly which is uh which which is an open source thing uh which doesn't include the the doesn't you don't need the license the paid license for that uh or you can use uh Docker from minicube or you can use kolima uh or Venture desktop there are a number of opportunities a number of projects which provide to you with the docker compatible uh environment so we just need the API because we want to have the direct control and saying like imperatively say create this network run this container pull that image configure this with those files like that so A Very imperative approach to giving commands to this API or clean up cleaning up the containers which is a big thing uh we we we want to say be able to say like all the containers that were started from this session please remove those uh and this is something that uh that use the docker API uh so I don't think other targets uh are well in the the in the active development at least so uh yeah but there are a number of options right that was a long list so the others are more of common so I'll let you move forward with your session yes thank you very much so what I have here is a very simple application it's a spring application it consists of uh of uh a controller and the the business logic of this application is fairly uh trivial it takes in HTTP requests and then uh it sort of deals with assigning ratings to sessions at conferences but for our purposes what's important is that it uses Kafka via the Kafka template and the ratings repository here will use redis templates so it means that we'll need to have redis to run this and the talks repository will talk to our normal relational database using the jdbc template so we need those three things to run this application uh uh and that environment needs to be there so uh uh what else I have I have Mala would be like you are doing a little too much too much clicking sorry I think it's just for the demos I usually do this let's remove that and what I have here is in the test directory I have a collection of tests there are some that are fairly straightforward it's just the context load test which checks whether my application wires it together uh and that the uh the data is exists in the sample database there is the check that our actuator for spring application actually uses and returns 200 and we are using the rest assured Library which is very very cool uh and it gives us the fluent API to create HTTP requests so those those requests that go through the HTTP they are doing actual request generation and it actually send the HTTP request there so we do that there is the test ratings uh check which actually sends the data to the application and awaits until it comes out from the other side so that's that technically through the application business logic means that it do it does the whole round trip into the database out of the database into Kafka to redis everything and then you respond back we will use availability which is another great library for dealing with any asynchronous events we use that internally in test containers Java quite a bit for waiting for various things like for example when you start the container and in the container you start some service say you start you start the database where you start your uh IntelliJ IDEA backend uh and then you need to wait for it to become ready right so database to accept uh become uh like get to the point where it can accept the queries or where the indexes are downloaded and applied uh for your ID you need to wait for that and only then return the control to the tests so there is this as in quantity that you need to work with and availability is a great library for the tests and we use that internally to wait for like container started events and so on so we do the the checks for the test we also test that there is the negative use cases which very often people sort of forget in their test Suites or not only just happy past but uh we're better than that in our sample projects here and we test the negative workflows as well so this is demo project of course I can run those tests uh they they will fail currently because there is no environment that I can run if I if I wait I don't want to look there where they want to look if I if I go into my terminal right and if I do drps uh it gives me I think it should tell me that my Locker docker Ocker is down yes right we'll see um right oh this is getting this is getting intense right we're gonna there are no containers running uh there are no containers running uh we can create that environment for ourselves we're going to use this after configuration test class here we're going to specify the containers that we want uh and we're gonna do three things right we need the life cycle control the configuration of the services and the sort of integration back into the application so we kind of just run the database or something we need to make sure that our application knows where the database is located so how do you start with test containers this is very simple and this is for people who are not experienced is that uh you just add a bunch of dependencies right you add the the you can add in all the modules that you want so we have Kafka test containers postgres and I think all the rest as well right uh a bunch of them and we add the integration with uh junit which will pull in the core as well uh so let's go back to the test and what we can do we can start writing the uh the things so for example we can do new generic container and we can do uh wait uh during container and what we need to give it we need to give the docker image name and tag we're going to start with redis we're going to start with redis and we're going to do the field we're going to do the static field here uh generic container and then we do the redis and equal signature computer very good right and so here we can give the the tag for and the docker image name and the tag for what we want we're going to pull right as 6 Alpine uh this image of course if we don't don't have it locally it will be pulled from the docker hub right so uh this is the official redis uh image that we're going to be using in our application and what we're gonna do so now this is the the container right so this is this is the object that we can work with so we can say anything that we want we can start or stop uh the container so we have the lifecycle configuration methods what we can do also we can do all kinds of configuration we can specify the environment we can specify what ports need to be exposed from the container to the host what network it belongs commands and so on we can overwrite the docker image commands if we want to have a specific ones we can copy uh files to The Container or after the lifecycle out of the container we can we can we can execute commands in the container when they started so anything that you can do with a Docker container you get the API here to do that from your favorite programming language which is great because you can just that's what I said oh this is my probably even better feature that I love right it just I can type Dot and I can see exactly what I want to do what I can do which is absolutely incredible uh so we're not going to do this what we're going to do we're gonna say we want to expose the port radius will be accessed through the network so check three seven six three seven nine use the port which redis uses internally so this is the internal Port within the container and it will be mapped to a high level random port on my host machine which is which is absolutely incredible uh and this is it right so now what I need to do this is the my configuration and I my past declares that it needs radius to work right with my test runs with Docker boom done my application is there and I can use it to run any Docker image possible right I can pull it from private Registries like internal Registries in my VPN if it's my local things I can do anything and I have an API to do that so I showed you the lifecycle methods uh and then there are a bunch of things that you can do right so one thing is you can use the gradient integration so you can mark the test containers annotation on the class and you can mark the oldest charitable uh startuples with the container annotation like this and those two guys those two friends will will what they will do they will tie the life cycle of this controller of that container to the life cycle of your unit pass this is the junit five annotations and they will literally uh start and stop the container for you so if it is a static field they will act as a sort of before all and after all so they will start the containers once per all the tests method in the class and if this is an instance field like this they will start the new container for every test method in your class I see a question from Ani about how to start the redis cluster uh this is a good question you need you can do uh you can I I don't have a exact configuration here but what's cool about this situation let's move those if I want to do like a bunch of radiuses I can I can I can do that there are no conflicts with the ports or anything because the configuration for the exposing things are the configuration is dynamic so I can start two of those and then put them on the same network so like oh sorry I can do static network uh Network equals new network new network right and then I can configure them with network Network right and then surprisingly with network network they will be on the same network and then given enough prowess with the configuring redis that I can just uh I can just uh Bunch them in the cluster uh I hope that answers the question I just I'm sorry I looked at the chat I saw that one question I responded to that uh but we we don't need to write assist here so what we can do we can just say like oh we did one redis here uh now we need to start it and we need to like put the information into our uh spring application what we're going to do we're going to create the method uh annotated with Dynamic Property Source this is the spring uh facility so this is the the spring solution but essentially it shows what you need to do um uh I need the static method right uh avoid set up things and then I will give you the Dynamic Property registry registry uh and then this registry this method marked with Dynamic Property Source will be executed before my test start and whatever I can do with this registry I can put data in it I can put properties in it and then it will be used to overwrite whatever static configuration I have so I can be a spring redis host and then put the host here and the host from my Reddit container is of course I don't know where it's running so the easiest is to get host um because my my container here right my object knows where where I have it uh so and then the same report right I can do it uh I don't know what port I need to have because this is uh randomly dynamically generated runtime but I can do the get first map Port so we map Port 6379 internally into a random port and we can just ask you from here that's what the dynamic API gives me it just runs this so this is the configuration this is how we tie it into the spring and then we just need to start it uh so we're going to say ready start um and then oh wait uh just join join the thing right so this is just the utility method from test containers that uh starts multiple containers it can start multiple containers in parallel or the chains of containers if I have like depends on depend on other containers being started and then they can build that complex hierarchies of things so this will this will give me the radius in one two three four five six seven lines five lines actually if I don't do the formatting nicely which you should which you should everyone should format their code nicely otherwise you are not doing everything you can to make other people's life easier so we have redis and as you can see even if you have internal things that for which you don't have the test containers module you can do the configuration very very easily your ID supports it um and it all works with just the API and here we will take care of the life cycle ourselves so we are not tying the life cycle of this radius to the life cycle of a unit tests um because we for our tested we don't need that kind of isolation that parts of the test use one configuration parts of the test is another configuration we can just be like fairly uh we we have a small test suit here so we are not going to start the containers again and again uh we're going to just start them once and we're not getting gonna even care about stopping them uh test containers features a resource Reaper mechanism we use a Docker image called The Rook which actually like I think yesterday just passed the 100 million pools on on Docker hub which is 100 million is pretty pretty sweet uh I think so and that Docker container that will start it will record all the tests all the containers that we run from the test session and then even if you test crash and even if your your JM process like just disappears uh with a sec fault or something um or if you're running with the remote Docker and your uh your machine crashes there is no internet after a period of connectivity it will just clean up everything that ensures that insurers not just that the resources are feed but it also ensures that there are no lingering things in your system that means that you can run tests again and again and then again and again and again uh and then and then it will run James has a question if you say have five integration test classes and you used before test and after test one that starts Atop The Container five different times uh that is an interesting question uh if you it depends like if you use before all and after all or before each and after each and then it will start and stop the containers if you use the annotations uh it will start to store the conditions according to those before each were before all rules I'm not sure Jun is at least exposes annotations uh for maybe are you even talking about Gene maybe not hopefully uh this is the J unit 4 maybe worry even something else anyway uh what I wanted to say is you have the full control you can manually control when you start the containers and you can build any complex hierarchies of uh when you start and stop them right so we have radius let's do Kafka Kafka is even easier because they can do Kafka container because I have my Kafka dependency and they can just say Kafka container equals new Kafka container I've got container and then I just need the docker image name so you can also have the the full Docker image name parse uh functionality if you want to parse the images uh yourself with a more complicated strategies so let's do the content Inc this is the default one right CP Kafka uh and then you know five four six right I think this is the version that uh exists there let me double check uh let me double check yeah Kafka CP Kafka yes very very good this is very cool I know I know this I kind of take credit for this but this is this is idea doing the the work right there is a Content assist there is a Content assist on strings within test containers API and I think some other context where Docker image names strings are Auto completed which is a recent feature and Kudos this this makes my life demoing so much better because very often uh very often those are strings right nobody nobody likes strings because that's where you make typos and then uh this is just this is just brilliant you can also pull things there if you want uh just from the context action right so we do Kafka and we don't really need to configure anything because everything varies pre-configured already which is sweet right so we can just we need to start our Kafka right and then we just need to configure Kafka right so this is string Kafka and it's called bootstrap servers I know this because this is the configuration property how what what it looks like I don't know but I can just use the API here can you see that get bootstrap servers I don't need to know how to configure things I can just I can just rely on somebody understanding how to do that for for me right so we did Kafka now and this is very very simple we started we did everything and then let's do the database so let's do it very quickly database static postgres container first thing I'm going to show you the website I'm not going to type it in but I'm going to show you the website so test containers uh com or test containers work which is for Java and then there's a called The jdbc Thing a database support I could type it in but I don't want to I'm going to show it for you so for the jdbc URLs for the jdbc's uh jdbc databases databases using jdbc like postgres or MySQL and whatnot what we can do we can have a special there is a special driver that you can use you can take the normal jdbc URL and you can insert the TC prefix in there suffix middle part into the thing and then the control will begin to the JVC test container driver and it will create the database based on the docker image name and the tag that you provide and if you wire it back into your application uh so you can just specify this one property like sort of sort of like this uh but I will not just run it for but like somewhere I don't know here right so you can do spring spring data source URL like that it's both of me to say that I'm not going to do this and then do it but like this is enough to create a database this is not the most flexible method but this is the easiest if you just want to sprinkle test containers into your into your application but we're going to do this like manually uh so I'm going to do postgres and I'm going to postgres container and I'm going to say new pause dress uh post address container and this is I think postgres 14 Alpine yeah supporting an Alpine I don't need to do any other configuration because it will be created with some default database name and username name and other things and it will also specify what to expose and it will specify what does it mean for the container to be ready this is what we said in before about talking about asynchronity of starting things uh but what what I just need to do I just need to configure that process and then they can do the same things right we just need to registry at spring data source URL and then review postgres yeah jdbc URL right and this is just perfect get username and then get password and then here we say username password right so I think this is it we start the containers we've run the things and we configure the things and now I think if I run those tests I think they should work do you think they work okay so let's see what happened well let's see what happened in our logs here when we run this the test enters found the docker environment and connected to the docker environment uh and then it because it's a it's a clean Cloud VM it will pull the images for me once and then if the images are in your local cache they will be of course using the local cache uh and then will not pull the images again and again again and then it will start those containers that we wanted so if we do here for example if we find here Target so for example some can hear Rook this is the resource Reaper with a very aptly named uh Docker image name starts very fast and then the rest we need to the rest we need to pull so they will start a little bit slower but if we rerun the test and then very quickly you will see that at containers start actually very very fast uh so postgres starts in like three seconds uh Kafka starts in in in less than less than 10. so if you run them for your integration tasks then uh you can run the test very very quickly uh the test rating checks the whole suit of your application here uh so that why it takes a few seconds to run because it's the longest test but it runs and it works so now what I can do I can do now a few things that the typical question do you have any questions here um headache um Shiva is answering a lot of questions so there's one question from Anil which I wanted to ask he posted that question initially and he said will you cover the Boardman setup just like Docker because a lot of companies use odd man so would you include that in your demo podman I will not include in My Demo podman I know that's a great question that is a very interesting question podman project is I think a it tries to be uh like a Docker compatible substitute uh that goes from a little bit different security and the usage practices and it's an open source project and I know that there are some recipes where people made test containers tests work respond Manuel uh last time we tried to run test containers Java test suit with podman uh unfortunately it broke uh and there were some incompatibilities right so Portland might be very compatible on the common line level for example I'm not sure I will not claim and I don't want to claim where they it is or not uh but since they they positioned as a substitute for Docker I assume that they they are but uh for test containers goes and talks directly through the API which is a more powerful and more powerful uh kind of sort of surface right and then some things are there that might be missing or might behave in a different way and we we're working with the with the problem team both projects are open source test containers and uh Boardman and I just know that there are some issues that you can stumble upon them right whether you will or whether you will like figure out a configuration where uh your test your particular tests uh work with podman uh you might very well figure it out uh and I know people who did but uh I would not I would not claim that it is uh like a like a sort of an out of the box solution uh yeah so for example if you if you're asking I think trying minicube which is local kubernetes installation that exposes Docker API right I think with that one you would have easier Time Out of the Box on your local development machine uh yeah or I don't know test containers cloud sorry uh no yeah so what man is uh is interesting and I hope uh this will all work out but uh and then it will be compatible uh but there is like there is currently no problem specific code in test containers projects because we work with any compatible Docker API right so and the there are some issues that are open on GitHub so that and seshiva is already asking a lot of questions one quick question do you already have this code up on your GitHub account so that anyone who's watching can follow along or would you post that after the session no no this the quote is there uh I have a link to that later but like if you go to the test containers slash I think Workshop uh so this is this is actually like a the whole Workshop in which you can create this application that I'm running okay uh and then if you go if you if you go to Safe Life and I think it's called tascinators red panda which is a uh it's it's a repository that they used for a different piece of content a blog post about running red panda which is sort of Kafka implementation writing test and this one is sort of like a a cheat code for the workshop because it contains the application in its uh close to final form so if you go to to that repository you can just run it and this is uh the final version which is I think very close to what we currently have here instead of using red planet container instead of the Kafka and the consonants Kafka right so I'll I'll let you move forward with your session because we are kind of running out of time a time and I'll ask the rest of the questions later then right so we're running a little bit out of time so what I want to show what I could show you as well right is how to configure things for our database right so currently our I think uh application properties specified that our spring application initializes the database so this is never this is never usually what you do in production so this is not something that you would do in your tests as well right so in production you would probably have the database migration story uh with Flyway or liquid base and you probably should have the same setup in your integration test to check both the migrations are fine and then to check that uh well your infrastructure supports that and it runs normally so I would love to run the migrations but I will show you a more even more generic way of uh doing things so what I can do here with my database control can container I can copy files to the container and I will just do from the class pass resource file which is the uh see my SQL right schema SQL which is the schema that is for my tests and I need to put it in a press in a directory in my postgres container for the postgres to digest it in so scripts uh installation scripts most containers have some sort of most images have some sort of configuration like this so it's not just database specific usually it's like if you have like I don't know like a a conch Ingress uh Docker container that you want to run you will put a configuration file in there in a specific location it will pick it up during startup and it will run those uh so for databases this is one of the most convenient ways to initialize the database you can just put the whole schema into this predefined location so we're gonna do this just just like that right and then that will mean that during the container creation this file will be copied in that and then postgres with initial initialize the database by itself so if for example I run just once this simple test I don't want to run all of them uh even without spring initializing data it will pass because now postgres will initialize the data for me and this is how you can configure all the Technologies right what wait oh my God this was not supposed to happen let's see couldn't create start container what is it right schema SQL create tables and contacts there's nothing why let me run this right we create the container and with copy oh copy file to container sorry so now it should run right anyway uh right uh this is how you can configure all or anything that runs in the container this is how you container configure your applications or look it's green now or looking at the green tests check this out check this out this is one of the rare moments of joy and happiness in any developer's life because next you're going to commit the code you need to clean it up you need to do a bunch of chores you need to push it and then you take the next task and it also starts with uh looking at the red tests so this is this is nice if this isn't nice I don't know what is right so uh let's remove this what I want to show I I cannot talk about toxic proxy and not show proxy proxy right so very quickly static proxy proxy container and this is brilliant oxy equals new toxic proxy container and it goes from Shopify and then this is uh epoxy proxy oxy proxy to zero Q10 I think Q10 now this needs to be put on the same network uh with network this network Network and then we're gonna experiment on redis we're gonna say with network alias uh I'm not sure actually if this isn't needed in the that needs much that's what she needs or not and then we're going to start the toxic proxy proxy proxy and then we're gonna say proxy that proxy for redis and then four port six three seven nine because that's where the radius is what is the radius exposing this is our proxy and then we're gonna do the proxy here so instead of feeding into Spring by configuration of this uh get container IP address and then proxy uh get proxy port right so now our application will go into proxy and now our application will go into uh talk to redis Via the proxy so let's run this uh one test ratings test just I want to show you very quickly how it runs and while it runs because I mostly just want to show you the timing of the block of the of the test here what I can do now with toxic proxy and this is where the fund begins I can say toxic Optics six oh no that's not not the toxic proxy you can see where you can see and then we won't say Downstream every latency Downstream is about two seconds long so every time we want to Downstream from uh yeah yeah exception exception exception right so now now we introduce the additional latency and maybe even uh F wait we'll set Jitter said Jitter like another hundred right so every every interaction will be two seconds so now the tests that Run in under 10 seconds is going to be so much longer it's going to be incredible so much longer right we're running almost out of time we're running almost out of time the test will run in more than 20 seconds if I put the latency here five seconds it will actually break because there is a five second time the full timeout to accessing redis in the template so we will like it will just fail because it some interactions are longer which is exactly what case the case testing can be doing look 20 seconds twice as long twice as long for every interaction and the best part is about the this being all API driven what you can do as your as a developer you can take the configuration for the container you can take the toxic proxy or any particular Docker configuration and you can package it in the module of your own you can create an abstraction for your peers to use at your company right and it will it will use the particular things right so it will use the particular setup you can make everyone do tests with the chaos testing practices or like with networks cut intermittently uh here and there and they will not even know that out of the box uh right um right right right right uh and yeah sorry were you waiting for me to kind of hop into the stream I'm sorry it's the time but like what I wanted just to say is that there are like a bunch of things right you can do more things like you can do complex topologies you can waking on the Fly you can create images from the docker files you can programmatically use build packs uh which I have an example but we don't have time uh you can do log files you can copy the commands you can set the resource limits to I don't know check that your application works with a particular garbage collection that JDM picks by default when there is like not enough memory and whatnot right you can also do local development so modern application Frameworks in Java support test containers quarkus has Dev Services uh Micron has test resources spring currently has this issue that we would like to implement uh the auto services initiative there or explore how these test countries for local development but in the natural what they do automatically if your application starts and you don't have the database or your Kafka or your other things configured you can just it will start one for you right the framework will start one for you so you don't have to bother with anything you can just clone the project run the project and it will run out of the box as long as you have your Java and a Docker environment uh available and then they're just a bunch of URLs so you can go on the website you can go check out the actual implementation and uh contribute fixes and issues and questions and join the slack if you would like because that's where that's where we talk with the community foreign I can show something else maybe as well but like I think we are at the cutoff time uh yes uh so first of all amazing presentation um I will quickly ask you a few questions because yes we're running out of time so one question is regarding the usage of new 4G test container one of the viewers he mentioned of course he mentioned that he enjoyed the session and he mentioned that they are using new 4G test container and they are experiencing adapt shutdown of this container so is there uh any known problem with this container or probably it's problem with their test code or something else neo4j uh uh that's gonna abrupt shutdown I I'm not sure I don't no I don't think I've seen any issues in particular neo4j uh developers are good friends and they are good citizens of the open source Community I know that they internally or like well internally the neo4j project uses test containers as the in the integration test sheet so they are running their integration test for the drivers um using test containers so it might be some sort of your configuration particular uh issue or might be something new uh but I'm sure it will be if you raise an issue or a discussion either on the test containers site or for neo4j I think maybe easier for in the test containers repositories then I'm sure this will be resolved pretty pretty quickly okay so the other question is um any uh pointers or tips about using test containers in CI CD because they were a couple of comments about having issues with using test containers or configuring it on using it with CI yes this is the good question so most of the issues in CI come from the way that your CI platform runs your tests uh it like the test was test containers need access to a Docker environment so you can uh usually if you're just already run in the container that means that you either need like a wormhole Wormhole pattern to allow Docker from the container to access the host to start containers and kill the containers uh or uh like other Docker and Docker setups um or the remote Docker setups so it might there might be a little bit of complexity to figure it out uh for the first time I think uh on wait where's the tuscaners on the website uh all the websites and the website you can see a bunch of uh various uh configurations that are maybe some suggestions what particularly do in certain environments so you can you can look at those you can also join the slack channel uh the slack workspace that you can get in the like get help and talk to us and talk to the other people in the community um who figure out the the similar problems and uh and the configuration for your particular CI platform and the second thing is sometimes if you start too many containers in the worker that is kind of sort of constrained by the resources right if you want to run your tests in a very small small container then running a lot of a lot of containers with heavy things might become a resource bottleneck so for that you want to have things like maybe a larger workers for your tests or offload the resources to things like task containers Cloud where you can uh well they test run locally but the containers run in the cloud I I use test connectors Cloud here look check this out I don't have my Docker running uh I don't have my Docker aligning I run my task containers Cloud which is a small application user space sits here in my uh uh in my taskbar right and it run all the tests all the containers were running in the cloud the tests run locally and it even didn't even notice which is I think it's a great Testament uh uh so yeah you can you can offload some resource to the cloud and then resolve the sort of the resource intensiveness uh of the workers that way so this is the main problems that I I've heard from people about CI so one last question I'll take is what do you already have the roadmap published of test containers on your website oh this is the this is the good question I uh like there are better people than me to respond to those um but I think there are what initials there are Milestones right there are a few Milestones uh which are next is completed but there is also like a 2.0 Milestone which comes with a bunch of improvements that we would like to tackle if we're talking about task and there's Java uh other projects of course other languages implementations they have their own uh particular roadmaps uh but I know that there are a number of things that we would like to even just clean up before the bigger like sort of major increase one of them is we test containers exposes junit 4 J units for uh in the public API in one place in one place we would love to remove that but since it is a API breaking change uh uh we cannot do that lightly and we don't want to do this lightly so what you need to do for now or don't need like what you can do you just can go into your intelligent idea and make sure that when you know you know when you import things you can just say like never import from that package and you can say you can you can say never import Jane it for classes uh and your ID will help you right and then and you you will never have that problem ever again all right okay and I would like um like to share some amazing uh comments about the amazing presentation that you had so we have amazing and very cool presentation the demo was thrilling um thank you for the new features great work and there's another one sorry and probably also watched your presentation at devox um nice thank you thank you very much great session and uh there was one more uh where are they quick presentation or leg and very informative session so thank you so much like that was an amazing session thank you so much for your time today for presenting with us I learned a lot thank you very much for having me and I would be happy to chat with anyone about any questions somewhere online and it's a special thanks to Shiva he was answering uh all a lot of questions on the chat so thanks to him yes yes he he recently joined and he's absolutely spectacular uh it's a pleasure to work with him yes so um and everyone who's watching you can follow uh Oleg and the Twitter handle is right here you can access the code which uh or they demonstrated on his GitHub account as he should and of course um uh follow our intelligent idea channel so that you can subscribe and get notified about the new content I almost forgot that line so sorry how much everyone for attending and uh we'll see you again um next month um or like before we chose the stream if you have any closing comments um you can share that now yes we're going into the winter holidays things so you can take this time to to learn maybe some things uh and then or recharge and uh whatever you do stay curious hopefully hopefully everything is good with you and your closed ones and uh yeah see out there in the open source community thank you so much and everyone's watching we'll see you again next month and until then bye

Channel: IntelliJ IDEA by JetBrains

Views: 7,367

Keywords: testcontainers, docker, testing, spring framework, integration tests, spring boot, java, intellij idea, webinar

Id: zfN8m9Dh9cs

Length: 71min 24sec (4284 seconds)

Published: Thu Dec 15 2022