Testcontainers have forever changed the way I write tests

if you're a rockstar ninja unicorn 10x developer then you're probably someone who can successfully test in prod me however I'm just a mere mortal one that likes to keep his job so whilst I may not test against prod I still like to test against Real services but doing so is easier said than done especially when trying to set up a test environment that works on more than just my own machine one common approach is to use Docker compose however this can come with its own problems such as Port collisions stale data a lack of automation and race conditions all of these causing what's known as the dreaded flaky test instead I prefer to use another approach one that can significantly reduce the flakiness property of your testing code this is the test containers package which has forever improved the way I test my code this framework allows you to easily automate the deployment of service containers for integration testing I actually mentioned test containers in my video on using Docker in unusual ways and it just so happens that the docker team acquired the project about a month before that video went live as a lot of the comments on that video expressed interest in test containers the docker team kindly asked if they could support a video therefore this video is sponsored by Docker for me to show how I use the test containers package in my own applications to show how I use it I have a simple project you can clone down yourself which has a few examples we're going to look at the first example is to get test containers working with a simple rate limiter that I've written that uses a key Value Store under the hood this project is written in go but the test containers package is also available for a number of other popular languages such as python Java rust and even JavaScript I'm choosing to use go because well I like it and that's the only reason you need to choose a language despite this the approach we're using here should be pretty similar to other languages as well and the test containers website also has great documentation if you need it okay back to the code this code is a naive implementation of a rate limiter this rate limiter has a method called add and check if exceeds which takes an IP increments its value in the key value store and checks whether it's exceeded the rate limit this is a pretty naive implementation but it will work for this demonstration to make sure the code works as expected I have a couple of integration tests set up these can be found in the rate limitor test file instead of using mocks these tests connect to a real redice instance which means we're testing the integration rather than a simulation however when I run these tests using the go test command then they fail this is because I don't have the correct redus environment set up fortunately as the considerate developer that I am I've included a Docker compose file in this project which contains the correct version of Redd to use version 7.2 which was also the last version of reddis that is open source however when I try to run the docker compose using the docker compose up command then I receive an error telling me the address being bound to is already in use this happens because I already have an existing redus service running on that Port therefore in order to run my tests I first need to stop my local redus in before running Docker compose up again with that everything should be set up to test our code if I run the go test command then everything works as expected and our tests pass hooray however we're still not out of the fleaky woods just yet if I clear my test cache and then run my go test command again the tests are now failing so what gives this happens because one of the tests isn't tidying up after it's done causing there to be stale data which affects the next time it's run now all of this is a bit of a dramatization but it does show some common problems that cause flaky tests when using something like Docker compose let's go ahead and make our tests more consistent using test containers first things first we need to add the test containers package to our project which we can do using the following goget command once it's complet we can then import it into our test file using the following two lines the First Line Imports the base test containers package and the Second Line Imports the weight module which we're going to need in a minute next we can head down to our single test function this is where we want to create our test container instance to do so we first instantiate a new instance of a container request this type will contain the properties of our container such as the image which in our case we want to be redis 7.2 the exposed ports of the container which for reddis is TCP 6379 and lastly a property called waiting for which is used as a weit condition to tell our request to block until that condition is met this is used to ensure the container is in a good State before our tests are allowed to proceeded however what should we wait for if we take a look at our Docker composed logs you'll notice that we receive an output of ready to accept connections when the container has started up this seems like a pretty good thing to wait for so let's go ahead and do that we can achieve this by using the for log function of the weight package which we imported earlier this function allows us to specify a log line that we want the container to wait for let's set this to the line we just identified ready to accept connections now that we have our container request defined we can now create the container using the generic container function of the test containers package first passing in a context. context followed by passing in a new generic container request struct inside of this struct let's go ahead and pass in our container request followed by setting the started property to be true which tells the container to automatically start lastly because this function can return an error then I'm going to add an assertion to make sure that this is nil with that our container should now be deploying when we run our test code let's go ahead and check it out to do so let's run the following watch Cod command which will show us when the docker containers are created then in the second shell I'll run the integration tests using the go test command this will still fail as we're using the existing Docker composed container however we should see two new containers being created the first of these is our redest test container which we can identify through the image name you'll also notice that the port has been dynamically assigned to 32781 this is how test containers prevents Port conflicts by dynamically assigning an open port to the exposed port Port we configured the second container that was created is known as the reaper which happens to be named after my favorite Shinigami the purpose of this container is to automatically clean up unused test containers which happens after around 10 seconds this is one of my favorite features of test containers as it not only helps you to make your tests more consistent it also prevents you from having a bunch of zombie containers lying around now that we have our test container deploying the next thing to do is configure our tests to use it in order to do so we just need to pass in the correct address for our reddest client to connect to however as we saw test containers dynamically assigns the port which means we're unable to hardcode the address value if we try to do this then it'll fail to connect fortunately the test containers package provides us with the Endo we need to connect with in order to obtain this we need to use the return value of the generic container function this return value is of the container type and contains a method called endpoint which we can use to obtain the dynamically generated address passing it to our reddis client with that our test code should be good to go if I go ahead and run them using the go test command you'll see that everything works as expected and just to double check let's go ahead and shut down the docker composed containers and then run our tests again with that we're almost complete with this simple example however we have one last thing I'd like to change as you're probably aware redis is no longer open source so let's go ahead and make sure it works with the Forerunner for Red's replacement valky which is not only maintained by a lot of the redis core contributors but also happens to have the support of the Linux foundation in order to use it let's first head over to Docker Hub and find the correct image and tag we want to use then back in our test code let's change the image of our container request from being redis to valky now we can check that everything works by running our test code again and with that we've successfully validated that our code will work with Red's successor whilst you're able to deploy any generic container that you like using test containers the project also provides a number of pre-built modules these modules are pre-configured implementations of various Services which help to simplify the setup code for your tests for example here I have some test code that makes use of a postgres test container you'll notice that compared to the reddest container this setup is a little more complex part of this complexity is the fact that we have to craft our connection URI hardcoding the authentication which we've already provided to our container whilst also having to interpolate The Container endpoint we can simplify this code by using the test containers postgres module which we can add to our project using the following goget command once added we can then remove our existing container setup code and begin to replace it with the module instead to do so we can use the Run container function of the postgres module followed by setting the container image which in this case is postres 16 next we'll add in the following lines to configure postres setting the database name the user's name and the password respectively next we can Define our wait for strategy which will wait for the following L line to occur twice and also setting a timeout of 5 Seconds so far this is pretty similar to setting up environment variables however the best part is Yet to Come rather than having to construct our own connection URI we can instead use the connection string method of the container passing in the additional parameter of SSL mode disable now if I run my tests I can see that it works as expected as well as postres test containers provide some other modules that I commonly use such as the module for Kafka which makes deploying an instance of it stupidly simple I mean seriously just ask anyone who's deployed a local instance of Kafka to know how difficult this normally is as well as Kafka another one of my favorite modules is the local stack module which allows you to easily test against AWS Services running locally I've typically used this module in the past to test S3 storage sqs message cues and AWS Kinesis streams modules and the test containers package itself makes it really simple to test code that would otherwise be difficult to do so for example here I have another test Suite one that's performing integration tests on a repository that manages wizard spells this repository is integrated with a post database which I've already set up with test containers when I run these tests you'll see they take a bit of time to complete around 10 seconds in total because these tests run sequentially one approach to reduce this time is to run them in parallel however when I do so it causes a bunch of Errors to occur this is because all of the tests are sharing a single database resource and so any modifications to that resource can affect other tests fortunately we can solve this using test containers rather than spinning up a single database instance for all of the tests will instead spin up many database instances one for each test this then isolates each test into its own environment preventing any of them from affecting each other whilst this is possible to do without test containers it's a lot more difficult requiring a lot of custom code to make it work this merely scratches the surface of all of the interesting ways you can use test containers with some other examples being compatibility testing where you can test your code against many different versions of these Services you depend on another approach I've seen is to use test containers to mock thirdparty web servers that your application May request protecting you from potentially getting your IP banned additionally test containers aren't just limited to tests either and can be used to automatically spin up a local development environment when you run your code the docker team have a great blog post on how to do this which I'll link in the description down below as well as making sure you have a consistent local test environment test containers also works really well in cicd pipelines such as GitHub actions which makes it super simple to set up your test to automatically run whenever you push up a code change overall test containers is a wonderful package one that enables me to write test code that hits real Services yet still remains consistent versatile and simple to use after having used test containers for a few years now I can confidently say that it has forever changed changed the way I test my code I want to give a big thank you to Docker for kindly sponsoring this video with a big thank you to everyone else for watching

Channel: Dreams of Code

Views: 111,931

Keywords: docker, testing, integration tests, containers, golang, go, developer, software development, coding, automated testing

Id: sNg0bnMF_qY

Length: 12min 11sec (731 seconds)

Published: Tue Apr 23 2024