Integration and end-to-end testing with TestContainers-Go | Nikolay Kuznetsov & Erdem Toraman

[Music] today today we would like to talk about integration testing with testing containers and God and I'm going to show you some sites and and the second half of the talk my colleague Adam I would show you a lot of go code and and live demo so and my name is Nikolai so both of us we work for London but in Helsinki office the one is famous to be here in Berlin to have headquarters but we we are located in Helsinki so we have a back NT team there where where we maintain about 20 micro services and all of them are mostly in go and today we would like to share our experience from the user perspective of this library we are not taking we are not taking the credits for maintain the library even though we have some PRS and some of them even went through to the master but we are not taking this credit we just sharing our experience so why at all we need integration tests of course we would like to avoid this kind of busier situation then we get to unit test pass but our system doesn't work in the way we expect it to work and for us the basic integration test is test against some database so in this case it's forceless database and in our project we use mostly postgis database for 90% of the time so this is yeah the basic scenario are some core service and testing its into interaction with suppose with database so and then how you actually get a database for testing of course you can you can start postgis locally like really starting it not in docker you can emulate positives with some in memory walk for the testing purposes but of course nowadays by default choices docker so doctor has a lot of advantages and especially for testing then so you can have a hundred percent compatible database in your test and even though you can use exactly the same version as you use in production you have and you get the same version in the test it's very convenient and it's more reliable and you can always expect that that database state is is empty then you start a new container and then your test is not wouldn't fail because it there is some other data maybe some corrupted data left in the database so eventually we want to test some go service and test it interaction with spas gates and the process runs inside docker container so this is the first approach like okay we just do a docker run we run the PostGIS and that's it then we run our tests against these positives inside the container so what what could go wrong okay there are certain things which could go wrong first of all we can end up with a host port conflict because we map a port inside the docker container - host port which is a fixed value and then if there is some other server and some other Postgres grunion on the same port for some reasons maybe from a different service yeah or maybe some deployed version of service then you get the host port conflict and then you your test cannot proceed another problem you might get that if your if you pause there so some other database inside container is not ready yet for example container is there is there but it takes some time to actually for the database to start and to be ready for the test and if you start your test earlier than that then you your test would fail and then you have to debug what's what's the reason why so if you kind of annoy you all the time then it was just invoking console command docker run the problem that after the test container is still there it keeps running and it takes your resources at your machine so you need to come up with some mechanism which would possibly a key or stop those containers after your test is is is done okay and another problem that if you try to reuse the same container in the in the future test then you get some stale data there so it's not empty container anymore and if your test depends on the on the on this data then it might be an expected behavior for the test and of course you need some kind of mechanism with in which you would be able to start the docker container before before your test so if you're on the local machine of course you can do that from console command and then run your tests from iid but for CI probably you need something else so yeah it's not so straightforward how to build those life cycles of the container and of the test and then you actually try to solve some of the problems like a kind of you detective container is there you restarted or you you reuse the same container and then if you use a shell script then your a shell script becomes like bigger and bigger but still some of the problems remain because we still use host port and we use here we use a fixed sleeping time out to be sure that positive is already started in the container and we can start our test and it's safe so this sleep v command is in some occasions it could be like we overcome it but in some occasions here if we have high CPU usage then it might be it's not enough and then our test would fail again so there should be some better way to do integration testing with docker containers so if we recap about door architecture then actually there is on our host machine or any say machine these days there is dr. Damon Runyon and dr. Damon expose REST API so then we invoke docker run it actually uses dr. Damon REST API under the hood so why I can walk docker run if we can invoke REST API and then we will be more flexible this is example for the documentation for docker REST API for example if you want to invoke a command inside the container if you exec this command and in this case it's date command then actually for for do that to do that first you need to create a command so it's not yet running but you just create that inside the docker then the next step you start the execution of the command and then you started but it's in a sink way you don't get the result immediately so you kind of next step is to inspect the result and then you actually get the result and maybe you get the fact that it's not a result is not yet available so just this kind of simple like then you tied opera dr exact then it actually at least free a guest goes to dr. Damon so there is test containers library which actually uses this idea to use docker demand API directly without invoking like docker run commands and actually this library exists in different flavors and different languages so today we are showing about go but originally it's all started with shower and Joe is the most these days the version for Java is the most mature and it has the highest community but a version for go is also are usable and you can get many benefits from that so under the hood this uses native dr. go client which exists which already exists and supported by by the by the docker company company provided by the docker company like for Joe it's not this case because there are only community versions to interact with dr. Damon so out of the box this library provides you features like host port randomization and other features that you get containers clean up after your test is completed and then there are available different waiting strategies for the container to assure that it's it's ready in a way you want it to be ready before you actually run the test so the API is simple like this now there are two entities one is container request and another entities or is the container itself ID container which is started so in this case we just want to run the latest version of docker of posters and then we want to expose specific port of that container to the host environment so if we run this container then can if you run this command and go then container gets started and we can see that the port from that standard positive support from inside container get exposed to our host environment to some random port it's usually start with 32,000 something and then there is actually API to get the host port back by the port of container so this is very handy because it actually prevents host port conflicts you can run multiple different positives containers or any other database and all of them would be independent so you can also run your test and parallel you can get parallel builds of of the same test of the different tests and you never get that a horse port conflict so and as a feature which comes out of the box is called ice containers clean up and it's implemented by by starting and you can a new help in container which is code right so there is a right Iraq as a character in Japanese movies which can kill people by just knowing the name so in the world of containers it's not the name but both labels so the idea is that and then you start the test container then you start the container for test container library then that container gets assigned some some specific labels and then test container library opens connection to the right demon to the right container it tells what are the specific labels and then it keeps the connection open in a separate go routine and once your test completes that goroutine is shut down this connection is shut down and then this is the trigger for the reg actually to start acting so it it stores those way labels and once it detects that those the connection is not a wife anymore it waits for more for 10 seconds after that it starts to Kio containers by those labels so in this robbery if you start the container you get always a separate right instance running next to it kind of sidecar and then this library provides you out-of-the-box different strategies in API you can wait for host port to be ready you can wait for HTTP some status as somebody some predicate for the body you can they for containers walk to some specific some specific a string to appear there you can of course implement your own custom custom waiting strategy for container to be ready there is one like there you can aggregate them into a single like you can use like multiple as a single waging strategy so if you want to wait for the host port to be ready and this is done this way then we create a container request we specify what waiting strategy we want so by default this whole sport is has time out of 60 seconds and what is important it does the check from outside of the container at the host environment and it also checks the port inside container because now it in systems then you start the docker container your port is open at the host environment like almost immediately so if you check just by some Linux command if the port is open yes it's open immediately but the service inside container su is still starting so it's important also to check for the port inside the container and that the library does it for you so you didn't have to care that much about it and this is one more example for HTTP waiting strategy you can wait you can wait on the specific endpoint and the specific port and then you can wait for specific status and even though you can wait for the body you can get the body response and then you can you can assert if it has some specific sink or not okay now it's demo time so I hand over to my colleague Adam hello can you okay you can hear me now my name is Adam and I work at the same place as Nikolai does so in this demo tation I'll have to use case scenarios one of which will be super simple a basic Postgres database will be started spinned of a container then we will run our test against it after understanding the basics of how this test containers library works and how we can actually ask some different containers and how we can customize our waiting strategies that I will go and deep dive into a bigger and arguably more complex use case so this is the initial use case we have a post case container here then we will have our own depository functions then within the same test main function we will start and initiate everything and run everything eventually so let's start ok so I hope everything is visible for you should I increase the font everything sizzle ok no reactions great so we have this users struct which we store in our database we have the user repo stroke which has a connection to SQL database and have three extremely difficult functions of creating and getting the users by just SQL queries and one function to run the migrations here I hope you can see from my head then let's start with it actual test so ok so the test is nico i showed we will initiate a podcast container right away from here a container can be initiated by a direct the image name which stores at some sort of image registry then you can okay you you can either start from an image then define the expose port your environment variables and you're waiting stategies here i just for demonstration purposes put two different strategies and combining all of them into a multi-strategy so waiting for every single one of them and this waiting for listening port we listen to our base part of five four three two then once everything is ready which we asked test container to do by setting this flag true then we will have a podcast container then eventually after everything is ready we will get the port that smell to to our host port then generate our URL connect it and run the tests with our reporter stack but let's jump into running did ok for this maybe i can first it's small we have I have this watch containers function so this we're just watching all the containers running with some formatting in every two seconds now when I run the okay okay now we are there we are about to ask for the container and then we ask there is no container created yet then we will jump into right away to this line or this line so let's watch this okay it's there created as you can see two different containers are created the container that we asked and it's randomly mapped into airport this random port and it creates this rike container as well which is responsible for killing this after our tests are finalized then we will continue here and we got the host port which is the same thing than generate everything then let's continue we have the con okay we have the connection no errors we run the migrations then we will run the tests then once the tests are run and we pass all of them we go there okay it hasn't been cleaned up yet but because it takes approximately 10 seconds for it for write to believe that everything is gone so it first could kills the post waste container then kiss itself and everything is cleaned up so the advantage of this has been okay maybe I can discuss it here because I made a slide for this just just with a single comment of go test we managed to run a random post quest containers mapped it to a random post random port without actually needing to know which ports are available then general test and everything is cleaned up afterwards so it was as simple as just saying go test but I clicked on debug for showing things but what can we use this actually for for more advanced use cases so I created this second demo for end-to-end testing where we have services one of which depends on the other one because of some use case and the user service here will be a data service basically creating responsible for creating service creating users and serving them and the ticket service is also responsible for tickets but it will also know about the users and users data so our tests are going to run and HTTP tests and all of these services that you see a ticket service and user service on the Postgres actually as well there will be all their own containers running in the same local network and they will also expose some ports the outside world so we can actually debug and test things as well so if you want to go to debt them off yeah okay let's close this as well perfect so for this demo I okay we have the user service if you if you quickly want to go over the user save service it's just like 38 lines what does it do it creates the connection to Postgres has three handlers which is which is just a basic house handlers the others are creating like get and post users and let's quickly look at them as well so we there are no questions left read through the request body creates the user in database and return the same users or just get the ID find it from the database and just serve the database and this is going to be run on this docker file and the docker file uses a builder Dockers and copies everything even in the context and just runs go build and run the application alpine image ticket service is also as simple as this one with one exception it will not require a post quest connection it will just need to use a service URL of course it would also need some database but for the sake of simplicity I decided to have a in memory database which is just a sync map so it has to use the service URL so it should know about the user service and should be able to call it within the same network within the same local network so let's look at it tickets you create a ticket post then you get it and the ticket is basically a user ID and movie name and when you want to get a ticket it should also have the full user data including the user's name so it will fetch it from the user service but so once we have these let's focus on our actual tests so in our test we have to create them one by one sort off because the user service will require the post cast container within the same token network then you ticket service will require the users internal URL so the main test is here for this for all of these to happen we need to first create a token that work which we can do do with this library which okay you can see we are trying to get this network here in this get net Network function if it is not there than we created then we want to have it okay then let's talk about this how to handle the network and the network elias's so I wrote this start container functions here so for every containers or sub entity within our doctor network these containers are created and they return one internal URL which will be used within the network and one map to our out that's for us to be able to debug it in like more conveniently so in addition to the previous podcasts we here have the network name which was created in the test main and we provide our own network Elias names for this user service process it is used service pro square so it will instead of localhost it will have used a service for specialized then here I also wrote as small for SQL waiting strategy in addition to those pre-built library ones you can always have your own waiting strategy you just need to implement this function because it's just interface which basically holds two thread or a host at routine until you feel like it's ready and what we did here was we just wait until select one returns something like ok no errors we repeat it in a ticket and as long as it it returns something then we are friend if it doesn't return anything and it will just return an error okay then we have users or users container starting it says the same thing it runs but here the difference is now instead of having a prebuilt image we had we wanted to start from the docker file so thankfully the container request has such an option from docker file and this docker file actually requires either the context of where docker file is or a context archive which you can feed as a reader the docker file content and the docker files name what all of us all of us use the default docker file name anyways so in this case we since this is a different the directory is just like joining the director and find the right context for this specific then it will actually run the dockerfile build the image I run the container and find the right port and return the internal and external euro and maybe we can now start actually running this let's start with this okay we are the urge this ok let's create it Network is both ok now network should we created or ok integration test network is there now now let's watch the containers again I also created false christs apparently let's go to console okay Postgres is running now let's go to ticket service as well and here okay now user service is running at this port and if we click on this there same of course there is not nothing but if you just go to house it should return something and where did this go okay it was behind actually us and if we checked this then post-crisis there and this random thing is because it said just a docker file without a name so it's just random thing it generated and user services there and if we go and create a ticket service as well okay ticket service is here then we can test this as well just to be sure it's there okay it's also running and so again behind so let's look at our tests we in our test we create a user first using the user service post request and the name of the user is Berliner then we assert that when we get its its - let's assert it so the user ID is one [Music] yeah so the user is here then the tickets will be created yeah and think it is going to be created then once we want to actually get the tickets okay we got the ticket I guess the got ticket should have the same user as like yeah it's very hard probably for you to see but it has the user from the user service yes no it should test everything and once we look at this this should be cleaned up okay everything's up okay so let's go back to our presentation what fantastic internet access presents you can show in this way yeah I think I will shop this way if this doesn't work okay anyways so we managed to run everything in the same local context on the KU Network and managed to actually pour time as well so we can test out ourselves and if for some reason some of them fails then you can go and see the see the post press and from your favorite post girls clients just go and check your data and in addition to that we actually build everything in every single time if you make a change of course because dr. Cash's things but if you make a change in one of those services and run something it will be immediately reflected in this place as well so we will actually you can actually test it in your local without having to deploy anything so let's go to takeaways so this is the website to go and check that's the addictive goes to Java version but under the same github repository you can see the go over the good part of it the code is very easy to read you can always contribute and see the peers and like if you understand go you will pretty much understand everything it says it is great for integration tests what we call integration tests in this context is this Postgres kind of very small way that you can run as in your local very simply without actually needing any complex mapping but you can even use this library for end-to-end testing where you have multiple servers running and you can try the full end-to-end testing in your local machine yes that's all and yeah thank you very much thank you do you have questions yeah if you have questions please shoot doesn't work yeah so do you have any experience with running with stuff and get raps yeah either we don't use github steai but as long as you have a running docker it should be fine you don't need anything else that you must just need to have a docker server server behind this so you can probably yeah I heard that it doesn't allow to to start your docker container so yeah you have to give it in you to have so it might be tricky but I was running that successfully on Jenkins in the previous company and Enzo andhe we have our own CI and it doesn't limit us not like not just our containers from the test don't by the way yes for example the Doozers every entity we're going to initialize a new if you need then yes like you can do this or you can use the same four squares if you have inator it's all up to you you can configure in this way that you initialize a single Postgres then map the same URLs to every single service you have as an environment variable and they will run their own migrations in different databases so you can optimize it this way I just wanted to show everything in a very direct and straightforward face so it's like the go away I have not tried this but I see no reason not to you just probably again need to handle your networks in the right way and oh yeah yeah of course if you want to run your tests in a different container then you just have to create a new container running these tests and you can do this decide to close to correctly even as I can it try also transfer I think yes you can do that but yeah I didn't see you like much proper to do that because if you try to debug it vocally then I think this approach gives you more like drawbacks than because then you need like your test has to be in the same docker network like if you're not yeah like if you don't want to use host ports then it has to be the same docker network and how you do it like if you debug tests from ID it's more difficult in this way I think but yeah nothing prevents you to do that I thank you and do you have a way to get the locks of the trash container because if your container I mean if you're only at the post press it simply but if you have like five containers and one of them crashes and it's removed immediately you don't get a fresh locks yes that I think in Java library it's very easy to see all the locks and this go version of the library is like relatively new so I think it's one of the plans to add more logs of the actual port direct every single line of log-in within the containers to outside world so we can actually see everything but right now it's not possible I think if you have if you can make that peer we can review all together I can add to your question that you can configure like to expose the work from some directory from inside container to your host machine and then you would get them there it's one thing and another thing that yes in Java library you can configure that the walks are streamed like from each container inside your test but it's not yet there I saw recently on Twitter that they started discussion actually to add this functionality to go version of the library so thanks for the presentation is the question regarding the the like creating the database for example I understand that your your example was was like a very straightforward example just to let us know how it works but they would like to understand how you guys managed like in application that has different packets and usually you have like multiple layers in these packages for example we have like two repositories inside these two different packets how do you manage to create for example this database just once or two usually create one database per package so if I understand the question correctly you're asking if we have a single Postgres container then how we manage to run the migrations and create different they databases for every single service is it the question no then maybe i didn't elicit package meaning services i guess yes yes and so you have package a and B and inside these packages you have like two different business logic so I said okay repositories yes geez these two repositories that rely on this database so do you create like one database for each package inside like this package and have an integration test or yes that that is that would be my preferred way yes so you can run the same so for every single package you can have the test and if your package requires some postcards integration that in the test main you initialize this but thankfully you can run this in parallel so it doesn't take that much time you can the data that was involved is not used actual that you can create different post classes at the same time because the post conflicts and they are like totally isolated so that that's that's what we might prefer bias Thanks okay to meet environment information and taste from my point of view some kind of complex environments you mix before about environment about docker container Suites rather huge block of code yes integration tests it is not easy to the way that I am used to is that we always test the variables as interim random variables anyways so or you don't have to change your main service for this for the testing so if your service requires the post case URL and all kind of settings and configurations as environment variables then we do the test yes it will take longer lines to write this inside your test but I think it's better than changing your own service for the sake of test so the service is clean but the tests are yes the a bit more cumbersome instead of the other way around radio yeah that's yeah I can add to this question to this question that like you can extract that code somewhere to like common library where you start the poor girls like then this at least this part of code goes away of course for your like HTTP service you need to customize that to provide some URLs and yeah that that's that could be a lot of code but on the other on the other hand it's go code it's not some Bosch or it's not some docker compose not chase and not llamo it's go code which I used to which you used to work with and then for for the Java version they have some kind of already baked like classes and Java which can start like positives like you don't need to provide your kind of custom waiting strategy it's already there so instead of using some generic container you use like Fazio's container and it's like one line of code in go version it's not it's not yet there and probably it's not gonna be there there is some discussion in community about it yeah but you can organize your code so so your test looks cleaner like you can separate that by putting different files different packages probably one my point is it okay that you starts too much containers for yeah but but integration test what we call integration tests us check in a single service interaction with databases and usually it's not more than three I would say the maximum we get Postgres elasticsearch and radius maybe that a twice and how is it code walk out walk out stack for emulating a double years so it would be for at maximum and that's not that that expensive even for walk on machine and for shy environment which usually has more CPU that an end memory that's fine though those like end-to-end test depending on how many services you have yeah they might be expensive and especially in Java word they are because for each services a Java service usually starts much much longer than go service so for go it's also not the big issue because they start quite quite fast Li and they don't consume that much memory comparing to tower okay any more questions okay okay thank you very much thank you very much [Music]

Channel: GoDays

Views: 1,019

Rating: 5 out of 5

Keywords: #GoDays20, Talk, goconference, golang, Berlin

Id: e4aU2bpERbo

Length: 41min 42sec (2502 seconds)

Published: Fri Feb 07 2020