Testing your K8s apps with KIND - Benjamin Elder, Google & James Munnelly, Jetstack.io

hello good morning good morning good to see y'all here nearly full room I hope you enjoy the the next talk it is by James Minelli the author of cert manager do you know cert manager many of you use it until kind do you use kind been elder is the author of kind and the ban and James are gonna talk about testing was kind some of it will be a little bit about what do you do for certain manager regimes yep yeah yeah so just just just wondering like how many of you use kubernetes in production and think that you have your testing store stories sorted out okay yeah but like generally how many of you use it in production and such alright okay cool and how many of you first time at cube con Berlin okay cool well welcome all the new timers and the old timers and over to James and then you go yeah thanks very much for coming today this is a very very full room I can still see more and more people coming in we're gonna be talking about testing kubernetes applications using kind today and a bit more generally too so first of all what is a kubernetes application I realized that it's kind of a very loose concept yeah we could be talking about a thing so it's pretty much anything that runs in kubernetes but we're going to be putting a big focus on extensions and controllers today in this talk so who here has kind of looked at running operators extending kubernetes anything like that yeah nice like cute builder and operator SDKs yeah so this is a good tool for you so yeah different types of tests first of all is something I'd like to define because I think the difference between unit tests integration tests end-to-end tests it's kind of a very undefined thing I've never really come up with a good strict definition so I'll start by saying unit tests they typically would make use of something called the fake client in client go they are it's effectively an in-memory implementation of the kubernetes api server and client it works great it does have a lot of caveats it's very simple you don't get the full admission control chain on the API server side you don't have additional controllers running you're missing quite a lot but it's great for quick simple unit testing yeah one big thing that I found is that race conditions timing issues with the actual controller reconciling and having to run multiple times you don't pick up on that so yeah you it's great it's very good for testing kind of state machine style actions you know given this input state what is the output but it doesn't really test the whole system as a whole something like this this is an example from cert manager's code base we obviously build more packages around it but in short you can see here the difference is we've got this challenge presented true is the starting state and then we actually check we have expected actions so here I don't know if I've might have a pointer nope I don't have a pointer but you can see here we've got the new update action so we're expecting an update to be we're expecting an update to happen that's like against the API server and we're expecting that state to be set to false so you can see here given that input state we expect one running the reconcile function once that will end out with a different output state so then integration test so who's used cube builder here or looked at it whew okay so within cube builder they actually provide a load of extra testing functionality so it will give you a bit of a framework that will start up at CD and an API server and optionally I think even a controller manager on your local machine compared to unit testing that does give you admission control we now have an actual API server running we can pick up on timing issues because this is kind of a bare-bones so I'm not going to say Kiva Nettie's cluster but bare bones keep Nettie's control plane right there so that's yeah they work brilliant and it gets you to the next next stage but there are more things out there to test then well there's more to it than that sometimes with all these kind of integration unit tests you find yourself tying itself to the implementation of your particular controller rather than the actual behavior that you're looking for so that's where end-to-end tests sort of come in and I'm sure we've all heard the term end-to-end tests what we mean here is actually bringing up a cuba Nettie's cluster deploying your application and actually encoding what we expect and want users to be doing into our tests so saying if I apply this I expect that after some amount of time two pods won't exist and the kubernetes code base is full of end-to-end tests you've got conformance tests all sorts and these are very blackbox you know you you go and apply something and then you wait to see what happens and if it doesn't do that then that's a failed test it gives us a lot of flexibility you're really looking at this from the users point of view and they're really valuable to have yeah it's black box and everything else the one downside here is we need a whole cube Nettie's cluster and there are lots of different ways to start cube Nettie's run kubernetes you could spin up a cluster on economy on a cloud provider or something like that and run tests against there but over time if you're doing it's on every pull request or every change that gets more and more expensive and I think kubernetes project itself does so by running standing up GC clusters for a while which again costs money all of its slow I think then you probably know it's probably Bana was it an hour hour and 15 to actually go through that whole process yeah most tests on Cabernets pull requests take about an hour plus to run yep so you can see that's kind of if you want the open your fast iterative feedback loop that doesn't quite cut it as a developer I have never run that test suite from my own local machine because like I don't really want to dig into how to do it and I also don't really want to run some cloud some cloud VMS for that period of time and also yeah who pays for that so yeah why and to us though and to end tests though some of these edge cases some of these problems only come up in a real environment so you don't actually get to see issues with say running out of memory because your applications done so like there's all sorts in there that you might not pick up on and yeah the way that other components in kubernetes interact with your own application as well can make a massive difference something that passes unit and integration tests in isolation all of a sudden you realize that one other cool kubernetes component is fighting against you and so you end up with this flapping back and forth and that can go on forever and that'll either generate tons of logs use nodes of memory or it'll just it's just doing the wrong thing we're just causing excessive load on our clusters so without end-to-end tests at least I've found everything looks like theoretically it's perfect but users are opening issues saying like you kind of just killed my API server nothing will continue running I can't start pods anymore so we have an example of where this came in really helpful in certain manager recently we've gone and refactored some API types we needed to make sure that as part of that refactoring we didn't break the old API format so I started out by implementing the changes so actually implementing the new format of API I didn't touch a single thing in our end-to-end test suite and pushed all that code up or ran it locally run the test suite locally and made sure that those tests still passed and because they are blackbox they don't understand the actual implementation what's going on if they pass from my point of view that's like I just ran users like users just sat there and ran this same flow and the fact that that passed gave me confidence to know that I could actually move forward with doing this of course I then went and added new tests at the end because you can't just add new functionality but yeah you gradually grow and grow your test suite and as you're doing that it bill your confidence and it just means you can actually ship faster we've done this countless times before before the test this week and where you think everything's perfect it turns out it's not quite so much yeah because it says they give you confidence to ship without it who knows what will happen so that leads us onto kind so we've been talking a lot about how difficult it is to get these clusters then I think you've got today so hopefully James has convinced you that into and tests are definitely something you should be thinking about and if you're thinking about into and tests you should think about kind kind is kubernetes and docker so we run docker containers to simulate nodes it does multi node multi cluster works offline if you pull the node image first and if you're putting a single node with the image pre-poll should take about 30 seconds to boot up a node so this takes away a lot of those problems and it hits these key goals a one if you're gonna be doing testing ideally as much as possible you want it to be fast so you get that iteration cycle you want to be lightweight so that you don't mind running it all the time you want to be cross-platform so developers can run it on their machines doesn't matter where they're at we're gonna be hermetic which is a term I mostly heard at Google but the idea being that we don't want external dependencies because those will decrease the reliability of your tests you just want to be testing your application not the internet so kinds of testing tool typical workflow Buddha cluster build your application and inside load application images we're gonna come back to that you don't have to do it this way we're gonna recommend that then you run your tests do again creating a cluster is just kinda create cluster you need the kind binary and you need docker that's it sometimes you might want custom configuration if you're doing this with kind you can configure the entire QP item interface it's Q beta underneath we have customized caches which we're going to show in a second and so you can configure anything you need to make kubernetes look like how you want it to look and that can just includes things like changing the service cider or enabling helpful features anything that you would expect to be able to do with a fairly real cluster so this example actually from some assert managers testing so what you're seeing here is kind zone configuration so we've got this just configure object I believe it's cluster now and so we've got one node and on that node we're saying we need to patch the Kuban config the set of particular service net because kindness herb sorry cert manager is doing some fun things with networking because they're doing like DNS challenges and whatnot so they're very particular about which subnets things are on so you can just configure like that very simple or if you want a bunch of nodes you can have multiple control planes will set up a load balancer container and do H a and you can have workers and you can simulate doing like rolling things across your cluster between nodes and if you want to do things like okay kind ships its own lightweight CNI but I wanted to integrate with weave then you can disable the default c9 deploy your own so now that you got your cluster up and ready to go what you're gonna want to do is load your image into it and the idea here is again avoiding flakiness avoiding being not hermetic no dependency on the internet instead of pushing your image to a registry in your testing flow what you do is you can build it and then you can tell kinda load it into all the nodes and so basically it's just going to take a docker archive of your images and then stream those to all the nodes and load it in so that when you tell kubernetes to run your image in a deployment it's already present it doesn't need a full you don't need to use the Internet you can do this offline so now you're gonna get the cute config kind generates a unique cube config for each cluster so you can ask kind where the cube config is point cube config environment variable at that run your tests now dreams gonna do a live demo sure this will go smoothly yeah cool let's see hopefully go a little bit bigger hopefully at the battery web see so in this demo I'm actually going to deploy the sample controller from the cupola to use code base so this is just a really basic example I've put together a very very basic run test script so here we can see we do first of all a little bit of cleanup because I had to do a bit of testing here and then we're using this client create cluster command that been mentioned earlier well go into and have a look at that config file in a second but once that's run just like Ben showed we go and get the client a cube config path notice here that we're passing a name variable this is because we can run multiple clusters at once so for example the federation v2 project they utilize kind to actually test federating multiple clusters together they pass different names for each one and yeah and that all runs on one machine here we then go and build our application so obviously depends on what you're doing here but in our example we're doing sample controller we've built a docker image for it and then just like Ben said we load the docker image into the node now if there were multiple nodes I think in this example rowing into a single one that would actually get loaded into every single one of those nodes so it's available for use we deployed the sample controller and then we'll start using it so enough looking at scripts let's bite the bullet and see how this goes so do a bit of cleanup and here we go kind is going along and it's creating a cluster so we're booting version one point fourteen point two this is just using one of the prepackaged ones that Ben has pushed up forever on it prepares the nodes first of all so that's creating the containers it then goes and initialize queue initializes cube Adam it applies all those patches that we've specified onto a base config and then it will start actually initializing so that under the hood is running cube Adam in it within a container we've already gone and configured systemd docker and the like and there we go we install see and the class is up and running so we're on to the next bit our script going we've built sample controller there it's getting a little bit too quick and we're building a docker image containing it and you can see here we've gone unloaded that image into the client container and we've gone ahead and actually deployed the sample controller so in your application here obviously the manifest you use the commands you run will be slightly different we're just trying to illustrate what a typical workflow would look like that at the end we've actually gone and managed to deploy one of our few resources this is an example it's not particularly useful and you can see we've got our command to run here and this is as been said the hopefully you can see at the back this here is the cube cube config file that kind is created this is typical works with everything everything I worked with keeps ETL and we can now go ahead and check our cluster there we go so you can see just like any other kubernetes cluster that you're used to we've got all the different components we've got our CNI here we actually run a custom lightweight CNI that's perfect just for testing on a single host because we know where we're running we can understand the network topology better and we've actually got our example application here our sample controller here so as you kind of continue to build now that we've got a cluster up and running you can continuously run that kind load docker image command and load more and more images into your nodes when you roll out changes and then just restart that pod and off you go so this gives us not only a testing environment but also a development environment here I'm running on my laptop but this could have run under CI and it will be identical the fact that it is identical is so valuable like if you have varying if you have varying different environments between different developers and everything else it's frustrating we go you know you see your tests are failing but you can't possibly actually debug it because you don't know where to start you don't know how to actually get this up and running so we haven't I will clarify we've not written a full test suite here this isn't telling you how to right tests in bash please don't write your tests in bash anyone really doing please yeah so looks like didn't go wrong and yeah so we are running here kind in on my on my laptop we do support running kind obviously in various different CI platforms just put together a kind CI slash examples repository so that's coming together gradually we currently have some examples for running kind on circle CI and Travis gradually adding more there but technically anything that's Linux your pretty good to go or Linux OS X or Windows I should say probably the most one that's not in this repo yet is kubernetes its own CI which is of course running on kubernetes so we run kubernetes on kubernetes because who wouldn't want to do that yeah you can go many layers deep yeah and one extra thing to note here we've kind of used the CLI side of kind we do also try and like we want kind of itself to be consumable as a library within go line so here's an example where we actually go and create a cluster all in code now anyone who's familiar with golang testing you can just go and chuck this into your test functions and stand up a cluster that's used for a single test and that I mean you can see the clusters boot within about 30 seconds but we can actually just bring up a cluster for each individual test which means that you know that if it passes once it should pass again and again because the cluster bring up is as Ben said hermetic so we know we don't depend on any external sources yeah no more bash so I'm gonna talk about what's next we're hoping to get better network management we found that you know we want it to work perfectly everywhere but one thing that is actually different for everyone is your local DNS on your host and that gets picked up by kind right now we're figure out what to do we'd love to hear ideas ipv6 support and testing so kind was actually originally written for kubernetes testing of kubernetes the project and we don't have any ipv6 CI so we're working on getting ipv6 worked out in kind so we can get single stack and then next will be dual stack as kubernetes starts to grow that we want to make note ports easier to access on Mac and Windows this is a bit of a caveat for application development now if you're on Linux all of the nodes have IPS as you would expect that you can reach because they're just docker containers on the docker bridge but on Mac and Windows they're inside like the docker for Mac theam which doesn't expose one so you'll have to port forward we're still figuring out the best option there you love to hear feedback there we want to make it easier for you to build like an exact kubernetes version but chiefly kubernetes on CI is building tar balls all the time that you can just download it like whatever version you want so we want to just take that pack it into a kind image then you can run it we're always trying to make things faster we have few more ideas there and we're trying to move from alpha to beta very soon hopefully but if you use a release version we probably won't break you and I'd really love to hear from interview your use cases and like what features you'd like to see because it's a small project right now but there seems to be a lot of attention we're trying to figure out what a prioritize so appreciate that here's where you can contact us this is under the sink testing and the kubernetes sub project we have a bi-weekly project meeting on zoom' like the most kubernetes projects we're on the sig testing mailing list and we have our own slack channel on the kubernetes slack you gonna invite that's like buckets do the repo at 6:00 our kids do / kind which is also you can just go get that if you have go modules on and the docs are it kind of sig stuck he'd say oh thanks for coming and I want to do a special shout out to Liz frost and hep tio who actually put together the CFP but couldn't come and James for jumping in and helping square this away rework the talk of it and bitching and thanks everyone for coming thank you very much thanks for making kind such an awesome project and well we do have plenty of time for questions actually and yeah please raise your hands and I'll run over to you as a mic okay so we got two hands here one there let's start here yeah um ipv6 support hasn't been merged already in the kind or it still exists as a extra PR it's in a PR it's almost there I'm gonna be reviewing that some more when I get back I mean could you be a bit more specific for the time when it's gonna get merged it's kind of important and we're he was at the same state about month ago so right it was for about two months we had a bunch of other chaos going on and we got back to that we've had a lot of progress on it recently but we're not quite there were very close it turns out one of the components we're using is not working in ipv6 properly we're gonna swap that out and then I think we're good to go but she's a bit of a filling one we're hope when it's ready we're hoping for another release about a month from the end of cube con we try to do monthly releases we did release just before cube con so hopefully that one okay and sorry second one very shortly let's say if my PRS actually changes the API server what would be my workflow to use when basically to use the custom API server image when it when your lustre comes up kubernetes PR yeah you just need to do slash test pull kubernetes kind et with hyphens between them and we've got an optional pre cement now we're hoping to probably make that a default one soon do you have a plan to support creating image retreats and share the volumes such as universe or ice cozy around with creatives across sir I'm sorry I'm not sure I understood the question so I create Cresta I want to create a registry as well so I can just pushing images from humanities using dictor or 1080 wheel and I also want to create shared volumes such as I knew it NFS or Isis target I wrote with the cross so we we do have support for adding extra volumes to the notes to pass through things we do not have any special support for registries right now but it is something we're tracking just go in order 1 - is it possible to use card with another container real-time like competitor de or cry or something like this using kind of the other controllers not yet we're looking at that we so that's a big thing that we want to do probably after we finish the ipv6 stuff is a refactor out so that we're not talking to Dockers in many places and we have like a little kind of CRA Ashim for unit testing and pod mana will be the next one we're targeting CRI functionality as much as possible like we don't use excessively docker specific things where we can avoid it but it will take some time we're big fans that we're running container D on the nodes now because we've been able to do a number of optimizations around how the food up happens because of that and I'd like to see more of that all right one more question here ok since you're suggesting that we should use it as a library you not use bash and you're still alpha do you have any plans on how you're doing versioning with kubernetes for example you have a project using 1/12 typically if you're in education or something you are a little bit behind so any plans on how to do that in in future possibly using replace um that's a really tough one and that's part of why we focus on the command line first because yeah that one's tricky why not one solution is to just have your test suite import this and hopefully your test suite can be decoupled enough to just talk to your application and then you can version that separately yeah I think I think that comes about kind of from the blackbox nature for example or our test suite never really touches any core application code it's it is basically its own application we compile that to its own binary I would like to ask is it possible to deploy images using hell you mean the the docker images or deploying the applications on it yes so I mean you can yeah you can use helm against the cluster helm and teller against the cluster that you're deploying to in fact manager does exactly that so we can verify that our helm charts also work as part of our end-to-end testing flow the docker image side of things I don't think that's something that helm does generally anyway publishing images using helm I think it's more as the deployment although lots of things change in this space quite rapidly so I'll disclaimer that okay thank you okay time for questions still any more questions on this sides one down at the front one down in front all right if we already use prowl at the CIE would you recommend a starting path to use kind you should send one of us a ping we do have that set up but it is a little bit hairy might give you a hand with that to clarify we use prowl as well and yeah I've been running kinda nerf quite a while now you just need to make sure you make your pods privileged enough if nothing else kinds own pre submits run probes so we can point to that another question here okay hi any plans or work already being dumped to use kind as a library from other languages I I think not I'm hoping that I mean so the command line is meant to be used from these things but that's an interesting topic there's a bit of logic in the kind library to do some of the setup we have to do things like generate the config or like we used to do more things during boot up to control that we've moved a lot more of it to be basically just running the image but there are a few things that we have to set up and configure and whatnot we basically probably have to like rewrite the kind aspect in another language and I think maybe over time it could be more possible if we manage to reduce the amount of stuff that kind itself has to do until we can just get it to the point of a docker run I mean I guess you can rap you know right docker run in Python or something perhaps I was thinking maybe that there's the kind cluster API provider which turns kind into something that you can access with cluster API and after that he could use any other any any client any kubernetes client attention to to to to do things with it I can't speak of the maturity of the project or anything like that we have actually discussed that it gets a little goofy because right now cluster API uses kind to bootstrap by default so would you like bootstrap in kind with kind but sure theoretical purely theoretical idea so there's another question here yes I was wondering how do you approach or if you have any plan to approach the problem of external dependencies for instance let's say the case I'm I declared a PVC using a storage class which is a natively a storage class because I've got an e KS managed cluster or I'm using an ALP ingress controller or even I'm using an external database let's say dynamo DB so - so you would want to sort of mock it out its storage is a particularly tricky one with kind I think you can install a fake storage class that pretends to be the same but like some level when you start to get pretty far in those things I would recommend like you have a QA pass at some point further in the dev cycle where you actually run in a more broad like environment and make sure that everything works out is expect there because storage is one of those points where like if you're not specifying many things it's gonna work great and be exactly same everywhere if you start tuning things and whatnot is it's gonna be a little bit different that's hard to avoid more questions please we've got five more minutes okay one over there I'll try to come over all right okay somebody hello I have a question more about testing operators and how you do this internally with kind and particularly with eventual consistency so the problem we are having as our operators are just eventually consistent so you run into like flapping tests because you have a time limit set how do we handle this internally or do you have any wise advice to do with this yeah that's a great question that's we had a good discussion about this the other day in some colleagues so like you say unit tests don't really pick that up because you run a reconcile function once and actually things are eventually consistent we need to keep going this is where I'd say kind really shines because obviously you are running your controller you are doing that whole thing you do need to be careful to maybe count the number of times it takes to get to your finally consistent state or how long and that can generally be handled quite well with test timeouts and the like I think some of the other strategies on a more like a unit testing level could be considering running kind of the reconcile function in a for loop up to 10 times 15 times 20 times whatever is appropriate for your code yeah that that's what I'd say for the eventually consistent things but with kind itself you do kind of want to do this in a larger cluster because it may be that you're not actually getting to your consistent state because something else is fighting you basically yeah does kind give you an ability to restart your entire kubernetes environment between tests or between tests Suites do you mean reusing the same cluster all like let's suppose that I want to clear my state between the test suites and maybe clear some kind of database state would it be possible to restart it so grammatically we actually recommend just throwing away the cluster and making another one it'll take you about 30 seconds to boot occult but like would it be done in an external script or is it like can it be coded into my test like in like in your test harness like okay so this test these set of tests are gonna run with this cluster and then this set of tests are gonna use a new one just spend one up tear down tear down is near instantaneous it'll take you a second or two and up is about 30 seconds so if you really want that clean state there's no better way than just actually having a clean slate Thanks someone he had a question is one of you alright if that's all the question oh there's one there yeah I can see so could you raise your hand again please [Music] so so so time to ask really quickly I want a heist on stage here anyone who has sent us a PR would you come up I know they're a bunch of people are here I see you Antonio okay the question would be you mentioned that you also run in Windows so using kind as a deaf environment for like a prod like environment or to work on Cuba natives on Windows what is it possible or do you what what are the major problems surrounding this when using kind it should run fine because it's just using docker containers that said we don't support Windows nodes yet but we do support Windows as a host it's a little bit not quite as well supported because we don't have CI for it yet and I don't develop from it but I do periodically check and make sure and we avoid anything that's operating system specific on the host side Thanks thank you Thanks thanks very much everyone [Applause]

Channel: CNCF [Cloud Native Computing Foundation]

Views: 5,601

Rating: 5 out of 5

Keywords:

Id: 8KtmevMFfxA

Length: 36min 17sec (2177 seconds)

Published: Fri May 24 2019

Reddit Comments

[u/mikeroySoft]

Not sure why the poster didn’t just share it in the Kubernetes subreddit. If you look at /r/k8s it’s all by the same person... why splinter a community instead of sharing to the one that already exists?

[u/lerrigatto]

Wait. Kubecon eu videos are already online? I am not even back home yet...

[u/gjcarneiro]

A nice tool to have in the toolbox, but 30 seconds to boot a test cluster still seems rather slow when compared to docker-compose. I will avoid this if I can.