An introduction to performance testing with open source tools, with Simon Aronsson

exciting to see so many of you joining uh 24 that's uh quite a bit especially for such a niche topic so my name is simon ironson and i'm a developer advocate with the company k6 and over the last couple of years i've spent a lot of time on testing test automation practices and building robust engineering cultures where testing is not considered an afterthought but a natural reoccurring activity in the software development workflow i'm a developer myself but i've been testing quite a bit for the last couple of years and today we're going to talk about load testing or performance testing in general and load testing in particular and we'll have a look at how to use open source tools to build execute automate and visualize performance tests mainly load tests and before we start i just like to speak a little bit about the topic of load testing and what it is because i assume not everyone joining today have actually practiced load testing before and load testing is a practice that is all about generating load on a system that resembles production load from actual users and we do this by executing load tests using different load profiles or load models to simulate real-world scenarios that we want to build confidence around for instance what would happen if our product or service suddenly hit the front page of hacker news or started to trend in a popular subreddit or what if we want to prepare for black black friday for instance uh we might have an average of say 100 users at any given time but during black friday that we we might see that number increased tenfold without any prior warning so by using load testing we'll be able to assess what would happen if that would actually be the case and it can also be used to verify other things like whether or not our automated scaling works or to analyze the effects of the of graceful gracefully failing nodes in our cluster or in a distributed system so what would for instance happen if one of our web frontends were to go down during peak traffic would we still be able to serve all our users using our other nodes or would we suddenly start to see failures where users would get bounced due to due to our service trying to recover and in this talk or we will mainly be doing a demo and when we're done with this demo then we want the to have integrated a load test into a ci workflow where we either can trigger the by doing commits on the master branch or by triggering it manually from a web ui we also want to want the ci to tell us if we suddenly stop fulfilling our service level objective or if we fail a performance threshold that we've that we have defined and we also want to be able to monitor the performance visually without having to run it ourselves so for instance if someone were to push a commit to master we would want our performance test to be run and our results to be visualized in in our visualization tool and all of this can actually be performed with open source tools we don't actually have to pay anything to do this so uh we'll be using an open source stack mainly uh with one exception for this demo we'll be using github actions which is the only tool we'll use that's not open source but it's available for free so we'll we'll have to make do with that but you could just as well use jenkins or any other ci you prefer and we'll use github actions to automate the execution of our load test hello we'll then use k6 to generate the load uh as well as making sure we're staying within our acceptable threshold or slo and the metrics we generate from the load test will be sent to influxdb which is a time series database to finally be presented in grafana which is a visualization tool to make to make this as easy as possible we'll be using docker to run most of our tools and so we don't have to install anything locally on our computers and sorry i got a notification i just have to turn it off there we go all right so when we build load tests a common approach for doing that is to use some kind of browser recorder or proxy to capture the traffic that occurs within your system or when someone uses your system and while this is a great way to get up and running quickly and with zero to little coding involved we usually recommend users to write their load tests themselves using javascript this has in our experience proven to be a lot more maintainable over time especially when we integrate it with our ci so let me just switch my screen sharing all right you should all be able to see my editor now and i'll turn off my webcam as well to make this to make it a little less in the way so uh well let's start off by creating a actual javascript file where we will place our actual test while this is actually programming the syntax for writing load tests in k6 is actually quite simple or quite scaled down so with the help of the documentation you should be able to write a decent performance test even without prior programming knowledge so we'll start off by defining our vu function and this function will be executed once for every iteration that our vu makes a vu when when in load testing terms is a virtual user that is a simulated user that performs something that a normal user would and this function that we define it will be executed synchronously from top to bottom one statement at a time and for the test to actually do something we'll now add a request using the http client that is built into k6 we'll then save the response of that request and do a check to make sure that our assumption about the outcome are correct so we'll start by importing the http client as well as importing the check we'll then do our http call in this case we'll do it against a test api that is available on the k6 website and we'll then define our check based on the result so for every request that we make we want to assert that the http response status code is 200. so we we write a label http response status code is 200 and we then create an expression that evaluates whether that is the case so in this case we'll go ahead and check that the status of the result object is 200 let's run this just to make sure it works so far all right so we can see here in the terminal that it actually went well we did one http request and we also get a lot of statistics or data about how this went so we can for instance say see that the request duration for this request was 108.45 milliseconds which i think is acceptable for a demo api we can also see that a hundred percent of our checks actually passed so uh a hundred percent of our one call had the http status code 200 awesome so so far it works yeah now we need to make this fail on errors as well so let's set up a threshold saying that we expect all requests to finish without errors and let's also add a threshold for the request duration or for the 95th percentile of the request duration we'll express this in k-6 asset threshold this could be any of the performance slos that you have defined for your team or for the service that you're maintaining or operating so we'll start by importing a metric object which is also built into k6 and in this case we want to do a rate then create a rate object out of that and let's name it failed requests we'll then also export a variable called options where we can set some options for the uh for the test run or for k6 to to take in during runtime and in this case we'll add a threshold or we will add two thresholds one called failed requests where we express express a condition for this to be considered past in this case we want it to be we want the rate to be less to or equal to zero less than or equal to zero and we also want to set up a threshold for the request duration and we'll say that the 95th percentile should have a duration less than 500 milliseconds well then also make sure that if the status code is not 200 then we'll add one to our failure counter otherwise we'll just add a zero so failures add result.status is not equal to 200 and if we now go ahead and run this again we'll see that another metric has appeared we now have failed requests as one of our of our metrics in the summary and we can see that zero percent of our of our requests were actually failed so that's great we're able to do one request against our test api without it failing but that's not really that helpful right because one request hopefully all systems can handle that uh but we need to bump up the volume a bit and add some more virtual users in a longer duration than than one iteration so we'll go ahead and do that we'll add to the options object that the vu count or the virtual user count should be 10 instead and that the duration for our test should be 10 seconds if we now run this again we should have 10 virtual users trying to do as many iterations of our test function as possible during 10 seconds hopefully we'll land around at least a hundred but we'll see what we get out of this so with that let's run it again k6 run test.js that was not 10 seconds let's see what happened here my bad options there we go it will now run for 10 seconds with 10 vu's and as you can see it was a whole lot more than 100 iterations that i thought it was actually 856 during this 10 seconds and now our our metrics that we get in the summary make a whole lot more sense since they don't all just show the same we can for instance see that the average http request duration was 111 milliseconds while the lowest was 103 the median was 109 and maximum was 222 we can also see that for the 95th percentile the duration was 120 ms which is more than acceptable in my book at least for our demo service we can also see that all the checks passed and that we had no failed requests and that doing this test consumed 10 megabytes of data or rather we received 10 megabytes of data while we sent out 115 kilobytes all good now that we have that up and running we should probably go ahead and add that to our ci as well and just to make things a little simpler or for the sake of time mainly i've gone ahead and already created a repository that we will use for this so just let me switch my screen again so we we have a repository on github called passion for testing and what we want to do here is push our code to this repository and then have it run whenever it detects any change to the master branch and when we do that we'll see something happening in the actions tab let's see there we go so let's take our change test.js change commit that and push it to github all right if we now refresh our window we'll see that the changes have appeared in our repository we have the test.js file here but we'll also see that since we haven't set up any action nothing will really run on for now so we'll go back to the code editor and add that script as well so we'll create a new folder called github and in that we'll create a folder called workflows and a file called main.yml or yaml so github actions is quite simple to use it uses almost the same format as most other ci vendors so if you've used jenkins before with jenkins files or if you use bitbucket pipelines or azure devops or anything like that you should probably feel quite at home with github actions as well so we'll start off by giving our workflow a name let's call it main or rather call it load testing workflow we'll define what we want this workflow to trigger on we want it to trigger on pushes and we also want to be able to manually dispatch a test run so we'll be able to trigger it from the ui using a button basically and the jobs we want to be included in this workflow we'll just have a simple one-step job called test and the label for that will be run load test we will run our test on the latest version of ubuntu we want the steps to be first checking out the code we'll need that to be able to run our test and then we run it using a k6 action available on hub actions so we don't have to actually set that up ourselves and we want the file name for our test to be test.js which corresponds to what we've set up here and that should be it so if we now go ahead and push again we should be able to see that the action has actually been executed in the github ui okay we can now see that our new file is present as well and if we go to actions and enable actions then we can see that our workflow is actually present here so let's go ahead and run it here so what happens now well we're gonna run our load test on a um on a cloud test runner provided by github so we don't actually have to care about setting this up ourselves or having a dedicated testing machine so whenever you want to run a test we can just go to the actions tab and click run workflow and it will spin up the machines needed and execute the test and we can also follow the logs so it starts by checking out our data and then it runs the load test just as it did locally and we can browse through the end of test summary just as we did in the terminal awesome great so now whenever whenever we would push a change to our repository these tests would be executed so we can build that into our regular quality assurance pipeline to make sure that we don't have any performance regressions and in case we do the workflow will actually fail whenever we pass any of our thresholds that we defined so for instance if we would have errors then the workflow would fail and uh would get notified immediately that we have something we need to take care of the same goes if there's suddenly a request duration change that makes the all operations against the server uh slower than they usually are so this is a great way to be able to set up performance goals and then forget about them and be reminded whenever there is an issue that we need to take care of i'll just go ahead and switch the screen sharing again all right so with that up uh we now need to set up our grafana instance as well as influx db which we'll use to store the performance metrics that we generate with our load test and in the k6 repository there is actually already a docker compose file available so the only thing we'll have to do to be able to do this is to run docker compose up dash d influx db graphona this of course requires you to already have docker installed on your computer but on at least on windows and on mac os it's fairly easy to install docker and you can usually do it without that much hassle so um if you want to try this after the talk make sure you have docker installed before you start all right so we can now see that we have a grafana container as well as the influx db container we now need to make the influx db container available over the public internet to be able to send metrics to it from our load test so let's expose that using ng rock in in a real scenario or in a real production environment we would probably do this by configuring our firewall to to port forward to our server in this particular case but to keep the scope as small as possible for the demo we'll just use ng rock which does all of this for us so angie rock http 8086 which is the port of influx db ah apparently i had that open somewhere else one second let's try again all right we can now see that ngroc has created a url for us that is publicly accessible and that will lead to our influx db container so we'll take this url and we'll add this to our workflow um using the flags property so here we can tell k6 or we can use whatever flags we would use in the cli just the same so we'll tell it that we want to configure an output and that that output should be influx db which is available on this url and once we push this the load test should start and we should also be able to go into grafana and see that we get performance data so let's push this and then switch back to the github window as you can see our latest commit actually triggered a test run and it has the same message as we used in our commit message and if we if we open it up we'll be able to see the logs just as before we can now see that it actually is configured to use angu rock as its output in the summary as well and if we now switch to uh if we now go to grafana which we have available at our localhost we'll be able to open a pre-configured dashboard that actually shows us the metrics generated from our load test i'll go ahead and up the duration of our test to one minute just so we'll be able to see some more data in real time all right our test is running and we should now be able to see data coming in let's turn down the range a bit to five minutes or so and here we go in about 30 minutes we've created our first testing load test script we'll set up our ci to use that and execute it automatically and we've also been able to create visualization that will show us what is going on in a more easily digestible or understandable uh form and if we wanted to elaborate on this or rather or or rather expand on it then we could for instance create a load test that simulates a whole user journey say for instance that you work for a e-retailer and you want to test the web shop then you could for instance log in search for something from from the product catalog add that to your shopping cart and check out and make sure that um that is done at scale maybe a hundred or a thousand virtual users concurrently and do that for an hour and see what happens and by doing that we'll be able to assess what would happen for instance if uh we will get a traffic surge during black friday and let's see if there was anything else i wanted to mention yeah one last thing if you are thinking of running this in production which i strongly encourage testing in production or the actual environment where your users will be in is really great it gives you a much better picture of what the actual outcome will be i mean regardless of how much effort we put into creating production like staging environments it will still not be the real deal and as such the results will not necessarily be representative and but just as you wouldn't learn math from scratch by trying to solve the traveling salesman problem we should learn to walk before we try to run so us testers of course already know this but we should always start by testing in a non-production environment make sure we get what is happening and that nothing breaks in a way that we've not anticipated and once we feel confident with that we may then move into production repeating the exercise with a fairly low risk as compared to going directly to production and last applications create data so if you test in production this data should be carefully segmented so that it never crosses the boundary into real user land for instance if you have a scoreboard showing the most most active users of your app then you probably don't want the number one spot to be your test user so you need to make sure that you plan ahead and that you make the blast radius as small as possible to avoid degrading the experience of your actual users and that concludes my talk so i'll let jonas on again and we'll probably see if there are any questions yeah thanks everyone it was great talk yeah yeah so are there any questions you could raise your hand so since there are so many can i give you the word yeah mike i have a question sure hi hi uh what about the performance counters from the the target servers can you uh integrate that in graphana too yeah for sure you could for instance uh set up distributed tracing or observability or whatever you you want and have that imported into grafana as well if that's something you're interested in i'd recommend you to actually a little bit of hidden ad here but to attend a a webinar by new relic next week where we will go into detail on how to how to integrate distributed tracing and observability into a load test so you can correlate this data but it's certainly possible you can do it with grafana you can do it with datadog or new relic or whatever apm software you use for sure okay because you you want uh if you have a slow response time you want to have the dig in the region for the lowest response time for sure that is really helpful right if you for instance were to hit the main end endpoint that you're testing and suddenly you get a http request duration of say two seconds then you'd probably want to know what action is causing that delay and especially if you're in a distributed environment where where services communicate with each other over http behind some kind of api gateway then you would probably want to correlate that with whatever handover time you have between those services behind the api gateway so for sure that makes total sense and it's and it is possible yes to answer the question yeah it is it's a matter of uh if you have the rights to proportion and where the environment is and so on of course yeah for sure and also it also usually depending on if you're on a cloud provider like aws they usually have it built in but if you host yourself then you probably would need to instrument your application as well to make sure that it actually sends out these metrics as as things occur in the application yeah yeah of course hi simon i have a question related to the virtual users if we want to kind of if the users required valid logins to be able to access the environment um do we need to set up all virtual users with valid accounts beforehand i mean if i would like to scale up the amount of users that would certainly be one way to do it to create test user that covers all the virtual users that you will be that you will be running but another way to do that would be to actually build in say the sign up flow for instance into your load test and have the users actually create an account for themselves run the load test and uh using that account but for sure if if you have to log in to test then you either need a api token or you need to do some kind of data you need to do some kind of correlation of data to make sure that you get that token into your request because while you can record your user journey that would uh create a static login token right so once that token expires then the load test will fail so you'll have to do that in some dynamic way for sure um and with the test recorder that we provide for for chrome that is to some extent built in so it does the correlation automatically for you and so that's absolutely possible or a way to solve it but either you'd need to create enough user accounts to cover the virtual users or you would need to have the users sign up themselves okay thank you are there any more questions so hello uh uh i'm just curious i didn't quite follow what's the setup with the server the controller driving the tests and the possible load test clients that are serving the load towards the server i assumed you run everything on the same machine here or did you use yeah for sure with k6 you usually don't have to to create a distributed uh or you don't have to create distributed load generator so you but you rather be able to run the whole load test on one machine we've been able to simulate up to 30 40 000 users on one machine so for most use cases it should be enough with one machine so no additional server should be needed but we als we also did this in two steps the first step we run it locally on my machine and then we run it on a smaller uh virtual server or container provided by github and the exact specifications of that machine i'm not really sure of but i know of companies or organizations that have run at least a thousand virtual users on those as well so the ubuntu step you did was that was served by github that liam or whatever you should call it exactly so so they take care of the provisioning of that and we just get to run our test and if you want to get different data from different parts of the world so like you want to simulate some users from south america and some from australia and so on to see if there is a some sort of behavioral difference that's certainly possible we have a project on github called the k6 operator that would use kubernetes to do this where you can force parts of the test to run on different nodes and you can place those nodes wherever you want in the world but that's quite bothersome so in that case if your use case requires geographic distribution then i would actually recommend our managed cloud service where we do that for you so you just you just specify the regions and it will populate servers equal to the distribution you set up in these regions cool anyone else i think matt's your son raised his hand do you have any questions he solved that by disconnecting instead so i think it's it's fine all right and i have been recording the session as well so i will make that available of the meet up and send out the link to it and i'll keep the repository the demo repository on github so if you go to my github account github.com then you can see the files we pushed there and you can even fork it and try it yourself and you have my twitter as well if you have any further questions or if you want to get in touch and talk more about performance testing or chaos engineering or any other topic related to reliability so i don't know more questions or otherwise i think oh it doesn't seem like it and then i think we usually see him on our radar upload and thank him for today thank you so much for having me it was a pleasure yeah thank you very much thank you thank you so much thank you thank you thank you thank you very much

Channel: k6

Views: 1,894

Keywords: development, testing, performance testing, load testing, k6, open-source, test-automation, automation, software testing

Id: Xyq6GItCAvY

Length: 41min 51sec (2511 seconds)

Published: Fri Jan 15 2021