Intro to load testing with k6 and Grafana (k6 data source plugin and Prometheus Remote Write)

hello everyone thanks for coming good evening for me in my current home in the netherlands but i know that it's a different time of day for many of you welcome everybody this is intro to load testing with k6 and grafana and i'm nicole van der hoeven so before we start a few notes we are recording this webinar and if you've already registered you'll get a recording of it later during the talk please feel free to type out your questions in the q a i will be leaving time to go through them after after the presentation and i'm also going to be joined for the q a portion by robin gustafsson who is the kasich's ceo so in case you have any questions about the future of case x or the road map or the acquisition that i can't really talk about it's also your opportunity to talk to him so i am nicole mendoza van de ja i've been in performance testing for over 10 years now i've worked as a consultant for most of that time particularly in australia and in europe using many different types of load testing tools i'm also a developer advocate at k6 and that means that it's my job to learn cool new things and then tell everybody how i did it so that you can do it too and that's what i'm going to do today i will start off by talking about why it's still worthwhile to test in pre-production a bit of a minority opinion in an industry that has swung from testing in pre-pride to testing in production in recent years then i'll talk about what k-6 even is and the reasons that i decided to join the k-6 team 10 months ago after spending quite a lot of time in my career using tools like jmeter gatling and loadrunner and then the meat of this presentation will be about why kasich and grafana make sense together and also how to integrate the two i'll be switching from slides to live demos so that i can show you exactly what i'm talking about and how you can do it too if you've been to k-6 dot io you might have already noticed our strange obsession with crocodiles they're everywhere crocs are an important nod to k-6's history kasich started out as a company called gator hole that built an mmo rpg i wasn't around for this but apparently they also had two real-life crocodiles in tanks in the original offices in stockholm sweden but fast forwarding to today i think that the crocs mascots still really fit the current business of the company i do think that load tests are a bit like crocodiles themselves because these crocs might look all cute and inclusive here but crocodiles in the wild can often look like this crocodiles are vicious and destructive load tests can be dangerous too after all the only things separating a load test from a ddos attack are intent and consent if you're not careful you absolutely can wreak havoc in an application with a load test that's kind of what makes it fun too however for the most part this reputation of load tests being destructive is undeserved at least if you're doing it the right way so why does it get such a bad rep in some teams load testing is a bit like winter you know it's coming you know it has to happen but you never enjoy it when it's here let's talk about some of the myths that make load testing truly disruptive in some situations the first myth is that load testing is about breaking an application now the purpose of a load test can be to find the breaking point but it could also be to recreate normal production traffic so this isn't necessarily true the second myth is that load testing is the same as sending a bunch of api calls many times there's a lot more to to load testing than just volume and in fact when a team doesn't focus enough on workload modeling load testing can be pretty destructive and the third myth is that load testing is expensive but a load test should start small then you can ramp up gradually that way you're saving on environment and support costs during your tests as well and the fourth myth that makes load testing dangerous is the myth that load testing can or should be done only in production i say there's a place for it and but it has to be in a certain way there's a place and a time and a way to do it and for many use cases testing in pre-production environments is just the most prudent thing to do so if you're here then i'm betting that most of you are already pretty cluey when it comes to observability observability is a huge factor in building performant and reliable systems but it can't exist in a vacuum the other half of observability is data something to observe so in production this is data this is traffic from real users accessing your application they visit pages that show up in google analytics create data like user accounts or orders that must be saved in a database and so they change the system in very real ways this need for data is also why many teams use observability in production observability has to go where the data is and in contrast test environments often look more like this rows and rows of empty seats except maybe for some testers trying out a weird combination of features not exactly a great proxy for production yet there are still some good reasons to observe or test in pre-production right it's cheaper it's less destructive we can do it earlier so there are fewer dependencies on other components and we can avoid having our users become our testers as much as we can so the real problem is not whether or not we should be testing in pre-prod the real problem is how to make this empty room more realistic in mid-june just three months ago k-6 joined the grafana labs family i only found out a few days before that myself but it immediately made sense to me i used graffana way before i even used k6 and that's because while it's true that load testing gives observability data to observe in non-production environments it's also true that without observability load testing is a little like flying in the dark so if you've read google's site reliability engineering book they talk about four golden signals that we should measure in a system it's latency traffic errors and saturation so load testing does three of those but it doesn't do saturation and it doesn't give us visibility into the system so there's still a missing piece so grafana brings observability and casex brings data so they work really well together which is why we've been using each other's products for a few years now before there was ever any talk of acquisition so let's talk about k6 when i joined k6 it was after i had specifically reached out to them because i was intrigued by the promise and the premise of k6 here are some of the things that drew me to it after spending my entire career using other tools it's developer and tester friendly as a tester i was relieved to find out that k6 was written in javascript because it's a language that i'd already learned from using tools like selenium webdriver or puppeteer cypress or playwright and then for a developer javascript is also pretty ubiquitous so it's a really good scripting language from that point of view and then there's the fact that almost the entire k6 team is made up of engineers and you'll get to find this out from when you talk to our ceo robin who often gets his hands dirty coding new features in fact maybe you can ask him to share what he's been working on later so as engineers we've built a tool that we like to use so there's a k6 vs code plugin because that's what we use and lots of script creation options and it works with many of our favorite tools k6 is also performant it's always struck me as a bit weird that performance that some performance testing tools are huge bears to install and execute and maintain some of them are so resource intensive that they themselves are the performance bottlenecks in the system so who's testing the test tools right because k6 is written in go so there's none of the additional overhead that java based tools are known for java was created in 1991 by the way so 30 years later were still using it to test modern applications kasich's is designed with performance in mind k6 is also easy to scale you can start a test from the cloud ui or from the cli and then you can switch between the two seamlessly so you can go from running a shakeout test on your local machine to running at scale on the cloud using different geographical locations another other cool things about k6 you can run up to 40 000 users on one load generator so our cto powell tested this and was blown away when i find out obviously this is dependent on the script and the application that you're testing but just the fact that this is possible is pretty is pretty amazing k6 integrates with a bunch of tools for cicd apm notifications alerting etc casex also has a similar business model to grafana where we have a locally hosted or locally executed tests using k6 itself and then we and that part is free in open source and then we also have a sas platform around it using the same tool but then adding convenience and enterprise features so just like with grafana you can get started for free using the open source version of k6 and upgrade if it's necessary and some users never need to upgrade either so this is a simple k6 script after you install k6 which for mac os users like me you can do by executing brew install k6 you can copy this code save that as a javascript file and execute your first test from your laptop so this script involves importing the http library from the k6 and then defining a function that includes an http get request for the url of the site that you want to test and a lot of tools stop here but we don't let me show you what this looks like in my ide this is vs code this is a simple the same script that you saw earlier it's just i i changed the url all this does is it gets information about a pokemon in this case pikachu so let me show you how to run it this is my terminal and this is just you just have to type k6 run and this is the name of the the test this is k6 this is what it looks like when you run it locally you see execution is local the script that you ran and then you get a bunch of metrics all of this is also configurable so if you want this to look different or you want custom metrics you can do that as well and the primary thing that we'll mostly be looking at is the http rec duration which is the response time and the average in this case was well there was only one iteration so it was 594 milliseconds which is not bad considering this is just a an application that i've got running and isn't really meant to be performant but this the problem with this is that this is still really basic and you will run into problems if you try to run this at scale here's what i mean this is the same script with a little bit more added to it so first we have the options here so now instead of this script where i'm just running it once with a single user now now this is where i can set it so that it runs five users and then set a duration for it too so that's part of of having a load testing tool being able to create a load profile around it and then this is the same request here but you can see that i'm i'm tagging it with a name and i'm including this check here now this is a good point that i want to stress because i still see a lot of of load tests that don't have checks and i just think that that's a wasted opportunity to verify that you are where you think you are or where the script is in the application so in this scenario i'm checking that the response code is an http 200 but i'm also checking for something very specific in the body of the response that's returned in this case i if i'm if i'm getting information about pikachu then i definitely want the word pikachu in the response because it's still possible to get an http 200 but then you get an error in the response and then this is a think time and the reason that i'm doing it this way is so that it's not um the same amount of thing time every time this is just a delay that'll go from zero to five so we can go further from here because you might say well it's not so great to have to always be be fetching the information for pikachu right what if we want other pokemons so this is same script again but further along and now there's a bit to go through but let's let's take it step by step so this is the same function that i've been showing you i've shown you two versions of but this time i am using a csv file full of pokemon names because apparently there are hundreds of them now so instead of always looking for pikachu and this one i'm i'm getting something from that csv randomly and that's also the same one that i'm looking for that i'm looking for that text in the response that's returned you still get the the thing time there and in this script i also added a bit more of a scenario here so now there are two scenarios that i'm setting the first is the one that that gets the pokemon that we've been looking at and what's cool is that we can set stages here so the way it's set up right now for the first minute it's going to be ramping up from 0 to 10 users and then for the next five minutes it's going to hold that steady state and i can change that to whatever value i want but you can also see that this scenario has a different executor from this scenario and it's not important to know exactly what each executor means because there are we have several of them in k6 but what is important is that this one has a completely different load profile now this function is set up so that it runs once with one user and it starts five seconds after the the test begins so it's kind of cool to be able to to manage which transactions and which functions go when and then there's also the thresholds here so right now i've got an error rate threshold and then i've i also have one for load generators because it's a really good idea to be watching the cpu and memory utilization of your load generators themselves so that you're sure that they aren't the bottleneck of your tests and then this is the response time i am checking whether the 95th percentile response time is less than or equal to 5 seconds so all of these things make make it make your test more realistic than this script so let's go back to the presentation and the difference between a request sender and a load testing tool is that a good load testing tool should have features that enable you to fill that room that we were talking about so to speak we're trying to generate traffic that replicates user behavior so we want our crocs to be smart ones or at least as smart as you want your you think your users are so here are the some of the things that i showed you in that quick demo scenarios are user flows paths through your system that you think users might take this involves multiple requests and maybe of different types adding a think time to represent the time that a real user would take to read text on the screen also makes a load test more realistic thresholds are sometimes called requirements and other times they're called slos all three mean the same thing they are boundaries that you set for what's acceptable in your load test a good load testing tool should also let you use test data so that you're not passing the same static values as parameters in your requests so if caching is turned on for an application server and you're requesting the same resource every time you may be getting faster response times than what you would really expect in production which is not a good thing so i've already shown you how you can run load tests using k6 and i've said that you can run up to 40 000 users why would you need k6 cloud well you don't always need k6 cloud and many of our users do find that the features of ksx oss are enough for their circumstances however if you do find yourself wishing that you could run on the cloud without having to provision instances using your own cloud provider or if you just want an easy way to get started that doesn't involve scripting then you might want to look at k6 cloud so i'm going to demonstrate that for you now as well so this is k6 cloud or my account anyway now you can create a new test from here but first i want to show you how to run that test that we that i showed you earlier on k6 cloud so i'll run this this test here um just clear that and first you can run your this test locally and then just output the results to k6 cloud that's valid as well because while you're sending the results there you may not necessarily want to be using you know the the actual cloud generators for maybe like a shakeout test so to do that you just do k6 run simple.js and then you add this flag so dash o cloud and you should see in a second that the execution is still local but its output is the cloud so this should be running in the cloud now if we have a look okay oh there it is so you see the simple.js and that's the one that's running let's see what it looks like so it confirms that this is a local execution it will run for five and a half minutes and it has five virtual users and we're starting to see the the requests come in now so you can see that if we had just looked at k6 the oss version there's not as much feedback as to what's happening so that's a situation where being able to output it to the cloud and seeing exactly you know how many requests are made and you can also scroll down and see this get pokemon uh function this is the one that's returning a redirect so you can also see the http response codes and any checks that you might have set will be in here as well but let's stop this for a second and i'll show you how to run it on the cloud so we're starting off the execution in the cli but then we're running it on the cloud so to do that it you just do k6 cloud and then the the file name of the test so you do that it does the same thing it's it's transferring over the script but this time the execution is in the cloud so we'll go and see that here as well and that's where it is and we'll also see this um this diamond here that shows us that there's one test that's running on the account if we click on that and the test is starting here as well except that the load zone is now ashburn which is the default now what if you want to run your own test on the cloud ui so you just click create new test and i'll just use test builder because we kind of if you want to you can just copy the script that you had and paste it into the script editor but i'll show you the test builder as well because it's pretty cool so i'll add a request here i will say this one is home and i always like to number my my requests and then i'll just hit the test k6 page and always add a sleep and then i can also set here which load zone i want it to run from so there's a lot here and you can also add different ones um ramping vu's let's say let's just do 10 i don't this is a test site um and so this way you can also select the stages so you can add as many as you want you can say now after this i want 20 users and i want to run that for 5 minutes like this so now it's ramping up for for that time and then you can say this is how you can have like steps to your to your test you get the idea and the cool thing is you can also use a test builder as a way to generate the script so if i click here then you can copy the script and you can run it in the cli if you want to as well so for now we will save that and i also wanted to show you this is kind of something that's better to do with tests i've already run but if we go into this project that i've got i also have this nice list of all of my executions and i can have my notes here so that i know which one like what configuration i ran with and this star is saying that this test was the baseline so i really love that when you go into a test you can click here and then set it as the baseline or you can compare it to any other test that you want so you can do that this is good to do if you're comparing it with a baseline as well and any any test that you run on case x cloud if you go into it you can also share it so here it'll do share test results you can copy that and now anyone that you give these this link to will be able to see the results but not be able to run on your account so i mean we all have too many accounts as it is already right okay so in a nutshell here are some reasons why you might want to use case x cloud k6 cloud k6 cloud is convenient using it means that you can switch from local executions to cloud executions and scale up your test as you go many people who use k6 cloud use it for ease of collaboration within a team so when you run a test on the cloud you can share your results and that's even if they don't have a case x cloud account then if you do want people to be able to run tests with you then you can invite them to your organization and manage the permissions that they have over your projects and tests and then k6 cloud also makes it easy to implement continuous testing so i showed you that you can run baseline tests and compare new tests to the baseline you can also schedule test tests to be executed regularly and send the results as messages on slack okay so that's an intro to k6 but what does this have to do with grafana well if you're using k6 oss the standard cli output is not an ideal way to analyze a test or report on it right and even if you're using k6 cloud it might be valuable to also have your load testing results in the same tool that your team is already looking at for server monitoring like grafana so k6 can be integrated with grafana in two ways first there's the k6 data source plugin which sets up k6 cloud in particular as a data source in grafana it works with any instance of grafana but it doesn't work with k6 oss and secondly there's prometheus remote right casex doesn't yet expose load testing results in prometheus format but this way will let you utilize the remote right feature of prometheus to effectively push the results to prometheus anyway this works with both k6 oss and k6 cloud and with any instance of prometheus so let me show you how both of these work first we'll start with the k6 data source plugin what you will want to do is go to this page while you're logged into your grafana cloud account and by the way in for this demo part i'm going to show the integration between k6 cloud and grafana cloud just because that's the easiest way to set it up but you could do this with locally hosted grafana and casex as well so you just have to go here and click on install plug-in uh a note before if you've already had this installed before this presentation upgrade it because i just pushed a fix to the dashboards earlier this week and you can just re-import if you go here you'll also be able to see an option to upgrade if you already had it installed and then once you've installed it you can go into grafana and you get to the screen by going to the cog and then data sources and then here you can just search for k6 cloud and that'll ask you for an api token which you can get from your k6 cloud account now i won't do it here because i don't want to expose my api token to a few hundred users but um you just click here and then click api token that will give you a string that you copy into grafana cloud so once that is set up you click save and test and then don't forget to click on over here to the dashboards this is how you also re-imported them if you already have it installed but if you don't have it installed you'll just see import here but let me show you what these look like so this is the list of tests that you've run on your cloud account but also note that there but you can select between different organizations here so i happen to be part of three different ones and also by project so i think that the one that we ran is the default one so we'll see that has a bit more data and and this is every every test that you've run on that project as well so oh it looks like that test that we ran already finished so let's click on that from here and this is the dashboard that you get by default that was imported in with the plugins so scrolling down here you can see the response time and throughput here if we had any threshold you'd be able to see it here as well and then the checks so this is recreating the experience that we would see here on k6 cloud so that way you don't even really have to go to 2k6 cloud at all if you just want some some teams or some engineers to see it so the next way that you can integrate is by using the prometheus remote right feature so let me show you that as well on k6 cloud you can go to this cloud apm section and i'll just show you how to and click create new configuration here i'll just show you the options so you can integrate with any of these providers but let's look at the grafana cloud one you can set which metrics you're going to want to to pass through to prometheus and to your grafana dashboard and to get this information the credentials and the remote right url you'll need to get those in your in your portal so if you go to my account on grafana cloud then you'll see everything that you do have for your subscription then you click on details for prometheus and then the prometheus remote right endpoint will be this you can go ahead and copy that and use this as the username and you'll have to generate a new password or api key so you put that information here and click save but it's not done yet because that's just it just that's just adding the configuration for the entire account you may not want all of your tests to all of your test results to go through to prometheus so when you're creating a test so let's go back to the one that we were creating oh i think we all yeah that's right we already ran it so here's a another one that that i did this is just a test for the cloud apm and then when you go to the cloud apm section here you can it'll be disabled by default and then you can just enable that and you can also configure it a bit so if this particular test has some custom metrics then you can put it here and then you can just save it and run it in fact i'm just gonna do that now okay now the other way that you can use this prometheus remote right feature is by doing it in the script so i'm not going to show you what what that looks like but there is a way if you don't if you want to bypass the ui and you want to do it from your cli for example there's just a an option that you can set so that you can you can also specify your remote right endpoint for prometheus over there the whole point of this is to be able to put your metrics all into prometheus that's handy if you're already using prometheus right for server monitoring and okay the test has started now and when this ends you'll see that there will be like a prometheus um thing here text here as well but let's just go ahead and see what that looks like in grafina cloud this is a dashboard showing the results from a previous test that i ran and it is a dashboard that i made myself so let me just show you what that looks like if you edit one of them you see i have the prometheus data source selected and in metrics browser there's a whole bunch of case x metrics so you can really do with this whatever you want i don't have anything else going into prometheus right now so that's the only thing that's there go back here and we should actually be able to see from here too that test that we just kicked off so let's go to the last five minutes yep there it is so this this demo app is is not really very performant as you can see but there's a lot of information here that you can tailor to however you want depending on what your team is really looking at go back to the presentation okay so when should you use the k6 data source plugin and when should you use prometheus remote right well the k6 data source plugin is better for recreating k6 cloud dashboards so if you like what you see on k6 cloud but just want to bring that experience over to grafana as well well this is the easiest way of doing that and if you have many past tests on your k6 cloud account that you'd like to see on grafana the datasource plugin will also bring all of those over so you'll be able to select any of the tests on your account and view those results on grafana in grafana and then the plugin is also good because it comes with pre-built dashboards so you just import those instead of having to create your own now the prometheus remote right integration is better for it's better if you're using k6 oss for instance and not k6 cloud i didn't show that way for the sake of time but what you can do is you can output k6 results to telegraph which will then format the metrics which will then forward the metrics to prometheus so we have more information on how to do that in our documentation that's casex dot io slash docs but you can also just do a search for it there so if you're already using prometheus incorporating k6 using remote write is probably the way to go that way you'll have your load testing results where the rest of your metrics are too and then if you're using multiple apm providers such as is the case in larger organizations where different teams use different tools for example this way we'll work with any providers that support prometheus which is becoming an industry standard i also think that prometheus remote right provides the easiest setup if you're using both k6 cloud and grafana cloud however nothing stopping you from using both i've personally installed both and i find that i use them for slightly different reasons so i like that the data source plug-in lets me do it by test instead of by time and i've been using k6 longer than i've been using the grafana integration right so i like that the that through the data source plug-in i have all of my tests even if i didn't think to set up the integration beforehand and then i like prometheus remote right because i think the integration is a bit better for real-time monitoring because i can also set up my own dashboards and only see exactly what i want to see but whichever way you choose having your results your testing results in grafana gives you the visibility into what's happening because otherwise testing in production often feels like this don't get me wrong there are still good reasons to test in production and good ways to test in production that don't result in this however sometimes it seems to me like the trend towards testing in production is almost taken as a license to test only in production and those are very different things in contrast doing load tests in pre-production environments especially with k-6 and grafana is more like this testing in pre-prod lets you validate an application's performance early so that you can identify and address the effects faster it lets you find performance bottlenecks that without it coming at a cost of a reputation or potential customers who are annoyed because they have to spend so much time on your application and testing in pre-prod also gives you insights on how close your application is actually getting to breaching your slos but these benefits aren't fully realized unless you have both sides of the puzzle observability tools to see what's happening and a testing tool to generate the data thank you for listening i'm happy to take any questions now on this slide you see a bunch of resources that you can use for recreating anything that you saw in this presentation or finding out more thanks for your attention and robin if you want to come on now as well feel free to ask him any questions

Channel: k6

Views: 1,179

Keywords:

Id: tFsIgbqXbxM

Length: 39min 45sec (2385 seconds)

Published: Tue Sep 28 2021