The Best Performance And Load Testing Tool? k6 By Grafana Labs

and now something completely different let's talk about performance testing and load testing and other types of testing that requires a big volume of data a big number of requests a simulation of many users [Music] typically i would be using j meter or gatling those are the tools that i've been using for 20 years now or something like that and they're great tools with huge support massive number of contributors and so on and so forth but this is not 1999 anymore the year is 2021 i wanted to find a tool that is a bit more modern a bit more different a bit better than those there are quite a few tools in that area but they are all limiting i'm not going to name names now i don't want to shame anybody which is something new to me i tend to shame a lot of different providers and vendors and tools but i'm going to avoid doing that and i'm only going to say that most of the new tools are either 100 cli based and that's not really good because i want to define what my tests are some of them are yaml based and yaml is not flexible enough for me to be able to describe the scenarios i need so i kept looking and looking and looking and trying different things and then i stumbled upon the one that is absolutely amazing and it happens to come from a company i admire from a company that has products i really really really like and that company is grafana labs within their portfolio there is a tool called k6 which i love i think it's absolutely amazing it does everything i need the load testing or performance testing type of tool to do and i wanted to share that with you so let's explore k6 as the go-to tool for load testing and performance testing and other types of tests that require huge volumes of requests or user simulations and stuff like that if you're not familiar with load testing and performance testing first of all you should be ashamed because we should be load testing our tools to see how it will behave once it reach production and starts being used by thousands or millions of users because that's what we all want we want a lot of people to use our software so we need to validate to test how our software will behave when attacked by a huge number of concurrent users and many people are using it at the same time so it's all about putting a demand on our system and seeing how it behaves whether it can handle what we throw at it that's what load testing is it is about validating better our system can continue working properly when under pressure you know many people using it at the same time i already installed k6 cli i will not bother you with installation because that's boring and i already cloned the repo with some sample manifest sample code that i will be using and if you want to follow along and reproduce what i'm doing the instructions with all the commands and references are in a gist and the link to the list is in the description so go and get it and let's play together let's start with a simple example actually everything is relatively simple with k6 but we are going to go simpler than simple at the very beginning the simplest possible so that you get the feeling of how it works so we have a couple of imports i'm importing http from the k6 http library and i'm importing sleep as well you will see why i need sleep in a second and then i'm exporting the default function think of it as main function the function that will be executed that contains all the code all the tests that we need it could be also references to other functions but nevertheless that is the one that k6 will execute and in this case i'm sending a get request to devopstoolkitseries.com and then i'm going to sleep for one second sleeping is important first of all for your mental health but in the context of flow testing sleep matters because if you're simulating real users many real users using our application we humans do not go from one page to another and click one thing and then another without any pause in between and sleeping for one second there is a good simulation of hey click this wait for a second click that wait for a second so it's closer to what we would do as people when using an application the important thing to notice here for now is that this is javascript but we are not going to run this javascript as javascript because javascript like node would be extremely inefficient node.js cannot handle thousands of users effectively but you know what can handle go so here's what k6 does it allows us to define our tests as javascript because javascript is easier for many people because people generally are more familiar with javascript than let's say go or c or other more efficient languages so we are defining our tests as javascript because that's easier but k6 is going to convert the javascript into go because go is more efficient it's a combination of two languages one javascript meant to provide easier interface and go being more efficient at running many concurrent users many threads in parallel then javascript then node.js so long story short we write our tests in javascript and k6 converts the javascript into go and runs it like there is no tomorrow and you will see soon that actually it is extremely efficient but we'll get there now that i have my tests this is a silly example we're going to go through more complex examples later i can run it by executing k6 run and then the path to the file that contains the test which is simple.js and let's see what we'll get the important thing to note to begin with is that the tests were executed with one we use or virtual users didn't say anything in k6 thought hey you want to run a sealed test this is a silly test i'm going to give you one user we're going to change that in a couple of moments for now i executed my load test based on one virtual user that is a simulation of a real user and that's not really load testing but it's a good start so let's see the metrics metrics are what really really really matters here we can see how much data we received how much data was sent the amount of time requests were blocked connected the duration of requests how many of them failed receiving sending tls handshaking dating the number of requests in total the duration of each iteration which is one second of sleep time plus whatever the amount of time was required for that single request the number of iterations the number of virtual users and the maximum number of virtual users there's a bunch of metrics over here and that's only the beginning because there are many more metrics that we might want to use think of those as default metrics or the metrics that are important and come out of the box but there's much more and we can define our own metrics but i'll get to that in a second for now what matters is to distinguish the built-in metrics those are the metrics that are built already in k6 and then we have custom metrics that means that we can create any metrics we want and use those metrics for assertions and the tests that we need or just simply data and those custom metrics actually both custom metrics and build metrics are based on few metric types we can have counters and gouge and rates and trends now if this sounds familiar that's probably because hopefully you already used prometheus those are very similar if not the same types of metrics that we can use in prometheus and you can probably see the connection you know grafana labs that has graffana which is a lot about logging and metrics and works very well with promote use and so on and so forth so you can see the connection here if you don't see the connection that means that you haven't used prometheus and then you should be ashamed you should try prometheus because promises is absolutely awesome but this is not the subject of this video and i'm going to move on because i realize that i'm losing it i don't know what i'm talking about anymore so let's try to spice up those metrics let's see how we can increase the number of virtual users the simulation of the users that we are using and luckily for you and me that's extremely easy we can just add a couple of arguments to k6 run command so let's do just that let's increase the number of users to let's say 10 right let's not go crazy yet we're gonna go crazy in a couple of moments but for now let's see how we can run that same test with 10 10 here 10 users so i'm going to tell k6 to run something and that something should be based on 10 virtual users so 10 concurrent users attacking my system and then they should do that for 30 seconds and that something is the same simple.js file as before we are still doing simple scenarios so the same thing as before but with more users 10 of them and during a longer period of time 30 seconds and now it's running and it's running and it will continue running for 30 seconds so let me fast forward to the end of those 30 seconds 30 something seconds and see what we'll get and the output is in the same format because i did not yet modify the output i did not deal with custom metrics yet or built in metrics so the output is the same but the data in the output is different this time we have a simulation of 10 users 10 virtual users attacking the system during 30 seconds and then we got the outcome so far everything is good i have no red flag anywhere nothing really failed everything is fine and since i do not have real tests over there i would not even know whether it's fine or not unless i manually visually inspect the data and say hey the average request duration is 151 milliseconds and the minimum is this and the median is that and maximum is this this this fast is this slow is this just as what to expect but that's not really the best way to use it because we should have more automation around it but i'm getting to that so bear with me so far you saw that it's relatively easy and straightforward to define simple lotus we're going to see more complicated later but for now i want to talk about performance and performance of k6 not of my applications in this case of k6 is one of the main reasons why i think that this tool is absolutely amazing it can handle 30 to 40 000 virtual users and that's a single k6 instance so 1k6 instance can simulate 30 to 40 000 concurrent users and that's a lot that's really really a lot especially when compared with other tools in the market to give you a perspective with 30 000 concurrent users we can reach around 300 000 requests per second or somewhere between 6 and 12 million requests per minute that's big especially when considering that we are talking about one k6 instance and if you need more than that more than hundreds of thousands of requests per second or tens of thousands of concurrent users then we can run distributed testing we can run k6 in distributed mode maybe in a kubernetes cluster and spread those tests across many different pods running in different nodes and then we can increase that number to something really really huge right kind of google size type of number of concurrent users but not everything is that simple because there are some operating system level limitations so you're unlikely ever to reach those numbers just by running k6 you need to modify your operating system to be able to handle those numbers and the reason is very simple because the number of concurrent threads is limited in operating systems by default and you need to tweak it a bit you have instructions in the official documentation so if you're serious about load testing and you want to reach really high numbers you might need to tweak your operating system and if you don't like if you run it from your computer from i have mac for example then you can reach a couple of thousand users couple of thousand concurrent users not much more which is still a huge number it's still a lot but not as much as what k6 can offer finally there are certain memory requirements i think that one virtual users needs around one to five megabytes of ram so for 1000 virtual users we're looking around one to five gigabytes of ram of memory so there are certain limitations imposed not only by your operating system but also by the amount of ram that you have cpu is not really a problem it's very good at handling cpu you don't need a lot of it but you might need a lot of ram depending on what you're trying to do and how many virtual users you want to have how many concurrent virtual users you want to have now let's go back to examples and one of the things i want to show you is options because i do not really want to specify true arguments the number of virtual users and the duration of the tests that should all be in the test itself and if i want to do that if i want to specify more things inside of the code i can use options so this code is almost the same as the previous code as simple.js with addition of options i'm exporting options and i'm saying hey i would like to have 100 virtual users because 100 is better than 10 and the duration should be 30 seconds and what those options allow me is to forget about arguments i do not need to specify the arguments hey this is the number of this this is amount of that and so on and so forth i can just specify all that as options inside of the code and i can just say hey k6 you should run whatever is defined in that file i'm too old to remember the arguments so let's see the result let's see how it behaves with 100 concurrent users and since this takes 30 seconds that's a lot of time right for watching a video let's fast forward to the end of the process and see the results the results are still fine nothing extraordinary my application is not misbehaving it is relatively responsive the maximum response time was 300 milliseconds which is not extremely fast but this application this is my personal site it runs on google running that's actually decent since i'm on a free tier and google run had to scale application during the execution of those tests but this is not a video about google cloud run so forget about it let's go back to k6 the tests that i defined and executed so far were not really good they were too simple for a couple of reasons in a real broad situation the number of users is not constant it is fluctuating right in the morning we might have less and then more users and then less again and then massive amount of users because it's christmas and so on and so forth and we might want to include those variations into the tests themselves and we can do that through stages we can say hey there are a couple of stages in the execution of the load testing and the first one should last for 30 seconds and target 25 users that means that it will start from nothing and go throughout those 30 seconds up to 25 users and that's not enough that's like foreplay and after that we should go for one minute towards 50 users and then we will be dropping towards zero users throughout 20 seconds so we have those variations hey we increase the number of users decrease the number of users this is one duration another duration those are the stages and we can specify other things in the stages but for now this should be enough to show you that we can have multiple stages and the second thing that was horrible in the previous test is that we had only one page test for attacking the home page only and that's probably not what you want to do and given that this is javascript i can write any javascript code more or less i will define a couple of different pages as a variable and then i'm going to loop through those pages and go through them all so first it will go to the home screen and then to post youtube and then to pause catalog and finally to the page with the address this does not exist and you can probably guess that that page does not exist and i'm doing that intentionally because i want to see what happens when something bad happens i want the simulation of an issue that would be potentially uncovered through load testing so i'm including a non-existent page to the list of the pages of my website and the third thing that was greatly missing is that there were no checks no assertions we could not validate whether everything is okay or no and i'm going to change that by checking whether the status the response of those requests is 200 or something else and also whether the duration is less than 200 milliseconds so the assumption is to begin with if the response is not 200 there's something wrong and the second assumption is that all the requests should be less than 200 milliseconds actually equal to or less than 200 milliseconds so let's run that and see what we'll get and while that is happening this is a great opportunity to tell you that you must absolutely must subscribe to this channel and like this video and join the channel because that helps me a lot keeping this channel alive i mean it's not a must you don't have to do that but it would be nice so think about it consider joining your channel and if you don't want to join the channel please like this video that helps the algorithm an algorithm helps me and i help the channel and the channel grows and all the stuff and i'll wait until you do that i shouldn't wait i'll fast forward to the end of the process as expected some of the checks failed 2106 requests responded with 200 which is great but i had 702 requests responding with something else something that is not 200 http response and that's normal because i included the page that doesn't exist but i wanted to show you that you can have different types of checks and one of those types can be response status and the other one was the duration 2805 requests responded within 200 milliseconds which is great but three of them three of the requests needed more than 200 milliseconds and that within the context of this test is bad and it means that that check failed as well and below it you have the same metrics as before i will not go through them again explore them on your own what matters is that you can see all the data you need and if that's not all the data as i said before we can do custom metrics and i think that i will explore custom metrics in a couple of moments now there is one important thing that you should know about checks and i will not tell you right away what that is instead i'm going to output the last exit code the exit code of the previous command which is k6 run and let's see what we'll get let's see what is the exit code of the execution we just run it's horribly zero now if you don't understand why that is bad you should because zero means it's okay and that means that if you executed this load test in your pipeline for example in jenkins or circle ci or argo workflows or tecton or whatever you're using the pipeline would think hey everything is perfect everything is okay so there is no need for me to stop anything there is no need for me to send your notification because those sets were great they were successful even though you can visually see that that's not the case or is it in a world of flow testing you should not consider every failure of failure because the situations of having only three out of thousands of requests last longer than what we would like it to be is not necessarily the reason to panic so k6 has the concept of checks which are just a visual representation of the results of the failures and successes of our tests of our validations but there is a different way there is something called thresholds that we can use to decide whether something is overall successful or no so checks are for your eyes only and thresholds is what they can use to instruct the machines to abort to consider something a failure so let's include those mysterious thresholds and through them i can show you how and what you should do if you want to run load testing as part of your automated processes instead of just running it manually and as i already said if we do want to run it automatically through pipelines or something like that we need exit code to be non-zero to be something else and we can do that with thresholds so i did threshold section but thresholds cannot be hey if you have one non 200 response you should fail no we need to be more intelligent than that we need to deal with percentages and averages and so on and so forth because we do not want to panic and abort what we're doing because there is one failed request we want to panic when some average or certain bucket of things is misbehaving an example could be hey i want two thresholds and one of those thresholds will be based on the request duration and i want 90 of my requests to be less than 200 milliseconds so i can have more than 200 milliseconds but at least 90 must be less than that and i want 95 percent to be less than 300 milliseconds and if there is five percent that takes longer that's okay and the second threshold will use tags and it will apply only to requests that are tagged with the key what set to home so that one will apply only to the request going to the home screen to a home page and it says hey home is more important than rest of the pages and 95 of the requests going to the home screen need to be less than 100 milliseconds and if that's not the case and this is now important i want you to abort if we do not say that we want to abort on failure then everything will be executed until the end and we will know whether it's successful or failed only after everything is finished but we can say abortion failure and delay abortion for 10 seconds what all that means what that whole threshold means is that 95 of the requests need to be less than 100 seconds and if that's not the case abort the whole execution stop what you're doing only if that is happening for at least 10 seconds and then within the default function the main function i'm iterating through different pages and for each of those iterations i'm going to the home screen and then going to that specific page so home screen that page home screen that's page but the home screen is stuck with the keyboard set to home which is the one that we are using in the threshold so that home screen is special 95 of the requests need to be below 100 milliseconds or the threshold will be reached and then it will abort and i can prove that this is a cicd pipelines type of friendly thing that can be understood by the machines by outputting the exit code of the previous command and this time the exit code is 99 and machines know that if it's not zero it's bad and they should stop doing whatever they're doing because that's how automation works it is mostly based on exit codes and with thresholds we have non-zero exits if we reach the thresholds and we did and another important thing to notice here is that the duration was only 10 seconds it wasn't less than 10 seconds because that's the grace period that i set in the code but as soon as it reached 10 seconds going over the threshold it stopped there was no need or at least the code defined that there is no need to continue running load tests after the threshold that specific threshold was reached now so far i was running everything locally and you have two options to choose from you can run things locally or in your servers you can run things yourself and you're free to do that because k6 is open source but you might want to consider using cloud options so instead of you setting up your servers and dealing with the managing and maintaining all that you can run things in k6 cloud so let's take a very quick look at what we get if we choose to use k6 cloud it all starts by signing up you need to sign up i will not sign up because i'm already signed but you should especially since there is free tiers so you can try it for free no questions asked so let's take a look at how that looks like after you sign up so go and sign up i'm not getting any money from k6 this is not commercial i'm not telling you to sign up because i get anything out of it but because you should try it out once you do sign up click on your users and then on the api token and copy the api token because we are about to use it we need to log in from our cli to k6 cloud and to do that we need a token so copy the token and then let's go and execute k6 login cloud we want to log into cloud and pass the token whatever the token is and now we can choose to run tests just as before on our own or we can use the cloud option and we have 50 concurrent users for free that doesn't sound like much but actually 50 concurrent users is pretty decent amount remember that's concurrent not users in total for small businesses and if you're not small business then hey you can spend a couple of bucks anyways let's run it for free for now and later on i will show you the pricing and you might freak out or not we'll see we'll get to that for now i want to run it i want to run it in cloud instead of on my laptop and i can do that by executing k6 cloud so instead of run i'm doing cloud right now which is a bit misleading but anyways k6 cloud and then the path to the file with the code which in this case is cijs so now the tests are not running on my machine they're not wasting my cpu and memory they're not wasting cycles on my machine but somebody's else's machine and i can see that it is initializing right now so let me click on that tiny graph the only run that i have so far and see what's going on it will take a couple of moments until k6 sets up everything in their own infrastructure but after those couple of moments i should be able to see the progress of my tests at real time and look at it pretty graphs right if you like colors then cloud is for you because you get blues and greens and reds and you know people don't like terminals and if you don't then this is the thing for you one important thing to notice is that if you run it in cloud then it will complete the test fully no matter the threshold so the abort and failure type of stuff that is important when we want to abort fast is not really being applied here when we run it in k6 cloud so it will run all the tests to completion even though we set the threshold and said that it should fail after 10 seconds if things go wrong and this is arguably a nicer a better view of the same things that we saw in the terminal we have as i said before pretty colors and knives graphs and we have here the list of all the different pages that were executed and we can see that this does not exist is failing with status 404 which is to be expected because i'm faking it and so on and so forth so what are you getting or what did we see that we're getting so far is somebody else runs it instead of us so no need to manage servers or whatever you need to run those load tests and prettier representation of the results now we are not limited only to terminal output and k6 cloud we can export the results to many different places it can go to amazon cloud watch apache kafka basics cloud csv data dog grafana influx db json net data new relic and i think there was one more which one is it oh yes that's the statsd is the last export that i know of now you might be asking hey why would i export those things there but you should because if you're using prometheus or new relic or data to monitor your system you want to probably apply the same logic to load testing as to real production monitoring and so on and so forth and probably leverage the same dashboards and same criteria and so on and so forth because load testing is really simulation of what will happen in production so whichever way we are using to monitor real traffic in production and to get notification when something is terribly wrong then we should probably apply the same logic to load testing because as i said before it's a simulation of production so if production is permit use try to use promote use for load testing if production is something else then try to use something else and try to apply the same logic to the staging environment or whichever environment is running your load testing and production and finally money money money money money money you want to know how much it costs whether it takes expensive or no well it's not cheap let me tell you that it's not cheap you get 50 users for free but that is usually not sufficient for anything but small companies so you might have to pay or run it yourself if you do choose to pay it's 400 bucks for thousand virtual users and thousand mutual users is a decent figure and if you need more than that then we are talking about 1 200 for 5 000 virtual users and if that's not enough you need to go for more you need to pay more but that's the thing if you're having millions of concurrent users then you shouldn't have trouble paying a couple of thousand bucks if you don't have many users then there is a free option it's a fair deal you're small you get it for free as you grow bigger you need to pay something for a service and services let me tell you that their service is awesome k6 itself is amazing i love it and using cloud is saving you from setting up infrastructure yourself if you're running in aws or azure or whatever you need to pay for that infrastructure anyway and you would save yourself trouble of managing all that and fine-tuning all the parameters to reach those numbers because hey remember what i said in the very beginning you need to fine-tune your operating system to be able to handle this number of concurrent threats so let's talk about k6 should you use it or should you not use it what are the pros what are the cons every single time i reviewed something and if i remember to do pros and cons at the very end which i do not always but okay that's life it's complicated anyways what i'm trying to say is that this is the first time probably the first time that i have nothing negative to say this is this is working really great this is absolutely amazing there are no cons maybe the price but it's fair to pay for the service so that's not really a negative thing either and there are many positive things first of all it's not declarative i like yaml i like decorative format but i like declarative format to specify the state of something that's why i like declarative formats to specify the definitions of my applications of infrastructure and so on and so forth declarative format is ideal for specifying the desired state but this is not the desired state these are the tests and i really do not think that you should be using any tool that allows you to specify tests of any kind including load testing in yaml or json or something like that that's not how we write tests and i'm really happy that k6 avoided the temptation to define everything example this is not a good use case for declarative form and k6 did well by leveraging javascript it is easy to learn the syntax it is easy to create simple things it is easy to create complicated things i love it i really really really like that it is using javascript and i like even more that it is not running javascript so javascript is the interface for you for defining stuff but it is running it through code and that means that it is really performant it can handle a lot of virtual users second it is very extensive you can do almost anything you want it can be as simple or as complicated as you want it to be and there is almost no limit to what we can or cannot do so that's really great it's extensive framework the only thing maybe potentially missing are additional libraries out of the box but hey its programming language is javascript you can figure it out you can import libraries you need and so on and so forth the performance is amazing i haven't used the tool in that area that could handle that many virtual users in parallel as soon as i would start reaching three digit numbers especially high three digit numbers or thousands maybe of users other tools would just collapse kind of like i don't know what to do i cannot do i cannot work with this or they would fake it they would try to convince me that they're running all those things in parallel while they were not really doing that and so on and so forth so performance is absolutely amazing i like that you can run it yourself or it can be sas you have freedom to choose whether you will do it yourself or you will leverage the benefits of using software as a service using k6 cloud it's up to you it's open source it can be as free as you want it to be or you can pay for the service and that's great and finally exports rock i did not show you how to export the data but you can check it from documentation and that rocks i think it's very important that we can export the results of our load tests to the same tool that we use to monitor production whatever that tool is mine is prometheus yours might be data dog or cloud watch it's really really really important that we can export to those tools and treat that data the same ways we treat other data and use the same type of alerting system and dashboards and it's absolutely amazing that we can export to all the major formats or the tools we use for monitoring that's k6 i love it that's my preferable tool that's my tool of choice for running load testing and performance testing and all other types of testings that require uh volume to be thrown at our system use it use it now immediately try it out you really really really should try it out if you have the need to run load tests and if you do not have such a need then you should you really really should everybody should slow test their applications you

Channel: DevOps Toolkit

Views: 39,679

Keywords: devops, devops toolkit, review, tutorial, viktor farcic, k6, k6 grafana, load testing, performance testing, performance testing tutorial for beginners, performance testing tools, performance testing tutorial, performance testing in software testing, k6 load testing, k6 load testing tutorial, k6 load testing example, k6 load testing review, k6 load test script, k6 performance testing, k6 performance testing tool, k6 performance testing tutorial, k6 performance, k6 performance tool

Id: 5OgQuVAR14I

Length: 36min 8sec (2168 seconds)

Published: Mon Dec 27 2021