JMeter vs k6: Comparing two popular open-source load testing tools

hi everyone i'm nicole van der hoeven and i love j meter but wait you're on the right channel let me explain as load testers we pride ourselves on being tool agnostic you know we want to say that we always choose the right tool for the job for the project for the client the reality is that that can be really difficult unless you're constantly putting yourself out there to learn the new tools one month ago i joined k6 and i want to talk a little bit more about why but first let's talk about things that j meter is traditionally good at first jmeter has a gui if your team has doesn't have that much experience in coding or if you're newer load testers and you just want something that's kind of like a drag and drop interface it's going to have to be j meter k6 doesn't have that at least not in the open source component of the tool the sas platform k6 cloud does have that but we're not going to be talking about that in this comparison we're just comparing open source tool versus open source tool secondly jamino's been around for a long time i actually had to look it up and it was first released in 1998 which means that it's had over 20 years of iterating and reiterating over the same core concepts and it shows jmeter supports a lot of protocols just out of the box and it has a lot of features you can really see the 20 years of development effort that's been put into it by contrast k6 is new it was released in 2017 so if your project requires a protocol that k6 doesn't support jmeter probably does thirdly jmeter's head start also means that it has a way more mature community there are so many people out there who are creating tutorials and courses and books and a lot of developers are also creating plugins for jmeter in comparison the k6 community is growing but small compared to j-meters both sides have extensive documentation but just because of the sheer amount of third-party materials out there i would give the edge definitely to j meter the fourth point is a huge selling point for j meter i think and it's distributed load testing jmeter comes natively with a way to kind of select a controller and you can install jmeter agents on many different machines to make them load generators that talk to the main controller that way you can kind of orchestrate far larger load tests than you would be able to on a single machine k6 doesn't natively have that you'd have to go to a premium service this is a theme that you're going to see in other open source load testing tools as well jmeter is pretty unique in that it comes with this mode and the fifth point is jmeter's built-in reports i think analysis is probably the most important part in a load test njmeter definitely makes it easy it has a lot of listeners that come with it already plus a lot of listener plug-ins that other people have developed for it on top of that it does recently i think it came in jmeter 4 it now has html reports that you can generate now it's a little it's still a little fiddly but it's functional that's not something k6 can do you pretty much would have to take the results and analyze it yourself i think it should be pretty obvious by now that i'm a j-meter fan girl but even i have to admit that j-meter does have its limitations the reason that i joined k6 is i actually think that k6 is pretty strong in areas where j meter is weak here's what i think k6 is strong in first is installation k6's installation was a breeze i mean being able to just do it in a command line it was just brew install k6 and it was there in seconds there were no external dependencies and it was only then that i realized how difficult it can be for a new jmeter user to get started with jmeter you actually have to install java first and you might not know which version you need because do you need an sdk or a jdk or jre or now oracle is flexing its muscles a bit on the licensing so some licenses are paid for but others are still free now it's just a little bit more confusing secondly k-6 has most of the features that you're going to need in load testing built-in no plug-ins necessary now jmeter does support them but i think every jmeter tester knows there's like a standard set of plug-ins that you really should download when before you use jmeter for the first time because there's just a lot of functionality that isn't there if you're not including plug-ins so here's a few things that you won't have if you stick to a no plug-in vanilla version of j-meter custom load profile stages rendezvous you can't even parallelly execute requests because by default jmeter is going to execute them sequentially there's no http 2 support you can't randomize files within a csv there's no throughput shaping timer for setting throughput in like stages and these are things that k6 just does without plugins thirdly k6 is built with performance in mind see jmeter's limitation here is that it's built on java so if you run enough tests on on jmeter you're going to come across an error about running out of heat memory that that happens because now you can't just worry about the resources of the load generator you also have to worry about how much of those resources you're allocating to the jvm it adds another layer of complexity on top of that jmeter runs on a more traditional one thread is one virtual user model which is really pretty outdated and not very efficient on the other hand k6 runs on go so it's able to utilize the go scheduler's magic and being able to assign go routines to threads if you don't know what any of that means here's what it comes down to in jmeter you're going to be able to run a few thousand users probably on a reasonable load generator a reasonably spec'd load generator for k6 you're going to be able to run tens of thousands of virtual users same test just different tools number four is that k6 is better for collaboration jmeter strength is that it's gui driven but that's also a disadvantage because in order to open up a script or modify it the easiest way is to open it up in your copy of jmeter but there is also a script that's generated but it's an xml and it is really long-winded and the way that it's structured doesn't really lend itself well to just kind of quickly modifying something you're most likely going to have to download the script open it up in your copy of jmeter which makes it difficult when many of you are touching the same script in k6 this isn't a problem k6 is written in javascript so it's way less verbose and it's way easier to see which parts people have added or modified and lastly k6 is goal oriented one thing i never understood about j-meter is why it doesn't have thresholds at the test level so it does have response assertions but that works on a sampler level and you can have it apply to all samplers that's not a problem but it doesn't sort of aggregate it in k6 you can say you know fail the test if 90 of the response time is over you know three seconds or or whatever it is and that's like one line in k-6 so i think that it's really important to always have in mind the purpose for your load testing that's something that just is native to k6 so in conclusion jmeter and k6 are both excellent load testing tools that i will continue to use they're both open source and i think that they happen to be strong in areas where the other is weak so i think they should both be part of a load testers tools arsenal in the end there's no better tool there's just the better tool for your situation and your particular context so i will always encourage you to download them both try them out and see what works better for your team till next time

Channel: k6

Views: 3,618

Keywords:

Id: noZppBruOSY

Length: 9min 46sec (586 seconds)

Published: Thu Jan 28 2021