Browser testing and API load testing in the same script with k6 (k6 Office Hours #35)

hello everyone and welcome to another k-6 office hours as always i'm nicole van der hoeven i'm a developer advocate at k6 and i'm here today with a bunch of my other colleagues um robin do you want to introduce yourself first sure hi everyone i'm robin i'm the ceo here at k6 i'm mark i am the head of customer here at k6 and i'm tom working on the professional services team awesome so we're here to talk about something super interesting um but i also wanted to to talk a little bit about the difference first between front end and back end testing robin i know that you have like you've gotten a lot of people and probably mark and tom would be able to attest to this as well probably a lot of people have been asking for front end for a tool to to be able to test the front end but traditionally we've kind of just said that we're testing the back end with k6 what is what was your rationale for that yeah so so we basically from the from the start right from many years ago when we when we started with uh load impact before k6 and now k6 um we've always always gotten questions if if you know our solution can can run low test with real browsers and as you said we've sort of stayed away from real browsers just because we wanted to focus on the the testing of back ends right rather than testing sort of end to end through the front and then back-end and wanted to sort of like stabilize that kind of use case before tackling browsers because browsers is a whole whole other ballgame in a way and now we feel that we've come quite quite a long way with with k6 and the protocol level side of k6 so we wanted to to finally sort of tackle the the browser level uh side of of things both from both from the sort of perspective of low testing but also moving into some other areas of testing which has sort of been the the um strategy all along with k6 yeah i think that when people when most people talk about load testing they're only thinking about the back end but there's actually a whole layer of it that we're not testing at all and i know one of my personal like the way that i got started in performance and in testing in general was by reading steve sauder's book the um what is it called now better perform there are two of them when it's about web performance yeah and he was high-performing websites that's it and there's a second one that's like even faster something like that yeah i read those two and he says that 80 percent of the issues that are associated with performance bottlenecks are actually in the front end not in the back end so mark have you um how often has have has this topic come up in your discussions with customers yeah i guess to to give context around it over the the past six years i've been here working for the company uh it comes up often people say hey i want to test the front end i want to see what's going on on everywhere and we again up until now we always tested on the protocol level so we always told people like you you need to focus on the back end that's that's kind of where you'll you'll find the most uh exponentially scaling issues the front end is never under concurrency uh but over the years i guess even giving that reasoning for a while uh meaningful use cases have come up hey i wanna i wanna hammer the the back end and while that's happening i also wanna see what's happening on the front end in an automated fashion that that could really be useful for my testing hey when the back end is is sending all these resources slowly how long does it take until i get a meaningful paint on the or render on the the site itself so it it's a nice natural progression it i think makes sense so why haven't we done it until now robin well as i said it's we wanted to focus on sort of the protocol level side of things we've had so many so many features and functionalities that we wanted to implement first in k-6 before sort of going to the front-end side of things and now we finally have the maturity in the k-6 project with the whole sort of extension system and sort of a lot of the core functionalities in terms of scenarios executors thresholds and and the like that we felt that it was now time to to tackle the the browser level apis and and make a really good sort of k6 version version of that um to allow people to as mark said mix sort of the the back-end testing kinds of scenarios with the browser level scenario so you can get like the full picture of this is what it looks like and to end the end user experience looks like this while i'm exposing the or sending traffic to my backend systems to sort of stress them so i think another reason why we've been doing back-end performance testing is that when you're talking about simulating maybe hundreds or thousands or tens of thousands of users it is a lot more efficient to do back end load testing so testing at the protocol level is a lot more resource efficient and that's that has a direct impact on your bottom line on your cost because it costs to have to to provision machines on the cloud or or just to have them in your data center or whatnot i think just it's it's innately more resource hungry to to to have actual browsers that are being driven it comes with advantages but you also have to be careful to to keep that in in mind when you're running a test maybe it's more effective to do like a hybrid scenario where you generate a lot of your load on the protocol level and then you have a handful of users doing browser tests but tom i i know that you also have a performance testing background have you used other tools to simulate tests on the browser level before i have yeah on a number of occasions i ask like i don't know the answer yeah and this was with uh selenium as opposed to playwright which is what we're using in xk6 browser so and one of the reasons why it was attractive that might not be as obvious as some of some of the other reasons we've already talked about but you know some you know higher management would wouldn't really understand what um you know an error http response actually means as far as the end user experience goes and so when you're automating at the browser level you can capture screenshots when there's a timeout you're waiting for some element to appear it never does you get a screenshot and that screenshot will effectively be the same thing that a user might be seeing and it'll be some error page or something or maybe something uh simply not rendering on the page because it's it's loaded with uh some asynchronous javascript uh http call so being able to present that to someone makes a much more compelling uh this is bad news than if you just say you know this particular http request had an unexpected response yeah it's a good point because with protocol level tools when because you can download those javascripts but it's not you're not executing that because that has to be executed client-side in the browser or something so just getting just having a script and downloading it and getting the response time that is the that's how long it took to download that not how long it took to run which could be very different things right you're you can't really compare the two measurements um so before we go further robin would you like to actually announce it or re-announce it i re-announce it yeah that we finally have browser level apis available in k6 via an extension called xk6 browser so it requires requires a custom build of k6 together with this extension but you can find uh already pre-built versions for windows linux and mac os um on the on the release pages of of this repo that nicole has on the screen right now it's very exciting very exciting to finally have this available it's a public beta we should say also so it's not it's not sort of production ready yet we wanted to get this out early so we could get some feedback and and make sure that we take the product in the right direction um using sort of feedback from phone community and from customers etc yeah so i'm super excited about this because for the first time we'll be able to have uh browser level and protocol level scripts in the same well browser level and protocol level tests in the same script and we'll also be testing the front end and the back end but also kind of like what tom was saying right that if um if there are sites that maybe you could already test on the protocol level but it it's a little tricky or there's just a lot of correlations that could be a really good use case for for using the browser browser level tool right yeah particularly when you don't have high concurrent vu requirements uh it actually is quite feasible nowadays you know uh 10 years ago that would have been a different story or even five years ago but you know compute costs are continuing to to decrease and so the prospect of running actual browsers especially in the cloud environment is is becoming much more realistic and who knows maybe someday um it could even replace the need to do protocol level testing at all the key thing to be able to do and which i think xk6 browser is able to do is still give you http request level response times you know that's information that you would want to give to your developers uh to actually debug issues you know they're not going to be too happy seeing a screenshot of of an error page they probably want a bit more detailed information and i think yeah we're able to capture both that end-to-end response time as well as the individual http request breakdown now robin i know um by the way for those who don't know robin was the the first developer of this of this extension um he he was the one that did like sort of a proof of concept and then just kept building on it so i just always like to mention that because it's super cool that our ceo has built still builds features for kasich's but i know that one of the things that you you are you want or that you hope for for for xk6 browser is rough compatibility with playwright why did you choose playwright and not like puppeteer or selenium or or something like that one of the other tools that are available on the market yeah yeah so i guess we can start with selenium which is the the oldest one right in which most of the um both previously and i guess today as well companies building sort of solutions uh load testing solutions based on browsers they tend to use selenium as the underlying technology i think it's changing a little bit i guess more more usage of playwright et cetera today but uh selenium is based on another technology right web driver technology uh which has had some some issues and selenium's api is also prone to [Music] the creation of pretty flaky tests because of just how the api is it tends to be usage of a lot of like sleep kind of statements if you use k6 terminology uh to wait for for different things to to happen on the browser side and that can create quite quite flaky test cases um puppeteer which i guess is well came definitely after selenium right is based on a different technology chrome devtools protocol uh it's it was at least before when puppeteer came was a chromium specific technology and but it it allowed for more stability in in sort of the execution of test cases and then playwright is and this is as far as i know is the same or partially the same group of people but they moved from google to microsoft and basically re-implemented puppeteer but as playwright and fixed a lot of the things that they i guess felt they they could have done better in puppeteer and it also made the api sort of multi multi-browser and they they improved one of the most important aspects in my opinion they made the apis less flaky and focused on sort of like test stability right so they they have um they do a number of like checks when you do any sort of like input from a user perspective using the mouse or keyboards or clicking or entering text etc to make sure that the element that is the recipient of that action is actually ready to receive it right so make sure that the element is present in the in the dom make sure the element is visible make sure the element is not being affected by any sort like css animation going on on the screen and and that it can actually take the the input event that you want to send to it and that makes uh so baking that into the api makes for the test authoring to be much more robust right so you get you get test cases that fail less frequently which is super important of course for any sort of desire to automate testing because you don't want to have flaky tests in your automation pipelines etc so i really really like that aspect of playwright i thought it would from my perspective is a huge improvement compared to puppeteer and selenium and that they've added multi-browser support um for chrome devtools protocol based tools is also awesome i know they've modified and patched like firefox and webkit to make make that happen but i think it's an incredible sort of feat that they achieved this so that's why in my mind if i were to rank them i would say playwright is ahead of puppeteer which is head of selenium that would be my personal ranking which why which is why i think it made sense to based on the puppeteer sorry on the playwright apis uh because that there's there's clearly thought that have gone into sort of that api design which i i personally appreciate a lot yeah i'd agree with that but i also like not put them on the same scale i guess because playwright and puppeteer are like close i agree playwright is higher but selenium is a pain to run as a load test like i know when when i was doing some benchmarking on aws m5 extra large instances with selenium i could only get consistently about five users and and that's that's very little considering that with k6 with a protocol level tool you can get like up to 40 000 and then with um even with puppeteer which is what i use not playwright you can get like maybe 40 users it's it's a big difference right and then some people even use selenium not just on its own but like they use jmeter to drive selenium scripts which adds to the resource intensiveness of it so it's just at that point it's really like you must just want to waste money because it is a very inefficient way to to run a load test i mean do you get do you get people like that mark like wanting to run selenium yeah yeah i i guess going back to my comment from earlier it's come up i want to run a i want to run a real browser i want to test with the real browser um and my the default answer was always hey focus on the the protocol level um and also just the inefficiency you call that hey yeah you can run real browsers but browsers are are resource hungry and i share the at least when i joined k6 load impact at the time i was given the anecdotal uh comparison that hey we checked that out years ago running real browsers and uh whatever it was at the time we only got two on a single instance of of wherever we were running it and you can be more efficient and it's it's yeah more efficient to test at the protocol level yeah so robin you want to um do a demo yeah let's do it perfect all right so we're going to use this example uh woocommerce website that we have posted here on ecommerce.test.k6.io thanks to tom thanks to thanks to tom yeah so it's a it's an example uh eshop basically that has a bunch of items here so we're gonna do like a very simple scenario where we uh click on an add to cart button here and then we move to the cart to see what's in the cart and then we can do some some manipulation of the item in the cart so let's switch to my editor and hopefully this is visible to everyone so this is this is a k6 test case which is only using the browser level apis from xk6 browser so we have our module imports up here we have an import statement here importing the k6x browser module and this is because i have zoom in a little bit robin yeah absolutely that good too much yeah that's that's much better thank you okay perfect so i naturally have a custom built k6 version here on my local machine so i can actually do this import and down here we have some options so this is just going to use a single scenario it's going to use a shared iterations executor which means that the virtual users in this case would sort of share these iterations amongst them but i'm only going to run this with one virtual user and it's going to run in two iterations it's going to run through the the virtual user logic here defining this function down here twice right and then i have this threshold defined and the threshold is defined on one of the standard k6 metrics http rec duration which is the response time metric and this works uh exactly the same as it does with protocol level apis with browser level apis because we hook into the http i guess subsystem you can say of chromium so we can get these these timings and report them back into k6 sort of internal metrics processing system so that thresholds and other features that work on metrics work the same right so if we move down here to the test logic the first thing we have to do is actually launch a chromium process so again xk6 browser today only works with chromium based processors so not with firefox and webkit yet we can optionally specify some arguments here when we launch the browser so we're going to run this in a non-headless mode so we can actually see the the gui of the browser because that makes it a whole lot more interesting and we also have the slo-mo option which is just an option to slow down any interactions that we have with the browser in terms of like mouse input keyboard input etc so we can actually see things rather than just like speeding through it right it also makes makes the demo a little more interesting so once we have the browser launched we can interact some of the methods on that object we're going to first create a new page which effectively just creates a tab inside of the browser and once we have this page we're going to step through three different steps here i guess in in this test so the first step is just navigating to the front page of this e-commerce site the second step is adding an item to the shopping cart and then the third item is navigating to the cart itself and see what's in it basically so the first step here we use the go to method on the page object so we just provide url and a condition here for when we want this particular action to complete right so we will instruct the browser to navigate to this url and then wait for the network traffic to go to idle and that's when we consider this step to be completed and we can move on to the next step and the next step here while on the front page of the site we issue this query into the html dom using a css selector in this case we just look for a specific item in this case with a product id of 24 this query here returns an html element and then we call a query method on that element which effectively looks for a further element in the subtree of that first element that we find a little bit complex here so we're looking here for the add to cart button for that particular product and then we issue this click action on that button so we have to click on the button and we know that this click in uh click to add this this item to the shopping cart it will issue an ajax and actually takes hr requests in the background to actually perform this addition to the cart and once the addition has been completed we can see this class being added to our anchor element so a class called added so we just simply here wait for that this selector to become present in the in the dom and once it's it's present we consider this add to cart step complete so we can move on to the final step which is navigating to the cart so again we just issue a go to navigation action here on the page object now get to the cart again we wait until the network has gone idle and the final or second to final step here is modifying the quantity of this item that we added to the shopping cart so we're going to use this evaluate method here on the page object which allows us to basically inject a piece of javascript into the browser's javascript execution context so what we're doing here in this javascript snippet is we're querying dom again for a cart item element and beneath it we are looking for an input element with a class name of quantity or qty we use this element obviously object here to and change the value of its this is the input field right so we change the value from one to the number two and then the final step here is to take a screenshot of this page so we could verify after the fact and see that the cart was changed in this case might be interesting and that's the that's those are the three actions then we close the page and close the browser and then we sort of do the same thing again right so twice so if we go to my terminal here and i guess i have to increase the screen size here as well all right so if we just run k6 as normal now so k6 run and then we point to our demo script mine's called demo3.js i don't actually see it on the screen i think it's just taking a bit to we're seeing like this like the previous screen that you were on we still see it on oh you see the editor for some reason yeah obviously we have a few artifacts i can see you moving things around but there's it's it's strange okay maybe it's related to your laptop almost taking off [Laughter] yeah good very very much explain explain that um all right what uh what could we do then well you can see the k6 run command so just okay okay let's run it and we'll see what happens right so we're just going to execute this as we normally execute a k6 test and hopefully we're going to be able to see the browser window that's going to pop up now if the if the lag of the visuals is not too bad i guess you're not going to see anything all right so i'll have to just do a commentary i guess so we have a browser window open and we are navigating to this ecommerce.test.k6.io site we find this product that we're looking for and we click on the add to cart button of that product and then we navigate to the cart itself where would you like to would you like to try and reshare your screen but like share your whole screen instead of the window i'm sharing my whole screen the second screen but yeah i can i can try stopping and trying again let's see the good old turn it off and on again yeah i mean is is that not the first thing that you say even when i have an issue if you can't turn it off then you restart it or delete it and delete and apply for kubernetes let's see if i can do this the same thing um i do there is a question here that that maybe we can talk about as well does casex have any mechanism for storing screenshots and other artifacts and something like an s3 bucket thinking how this would relate to the cicd talk on now oh this was during the um this is during the observability con which is where we did the announcement so i don't believe that their xk6 browser doesn't add support for for being able to do it in an s3 bucket right robin no exactly so we haven't anything specifically built into xk6 browser for storing screenshots or any other kind of artifact from a k6 test it's something we're looking into because i mean today we can do screenshots but we intend to also add support for capturing videos and generating pdfs etc from pages so so we are actively looking into how we can do this efficiently both when you run through tests with one virtual user so effectively like a functional test or a very simple end-to-end test uh compared to when you do it as a load test with maybe thousands of users because you probably don't want thousands of users to create screenshots and videos etc so we're trying to find a good solution that works for for both those are like scales okay i think your screen looks better now let's give it all right again let's give it a go again then so it's a little bit laggy if i look here in the screen but okay so i don't know i don't know if you can see the the full command i typed but kasich's run demo3.js can you see that on my screen no no we just see very laggy yeah i think there's a extreme lag between my screen and what is actually shared let's see let's see if i execute this and if it runs otherwise i guess we'll have to um this demo demo idea okay well we're seeing it's running there's still this is like a lot of lag there though tom would you happen to have anything handy um robin if you can share the script i can i can try running it on my machine let's do it okay so one of those very visual things that you really need to see too yeah because because the cool thing is is that while you're running the script the browser actually pops up and there's like an invisible user using it you're still screen sharing robin by the way i can i can remove him so you can post it to i assume it is his laptop trying to take off it just can't handle everything at once so it's uh treating the screen shares the least important yeah yeah clearly it's not the least important um but i think that script only has like just the the browser test right because i think it would be super cool because once once it's in k6 well even now if you you have to build your own custom version of of k6 because right now it's an extension um but you can already have like browser tests and protocol tests in the same script which is which is so good because before this you would have had to have learned two tools at least and have very different test suites so not exactly the most efficient way to do it mustafa is blaming macbooks on it shame on you okay let me know when when you want to share it tom yeah mark have you have you like talked about this to to other people external people already and have you gotten some initial impressions the people have asked about it and we've been uh kind of teasing that it's something that we're working on is beta to to some of our customers and prospective customers for a while um when a one perspective customer sales team is currently speaking with i was able to send them the announcement and uh they seemed pretty excited about it because they they directly told us hey time to first paint is a meaningful metric for for our company they're an e-commerce company and that's that's not surprising that's that's a good thing with building on the devtools protocol right because like i know selenium has issues with not being able to intercept browser events as well but also chrome dev tools opens up all of these front-end metrics and even the lighthouse stuff later on that's not there yet but there's i think the the choice of of the dev tools protocol was a a really good one from a like future features perspective yeah all right i really like it it's are you ready tom yes i think so all right let's go for it hope this works better go windows oh yeah tom is actually on windows i would screen share but but i only have one screen it's a testament to you know it's all being very cross-platform if this runs so oh yeah inadvertently everything could you also increase your zoom please yeah electron visual studio code okay all right okay so i've i've got a k6 binary built in this directory so i just do k6 run and then woocommerce.js there's the browser popping up i did note it takes a while to load the main page um there we go this looks like it's added to cart now it's gonna navigate to the cart i believe okay i think that was it actually it uh closes as soon as it's taken that screenshot but in this folder now there there is a cart.png and this is what that looks like so we've got a quantity of two there which i think is what we're expecting to see that is awesome yeah we've even got some you know browser specific metrics that show up here as well so dom content loaded first content paint meaningful paint browser loaded as well so these are all really useful timings to to keep an eye on and their timings you wouldn't see uh necessarily um if you're doing this at the http level so it's quite useful straight away i haven't had to do anything unless it's in the script somewhere to have these guys appear here no i think those just happen automatically awesome yeah and you'll also be able to do custom metrics just like you you can with with k6 core um and by the way so the reason right now it is an extension that means that if you download k6 you're not going to be able to get the x the browser part already baked in and the reason that we do that is because we're really picky about what we do put in case x core mi hale from the k6 oss team was just here a few episodes ago and he was talking about this very thing that that he they're very rigorous like the k-6 oss team go through everything that's on that's included in that binary to strip away ever absolutely everything that's unnecessary because a performance testing tool should itself be performant so we are still very much at an early testing phase for for xk6 browser which is why it's an extension for now so it is usable but it is still not exactly where we want it to be before it gets pushed into k6 core do you agree with that robin uh i completely agree yeah that's that's the the reason so we decided this decided uh as part of like building the xk6 extension system that whenever we want to introduce any any sort of major capability or functionality into k6 we'll first build it as an extension so we can have it sort of grow up outside of kasich so we don't introduce unstable things into k6 core because we want k64 to be really stable it's it's sort of one of the the core priorities in you know everything we build around so testing that the tool should be very stable so that's that that's what this extension system allows us to do like it allows us to test things and stabilize them before we eventually perhaps include them into into core so that's what we're doing now with xp6 browser as well so if you want to try this out by the way i assumed that you had stopped sharing tom already let me know if you want to get back on but if you do want to try this out first you'll need to read into the xk6 extension system and i personally think that it's a sign of maturity of a tool or platform when when you're when they're able to open up to other people to do custom things with with that product so i really love that that xk6 exists it's basically a way to bundle your own custom things with into k-6 it's not technically a plug-in but it does fulfill the purpose of you being able to pick and choose essentially which features you want to get included in your version of k6 and one of them right now happens to be xk6 browser but that doesn't mean that that's the only thing that you could add you could just imagine like this ideal of having you know the api tests from from k6 core and then xk6 browser for functional and also browser based load testing and then you could add in xk6 chaos for for chaos engineering i really i got this comment on linkedin a few weeks ago from someone who said that kasich's is becoming an sre's swiss army knife because it can just do so many things sorry sorry for interrupting you i said pablo is going to love that the swiss army knife yeah rcto he uses that terminal it does okay yeah i think that's what it's turning out to be because now it's not just load testing that that we can get into and really it shouldn't matter what type of testing you're doing ideally you'd be able to do many different kinds because it all affects end user performance this also opens it up to things like and i'm looking in the future okay don't expect that it's there now but what about things like accessibility you know we currently don't have anything that that could test for that is that something that we at all get asked about i don't i haven't no i haven't heard that now because i really love how marie drake was was on early on in k-6 office hours talking about accessibility and how she does it with cyprus and i believe she uses the axe library for it or something but yeah that's the ideal to have one tool to do it all one test suite one language oh let's talk about the language actually so the scripts are in javascript as tom and robin showed you but it's actually written in go just like k6's so it comes with all of the same performance benefits that that made us choose go in the first place robin how about um so i don't want it all just to be about what's cool about it what doesn't work what are its limitations or what isn't available right now right so i think we mentioned this in the beginning but we we aim for having rough compatibility with playwright apis at least initially because and then the rationale for that is just to make it more familiar to people that we don't introduce a completely new api that people have to learn so it's easier to sort of try with the knowledge that you already have using playwright but we we don't we we don't support the full sort of breadth of the apis that player has it's a pretty pretty broad set of use cases that you can use playwright for and we don't support all of those apis and and probably all of them wouldn't make sense also for the sort of use cases that we are aiming for either and another big sort of roadblock for us in in adding support for all the apis is that k6 itself basic score does not support asynchronicity so promises uh in in javascript so that's something that's currently being explored i know by the k-6 open source team in terms of the extent to which we can in the short term support promises and asynchronous javascripts but currently we don't we can we can't implement some functionalities that exist in playwright basically uh today so those i would say are the biggest sort of limitations today right that we don't have the full breadth of apis that player tasks and we can't do some of the asynchronous api stuff great support as well as the multi-browser of course is a big major thing right where we focus today on the chromium based browsers and we'll look at firefox and webkits in the future i don't know if you mentioned that earlier and you're talking about playwright versus puppeteer but that's a very big selling point of playwright that it is it's built to do cross-browser testing so now we're getting into cross browser load testing potentially not that you you really need to to do it that way but i think that's very cool something some applications just don't run very well on certain browsers for sure for sure yeah nellis niku it says k6 a swiss army knife for building reliable systems i think this is ours i think this is pavo given the sneaker in the end oh is it oh that's right forget it i was going to be all excited but that that's our cto he's just coming up with a tagline there incognito yeah i think another cool thing once we can support different browsers i mean people can i mean replacing like a browser stack or similar that hey you can run some tests see how everything performs make sure everything's correct and and automate that as as part of a pipeline as well or a different step imagine running different k6 steps or steps for tests in different steps to to to really make sure before pushing further yeah that's awesome because we we already support multiple scenarios so to be able to say you know run run this api this function filled with api stuff like http requests at the protocol level and run it at this load and have this load profile and at the same time simultaneously have the scenario that starts at this time and only run 10 users of that but have it constant and like it's just it's just more information about the entire system what are you excited about tom for for the future of xk6 browser oh the prospect of running a uh i guess a load test consisting entirely of of browser-based vus is something that i'm i'm quite interested in having done it before so the the largest test i ever ran went up to 10 000 virtual users concurrent virtual users um i don't remember the exact number of vms that were needed to support that but it was i think it was at least 80 in in in the selenium world so it was quite a lot of chrome instances per load generator and um funnily enough there was no difference between running that headless or not so you know so i just had it all showing up and i could actually log on to these machines and see you know hundreds of chrome if not thousands of chrome processors you know there would be hundreds of actual windows but behind the scenes uh you know chrome actually launches multiple executable processes for each instance of chrome and it was just like a massive long list and it worked surprisingly well and there was a bit of work uh to remove some of that flakiness that robin alluded to exists with with selenium and um so i'm really looking forward to not having to worry about that flakiness and you know this is one of the reasons why we're in this beta phase is right now to iron out all those issues so that by the time it's uh you know fully released version one it'll be a very stable uh um way of running load tests basically and you know i was doing this for a customer it was a managed uh project and so you know the people i was reporting to are very very pleased and to be able to see the actual page that users would be seeing themselves when the problem started happening so and the other thing was the scripting would have been rather complex to do at the http level and the actual automation code itself was pretty easy to write in comparison and that's also a big a big factor particularly if you're new to load testing or automation in general you know doing that stuff at the http level can be very challenging and uh so you know it can open up load testing to a completely new set of users but then actually you know functional testers that are being asked to suddenly do non-functional testing you know that's a that's a big step and but but yeah it should be a lot easier to do with browser level automation yeah because i think that writing a protocol level script can be a little bit daunting depending on the application because if you're testing like a website or something and rather than like an api endpoint it might not be as simple as just doing a single http request sometimes you might have different headers sometimes there's correlation where you have to get something dynamic from the previous response and put it in the next one but when you're testing at the browser level you don't really have to care about how those things are being formulated because that script is driving the browser the way any user would when we use normal websites we're not we don't we're not exposed to what header is being sent or or the authorization token or anything we're just thinking oh there's a button let's click on that and that's what the browser level script is trying to replicate so you can see on on robin's script earlier it was way simpler and potentially maybe those those requests could have been really complicated but to a browser level script it's really the same and even when from a readability perspective if you're not used to reading load testing scripts that are on the protocol level it's way more intuitive to just see like browser.click you know rather than you know a post request or or something like that with like things that are that are encoded um like credentials or something so so that's a lot easier james leatherman has a comment just the savings of getting around octa at the protocol level will be worth it [Laughter] yeah that's very yeah for sure authentication is one of those things that happens very early on and so you know if you're new to load testing at the protocol level this is a major stumbling block and you know there's usually several redirect chains going on and requests going off to different hosts and that that's a lot to expect of someone to be able to do right off the bat so yeah that's a very good point i would also say that the the whole to like recording and playback uh becomes easier easier as well when you're when you sort of record down to browser level apis compared to protocol level apis but for one the the generated scripts in in the case of k6 would not contain all of the various like static assets right because you can just you can just keep it on the sort of document level navigation events because all the static assets are just like implicitly loaded as part of loading the page and you don't have to care if there's like view states that needs to be passed between requests and things like that or xhr requests or sorry like csr tokens and things like that they just work right because you just click on elements you're at the higher level so i look forward to that we have which is part of part of our role map right to add support for recording scenarios that record down to pro sort of browser level apis compared to the protocol of apps oh james actually just asked a question about that are we planning a recorder for this just to help my laziness it's not laziness though it's like wanting to be efficient with your time right exactly that's how i work smarter yep why do something that can be automated yeah exactly but also to your point robin i think that there's a lot of things that if you're if you have an api load testing script that if they just change the end point or you know if they change something on the page then your script may no longer work and you have to refactor them but if um if nothing changed on the ui like all of the selectors are the same then it doesn't the script wouldn't even be affected and vice versa it could be that something changed in the ui but it but the end point didn't change so then maybe the browser level wouldn't work but the protocol level would so having both of them is actually like a bit of a safety net as well to make sure that you're still automating something and applying load even if something changes yeah definitely and like protocol level and browser level also lends themselves to different use cases right like you would use protocol level obviously if you're testing like an api or microservice or something uh where a browser doesn't even make sense right whereas if you're testing a more complex maybe website that yeah has either like some sort of authentication like octa or or some other you know thing that makes it more difficult to do a protocol level test the browser level api is really shine so i think they they complement each other but you know are are effectively applied for different use cases so how is this going to change things for k-6 cloud what are the plans for having that having xk6 browser functionality available to scripts run on k6 cloud uh well so this this functionality is only for open source no i'm kidding of course we're going to of course implement this in the cloud it's it's just that we typically have the the way of implementing things first in open source before we take you to the cloud because again like we wanted to to mature and and make sure there is like that basic level of stability and reliability in using the tool and and then we will look at how we can implement this at scale for uh our customers of the cloud service and we also want to have it sort of like a more complete experience in terms of recording capabilities and the test authoring using our test builder feature and things like that before we we can launch something in the in the cloud but it's coming what do you think will people most people on the cloud most ex most k-6 cloud users will they want to just have both or or will they choose one or the other i think we'll see both uh so i think i mean we've heard from from customers that have requested this over the years that some some only want to have like a small number of uh virtual users using browser level apis just to capture some like the end end user experience so they can see what what it looks like in terms of like screenshots videos and and these like browser level metrics like first contentful paint and and things like that uh just so they have both but it's as you as you mentioned and whereas we've mentioned several times it's very inefficient to run say a 50k vu test with purely browser level users it can even be you know very uh expensive to do so which sort of prohibits doing that very frequently which which is bad for other reasons right so i think for larger tests we're going to see like a mix where you generate the base load with protocol level and then just have a few views using the mouse level apis but the two what like tom points earlier if you have a scenario where you only need a handful of virtual users and you have a very very complex sort of website that would be very you know horrendously slow from a productivity perspective to write protocol level tests for you would use browser level all the way through so i think we'll see both what's your wish list mark for for xk6 browser so i can i can scope creep our front end team yeah pretty much basically so obviously all the metrics time to first paint and and downloading like getting them right into the k6 cloud uh having i don't know some type of new section to show the screenshots we're taking and organizing them and then finally kind of like pushing that into our pdf creator so you can kind of have it all next to one another and when when my customers say hey i need to drop something on a senior executive's desk to to show them that we're going to stand up for the traffic for black friday or whatever it may be that they have that to give to them and makes them look good makes them look impressed upon that hey this is my whomever who created this report i can understand it i see all the information i need yeah and and i think that that's a very big part of the appeal of this that we're not reinventing everything we already know playwright works and it works well we already know k6 works and it works well so we're just using the same mechanisms like we already have we already know how to process metrics so we're just adding different ones which is why it's so it's so awesome and so exciting i this is like my favorite feature that we've announced since i joined yeah i just love these projects where you just like cobble together like uh the k6 tool right or platform now with extensions and and also the chrome devtools protocol and then you just like it's basically just like glued together with json messages going back and forth between the two that's that's really all there is to this extension uh i mean it's a little bit simplified but it's really powerful i think when you have this ability to extend the tool that you said before nicole and and you have these powerful technologies developed by i would imagine hundreds not thousands of developers on the chromium team uh it's really really cool in an earlier call tom mentioned something about mocking and that is one of the things that's on my wish list being able to get to the point where we can stub out certain parts of the application i actually mentioned on linkedin something like oh this is the first tool that i know of that's done both in the same tool and someone said well actually karate has done that and i'd never heard of karate i don't know if if any of you guys has but they also have this way where they can intercept requests or or events and then and then handle it a different way so i would love to see that in xk6 browser too yeah playwright class support for that right so it's uh and it's something we would want to have support for as well where you can like intercept requests change them um block it change the bodies change the headers etc so you can effectively do some sort of like mocking right and simulate different kinds of events happening um and see how your website or application handles it this year awesome okay so we actually announced this at observabilitycon you can still sign up and view the sessions on demand for free i think some of them you have to sign up for but it's it's like there's no charge for it some of them are just available like i think the keynote is so we have three sessions on there for k-6 since we're now part of grafana labs and the first one was i did have a small part in the keynote where i just go over the basics of getting started with k6 and how to use it with grafana and then we had a separate session with me robin and olha called intro to using k6 load testing with a grafana observability stack really that's the title that we used because we didn't want to spoil the announcement it is about xk6 browser and another cool feature which is prometheus remote right k6 can now export in in that format as well which is super awesome and there was another one that wasn't even an internal talk that is well worth watching it's called building performance tests into your ci pipeline with kasich and grafana and it's by matthew churcher from vonage and i think he demonstrates his use case advantage for 4k6 i believe they use github actions so that's pretty cool um thank you the for the three of you for for joining on a friday night was there anything that you wanted to say before we close off just that we are very happy to get any and all feedback we can if you give xk6 browser a try it is a public beta right now so we do expect there to be plenty of bugs so we would be very very happy if you report those when you find them as well as give us feedback on what features that you think are most highly prioritized for your particular use cases yeah so i actually just posted a bunch of links to observability con and then to the blog post about how to use xk6 because you'll use that to get started with xk6 browser and then the github repo which is now public for xk6 browser so give it a go and and let us know what you think tom yeah no that's uh that would be my suggestion as well try it out let us know how it goes and yeah give us give us feedback and mark if they want to give us feedback what's the best way to do uh for xk6 up probably through the the extension repo that's probably the best way to to centralize that yeah that's a really good one but what about if they people just have questions about how to use k-6 and k-6 cloud in general yeah i mean that's our our support team if you're if you've signed up for a trial a free account you can always email support k6.io um we also have our community forum our community slack which are very lively so don't hesitate to hop in there ask questions tom rob and myself are active in there from time to time along with a bunch of other people so lots of ways to to talk to us all right thank you everybody um and thanks for thanks for watching have a great weekend and we'll see you next time bye bye bye thanks much bye-bye

Channel: k6

Views: 274

Keywords:

Id: Y4qDAaJgxV0

Length: 60min 44sec (3644 seconds)

Published: Fri Nov 12 2021