Performance Testing Using Artillery and Playwright API

[Music] get ready to discover some of the most actionable devops techniques and tooling including performance and reliability for some of the world's smartest Engineers hey I'm Joe colantonio host of The devops Tool chain podcast and my goal is to help you create devops tool chain awesomeness hey it's Joe and welcome to another episode of the tuscule devops tool chain today we'll be talking about hassi all about performance testing using artillery there's been a lot of new features coming out with artillery really excited about the solution we featured them on a few times on our performance Guild so really excited to have the actual creator of artillery joining us hassie who is an open source developer former SRE and the creator of artillery.io if you want to upskill your performance testing efforts and devops and you don't know what to do or where to get started this is the place to go listen all the way to the end to hear how artillery can help you make better more performance software you don't want to miss this episode check it out hey if your app is slow it could be worse than an error it could be frustrating and in my experience frustrated users don't last long but since slow performance is a sudden it's hard for standard error monitoring tools to catch that's why bug snag won the best production visibility Solutions in industry has a way to automatically watch for these issues real user monitoring it detects and reports real user performance data in real time so you can quickly identify lags plus get the context of where the lags are and how to fix them don't rely on frustrated user feedback find out for yourself go to bugsnag.com and click on the free trial button no credit card required support the show and check them out hey hassy welcome to the guild uh very excited to be here Joe uh big sign of what you're doing I appreciate it I've been following you for a while it's great to finally have you uh on the show like I said you seem to be releasing more and more really cool new features so I really want to dive in but before we get into it I'm just curious to know uh someone that probably uh is busy why did you decide to create a solution like artillery that I owe uh personal pain that's what I want um artillery goes back uh quite a few years now I think the original release on GitHub is either 2015 or 2016 so we're talking you know seven or eight years and the origin story I mean it's the same for a lot of Open Source tools I think I was on a team that was working on a real-time chat app and we needed to load test it because we had um reasonable reasons to believe that would get 50 to 100 000 users on day one and nothing that existed at the time could do socket IO load testing so I put together you know just a bunch of scripts really uh it wasn't even like a tool that someone else could use um and then what happened uh I gave a talk about the actual architecture of the application that we're building at a conference and I just mentioned kind of as a by the way you know we built um these scripts to load test it and then most of the questions that I got afterwards were about the load testing scripts and not about the architecture of the app itself so I kind of thought okay you know I'm not I'm not the only one who's maybe struggling with the load testing tools that you know existed at the time and if we go back to 2015 2016 the landscape was quite quite different so artillery ended up being you know a fresh take at the time on load testing and we you know we're doing it again uh now with some of the features that were released recently and again I'm happy to talk about those as well awesome yeah so uh you know probably people around then J meter probably was was the biggest one and so what how did you design artillery then to be different than maybe some of those other tools that may have been around at that time yeah so jmeter was I think it still is by far the biggest load testing tool right it's pretty much the standard that everybody knows um I remember when I tried using jmeter you know uh I was really kind of turned off by the installation process you had to install Java you had to get all that set up um it was still is mostly a GUI First Tool as well and I kind of wanted something a lot more lightweight uh something that was easy to dockerize you know Docker was up and coming at the time already I wanted something that was really easy to run in cicd pipelines and I really wanted something that would be a source control friendly which jmeter wasn't at the time like there are some solutions that you know you can use today but that was the that was the main motivation I really like weight Source control friendly Tool uh and I also wanted something that um I could extend in JavaScript using node.js specifically just because the ecosystem is so big and you have a module to do pretty much you know anything um and you know anyone who's dumb kind of production grade serious load testing will know that you always end up writing some custom Logic for complex apps so I wanted something that I could use JavaScript with as well to make it easier nice so how hard is it to get started with I know some tools have come out recently that uh cater towards more developers where it actually plugs into the ecosystem uses the same language I'm trying to think about artillery we had some sessions at my conferences over the years I could have sworn to use it a yaml file but I'm not sure if that's true so how like how easy is it to get started with artillery I guess the question um it's pretty easy and you know we try to make it as easy as possible so you have a few options uh to write your first crypto artillery uh one of them is that yaml DSL the main specific language which ends up being a really readable representation of say HTTP request that you want artillery to make so you know initially writing get URL headers whatever else that you want to customize um you also have the options uh since fairly recently to write a playwright-based code and then run that as low tests and that's amazing for testing really Dynamic web apps and with that it means that you don't even need to learn anything or too much um that's artillery specific you write your playwright script and then you run it as a little test with artillery and then artillery also lets you drop down to just raw JavaScript and node so you can write your low test definitions that way as well and also these days with uh you know co-pilot and other AI helpers that we get as developers what we found just internal testing on the team is that um GitHub copilot and chat GPT with gtp4 especially it's actually really good at writing artillery scripts um it's kind of become my just go-to you know first um my go-to way to get a first draft of a load testing scripted artillery these days so can you explain that process then if someone goes what so what is your flow then if someone wants to try it for themselves um so if you if you have a GPT account uh especially with gpt4 enabled um GPT 3.5 is pretty good but not as good uh you can ask it's right on Taylor is good for you and you can say for example I have a Json I have an HTTP endpoint that returns uh some Json and I want to send a get request to this endpoint and I expect you know this type of Json to come back and it just writes a script for you that just works 90 of the time and then if you're using copilot in vs code it will autocomplete a lot of um artillery specific you know settings for you if you're working inside an existing artillery app test script nice that's very cool so uh you also mentioned playwright which has been really gaining speed and uh functional automation so how hard it sounded like it was pretty easy so I have an existing playwright test I want to make a a performance test out and I guess we need to go over what what do we mean by performance testing when we're using playwright compared to just straight artillery but say someone has a playwright test how do they instrument it to just run an artillery then yeah so the best maybe the best way to frame it is that um you can use the playwright API so the playwright API that you can use to work with page objects or Pages or you know the browser window in the browser context and you can write scripts using that playwright API using JavaScript and then you can use the artillery CLI to say hey auxiliary run you know 10 000 copies of this for me which means 10 000 browsers right so one thing that we released recently is support for running low tests using uh serverless Technologies on AWS so Lambda and fargate and our playwright support plugs directly into that so what it means is that let's say you're a q engineer and you want to run a production grade load test on a web app that you know you're looking up there you can write a short playwright script that goes you know go to the home page click on this button fill in this form field the entire user Journey and then you can spin up you know five or ten or Twenty Thousand browsers on AWS to run that for you with artillery and you don't need any devops knowledge artillery just does everything for you and it's free it's open source it's out there for anyone to use well okay so that's another question on my house after that but um so uh let's switch it into okay yeah so when would you use this approach I guess a lot of times people say oh uh because you're bypassing the browser normally well when I was starting off with performance testing we had tools like load Runner that kind of bypassed you caught the traffic and just bypass the browser this way you're leveraging the real browser so I assume you would probably catch is this more for like uh Java JavaScript front-end applications really heavy front ends that uh that give you a more realistic uh idea of what the performance is going to be for the end user on the desktop rather than the backend server um yes that's pretty much exactly it so I've I've done load testing on complex web apps you know using traditional tools so the two common approaches that you kind of would reach for today as a developer Q engineer is you either completely bypassed the front end and you try to replicate what the UI does using direct calls to backend apis right and it's really really time consuming to test the complex web app that way and it's also never really representative of the actual traffic patterns to the back end because the UI does its own thing you know a single action on the web page can result in 10 calls to backend apis and it's it's pretty much impossible to do that in a you know in a way that's truly representative of what's going on and you know you'll spend a lot of time trying to track down you know what's going on uh through the network Tab and um Chrome Dev tools uh the tests are not being really brittle they go out of date really really quickly so that's the first approach um nobody enjoys doing that uh the other one is um you can save hard files in a browser so um a complete kind of web archive format of everything on the page and then you can use uh load testing tools to do a replay using those hard files so that's a bit less time consuming but you're running to the same issue where your load test actually ends up being not really representative of what's happening in the real world in a real app so what you know we thought is hey it's 2023 right uh we all have access to the cloud we all have access to serverless compute on demand why not just run headless browsers this is not something that was possible even I don't know let's say five years ago but it just makes sense to do it today so that's you know that's that's the motivation and um by using playwright playwright is absolutely amazing Tech um it makes running headless browser so so easy and the apis are so nice and they're always releasing new features so what you can do now is uh apply the playwright script from the point of view of an end user and then run that as a load test and that ends up you know basically exercising all of the back end systems in the exact same way that a real user would so you get um you get you get the best of both worlds you've got your say you've got your backend apis hooked up to you know a monitoring system like data nowhere near Relic or whatever so you can track the performance of that in that system but then because you're running real browsers you also get uh user user experience metrics so core web vitals so you can actually track how things like um largest content paint or first input delay are affected by increasing load or by high load so maybe I'm an old greedy capitalist but uh why is this open source like how do like you spend a lot of time on it you're doing it just for fun like how is there a model where someone could pay for extra consulting or is there extra features that are more paid based on like a dashboard or something I don't know yeah um I'm a capitalist you know I wear a t-shirt capitalist Club um so the way you know the way that we look um by the way artillery is an open source project all of the stuff that I've talked about so far is free it's open source uh but there's also company behind artillery that's two years old at this point uh there's an entire team you know we obviously want to build a sustainable business around it so the way that we look at load testing is that load testing is one of the most valuable and Roi tools that you can have in your toolbox when it comes to Performance when it comes to reliability when it comes to making sure that you know your services can scale and can withstand Peak traffic right but it's really severely underused compared to other types of testing you know and that mostly comes down to how clunky or difficult the tools are that we have today um and how you know how limiting they are in many ways so pretty much every other tool that's out there today other than artillery is stuck in that pre-cloud native world you know where you can't just scale out and run your test from your own AWS account but that should be possible right it should be something that just is available today so what we want to do is we want to make load testing as easy to to do as unit tester integration testing and I want everybody to load test and I want everybody to do it continuously from you know on schedules or from their cicd pipelines um the way you know the way that we look at it is by doing that we help decrease the reliability of all of the software that runs in the world that you know we all depend on um and then where the commercial side of it comes into is that once a team runs those load tests on the schedule they run them from a cicd pipeline uh what you kind of end up needing is a way to collaborate on all of the performance testing data that is generated by those tests so we offer a completely opt-in optional add-on which is a cloud dashboard that you can configure the CLI to send all of those metrics and reports to and it turns that you know dense numeric data into actionable reports and it allows you to collaborate on those reports different teams or even different business units so that's um that's that's the business model behind artillery the company but the you know to answer your question the reason that we're given all of this stuff away for free is open source is because we want more people to run low tests and ultimately have more reliable and more resilient systems absolutely so I said load testing performance sucks in 1999 and uh it was very difficult and been trying to get people more on board seems like still a struggle but it seems like it is picking up and I'm wondering why is it because people are going Cloud native and that uh just the ability of the technology now to really be able to get real real deal back in the day like we test them a staging environment and then you go to production completely different nowadays I guess with the cloud it gets more more realistic so I don't know what is there a reason why why you feel this way is there an uptick in a wide performance testing is is so important nowadays absolutely yes um I think I think what we're realizing more and more as an industry is that the stuff that we work on is just incredibly complex and it keeps getting more and more complex every single year you know um things like things like the cloud things like microservice architectures things like external third-party dependencies that we use more and more for all sorts of you know non-core things to the product that we're building um all of that stuff means higher development velocity usually but with that comes uh increased rates of change and also just increased complexity and we love complexity on things like emergent behaviors um you know unforeseen interactions between components so I think that um what we're realizing or what the industry is realizing is that load testing is a great tool to kind of tame that complexity and to maybe understand it a bit better and to use load testing as almost um you know a fitness function and kind of evolutionary terms to help shape your system as it grows as it develops and to stay continuously ready for you know for Max traffic and um it kind of you know it kind of tracks with a couple of other trends that were seen in the past few years chaos testing is one of them like exactly the same motivations you know we've got this thing it works but there's no single person that can kind of understand the top to bottom anymore so we need to you know we need to actually induce failures and see how it reacts another trend is testing and production and again kind of the same you know motivation production systems are so complex we cannot hope to replicate them in staging so we need to test directly in production as well um so you know that's that's what I've been seeing absolutely and also observability it sounds like would play well into that also so you've been speaking about testing and production for a bit I think I saw a YouTube video from 2020 um so back in the day once again this was very difficult to do it people think you lost your mind if you're saving a load test in production so is it is it because we're able to scale up environments that are just like production that we're able to do this what do you mean by load testing and production so a lot of people still have that same reaction these days when you suggest low-text in production it's like well no we can't do this we can't do this you know it's almost a reflexive reaction because it um well it sounds it sounds dangerous um I've so you know you mentioned my background there's an SRE it's in devops um I started out as a software engineer and transition you know gradually transitioned over to more you know operational back-end type of um type of type of work um and I've done I've done a lot of production load testing over the years and um what I would say is it's probably it's probably the most value that you can get out of flow testing is to do it in production and 90 of the time it's actually really safe to do you know there's a few things that you need to do to make it safe uh but load testing in production does not mean making production down with your low test and um the main barriers to it are not actually technological they're more around people's perception and you know organizing and aligning different teams and different stakeholders just around the idea of being able to do it but every single time you know every single team that I've been on um every single customer or user of artillery who are doing production load testing it can be a long road to get to that point but once you've done it it's almost like you know it's it's a Magic Moment all of a sudden you realize oh hey um we're load testing in production so we know that our actual system can you know handle plus x amount of traffic or you know no matter how many times we deploy tomorrow we know that All-State performance that you know our slos are going to continue to be met um and yeah it's everybody's an infant convert once you make it happen that's what I've seen um absolutely so so once again nowadays we used to have a dedicated performance team that own performance uh but now I guess you're trying to make this easy enough for everyone to use in this name of the the podcast is devops tool chain so who who does performance nowadays you know when you talk to companies or other people is it sres is it still a performance uh group who's responsible for it typically it's a real mix um and I would say that so that you know there's kind of the reality on the ground today but there's also a trend that we're seeing that's emerging and the reality on the ground today is that most of the time load testing is done by some sort of uh SRE or SRE adjacent team or maybe the back end team maybe the operations team some companies do have dedicated performance and reliability testing teams um but you know I wasn't sap so I can say that um a lot of the times a lot of the times the SRE team is a giant bottleneck and often an impediment to um product velocity so there's always this tension between product teams and SRE teams and a lot of it is a lot of it is um you know inherent just to you know just it's just because of how things are if you're an SRE you are thinking in terms of reliability and that is often fundamental in odds with you know the right to change and throwing things into production but a lot of it is also kind of accidental you know SRE teams are always uh overworked uh and on their staff like at the same time um so anything that you can do to make SRE things self-service for developers that's usually a big win and that's usually something that you want to do so what we're seeing is that more and more companies that operate at scale that you know really want to invest into performance they build out internal load testing platforms for their Developers but if you do that every single one of those teams they run into kind of the same set of issues um so the first one is that if you want to run production grade law test uh across the company you need a way to run restricted load tests so that means you need to come up with some sort of an infrastructure-based solution where developers can create they'll distribute load tests you know it could be dedicated kubernetes clusters or maybe fleets of you know ec2 instances whatever that is and that's a lot of work the next problem that you need to solve is you know let's say um using geometer and you can run multiple instances of it on the kubernetes cluster that's great but how do you aggregate all of the network support that jamita produces for you and all of the logs right so you have to build something and how to do that now the next step is if you want developers to actually use this platform you have to offer good developer experience so that means building some sort of an internal CLI uh some sort of an internal dashboard to do it um and you know none of this is really rocket science but it takes months and years of personal time to do that and every single company that you know we're looking at is Reinventing you know the same set of wheels over and over and over again um and you know coincidentally not really right artillery takes care of all those things so if you're on that SRE team or a platform team you can grab artillery open source and it gives all that stuff for you out of the box so it can really cut down the time for you to build your internal developer load testing solution um and then to go back to the trend that we're observing as well we're seeing more and more activities that are traditionally associated with um SRE teams or you know production engineering teams moving more and more into product teams and by product teams you know by product teams I mean front-end Engineers full stack engineers and POI Engineers as well so things like load testing things like active monitoring we're seeing more and more of that being picked up by those teams because they tend to be the people who are the closest to the user visible surface of the product so there's this inherent kind of understanding of let's say why performance is really important the problem is a lot of the time those teams don't have the tools that work for them today so we you know where we want to make a big bet at artillery at making things easy for full stack and front-end and QA Engineers with load testing which also you know looks back to why we added support for playwright and why we want to make it really easy to run distributed load test with playwright love it so as I mentioned you've been it seems like maybe just because I've been following you more because I have a new show it seems like you've been releasing more and more features uh do you have anything on the roadmap they're excited about or any other features that have been released that we didn't cover and I think you did some with GitHub actions as well if I remember correctly yeah we just we just released um an official GitHub action for artillery so uh it's really easy now to integrate with GitHub and Trigger production grade load test from your GitHub actions um that run you know in your AWS account um which is something that's really difficult to do with more traditional tools uh we have a vs code extension on the way which will make it even easier to write new upload test script if you use vs code there's a bunch of other stuff in the works so today we're supporting um Lambda and fargate on AWS as options to scale out and distribute your load tests we're looking at adding more uh platforms not just on AWS as well a lot of our you know users have asked us about running artillery on Azure so we're hoping to add that before too long and we're also looking into production load testing and supporting it as a native load testing type in artillery and that's something that we have some really early pocs off um and we're really excited about there is no um you know formal release date set for it but hopefully as soon as possible awesome so I I guess one last question is uh what are the main things people look at now when they're doing performance testing once again I'm talking about back in the day I used to have CPU disk space you know uh throughput you know response time I don't know if that's even valid now with microservices and everything so when they use a solution like artillery I know some other ones like you have to come up with your own crazy like you said back end to kind of collect all this data how does it work with artillery and what are the main things that you usually would look at when you're doing a performance test we try to surface as many metrics and as many measurements that are directly relevant to user experience as possible so you know I mentioned uh core web vitals that we expose if using playwright so that's if you have a web app that's probably the best set of metrics for seeing performance from the user's perspective if you're load testing apis let's say you know graphql apis or Json apis there's a few other there's a couple of other things that you can do so we have built-in support for something called Optics which uh you know some of your listeners may they have come across it's an industry standard for kind of compressing and describing the performance of an API from a user's perspective as a single number uh it's an open standard there's a formula that lets you calculate it so in artillery you get that reported out of the box uh so you know a lot of the time you can just look at that single aptech score for a load test I'm just like okay this is good this is within the threshold you know I don't need to look at anything else um and then we also have a feature in artillery which lets you set up arbitrary checks and success conditions uh based on any combination of metrics in your load test that also gives you that you know green red pass fail score at the end of your load test run and the motivation there is the same you know you run your test unless something is broken you should need to really look at very detailed reports you should be able to see that single score so um that feature gives you a lot of flexibility to Define what that means for your application exactly and then it gives you that pass fail at the end yeah so I guess having a score then like 95 I guess it makes it easier to run continuously uh in your Sprint every two weeks you could say did it get worse it didn't get better easy thing someone could see right away you know it dropped by 10 let's look at it or it's about the same we don't have to worry about it sounds like using this yeah what happens a lot of the time right is um there's rarely in the real world there's rarely you know a single commit or a single configuration change that introduces a massive drop in performance usually what you have to do is you have to look at you know three or four or six months of data and then you start seeing you know latencies creeping up over time that kind of thing yeah so trends like absolutely and um the easier it is to see those Trends the more likely it is that someone will look at them and that's why we try to compress you know as much as possible into those really um digestible kind of single number metrics as possible that is not to say that that's all we report artillery gives you access literally to everything down to you know the time it takes to resolve a DNS name for example so it's all there if you want to dig into it but by default I don't think um you know you should need to look at it every single time awesome okay I'll see before we go is it one piece of actionable advice you can give to someone to help them with their devops performance efforts and what's the best way to find or contact you um so the best way to find me is probably Twitter uh um uh H belstra um or artillery's website is on auxiliary.io if you haven't seen it go check it out uh best piece of excellent advice I would say uh on a devops journey is Implement slos if you haven't um it's one of the best one of the easiest not easiest one of the best wins that you can make to align the entire company around kind of the same goals and then once you've done that start running load tests to use as a guard rail around those slos um it's one of the best next steps that you can take and for links of everything of value we've covered in this devops tool chain show head on over to testguild.com forward slash P one two four remember latency is the silent killer of your app don't rely on frustrated user feedback you can know exactly what's happening and how to fix it with bug snag from smartbear see it for yourself go to bugsnag.com and click the free trial button no credit card required so that's it for this episode of the devops tool chain show I'm Joe my mission is to help you see creating end-to-end full stack devops tool chain awesomeness as always test everything and keep the good cheers hey thanks again for listening if you're not already part of our awesome community of 27 000 of the smartest testers devops and automation Professionals in the world we'd love to have you join the fam at testguilds.com and if you're in the devops automation software testing space or your test tool provider and want to offer real world value that can improve the skills or solve a problem for the guild Community I'd love to hear from you head on over to tesco.info and let's make it happen [Music]

Channel: Automation Testing with Joe Colantonio

Views: 2,341

Keywords: What Is The Browser Context In Playwright?, Playwright With JavaScript, The browser context in Playwright, Playwright Browser Context, What is the difference between browser context and page?, How does Playwright communicate with browsers?, playwright API, browser contexts, artillery, playwright, browser context, how to use browser context in playwright, how to run playwright tests with different browser contexts, load testing with artillery

Id: WreQ_imn4FE

Length: 32min 11sec (1931 seconds)

Published: Mon Aug 28 2023