How to do Continuous Performance Testing with Lee Barnes (k6 Office Hours #26)

hi everyone and welcome to another k6 office hours as always i'm nicole van der hoeven and today i'm joined by lee barnes hello everyone thanks nicole for having me yep absolutely so i'm lee barnes cto of utopia solutions utopia solutions is a software quality and testing firm uh i personally do a lot of work in performance testing and test automation mostly in an advisory role helping organizations understand how they can best implement those disciplines within their delivery processes so how long have you been how long has utopia solutions been around oh utopia solutions has been around for about 21 years now so a long time i first got started in this space going back even longer than that probably 19 no 93 94. um shortly after that we became if anyone remembers uh mercury interactive's first partner uh one of the first partners if not the first uh back in 90 1995. so wow long long long history with this stuff for for better or worse i guess that's huge and have you um have you always done performance testing in particular or is it testing in general uh testing in general testing in general i uh even even performance testing and automation um i i approached primarily from a process centric uh perspective you know back back in the early days it was load runner even even before i think it was called loadrunner they had mercury had some tool on a unix platform that we were using to do load testing with concurrent x window sessions something crazy like that um uh so you know certainly back in the day folk more tool-centric because there what there wasn't that many tools but but i always had uh an interest in understanding what's the what's the right way to to do this what's the what's the right way to approach uh performance testing um and i i think we should probably say you know um before we get started uh there's a lot of terms thrown around i think generically for performance testing and load testing that um uh you know i this is general um you know multi-user testing or you know whatever the specific goal is right i think we can have specific terms for that but uh i i typically use performance testing more generically which which i think probably a lot of uh purists don't like but it um uh pretty well accepted i think yeah i i think it also um has evolved as the industry has evolved because before it was just performance testing was just the one thing performance testing was exactly the same right as load testing or band testing you know because that's all that existed but now it's certainly gotten more nuanced so now i think performance testing can can include front-end testing um like those browser-based or browser-driven tools that that measure um like elements on the page not just the response time from a client to a server and back exactly exactly and i think too the important thing when you get started is to understand what the objectives are if you get in if you call something on general performance test and don't really know what you're after well that that's a big issue but if you understand what your objectives are um you know what what you call it is is less important probably yeah and i i also might be sometimes i'm tempted to add in the definition of performance things like reliability or availability or scalability i i guess it's just a matter of semantics in the end because different people say it's it's a different area like maybe it's on the same level as performance but just for me in order to say that an application performs isn't it important that it's available or scalable oh and absolutely i think it's all very interrelated it is all very inter interrelated and i i think um a lot of organizations get in trouble because they don't understand that right they throw some concurrent load at a system maybe they've thought about it a little bit in terms of you know how that uh might map to what they expect to see in in production but there's not a whole lot of thought right everything's very compartmentalized and they check the box in terms of of the testing uh but they're not thinking about that that broader question of oh yeah it performs it performed in this test environment but is it even going to be available in in production to to perform and i don't remember if we talked about this the first time we spoke but i i also wanted to bring up the normal delineation between functional testing and non-functional testing because i just hate the term non-functional oh terrible terrible yeah it seems like it's not necessary if it's not functional yeah exactly exactly and i i sort of get where it came from right functional testing um you know has its definition and and and what performance or load or stress testing is is different from that so you know maybe that was just uh intuitive to maybe call it non-functional testing but it i don't like the connotation it has and i think you're right if it's non-functional why are we doing it yeah i i think that as things as the internet has only grown in reach and more and more companies especially during the pandemic have gone online it we've just gotten to the point where being able to handle higher load than normal or higher load than you expect and doing some sort of load testing to prepare for that is is a functional issue oh absolutely absolutely um if if it can't handle the load it can't function so let's let's just drop the delineation between those between those two things so let's talk about how we met um we met because we are both speaking in a conference next month it's going to be testcon europe and let me actually just post the the link to lee's speaker page at testcon can you tell us a bit about what you're going to be talking about sure um i'll be talking about continuous performance testing um and the the the purpose of the talk is really to identify that the way that many of us myself included have done performance testing for a long long time just doesn't apply in more modern iterative fast-paced delivery processes i spent years and years of my career helping organizations set up these kind of monolithic performance testing centers of excellence and and tests that were executed at the end of long waterfall release cycles and and that seemed to work uh but but the reality was um even if we found issues after this long development cycle it was way too late in the process for the organization to get their first uh first glimpse of of how the system performed um so so the talk points out uh the issues associated with that approach i call it the traditional approach um and and talks uh then goes into talk about okay what can we do at each stage of the pipeline to learn something about uh about performance um and spend some time talking about kind of what's missing right what um if you do some some research or even just talk to organizations not a lot of organizations are doing uh performance-related activities upstream in the the pipeline so i thought it was interesting to to dig in and and try to find out why that why that is so we we spent some time talking about that as well so why do you what do you think is the push towards incorporating performance into into devops or ci cd pipelines now i i think if if you try to take a traditional approach to performance testing and move it toward the end of the the product development cycle you're losing a lot of the the benefits and advantages of of delivering in a more modern iterative uh efficient way and uh or you're letting again you're letting your customers performance test it right as as the system is is released over time but if you push that upstream you're you're understanding performance at a much uh much earlier time and a much lower level where the remediations are obviously much more efficient to to make from a cost than a time perspective we've probably all been involved in situations where you know we've we've found some uh infrastructure architecture related performance issue right before the system's being released and and you know the choices are not very very good at that point it's either we release and we we take our medicine or we delay release a long time while we try to remediate that there are two comments here i wanted to highlight naveen kumar says performance makes an app function it's part of the function right it's it's an attribute of the functionality and james leatherman also says he would suggest that the old axiom that a defect costs 10x to fix in production versus development is no longer valid and hasn't been for several years and this is a follow on from that that also means that you can take advantage of production workloads to validate and optimize performance oh absolutely i i think um i i don't know if i agree 100 with uh with the cost to fix uh defects that that could have been found much earlier that make it into production um are not uh are not more but but evaluating performance should be a continuum and from the very beginning of a delivery process all the way out into production because the reality is there's things that um uh that you can't find in a in a a non-prod or um you know upstream environment i guess it also depends on on the methodology that you're following because if a company is still following like a waterfall type methodology then i still think that a defect in production is way way way more um expensive to fix than earlier on but for some teams that are structured in a more agile way that are doing continuous builds to to production several times a day sometimes yep i think that cost has gone down for sure oh absolutely absolutely the the increment of what you're releasing the production is so much smaller you know eight 18 months of work right that's that's just and now finding a performance issue is is a big big big issue you know 18 hours of work is is uh a big difference so from that standpoint i agree completely it was interesting that you were talking about monoliths because that's that's normally a term that's used to describe the application not the testing tool stack so i never quite thought of it that way but i i totally understand what you're saying and it makes sense and on that note as well um i it kind of reminded me of conway's law which is often a law that's discussed when talking about monoliths versus microservices and conway's law for those who don't know it says that there is a correlation it's it's an observation that there's a correlation between the architectural design of the system that you're building and the structure of your team and i'm thinking that perhaps that's also one reason why testing and performance testing has changed because teams are now agile it's not just one gigantic team and everybody's a part of it there's you know smaller teams of testers or developers or or maybe it's broken up into cross-functional teams depending on a feature and that lends itself more to a kind of performance testing that is more continuous in nature absolutely i the um the organizations and even some of them that i've helped set up their performance testing center of excellence they struggled trying to to fit that model trying to to have that be essentially an internal service bureau to all these agile agile delivery teams and they didn't see any improvement or really any success at all until they started to move that function into the delivery teams themselves and then get away from that kind of performance test team sitting up in the ivory tower and waiting for waiting for code to be delivered so why do you think more people or more teams aren't doing it why isn't everybody doing performance testing continuously well i think that it's just not something that that people have done because of the the traditional model of it's always performance is someone else's problem right we're going to give them the application in whatever state it's in and they're going to execute that uh that performance test and they're going to hand the results back so it's it's a way of thinking it's an approach uh it's uh tool sets it's uh processes around analysis um that that the delivery teams just haven't done before certainly nothing to do with their capability to do it it just it just it hasn't been part of their uh their work life you were saying earlier that you thought that not too long ago performance testing was very tool-centric would you say that it's still as tool-centric now um i i think it can be um i i don't think that's the right approach i think the biggest value from performance testing is in the analysis and if you don't design the right test based on the right objectives and build the tests that effectively uh implement those objectives the results you get and and therefore the analysis is is not going to be the right uh uh the value that that you expected so i think it's it's it's still very important to take a process-centric approach um the the tools though right they they have to support your objectives they have to work in your uh with your technology stack they have to be available to your uh to the users and that's changed a lot over the years as well right when when we had these these dedicated teams they could go very deep in a in a tool stack and as long as that tool supported their uh their technology environment that that worked but now we've got a much broader set of um set of users as we move performance testing into the delivery teams certainly not any less capable technically but you know cur tests that had big learning curves or uh very specific ways of of working or or of using them um there i've seen some difficulty with those types of tools being pushed into the delivery teams they need to be a lot more developer friendly yeah and i think it's it's funny because this is the kasich's office hours right so we are a tool builder and vendor and yet i don't think that k6 is what's going to make you good performance tester no tool can make you a good performance tester it can help you along that path absolutely but like you say if if it's not if the tool itself doesn't fit in with how you work or how you want to be working um that's that's one issue but another issue is if you if you don't know the fundamentals of performance testing or you haven't spent the time to understand what it is you're trying to figure out with your tests then yeah it doesn't matter if the tool is great it it it doesn't and um i think this is true with any tool right i see a lot of situations where they pick up a tool and they create some scripts and generate some load um and and call that a performance test or a load test or or whatever their uh objectives might be if they even have any and that there's really no value in that in in the end they might get lucky and they might find something you know putting the system under load and and uh you know happened to stumble into something but in terms of really understanding um the performance of the system under various conditions that are important right that's like finding a needle in a haystack with that approach yeah and what are some things so i guess when you when you talk about continuous performance what what does that entail well to me it's it's evaluating uh performance at the appropriate level and and to the extent you can at each stage of the pipeline so i and in general as you move uh further upstream it's looking at performance at at a much smaller componentized level one one of the the biggest problems that i see with traditional approach uh or with organizations trying to implement a traditional approach within sprint for for example uh is that they have this you know some number of of very um uh very brittle very sensitive scripts and as the application changes they can't keep up right they they can't get that test ready to run but if you're focused at a much uh much smaller level there's the the scripts typically there's less impact to individual scripts especially if those scripts are an api level versus an http or a ui level um uh but you're you're understanding something about the system at a much smaller much more modular componentized uh way now do those results mean anything to a production user no probably not so you're not you're not publishing those results in the same way you'd publish a uh you know some some response time analysis uh under load in a more production-like test uh but you can take those results and trend them over time and then if you see anomalies in those trends you can investigate them and and be able to trace uh those issues back to the commit or much more closely to the commit that caused them again versus you know end end of cycle tight performance testing where yeah we've got some problem now we've got to start digging in and and see where it is yeah so um i would also i think this might be a good time to share something that this is a way that i try to incorporate performance testing continuously let me share my screen so okay so i've got a site and this is just my personal site this isn't a k6 thing please don't load test it by the way but um but i am concerned about i i'm about the performance of this little blog that i've got and one way that i've tried to address that is that i've this is my i use hugo so the all of these are my blog posts right and right in the same repository that i have my blog posts i've also got a kasich script and it's it's a pretty simple one it goes to the home page so it goes to two different pages and has think time right and how i do that how i incorporate it is because my blog is on on github i just have an action that fires that test off every time i commit to the repository so every time i change something on my blog whether that's you know adding a new blog post or changing the theme or or adding another page i get these runs of that same test and the cool thing is you're talking about trending um this is the same thing on k6 cloud now because that's one of the advantages of working for k6 is that i i can set up these things on my personal blog and and call it work because it is i'm i'm seeing how how the response times change over time and because i set thresholds there are also some times where i can see it turn red and then i can get immediate feedback on what's working what's not working um and i've also so i've got two ways that it runs one is when i when i actually make a change and the second way is i schedule it so uh see it says here that that the next run is going to be at 4 45 oh that was actually i think i should refresh this because i think it just ran um so the next one's going to be at 8 45 p.m today so this is the just the last run slightly higher but i mean we're talking about about 40 milliseconds here yeah um so i so i think that that's also kind of what you're talking about right that it doesn't have to be i think that one of the things that that holds people back is that they think oh if i do performance tests all the time then that's you know thousand user tests every hour every day this is five users i'm using five users for mine but it's still enough to give me data absolutely a handful of iterations a handful of users typically at the component level and remember we're likely not going to going to be in environments that are anything like production upstream right so so the ability to run large user loads doesn't make a whole lot of sense uh a whole lot of sense anyway but but absolutely you're learning something about uh about performance and i think one of the what the the probably the most important point uh when when implementing uh some type of continuous performance testing is is start small and evolve right it i can't prescribe a one-size-fits-all solution i don't think there is one um whatever works in your organization my talk's gonna give some guidelines on how to approach that and how to think about it but ultimately uh you're going to do what what works for your organization and you know if a few iterations for a few users isn't enough for some reason you're not getting the data you want change it or the the type of test whether it's a you know an api call or something more at the ui level http request but maybe you know for whatever reason you have to pivot one way or the other but uh but absolutely um you know you you can't think about this old traditional huge load um uh kind of high fidelity performance test in upstream environments yeah you also i i think that the the few user tests gives you sometimes even more data than a single thousand user test that everybody was prepared for you know because if it's just five users you can run it across even you your an environment and it's it's not going to be noticeable or if it is then you have a bigger problem than the test right or the test right absolutely but that that lets you that makes you more flexible too um gives you more options for where you run it you can run it in pre-production environments and production if you're if you're okay with it just being a few users because if those the key thing is that those users have to be running with some regularity you know not just once a week but maybe every every day every hour sometimes it depends what what your use case says of course absolutely and how frequently you're you're committing or building absolutely one on the on the uh the other end of that uh one of the the biggest problems that i've seen where organizations have tried to um implement this is is just overwhelming the organization with data and you know they didn't start small they they built a bunch of these little tests maybe the tests were fine and each independently but there was so much data coming back to the organization they didn't know what to do with it and they haven't seen it before right so you've got to be able to understand the data that you're seeing and you've got to be able to take action on it if if necessary and one of the symptoms i see in that case is it's just kind of overwhelmed and they they try to keep up and then it's just like we're going to ignore it because it's just it's just too much so um one of the things i'll talk about is you've got to start small right and and learn understand the data that you're getting and then ask for more um and maybe the data you started with isn't isn't as valuable as you as you thought it was going to be well okay let's stop there and let's look somewhere else where we can get some more meaningful data yeah and it's also important when you're running these tests to have a clear understanding of what is acceptable and what isn't so thresholds are a really good way to to do that yes if you can set you know an acceptable response time and then you know maybe you can set up notifications so that your your team is informed when when the performance crushes that crosses that threshold then you're actually making use out of those tests because otherwise these tests might be running in the background but if if you're not getting that notification when things go wrong then you're you're missing out on a lot of the value yep yep absolutely and and it's just as important to to set those thresholds so they're not too sensitive uh as well because the the similar problem happens and you just start to start ignoring those alerts um yeah so that's the problem though that that sometimes depending on the tool sometimes you can't even set thresholds sometimes they just run the scenario and then that's it they don't really give you any insight as to whether or not that was good or bad no true true i i think um a lot of the organizations that i see that have implemented this successfully maybe aren't relying as much on the tool um they're taking the data and uh and importing it into uh you know maybe their own uh time series database and and setting their alerts that way rather than they're using the tool more as the um uh as the load generator and uh maybe the initial collector of data but they're they they may have multiple tools for for multiple reasons and and they uh try to um uh collect that data in a single repository and uh report out of that for example yeah so just before the just before we started broadcasting uh we were talking about the importance of observability and the difference between traditional monitoring and observability and it's that in traditional monitoring you you go and identify what you want to measure and then you set up ways to to measure those metrics in particular the problem is that sometimes you run a test and something goes wrong and you didn't measure the things that you now need to look at so observability is a little bit different because it it instruments a lot of things it it um sets it up so that many of the things that maybe you didn't even think you would need to look at are now being monitored and after the test it's more exploratory it's not just like check this okay done check that it's more like i wonder what what this metric was at this particular time you know and something having an observability platform set up is essential to performance testing these days i i agree 100 and even you mentioned you know look at a specific data point at a specific uh time the the traditional apm tools and they may say they have observability but but not true observability most of that data is aggregated so the ability to get down to a very specific data point is lost right and and or observability you can ask very specific questions um about data with high cardinality and get down to okay yeah this specific user had this specific problem um at this specific time as opposed to you know hundreds or thousands of data points that are aggregated um into a single point yeah the granularity of the data makes a huge difference actually um and that's why sometimes when when you get the uh performance testing report and it's all like the average response time across all of these transactions was this i don't actually find that useful at all because they're different transactions you know if you if you're averaging something then it means that you're assuming that all of those transactions are comparable and often they're not no well even even the same transaction the same process that you're executing over a period of time most performance and loan tests have a period of ramp up so you have some data points that were taken with relatively little load on the system and other data points that were taken with uh with high load the average doesn't tell you a whole lot and you know talking about you know the approach to performance testing you know looking at the 90th or 95th percentile of the data tells you a lot more about the actual response time than something like the min max or the average but you know uh very often that's the data that that i see reported and again um you know how meaningful and how valuable is it a lot of people don't understand that maybe there isn't a whole lot of value there this comes back to the monolith versus microservice in testing tools conversation because i think a lot of the the tools now are going towards this one-stop shop mentality where they just want everything packaged in the pretty bow from start to finish and i i get that that's i guess in some ways easier because easier for users because you only install it once but for me i don't know maybe i just i have wonderlust in my heart or something because i do not want to be tied down i want to be able to say well i don't like i don't like this results analysis tool i want to plug in my own and i hate being locked down because i i don't want to get into the situation where i i change tools and now all the previous baselines that i took i can't use anymore because you know it doesn't present the information in the same way yeah now that i think that's a good point and if you if you talk to a lot of uh seasoned performance testers i think you'll find that they're what i call tool smiths so they they understand how to find and use the tool that is most effective for whatever their specific problem is versus maybe newer performance testers there's nothing wrong with this but but they tend to become what i call tool jockeys and so instead of using tools for the appropriate purpose they grab one tool and they ride it and um they don't really and there's also a danger in that scenario where they they become very tool centric and don't really understand performance testing as an approach or a process but but they they look as the tool they look at the tool as the performance test yeah and i mean i i work at k-6 now but i haven't even been here a year yet i think it's ten nine months so far and before that i wasn't using k6 i was using j meter and gatling and selenium and puppeteer you know because i like to try different tools because if you don't know what's out there then how do you know that you're using the right one so i think that's a very big danger in performance testing to get pigeonholed because that's the tool that you know yeah absolutely and you don't know what's out there so how do you know if if what you have is is providing you the the value that you need because you really don't know anything else one of the things that sold me on k6 because when i wanted to change jobs i looked at k-6 in particular so it wasn't just happenstance i i liked the tool and one of the things that i liked is that they they cater specifically to developers i mean testers can use it too but most testing tools cater towards cater to testers but k6 has a developer bent and i thought that the concept of unit tests for performance was is something that not many tools have really explored no no and and i don't think many of organizations have explored it as part of my my presentation i did some research i haven't done a whole lot of uh just given my background a whole lot of unit level performance testing but there are uh a suite of frameworks i won't call them tools but frameworks out there for supporting uh performance testing at the unit level and um a lot of problems you wouldn't even think of for instance just just because uh the tests are so small and and it's important to to be uh you know very sensitive in response times some of these frameworks have capabilities to to kind of package up your tests and your code and put them on a a known um isolated environment because what whatever else is going on in the environment that you're executing the test could could have a huge impact on those on those results so yeah very interesting uh very interesting research there although i don't see a whole lot in practice yeah yeah i i think that the language is a big deal as well because the ideal is is a testing tool that uses the same language that you're developing in but there are so many languages that it's it's hard to pick one yeah yeah absolutely you have those frameworks that i that i found in my research they were almost all java based i suppose not surprisingly but yeah well a lot of load testing tools are java based too and i would like to see the industry move a bit away from that just because java is a pretty old language now is it like 30 years now and it's starting to show and i think having having used java based tools in the past it's it's really frustrating one of the things that i hate is that there's this whole jvm layer that you also have to tune so you have your script you have the application and then the jvm you have to know how much memory to allocate to it and this just you're adding a layer of complexity that i would rather not have and then and another important aspect of that is a lot of a lot a lot of users don't understand that right so they're they execute performance tests and don't understand that possibly their tool their tool setup are impacting the results that they're seeing and it comes back to who's testing the performance testing tools yeah exactly exactly because they need to be tested too you are going to be relying on this tool to tell you whether your application is performant or not so you better be sure that it's performant too because you don't want it to be the bottleneck yep i you know you mentioned the k6 being a developer friendly tool and um there was a trend it's it's not as bad as it was but there was a trend um in the commercial tools especially around ease of use and the idea was i i need someone to be able to create scripts as fast as possible and be able to play them back and and have that green check mark uh because there was this feeling that the the scripting was just too complex too difficult um and but what that what that resulted in was was very often um just kind of hiding what you really needed to understand about what you were doing from from the users and opening up to uh to a set of users that of course had good intentions but just didn't really have the the foundational knowledge to understand what they were doing yes they could get a a script to work but those ease of use features marketed by those commercial tools for a long time i felt anyway caused more problems than they solved just just by allowing people to check the box on yes i created a set of scripts i ran a test uh with really no uh you know no thought beyond that unfortunately yeah and i think that it's very telling that at kasich's we are almost i think almost all of us are engineers of of some sort and we use we use k-6 and so we care about about it being slow or being clunky we've made decisions against we we've decided not to include certain features just because it would make kasich's bloated so we we make a lot of interesting design decisions as a team that i think maybe if you were purely watching out for from a profit perspective that it might make more sense to include features all the features so you can say yes to everyone who wants that feature but the reality is you still those things come at a cost you still have to pay in overhead every time that you you add something to a tool yeah absolutely and and you know the number of use cases you know that require millions uh millions of concurrent uh users maybe are you know not as not as high but as those tools become more bloated their ability to support those use cases is becomes very very difficult we've we had a a project just this last year where a good handful of tools really couldn't get anywhere near the the generating the level of load yes they were cloud-based load generation and and uh but the infrastructure and the bloat as you described just just you know those tools started kind of rattling around uh you know in the hundreds of thousands of users and really had no no hope of getting to where we needed them to get so there's a trade-off for sure so so what did i can't remember if you you told me this earlier but what did you end up what did they end up going with um well the the customer uh changed their objectives unfortunately and uh uh didn't get to the concurrency um or the throughput volume that they wanted to which which was a problem right they they uh they had uh purchased a tool uh a subscription and um didn't make the the tool vendor prove that they could they could do what they said they could unfortunately so okay well um i do want to show this blog post take it with a grain of salt this is made by kasich so just understand that but i think it's actually a very um unbiased if that's possible review it's very thorough this was um this is a blog post that compares different tools and you know it's unbiased because k6 isn't the best for everything it's not the best performer or whatever so this definitely includes areas that we could improve and just put that link in in the chat as well um but awesome what i love is that it has um like the memory usage here and it goes through all of the tools and like i said k6 isn't isn't the best in in this regard right because this apache bench is still better in terms of the the memory usage um and and also the rps but it kind of gives you a more quantitative way to assess tools right to see whether or not you can adopt these tools absolutely absolutely so it's like a a bit of a trade-off and by the way i think the scripts or the tests that were done are are also linked to here so you can see here you can try it yourself awesome yeah so these are all reproducible tests all the methodology is on there so i actually love this because it this seems more real than like you know you see charts where this bit is cut off and k6 is the best but what's interesting is j meter and gatling java based tools um i know that gatling also uses scala but it is still based on on java uh but they are significantly more resource hungry than other tools yes that's a problem if you're talking about the million scale oh absolutely yeah absolutely absolutely so um just go back here so what do you think when people come to you do you have to convince them to do continuous performance testing or do they already have that idea in mind and you show them how to do it i wish they would come asking for help with continuous performance testing typically it's it's we've got this system we've we realized that we need to performance test it and we didn't put that in our plan or we already have issues can you help us uh run some tests so we can uh we can investigate those issues but so i just start talking to them about you know what the process could have been right and and what maybe they can start moving to in the future in terms of understanding uh performance throughout their their delivery process versus this kind of either you know someone had a check the box task at the end of the release cycle if they had to run a performance test or um you know they start getting nervous or already that they don't have one or that they've already got some issues that they have to uh have to look into um but but definitely thinking about this ahead of time and building it into the way you you develop software uh is going to be a much smoother approach than than trying to kind of fit it in after uh after the fact and it becomes a cultural thing as well like like i said their organizations aren't used or delivery teams typically aren't used to seeing this data but if they're the ones from the beginning helping to build the processes and bringing the technology around that um it becomes part of the way they uh they deliver software versus someone else trying to kind of shove it in um after the fact yeah so when you've seen companies to the successfully when they're actually at that point where they're doing continuous performance testing what are some things that they do that the ones that didn't get there have done um start small first of all and and with a focus on continuously improving and and making this this effort actually part of their some you know some organizations called their technical backlog so not their backlog for the app they're building but the backlog that kind of drives the way they build uh software uh because if this is just if this is off the books work right yeah we'll get to it someday or it's someone's job to to research it'll never get done so making that part of the way that they work uh certainly and starting small and and being being willing to to change right don't don't just march down a path and and uh don't stop till you until you get where you initially thought you were going um but be willing to to to evaluate what's working what's not working and and adjust accordingly so what's the first concrete step that you would suggest that a company take probably at the component at the component level and this is easier said than done in more of a microservice architecture versus a monolithic architecture but identify critical components that you can start running some small tests on as part of your integration process a part part of your build process and start getting that data back start trending that start have the organization start to see that value and then and then expand uh from there that's typically where i see organizations organizations start i think that's why i'm not entirely sold on the future of testing tools being browser-based because that's really still just addressing uh the end of yes of the cycle right when it's already you're already able to access it on a browser so what happens if it's not even integrated yet you should there's still tests that you can do at that level absolutely absolutely and that's that's part of the problem with trying to pro force that traditional approach where all you've done is test the end the end product into that well yeah you know this may be a new component you're building it's not integrated yet it doesn't have a ui but we can still access it right we can still we can still put it under some type of uh some type of load and you just gotta take yourself out of that traditional mindset and yeah there's only one way that i know how to create a test yeah i think yeah doing it at the component or or unit level is definitely um a good start and already that is that can be daunting as well because if you're talking about unit tests that's probably in most teams that's not a tester that's doing it in most teams that would be a developer so there's also an element of educating developers if they're not if they're not already doing that and talking to them about best practices for testing yeah i i agree in some of the organizations that we help transition from the performance testing center of excellence to more of a continuous performance test the the resources from that center of excellence were essentially distributed out into the um uh delivery teams uh but it was it was a two-way learning curve because the performance testers had to get out of that mindset and and but at the same time help the delivery teams understand well what how do i think about performance testing it's not just a it's just not a unit test and a big part of that was as as the work is being planned to start thinking about performance factors ahead of time or uh up front as opposed to at the end and now we're sitting around well what's you know what what are our objectives here what are our slas and uh i found that very few organizations are thinking about uh performance performance-related uh factors early on it doesn't have to be you know detailed acceptance criteria but thinking about hey i'm building this component what what type of performance factors might uh might factor into this i want to take the opportunity to also show off a new feature awesome um because i was surprised as well there's this thresholds list now so you've been able to do thresholds in k6 oss so the free tool and on cloud for a while but now there's also a screen that goes through all of your tests and you just are able to see it in one screen and i think and i really love this because you're talking about slas and it has different names right it's slas slos thresholds checks their requirements essentially they're just your objectives quantified not just you know i want it to be faster but like no i want you know the the error rate to be less than three percent in this case or you know this is about response time and when you see it in terms over time if you're running continuously then you start to see you start to pick up on patterns like certain tests always fail or they always fail on a certain day or at a certain time those things are not are not things that that your big thousand user test is normally going to pick up on because you're not running that every hour or every commit even or every release yeah absolutely that's uh another big issue with that type of testing right it's this big event it's typically a one-time event if you look at you know a relatively small period of time and there's a lot you can miss or there's some data you can get that has nothing to do with the with the application under the under test maybe something was running on a server you had no idea about and impacted your results so absolutely trending over time at a smaller level uh just gives you so much more valuable information i'm not saying there's not room for the traditional you know big bang system level performance test uh as the application um uh gets built and released at a you know at a much um much more mature level absolutely but let we shouldn't let that be the first indication of performance of that system it's almost like we're i feel like we're seeing a bit of a yo-yo on that front because the in the traditional way was always you should only ever run a test in test environments never in production and then for a while it was you should always run it in production because that you know you then you don't have to simulate load and and you know it's all about being agile and releasing frequently and now it's it's almost like i want to say well don't forget to also test before you release because there's so many other phases and so many ways that you can test it without necessarily requiring a full-blown production environment or even a production-like environment absolutely and and i think that's why it has to be a continuum it very interesting though that you mentioned the the testing production it totally was a was a yo-yo it was it you know back when i first started performance testing it was completely taboo you you don't do anything in production um and then the more uh say the more modern or you know liberal organizations would uh they embrace that but if you were regulated or certainly the old kind of conservative uh type businesses that had you know ran their their businesses on these out of these large it shops you couldn't touch production and now it's you know it's it's pretty uh ubiquitous yeah it's it's funny how everything goes back it's like monoliths to microservices and now there are some companies who are saying well actually we're going back to monoliths yes and it makes pendulum swings it makes me think of conway's law again and how we were saying you know the the structure of the application or in this case a testing tool stack mirrors subconsciously the structure of the team i wonder if as people start testing continuously as performance becomes an activity that's done throughout the entire process that i wonder if that's going to have an effect on the roles because if testing is done by everybody and everyone's a tester then that's adding a lot of a lot more responsibilities to all the other roles it is i think um in some of the large organizations that i work with that have implemented this they have a role of performance owner it's not the sole performance tester it's not the person solely responsible for performance testing but their their role is to kind of clear the way and support the other roles in what they need to do to to make continuous uh performance testing happen uh because you're right you can't just kind of wave your magic wand and say okay developers you're gonna now be running these component level performance tests there's a lot of work that goes into that to to you know set up the infrastructure and the plumbing to make that happen and and understand what results are et cetera so i i have seen some of the larger organizations be very successful with that performance owner role again not not uh the the kind of one-stop shop for performance testing but but someone that uh is really the the uh supporter of continuous performance testing so we we still have a few minutes here but we should probably wrap it up and let people get to their weekends um is there anything that you want to plug before we end uh just excited for my my talk at tescon and yours as well by the way i'm very interested uh to see that from uh utopia solutions standpoint you know we're uh we're always here to help anyone who wants to learn more about how they can incorporate these practices specifically uh around uh performance testing and test automation into their into their delivery cycles so but thank you nicole for having me and of course thank you to k6 as well i also posted the link to utopia solutions in the chat um check them out and also because i forgot it last time we're hiring we want good people we have a bunch of job openings with k6 and with grafana if there's nothing in k-6 that that fits your needs we are looking for good people honestly we want people that we can work with first and then we'll talk about the skills so if you think you might fit into our culture we want to hear from you and i i think that's that's about it for me i did post links to the the speaker page for from me and my normal co-host simon aronson the three of us are speaking at this one conference about different things and so that that's definitely worth checking out it's going to be completely virtual which i'm a little sad about honestly but i guess it's going to at least be easier to access us for most lithuania would have been an awesome place to go i agree all right well uh lee you can stay on but thank you everybody for watching have a great weekend and we'll see you next week bye thanks everyone bye

Channel: k6

Views: 266

Keywords:

Id: 5pd0G-jBOTo

Length: 59min 57sec (3597 seconds)

Published: Fri Aug 27 2021