Performance Testing using Playwright

uh yeah thanks for coming uh uh today we're here to talk about performance testing with playwright um my name is uh john hill um and a little bit about me um i'm a web ui test engineer uh here in palo alto california working at the nasa ames research center and i've been doing uh ui testing for about 10 years now and i have been doing a little bit of performance along the way but most recently i took a job to work at the aims facility out here to work on um web ui performance on the viper mission so i'm uh for the past year i've been working on openmct and uh the viper mission um and the viper mission um is part of the artemis program uh to send a lunar rover to the south pole of the moon and explore resource prospect and drill for water so water is most likely found as ice in the the shadows of craters um which is unfortunate because the uh our lunar rover lunar rover is solar-powered so it's always a race against time so it's a fascinating um uh project to be a part of um and it combines kind of my experience with web ui testing um performance and uh open source um i have i've worked uh on the for cloud bs and for red hat ansible um so it's it was a great opportunity to stay in that space and you can check out openmct um i've included the link here in the the slide notes and i'll provide other links of course and uh it's uh what openmct is is it's um an open mission control technology standard that's very pluggable and extensible and works with various backends um in the space industry in various space agencies and allows uh you know mission control operators to get complete situational awareness about what's going on during the mission and also allow scientists um to kind of view data um about the mission in real time and historically so it's all running in a browser and the ui designer is going to kill me because this version is over five years old and i'm sure uh your designers at home would also kill you if you showed a five-year-old version of the application um but he's on pto and i didn't get any clearance on providing any other version so this is what we're going to look at today but you can see kind of uh what we're what we're working with here so on the left are uh images streaming from the apollo program um in in this case uh but the lunar rover drivers will see representations of the moon um and if that's kind of juxtaposed with um a display layout that they create for themselves here that can have any sources of telemetry any sort of things that they're curious about um you know like the plan of what's going um to happen next all of that is made available and again all of this is browser-based um and you can see why there's uh um you know why performance would matter right this is a mission critical stuff um you know nation critical software running inside of a browser and so i'm going to switch back over to the slides so in this talk we're going to talk about client-side performance testing we're going to talk about what you can measure um and we're going to do this with playwright and all of the code to do all of this is found here um and it's not practical to follow along uh don't worry about it we'll get you know play with it later we're going to talk about the capabilities that playwright provides for performance testing um some of the problems that are inherent with uh using playwright and some of the solutions um and we'll end with kind of a state of the union and what's coming next um really exciting space no pun intended so in this talk uh we're not gonna talk about how to fix problems once you've found them um and this is kind of you know part one uh i don't want to undersell this in any way um i you know i haven't completed the journey the the mission launches until november 2023 um which is going to be here before i know it but not everything is completed and i don't want to present it as that so you can follow along by coming to this meetup group or follow me on you know on github for example to kind of keep track of what i'm doing and i'll be uh posting things on linkedin as i find them um and i hope to have maybe at the end of all of this kind of like a retrospective about what i learned along the way but just super quickly we'll talk about what client side performance is at least in the context as i as i've come to understand it it's you know a poorly it's a loose definition but it's kind of the user experience while using a web application in a web browser so kind of hand wavy not a good definition but um it's really best uh and easiest to contextualize when something goes wrong um that's the thing about web performance is everything it's one of those things you don't notice it until it's gone or until something goes wrong and so the three easiest ways that i know to describe the problems associated with client-side web performance are long loadings or things that are loading improperly or are visibly changing or loading slowly slow relative to what we perceive to be okay by by using web apps we also have jank or our freezing or animation that gets us away from the the gold standards of uh 60 frames a second so on the left you can see a firefox spinning with some jank and on the right you can see uh perfectly uh a firefox spinning with perfect fidelity at a rock solid 60 frames a second and then worst case right worst case is uh when your browser session totally crashes um due to either an error but often times uh due to some sort of performance problem that only manifests over time like a memory leak um so all three of those are kind of three things that kind of describe some of the problems associated with uh client's head performance but there's another another thing that is kind of maybe only applicable to web development teams maybe not the users themselves and this comes from a previous project that i worked on um this is another type of web performance problem which is um a junior developer broke um the react uh hooks mechanism on this particular page on the left which is kind of rendered out of screen um sorry if it's a little bit difficult to screen but the the point is it's it's done loading right it's it's finished loading but on the right you can see uh the network traffic for this given page and what this person did was created the world's um best load testing tool to take down our app because uh it can it couldn't resolve the the state of the react hooks and so it kept issuing more and more requests until eventually the back end gave out so this is another type of performance problem that can be caused by the front end that may be even invisible to the front end in our case the functional test didn't find them certainly because they're as far as the application was concerned it was performing well but there are other things that we need to be mindful of when we talk about client-side performance so what do we measure well when it comes to single-page applications or client-side measurements is really what we are most concerned about are things after the first load so after we have all of the assets on the initial page load um so you know in my in my day-to-day we don't care about necessarily how long it takes to get that initial load because once you have that all of the assets that you need it's really what you do with them um and in some cases it can you know our users are using the same page for many hours but the things to measure after the first load are network it still matters after the first load terms and latency the number of requests that you're sending what you're doing with those requests that still matters cpu and gpu right sort of the jank and freezing that we're talking about the frames per second um how efficient your code is uh the javascript itself is and how it's being rendered inside the browser um and lastly memory so memory is hugely important in some cases mission operators um will be using the same page for six hours i'm assuming there's a bathroom break but i can't be sure um but what you have to worry about in that case is when he's he or she is moving around um and using the app that that memory is released when no longer needed object references are released and that leaks don't form over time so where do these measurements really come from well they all thankfully come from chrome itself so chrome does a very good job the chrome dev team i should say at making all of these things that you would want to know about web performance available to the um to the dev developer or the tester and that's interested in this and it does it through two ways um it uses you know chrome is driving some of these web standards i think we're all aware of that and a lot of the the web apis and the browser apis are now standard sometimes they're experimental but they're built in to the chrome to chrome and and basically any browser you'd want to use but also chrome uh uses chrome dev tools to kind of expose additional pieces of functionality so the chrome dev tools will expose the standards but also kind of the devtools api's standards which is really great and so i'm going to break it down into those two first so the web and browser apis that we care about the most are available on the mozilla developer network but um resource timing apis how long does it take an individual resource request to um how long does that take and where does it spend its time so the resource timing apis give you a very insightful view of what was requested what was received and how long it took in between and what it could do with that information something that's worth measuring and playwright and we'll show an example of that the long tasks so a long task api is a great way for proxying cpu gpu and animation measurement um and it effectively what it does is it keeps track of where the main thread is locked for 50 milliseconds or more and that's really what kind of gives you the the appearance of jank that lets you it will break the illusion that things are animating completely and it becomes a very good proxy of kind of keeping track of general cpu gpu and animation based uh problems in your application if something doesn't finish within 50 milliseconds it's logged to the long task api and then you can do something with that information um and the last one we're going to talk about again there's many more but the last one we're going to talk about today is the user timing api specifically the performance mark and associated performance.measure again built into all browsers um and the most famous example that i'm aware of for this api is when twitter 10 years ago wanted to measure how long it took their user base to send their first tweet so let me turn off slack over here really quick so what they needed to do was they needed a way to kind of keep track of the application to know um on the given page how long it took after all the assets had been rendered after the user had received everything they needed to actually make that tweet and that kind of gave them kind of the overall view of user-centric performance and so they i don't know if they called it but it was referred to as the user timing api so um the second half is all of the dev tools specific so the best way i know to do this is actually i'm just going to dive in and we're going to look at all of this together and we're going to use it uh we're going to do this together with um we're going to do this together with a google search demo and i'm going to see if canary is working for me today it is not so canary broke sometime in the last 24 hours so you can't see this or you can see that you can't see this i should say so i'm going to have to go and open up an incognito tab just to make sure that we can totally isolate the browser performance here so i'm going to lower this just a little bit all right yeah i'm not sure um they just pushed 104 um and now there are issues so let me do the first thing i need to do is i need to open up the chrome dev tools um and i and if for those of you who are not available it's available in the the kebab menu there um so we have let's see we have the performance tab which we're going to talk about today and then at the bottom um if you have the secondary part of the the devtools you have the performance monitor tab so again this is what we talked about earlier remember we talked about the cpu utilization um and we talked about um kind of the uh so we talked about cpu utilization um and what you also get here um are the kind of the javascript heap size dom nodes and event listeners and documents the combination of all four of these make up the sort of things that you care about when it comes to memory loops if these get larger over time without being reclaimed in a garbage collection cycle then eventually you'll exhaust all available memory your cpu will peg to 100 and you'll get that off snap that we that we saw earlier so what i'm going to do now is i'm going to start a recording uh using the performance tab so here we go i'm going to go to google.com and i'm going to select this and type in playlight hit enter and we've got the results we've got the results that we needed and what was generated when i was doing that recording is this artifact in this artifact i'm going to close this this artifact is the the performance timeline and this is where you get access to again your heap documents nodes listeners um and in this case gpu memory um that's kind of your memory that you care about you have your network information and the resource timings associated with each individual network request you have your timeline view of everything that occurred so you can see where when when all of the assets were available you can see when it became actionable and you can see when i started to interact with it when i went beyond the single page boundary and onto the second you can see and got the initial set of search results um but the main thing that i want you all to take away from this is that it's a lot it's a lot a lot there's a lot going on it's very dense there's a lot to kind of suss out here but just about anything that you would want to know in terms of web performance or client-side performance you can discern from this view this this timeline view and all of this is made available um as a artifact right um as a json file to import and export so you can look at look at different snapshots every time and i'm going to call out two more things here um you can see where the main thread you can see this you know harness the evolutionary part of your brain that says red is bad right red blood bad um this is bad that the the long task on the main thread took longer than 50 milliseconds and with enough uh you know contextual understanding of the web application and you know enough time i guess we could all figure out what exactly is wrong in making this call here um and maybe look at ways to optimize it i doubt it with google search but in any case um this is your long task and this is being called out explicitly and then the last thing i wanted to show and i hope i can do this quickly um are the user timing uh i think i lost oh here it is the um uh the the frames which relates to the fps um and then the timings tab so this is interesting so the timings tab gives you a view of all the individual points and time that um that are associated with given events on the page but specifically interestingly enough these individual slivers are actually contextual performance marks given to us by the chrome dev team um or the google search dev team so the google.com search team put these marks in place to keep track of things that are you know interesting uh to them when it comes to performance um and unfortunately they're obfuscated so we can't see exactly what they are but i could imagine they would be uh related to when i first interacted with a given page and and um what was associated with um that first interaction or the the drop down of the list box so i'm going to go ahead and just leave this in the background and again this is the performance tab um there's a lot going on um but it kind of completes the um kind of the apis that we need to understand kind of client-side performance anything that's not available in the official spec is exposed or made available in the dev tools itself so we have the network we have in this case these are timings related to this one in particular is the first paint we have the user timing or the performance marks here and then we have our long tasks all made apparent uh through this you know this noise of a very very noisy image but the question then becomes how do we get these from chrome and how do we get them programmatically and automatically um and the way to do that um one of the best ways to do that i should say is with a tool called playwright um so playwright itself um is more traditionally under known as a functional test automation tool um it's based off of puppeteer um and puppeteer is how the chrome dev team tests chrome created for themselves so that they could test chrome test the internals make sure that everything worked and what ended up happening as i understand it the some of the developers were poached from google to microsoft to effectively make playwright and playwright is very similar to puppeteer and some of the apis are interchangeable and we'll see that as we as we go through this but specifically what it was intended to do was support edge and so that's you know microsoft edge you know i think it makes sense for them to support edge chrome firefox safari and ios ish um so it's sort of sort of a ios device um and so puppeteer can only work with chrome in addition puppeteer initially at least only worked with javascript playwright supports java c-sharp python python async io javascript and typescript so you can use any language that you're familiar with or most of them i think the main one that's missing is go and ruby to interact with the browser but it provides a very very tight coupling to the browser internals much more than is traditionally available in selenium which um isn't necessarily important for you know a lot of test engineers they don't they don't really care too much about the browser internals they're more interested in you know the spectrum of coverage that they can that they can cover by writing a single test with selenium a single test selenium can cover many many many many many more platforms but in our case today what we want to know is actually something that's more specific to the internals of what's going on with chrome so we we're going to use playwright instead of selenium and that's because we kind of get these inside out performance measurements that we care about um so the easiest way to talk about playwright is actually just to kind of get started with it so to get going playwright you simply have to do these two commands npx install playwright and npx playwright codegen so i'm going to do mpx playwright codegen and just to show you how easy it is to write a test um i'm going to start this up and on the left you can see the the chrome browser and on the right you can see this inspector thing that's going to make sense in a second so i'm going to go and repeat what we just did google.com and immediately you can see what's going on here so on the right hand side you can see a weight page go to and then the url and on the left hand side you can see interactive view of what playwright has access to and what it would do to locate and interact with that element in that case i'm going to search for playwright as if i'm you know doing this manually which i am doing this manually and it's going to record all of those things and then along the way and then give me access to this code that i can then immediately take out of this uh recorded session and put into my um code base so i'm gonna go ahead and do that now i'm gonna stop the the running test go to vs code that i have here and i'm gonna go to my again this is the repo that's available online and you can see this this code block here is exactly what was copied out of um that record and playback functionality available in playwright so what i'm going to do now is i'm just going to run that just to prove that it's the same npm run test local basic is the name of the test so it's going to run this test that i just recorded um you know this is this is standard test automation stuff right uh i think it's cool but uh this isn't necessarily while we're here but what um what we just saw is how easy it is to kind of get started with something like playwright um and so i'm gonna go back over to the presentation and then talk about uh kind of the hardcore parts of playwright so we're going um into the capabilities of play right now that we kind of got our feet wet and got started i showed how easy it is to get started right but let's demonstrate kind of how powerful it is so the first thing i'm going to demonstrate is talking natively to the chrome dev tools and then getting access to the performance api so let's go back into our code and i'm going to open up this first test suite and put a dot only on it so that we only run this test and you can see that we take that same test from before it's right here this is the same test from before but all we're going to do differently is we're going to instantiate a connection to the running chrome dev tools protocol session and then we're going to send a command to that running session and just say enable performance metrics and then when we're done we're going to send git metrics to that running cdp session and then log them to screen we're just going to log into screen so same test as before but we're just going to see what information we get basically for free so instead of running this i'm going to run local capabilities so we took that same test launched it we're talking natively to the running from dev tools and bam look at this so this is all of the information that we got out of that test that ran earlier that's now available inside the test code that we can do whatever we want with so you can see some of the call outs here we have uh remember memory leaks so we have the number of event listeners on this page number of nodes on the page and what we really are interested in is the kind of the heap size so all of this information is available to the test all of this information is available from a easily written um test case that you may have written yourself or recorded for yourself and these are things that you can kind of keep track of and assert on if you'd like to so let's move on to the next um capability before we go there um a question from josh sure how different is this from puppeteer because as you page does the value to do this and things gets weird [Music] this seems a lot easier um yes so uh so it's uh some of the apis are identical um and uh the the thing about i'd say the main distinction between the two is um playwright is written um more for uh you know you know qa folks to get started with testing so going um and clicking on these certain elements and interacting with things is far more likely to work um because of some implicit actionability checks so in order for me to click on this aria label search on google.com it needs to be in the dom it needs to needs to be visible have the state of visible and it needs to have stopped animating for five milliseconds so um when we talk about kind of flakiness and we talk about the distinction between puppeteer playwright and selenium a lot of the great things that come out of playwright is all of these actionability checks that are autumn automatically built in to everything that you do so this is far more likely to succeed on multiple attempts and there's less weirdness that'll happen just in general um but what i don't wanna i don't wanna um bury my lead but i'm gonna talk about the performance evaluate in about 10 seconds um but it is it is um it is still you know working with raw um uh you know data structures from uh javascript and it is wonky um they're they're they're i've run into some issues but um you know i i i myself uh just ask the the development team about it and they'll usually um you know fill in the gaps of my knowledge um where possible yeah i'll let you go on with the restaurant we'll talk more about other questions sure yeah yeah yeah again it's just about action the main the main distinction is uh actionability um the actionability checks um let me go back um oh so the the next thing i wanted to show is um this one here um uh so this is the next capability i want to show you that uh we actually have access to that uh performance uh timeline trace the raw data and we can record it at any point in the test that we're interested in or not at all um but back to your evaluate wonkiness um because we have this tight coupling between the browser and the test framework i can actually just inject my own marks into the into the browser arbitrarily at any point so that that's one thing you can do you can evaluate basically anything inside the browser from your test and then do whatever you need to from that point so in my case i'm going to take that same test i'm going to start a trace i'm going to inject my own marks that mean something maybe only to me that don't modify the application code including my own measures but include them alongside all of the existing marks and measures that come from google and then include those on the performance timeline so let me do that really quick um so a one worker so running the same test and again this is going to get very boring i apologize it's not doing much but this in this case uh it's not doing much differently um under the hood um but what you see here are all of the uh the marks um both from the chrome dev team and uh from myself that i've kind of injected into the page um and then the measurement the measures between them so the measure is literally it's literally just um uh you know you come up with this measure a variable and then you you pick two um marks arbitrarily so you could even measure the difference between a google one and your own if you wanted to for example um i don't because they obfuscate it in a weird way and i i tried that once and the demo broke and then i was a little sad but uh what we can do here is um we can take that um file from the test run and uh go back to our session here and go and import it right here so this is what we just ran the test so that test generated exactly that same trace from before but the the main difference here is this artifact also includes our own marks that we added from the test right so these marks are included in timeline so you so you can do anything you want at this point um with these so if you wanted to measure your own things in the application you can if you want to make your own marks or or make your own measures totally up to you totally um you know this is using the same thing more more to that point this object this artifact is just a json um blob um and so you can um get pieces of information in and out of it um as needed um it's it's great um and then the last thing i'm gonna show is api interception and fps measurement so i'm going to go back into our test suite here get rid of the dot only collapse this test and i'm going to go to route image placement so i'm going to do two things there is an open bug that you can only modify routes by disabling the service worker so i have to when the page loads i have to delete the server service worker not a problem there's an open pr to fix that um i'm not sure when it's going to go in uh similar to before i'm going to connect to the running cdp session and i'm going to send a command to show the fps counter over top of the running page next i'm going to wait for um the google logo to come through and replace it with one on the file system for bing um they're their nemesis i guess i don't know they're probably pals i don't i don't really know um but all of this is going to happen um while the test is running um in the significance of this and i'm going to take a screenshot at the end the significance of this is now that you're in the browser um with performance tests you can modify any um any sort of api request or anything that you would want to modify on the fly so you can imagine like if you have some particular uh image or some particular api response um coming back from your uh you know a cdn in a weird way you can just you know stub that out effectively and keep a more um and keep a better view of web performance um for your specific team that you might be on so maybe you have uh one part of a page that you are worried about but you know this other team they're a bunch of jerks and they never have accurate resources you can just stub that out and continue on your way you can also step out individual parts of an http api response request and response and anything in between um you don't just have to do static assets so i'm going to go back to the tests open up this and go and show that while this test ran at the very end uh it was using this amount of uh gpu memory um and it had 27 frames a second so something something an artifact to kind of keep track of so on my team we have um certain displays that certain people certain scientists are going to use so we kind of keep track make sure it stays above a certain level a certain number of frames per second um uh during the course of the test okay so let me go back so those are those are really six capabilities smashed together um just for uh the speed at which i can execute things um i'm going to show two more like i said you know i'm on a journey here um and uh certainly not selling you all on anything um i have two more capabilities that are proven sorry these work but i'm not i'm not gonna stand behind these uh so uh yet so the second one is um the performance observer object is a um browser-based api that allows you to you instantiate this performance observer object and then you say hey be mindful of these events and measure and monitor these events over time and the one that we care about are is the long tasks api so i'm going to go into the code here and going to the add observer so what you would do with this is this is an example of the test that you would write you would add a nit script the location of this this function here and then when the test begins the page is initially loaded you instantiate the performance observer object and then you push things arbitrarily to it in this case i'm pushing long task entries pushing it to it and then at the end i read it off furthermore i'm i'm taking that one step further and i'm triggering more of these long tasks remember long task is the every time the cpu every time the main thread is pegged for 50 milliseconds i'm actually throttling the cpu to artificially trigger more of those tasks so that i can be more mindful of where they might occur with a less powerful machine for example and again this probably works like if you were to run this in a test it has an output but i don't know exactly yet whether or not this is really getting an accurate view of those long tasks and it's something that my team is going to um investigate further soon ah har file so uh the other thing you can do is in the same way that you can remember record the performance timeline you can actually record the har file which is the complete suite of network requests as they're being sent across um so you can then have this artifact that perfectly captures the api requests and response that you're generating from your user in your performance test your front end test and then you have this artifact that you can then do something like run it with k6 or jmeter to simulate many users so um uh so what that looks like that was generated in that test that we just ran so i'm gonna open up um uh k6 and i'm gonna just uh see i should be able to go back to test builder configure i'm going to delete this test i'm going to take this har file that was generated and then create a new test start building and then import the har file that was recorded while we were going to google.com right there and then i'm going to run this as a virtual user um and this is going to convert to a script and again i think most load testing tools per have this capability where you can take a hard file but what i'm really trying to demonstrate here is that you can take your existing uh playwright performance test record the har file along the way and then convert that into a virtual user so if you're going to run your performance tests uh your client side performance tests and you want to simulate many users what you can do is you can run your client-side test once capture the har file convert that into a virtual user for jmeter k6 whatever you're going to use then simultaneously generate as many users as you want for that given uh user journey that you have and simulate as many users as you want um because the cost of running a real browser is you know quite expensive yeah there you go we just load tested google.com and they did great good job google um anyways uh so yeah so that's that's one technique again i haven't done this yet so i don't know all of the you know all the problems that are going to occur i have used tar files for jameeter in previous projects i've worked on um so yeah worth calling out um so let's talk about some of the challenges associated in performance uh playwright testing um the main one is test drift so this comes from dynatrace's synthetic monitoring tool so this is a paid for product saying that they're generally difficult because whenever a minor application element such as a button change the corresponding test much must be too so i call this test drift in the same way you know there's configuration drift in um in it where somebody will modify something and then the it'll be out of sync the configuration is drifted from the the code this is kind of test drift um and so the way around it is to use something like playwright um and so this is this is something that's actively going on oh goodness this is something that's actively going on today um he may have even merged it in um but on my team so i've open sourced all of the um the performance tests downstream um and they run on every commit they don't assert on anything right and they don't record the the they don't push the data anywhere they just make sure that when i run my downstream tests um on on you know internal to the nasa network and everything that they more or less match the state of the application that is behind you know the closed source so you can see here i run the visual test functional test and also the performance tests on every commit and you can see this this person is dramatically improving our search capabilities um but with that we need to also change some of the locators so this this ensures that my performance tests my client-side performance tests continue to work downstream even as you know for example the class name you know changes so it's a technique um by leveraging playwright i'm actually able to share the same code base so all the developers are now totally capable of writing functional tests and by extension they could actually if they wanted to write performance tests so it's a it's a pretty great way if you take a look here you can see where i'm adding my own marks so the idea here is with time if this becomes an interesting performance measurement actually get this baked in to something about the search results so that's something that i hope to do tomorrow actually is get the get some of these performance marks built into the code so i can rip out the uh the page evaluates quick question about file how did you generate it um you just uh so you add it to the playwright config here uh where is it yeah you the record har comes with the browser context and you just provide the path to the given test so um you could um so this this is the playwright test framework that's um that's uh defining that the local chrome that all my performance tests are running on um are using these properties so the the viewport height the timeout values headless or not um at the same time you can uh you can send in that you want to record a har for every test that runs so that's where that's set and we another question was related to cyprus uh how that compares and like also some conversation about this asynchronous being able to run their synchronous calls and still get the results back something yeah so uh yeah exactly so philip uh so philip uh harik i think is his name i went to my talk uh in march um and he's a cypress ambassador um and he provided a example of exactly how to do that um with cyprus based off my talk so i can um after i'm done with this what i'll do is i'll i'll include a link to his work on how he implemented marx using cyprus asynchronously in exactly the same way again that's a that's a browser api that's nothing to do with chrome or chrome dev tools that's just something that's built into all browsers um so something to be mindful of and to speak more specifically to cypress i uh i actually implemented our first end-to-end suite functional and visual uh in cyprus and then uh had to rip it all out to put in playwright because it's um for for the reason for the challenge that i'm trying to solve here which is uh keeping the the kind of the the the performance tests uh working alongside development as the development changes it was just it was going to be too much to kind of keep cyprus and playwright so i just ripped it all out and just used playwright but you totally could do some of the same some of the same things with cyprus but not nearly to the same extent in terms of like the performance observer thing for example i don't think you could do and a quick follow-up question and heart generation will generate many uh will it overwrite the car file file if you run it multiple times um yeah yeah yeah so this this was updated yeah but will it do it for each test before uh or uh will it generate multiple files oh i see yeah so in that case you would want to um do you would want to do something like i've done here with the fixture file um and then you would want to [Music] extend the base browser class and you would want to do the base context class and then start and stop the har file from within a base fixture like this and rename it so instead of naming it har har you would name it the name of the test based off of this browser um context that you're extending um that's a good question again that would be something that i'll i think i'll eventually solve um i just haven't um we're just not there yet we talked about uh challenges challenges and solutions so network remember so we talked about network being one of the leading problems associated with web performance well the best thing to do is just get rid of it so you can get rid of it by running so the traditional problem with running some of these load testing tools especially some of the sas ones is that you need to run them um kind of outside of your internal network so you basically introduce the entire internet in between uh your synthetic user monitor and the web application so you have to do a lot to remove the the internet variable so um when you're doing something like playwright and doing a diy you can just run it inside your same network in some cases you can run it on the same host as either your back end front end uh you know any part of your um internal network that you would want to most minimize in terms of network delay then once you've removed it then you can control it so in the same way that you can use the chrome dev tools to um to inject network latency profiles you can just do the same thing in your test case right so in your test case you can just add 500 milliseconds of latency um so you can then create individual profiles once you get latency to zero then you can then totally control it and do whatever you need to to either simulate your users uh differently or simulate network conditions that would have um impacts on the front end um and then yeah it kind of just to talk about uh the the network control that we looked at earlier this is the code of routing and fulfilling the bing image into the search um but the more specific example um would be so um on my team uh one of the database calls um has everything to do with the database uh of the environment that you're in so like the qa environment production uh stable all of those things um have you know different size databases and different you know responses so what we've done is um uh just uh mocked the response payload um and made that the same so that the rest of the application and and associated calls um are more controlled um and so you can uh get a more precise sorry not precise more accurate view of the other parts of the application that you care about um what we haven't done yet which would be really great is to take that uh api response and make it much larger to start doing some of the proactive uh performance engineering tasks like see how the front end would work when you have thousands of a given folder type right all of these things and you can also inject your own latency to these um routing requests if necessary um the next thing to talk about is transferability um so a lot of the times people have like named performance machines that the metrics are only tied to the measurements are only tied to and you know your machine's not going to be the same as mine this machine is different from the one you know right next to it um so the best way to get around this is to use a tool called browserless which is effectively chrome as a service um and it does way more than we have time to talk about today it deserves its own series of talks um but it basically becomes the interface for performance testing because it's chrome and all of the essentials in the container which is very difficult for anyone that's managed to selenium grid it's not trivial but it exposes a generic cdp endpoint for your test to use that they connect to via a connect over cdp api almonds gives you transferability because you can lock it down with c group control so you have this browser that is the same everywhere presumably the same everywhere two cpus four gigabytes of memory something that is transferable um and that you can measure so what i have here is i instead of running it uh remotely i have it running locally um and so this is uh the web ui that it comes with so i'm going to hit the play button and this is a playwright script or puppeteer script i'm not sure but what you see here is the test on the left on the right you have a video of the test running and then access to the chrome dev tools associated with this browser that's being pumped out um so my two-year-old just got home so i'm gonna deflect them to go down to the pool um so um yeah so this is that test running and if you want to have direct access to it um you can get a video of it running in real time um or if you want to run your own tests you can go browserless capabilities so let's run that same test that we just ran a second ago but instead of running it locally with a local playwright install i'm gonna run that with um this containerized chrome instance um let's see if it runs it did oh crap okay so this is that test running um and then this is just that you know that same test is running uh in a chrome in a container and so what you could do is you could um you know keep a version of chrome running on your network basically attached to any part of your internal it network that you care about um and they connect remotely to it with the remote cdp so it basically looks like this um so you have you can basically move browser list as close as you need to to any individual part of your internal it infrastructure and then connect to it via the exposed connect over cdp interface and so effectively you start with zero latency and then you can kind of inject your own to create any of these network profiles and again this is a transferable thing and then the next thing to talk about with transferability is creating a machine baseline for these results so every every even with even with uh um you know controlling this with chrome as a service um there's still going to be times where you need to uh you know create another layer of transferability so with this one which you can see i've taken um a test and what i've done is i've basically injected html and then i check for that html that i've injected and that time between the two the console time is how long it takes this particular machine to run um and generate results so before i run any of the tests i just get a zero to baseline basically and there's still a lot of variance in this um and what this gives me is this is kind of like the thing that becomes the baseline of all measurements if i'm testing and measuring nothing um then i know how that compares to the rest of you know the web application that actually has you know real javascript um the last few things to call out um precision accuracy and variability um leveraging browser time from the performance apis is great um so instead of keeping the system time like you saw here uh it's great to use those performance marks to keep everything in the browser to keep a browser-based view of all the time metrics that you have because it's very precise the browser time is in the order of microseconds um and the next thing to call is you know playwright itself is going to change chrome is going to change here is a chrome um bug that was just fixed that impacted our functional tests just fixed i guess last week where basically they broke one of the sizing apis and it broke our performance test and our functional test because you know chrome changed right that's that's all i'm trying to illustrate but you're still gonna have other sources of variation you have run run over run variation and how do we know if the ten percent decrease in a certain metric is in the application change or just in the noise even after we've controlled as many variables as we can and that's to run your tests a lot a lot a lot and it becomes a data problem um so the way that we've solved this is with influx db they provide a uh it's a it's a time series database very similar to prometheus for those that have used it um but it provides a native javascript client that you can just add to your test um and what's great about influx versus the rest is you can modify the time stamp as you send it versus prometheus which uh its view of the world its view of time its view of measurement is the time at which it received a um a data point um so when we're when we're worried about accuracy this mat this matters um and so this is what this looks like right um you uh basically have your uh your test measurement that you care about some things about it and then the the date.now for example if that's what you're going to send so it keeps a very accurate view of of measurement uh yeah so that's uh in summary uh that's uh we talked about client-side performance uh we talked about what to measure uh we talked about playwright um and we automated a google search performance test to some degree um and then we talked about some of the problems and associated solutions um so with that being said i'm gonna spend just a minute talking about kind of what's next um so this is a really exciting space um and uh it's by coming to meetup groups like this that i'm able to keep aware of you know what all is going on in this world um one of the next tools coming up is xk6 browser so you saw k6.io which is a traditional you know api load testing tool well that team is creating their own kind of container by way of a go based performance measuring tool that's that's using playwright apis effectively to do what you know i'm doing today um it's in beta and it supports the connect over cdp if you're interested but what it solves um if is it basically allows you to generate as many measurements as you need and then tap into that huge k6 ecosystem to correlate any sort of events that you have in terms of your system monitoring or anything that's going on living in your system you can automatically just pump and get in by you tapping into that ecosystem and then for me i need to push everything into a performance observer and again i don't know if that's actually going to work and then someone asked about puppeteer and playwright there are two apis missing from playwright in with regard to performance um and one of them is the kind of the a great way to automate memory leak detection um so i'm gonna have to do this by hand um and then uh the the last thing is um there seems to be some performance timeline library parsers out there but they're from like 2017 and they don't seem to work with any of the off the shelf things that i'm doing um so i'm hoping that that comes into existence where i find the right uh performance timeline parsing library because then what that'll effectively allow me to do is anything that i generate during the test i can actually just request out of that performance profile timeline view and then have total access to that in the in kind of the test case um and then the last thing so the front end tests are all on playwright and we've started an initiative to open source everything um and so this is a great opportunity to contribute directly to nasa um so a traditional cla applies to our tests that applies to you know the regular web development um so it's as easy as a little kind of get started with playwright um i've onboarded four um open uh you know open testers and all you basically do is you go to this github discussion um put your name in and then we'll get you started um it's a great way to contribute um it's awesome uh and a lot of people have found it to be fun which is uh i think you know playwright is fun so i'll end it here thanks everybody awesome awesome thank you john looks like reflecting michael and i when i speak when i speak i need to mute you something like that okay it looks it works when it's perfect so um amazing i mean this is mind-blowing uh i just want to highlight that um when you uh came over to meet for speed uh i was like my mind was blown with how the problem is so different from what we usually do right um a lot of measurement we say well let's do your real user measurement in real time and and then we discussed how well for you real time is very precious and you have to do it way before you start the mission right so you cannot test any of this as mission runs and it was like very different world and uh obviously these kind of tools are like the only way to accomplish this and this is very different you know so uh if we if if anyone has any questions please post them in the chat i do have a question myself so i noticed that when you send those client requests through chrome developer protocol i guess cdp right you use some commands uh how do you figure out what they are i i use them actually that's with the technique that you showed which is the dev tools and dev tools approach um in one of the the meetup groups um that that's and some of them were just kind of found off of the web um you know a lot of what i've done came from a lot of the work from synthetic user monitoring teams um and so that's where i've got some of them um i am looking for a few more um with with regard to um getting getting uh you know more precise frame measurement is really what i'm looking for um i you know looking at inspecting screenshot images is kind of uh doesn't scale uh so i i would be interested if anyone out there has a better um you know frame per second measurement that's available in the devtools for example okay let's see if we have any other questions [Music] tanner's asking to simulate mobile device devices uh would you just use playwright throttling options or is there something like browserless for mobile devices or say integrated with webpagetest to simulate mobile devices um so uh um to simulate a a device you could so i can i can show you what we so one of the requirements uh for the mission is ipad um we've disabled this but what you can do is let's go to ci you can generate you know you can use an ipad which is basically using the open source webkit browser and then sending more than just it does a little bit more than just send a proper list of user agent stuff but it wouldn't for example it would give you the rendering that you need um in the view that you need for ipad but it wouldn't do anything more than what you see here so this this is effectively the the view of the world when it comes to using the playwright um devices for for mobile ozy aussie is asking about resizing and setting report size waiting for time i'm not sure i understand the question though so the way that i manage browser sizes uh is to use um profiles in the playwright test framework itself um so the playwright test framework uh like right here allows you to set certain mission sizes so like this one um this one is unique so we uh the so we we do know uh during the mission what um resolution people will be using for for the monitors um so like they were bought and paid for on the you know on the government i guess it's your tax dollars at work so we know what the the the viewport will be so we have the special profile for those monitors here um so i i that's how i do it um and then when i run the um functional tests i run them all at that resolution um but interestingly um i don't do that for the performance measurement so for the performance measurements i actually go into the lab and i take real measurements with a real device for kind of the final final thing um with with a real monitor um just just so i can get everything down including um you know the gpu itself um which is which is hard to do in containers yeah we have some problems with gpu testing uh in our work we're very very flabbergasted by that you know it's a problem um another question from josh uh so do you not trust using perform server api in here i know you said work in progress uh what is your main concern yeah exactly yeah so i i don't want to unders i don't want to undersell it i just haven't we haven't used it yet um but it's it's it's a way to um i don't know if the long tasks for example are being triggered because of the communication between um the browser context and the playwright context so that i'm getting back an accurate list of long tasks but i don't yet have the attribution associated with them and i'm not sure yet if it is uh due to just the interaction between playwright and the browser but if i wanted to measure things other than long tasks um uh like uh individual resources and everything like that i would totally trust that um but i'm uh i'm more concerned with that can you test that against uh yeah yeah that we just have i haven't done that yet exactly yeah so i i think i will i think i'll get there um i think um i could get there faster if you all contributed some open source and test yeah i just you know it's just dividing up my time between functional and performance you

Channel: NY Web Performance Meetup

Views: 17,691

Keywords: Web Performance Optimization, Web Performance, WPO, playwrite, testing

Id: IrK-XDH72bw

Length: 65min 46sec (3946 seconds)

Published: Wed Jun 08 2022