How Carvana Transformed their Testing with Cypress Test Analytics

so welcome to another cypress webinar with me are two guests from the same company called carvana tyler and eric uh tyler monty do you want to introduce yourself yeah so my name is tyler monteith i am a engineering team lead on the quality engineering team at carvana i've been with the company almost three years coming in march so just under three years it's it's good to have you as a guest and eric connor hannah uh how are you doing great thanks for having us uh my name is eric hannah engineer 2 in qe farvana currently on the finance team and been with carvana for exactly a year well nice to have you both and this is your webinar you prepared all the slides you want to tell us how carvan actually uses cypress test analytics to do with testing and debugging efficiently before we begin a little bit of housekeeping the slides the link is displayed there already public so you can follow the presentation along or you can send it and look at the slides later on we will get to the questions and the questions are from you our audience you can ask a question at slider.com by entering event code carvana big surprise you can also upload other people's questions so that when we get to the question section at the end the most interesting questions will be answered first as always this webinar is recorded and we'll send a recording and two slides to everyone who has registered so tyler why don't you take it away and let's begin all right so a little bit of background on carbona carvana is an online used car retailer we are proud to say that we are the leading e-commerce platform for buying and selling used vehicles offering as soon as next day delivery to 263 markets i think we just opened up another one across the united states the continental united states we don't offer uh service in alaska or hawaii and we also have a pickup option at our 26 patented signature car vending machines which you can see here after our poll i believe did i go the wrong way i'm already messing it up i forgot how to use this that's okay so there's our beautiful vending machine i think that one's in california um so yeah consumers have the option to browse through more than 20 000 vehicles on carbana.com with the ability to purchase and finance high quality used vehicles in as little as five minutes i've heard that some people actually do go through the process that quickly which is just absolutely phenomenal and they could do this all from the comfort and safety and home of their home or on the go via a mobile device so we offer a mobile platform via the app store or android google play store as well as a mobile browser option as well carbonic vehicles all come with a 7 day return policy no questions asked so if you get the vehicle delivered to you there's something you don't like about it since you bought it online you have the ability to return it and all of our vehicles go through a rigorous 150 point inspection and they've never been in a reported accident no flood damage and we take the quality of our vehicles very very very seriously since we know that people are buying them without seeing them first over the internet and with that in mind we have a question would you ever buy a car online this is a great question because tyler i do see carvana commercials on tv all the time and i even seen your car mobile you know driving around carrying a car it's a big purchase so it's interesting if you can actually convert it from you know physical experience to online experience oh i it seems like we've got a race we've got a race china it's it's not a race because only from carvana i counted us yes yeah yeah so i guess if we count that technically we are over i'll take that yeah it's an interesting um you know experience experiment if you will um trying to buy a vehicle you know that large of a purchase usually everyone buys one you know every five years every 10 years it's a significant experience so the founders of the company ernie garcia and a couple other people you know they had that going into this thinking this is extremely important um one of our core values is your next customer may be your mom and pretty much in everything we do we try to instill that kind of thought process everyone you know from answering the phone calls all the way up through ernie we try to do our absolute best to make sure we treat everyone you know with uh with you know experience intact and just give them the ability to feel comfortable through this kind of process answer questions along the way provide as much information about the vehicles as we possibly can so many people have answered and it seems like the online part is important you probably have a website or a web app i want to say you want to talk about that next yeah so here is our our home page and then to break that down a little bit let's make sure i do this right i'm going to mess it up okay so yeah we have we have a desktop application that has mobile functionality as well so if you're on a mobile device it will scrunch itself up really pretty that way you can view it correctly um and within that we have a lot of technologies we also have a lot of applications that our internal advocates use we have hundreds of them working behind the scenes tirelessly to move vehicles across the country a lot of logistics things that are happening as well and with that we have to support a lot of technologies that the teams are using to develop these applications that kind of all funnel into this one main point primarily we are still pretty dotnet heavy on our back end um recently in the last couple years we ported over to react on our front end i would say for the majority of our apps especially our customer facing applications and then we have a lot of stuff in between we support python applications java applications we're starting to dabble into javascript node backend applications as well doing some cool stuff there porting into kubernetes like just across the board like there's a bunch of stuff flying around that make all of this possible so within that you can see here's our search page i know this is a pretty basic shot i try to get a couple like gifs or videos going but they were too high-res and i had some issues with that but you know if you go to our our website carbana.com and start playing around on the search page or the financing or car finder pages you'll find that there are tons of apps and services feeding into these main points that allow you to search for you know thousands of vehicles with specific criteria applying your financing terms where it'll actually pluck out cars that don't meet your financing criteria if you choose to finance and even applying like a trade-in vehicle a lot of people will trade in their older vehicle when they buy a new one and you can actually apply that like as a down payment filter that gives you the ability to to filter out even more cars that way when you're going through the process of looking at vehicles on our site you generally will come to the search page you'll figure out which kind of cards you're looking for and once you have some cards that you want to look at in detail clicking on any of these search tiles will take you to our vehicle details page where you get the ability to click on the vehicle images and when you do this live you can actually spin the vehicle around in a full 360 picture view you have the ability to open all the doors zoom in look at any defects like if they're little paint chips like i said we purchase high quality vehicles but because they are pre-owned things happen when people are driving them and there's little minor defects and we want to make sure that we just make it as transparent as possible when people are searching and viewing vehicles and then you have the ability to actually go into the vehicle itself and use like a 360 panning option to look around the entire interior of the vehicle we also have the make it yours section on that page which ties back into that financing and trade-in and down payment option and there's an additional section too that allows you to look at the inspection reports the original windows sticker and then all the details about the vehicle itself so there's a lot of services a lot of dynamic data being loaded into every uh single page that we have on our our site and on top of that we have a native mobile app too so a lot of the native mobile app back end actually pulls from the same desktop app just to have a lot of unity um but the the actual app itself is completely custom they built all their own elements all of their own um descriptions and pulling everything in so that way you get the same experience but it's you know on a native app so across the board carvana is doing a lot of different stuff and there's a ton of services and a ton of teams and it makes it pretty challenging for the qe team to try and scale up with that it has been an interesting experience as the company's grown let's see so with that here is our team uh this photo you can see on the right that was taken 2019 pre-covered times back when it was fun and safe to go out into public and do cool stuff um this was actually shortly after we had purchased our carvana license and we got a lot of awesome swag from you guys so we thought it would be really funny to wear all of our shirts to a local like event area within phoenix um because we are hooligans and we knew that if we got in trouble we could just be like oh yeah we're with cypress like you know you know send the bill to them um but we've grown pretty quickly in the last couple years so arlano was founded only seven years ago and they didn't hire their first quality engineer until 2014 uh by 2019 we had grown to a team of 15 and now we're up to a team of 20 and you know looking into the future we hope to hire even more people as the company continues to grow as well and tyler can i quickly add if you're watching this webinar after a webinar we'll send an email to everyone who has registered and we'll have a link to the survey if you feel that survey you can get a cypress t-shirt and pretend but you work at carvan already so yeah yeah they're they're phenomenal t-shirts just to help sell that they're soft comfortable they fit well um and they're great for activewear and they do remind you how to install cypress on the back because they have mpms that's right they have the npm install command right there on the back perfect all right okay so let's let's kind of dive in a little bit of how we tested all of this stuff uh pre-cyprus back in the dark times um so we had a selenium framework that was built with csharp.net and we used the page object model to kind of classify and organize all of the information um the actual test environment that we used we use vsts pretty extensively it's now called azure devops within that space they have azure pipelines so that way we can define the build steps pull the code down run the actual test gather the results and post them somewhere at that time too in order to actually deploy the nodes out for the selenium framework uh we had a devops person build us a custom java app that would go grab virtual machines spin them up and then connect them to the the main hub of how the selenium network worked [Music] since then right before cyprus we actually jumped into a new platform this is our kind of like automation 2.0 free cyprus that we built this used kubernetes and it was a custom selenium image uh put out by a company called arrow cube i think and they had images that were called the moon pods they were pretty cool we were able to spin up up to like a thousand nodes on the fly which is a pretty big step up for us at carmona because at that point we were trying to do mass parallelization um but it was difficult because we only had about 100 test cases and that moon structure was was very flaky still and there were additional downsides as well that made things pretty difficult which included two hour execution time even with all that parallelization we had long running tests a lot of retries i'm talking like major flakiness across the board because we had some seriously long-running end-to-end tests and the debugging of that was just difficult we actually had a lot of challenges trying to get video off of these kubernetes pods and this java grid scaler that we had so like trying to figure out where tests fail why they failed what preconditions may or may not have happened um it was difficult to get back so we actually focused quite a bit more on the logging structure for that application um so that way we could try to get some sort of clear sense of what was going on but you know we weren't happy with it at all we knew the direction that the company wanted to go within in terms of growth and scaling and we knew that this was not going to work out long term there were some additional challenges too from like a process level it was just super difficult to debug our longest running end-to-end test took anywhere from 14 minutes without a retry if i recall correctly up to 30 45 minutes for that one single test case like i said using the page object model and net it had a higher barrier entry so our more junior qe um they had a difficult time jumping in and actually contributing and participating into the the code writing process for that that architecture and at that point we had a single test environment there was a dev environment as well but it's a no man's land generally speaking like you know i think a lot of places where the devs just go in there and do their thing so we had the test environment reserved for us but you know it was limited feedback we weren't able to go into prod and actually test buying cars because we would have chewed up inventory and caused a lot of things to happen um that wouldn't been you know good for the company so we were just focused right there in our test environment with our you know at best flaky uh test framework and this is probably one of the biggest setbacks that we had was just there was a lack of developer support within the org our front-end developers didn't really like the idea of having to spin up a super heavy selenium solution in c-sharp especially because they were using javascript they were all porting over to react and our back-end developers although used.net same thing they didn't want to spin up a web browser to do their back-end testing so we're like stuck in the middle and we had like a mountain of changes and a mountain of new features coming towards us i mean we knew pretty quickly probably within like eight or ten months of having this solution that we needed to make a change quickly so that's when we started looking around um and the difficult constraints of having to like scale up to so many tests to parallelize everything get the feedback that we needed to get the analytics that we wanted from a enterprise size so that way we could share with leadership like where everything was from a metrics and kpi perspective like all of that was lacking and we would have had to probably hire a whole dev team to take care of all of those things so we really started to shop around at that point say okay you know what's next so why we're here cyprus uh we looked at a couple options test cafe was one that comes to mind it you know it had some of the features that we were looking for but cyprus is the one that really really nailed it for us um it allowed us to rapidly develop and test in a uniform low barrier to entry environment you know it's all based in javascript and node it's derived from mocha which is super easy to interpret and read we love that about it our developers also love javascript kind of regardless of their domain or focus area the front end devs that's their natural playground they love it the back end devs have an easier time working with javascript when they're using a different language so it's easier for them to adopt and the dashboard is just like the shining jewel i think for our company having to have all of that enterprise support so having the parallelism taken care of out of the box the analytics of course that came a little bit later i think we purchased our license just prior to analytics really getting beefed up to where it is now um but we knew even then with what it was able to show us with you know how many how many uh nodes to use how many vms to fire up seeing the actual test cases and the um the outcome of those tests seeing where each one of them got ran being able to click on them and immediately see the video playback see the the screenshots that was like a huge win for us and it was it was super super easy to do beautiful so initial wins we had to port over all the tests that we had in selenium and as soon as we ported them over i think it took us like two and a half months roughly to get all the tests over it was pretty difficult getting through our old page object model and like you know transgressing through every single page and diving into every single definition that we had but once we got everything over and we realized that you know we could put a lot of that logic into the commands area of cyprus um and figured out how to set up the actual test runs we had an extremely clear idea of why and where a lot of our flaky tests were failing that we just kind of like knew they were failing and notated previously but now i could actually see super super clearly exactly why they were failing in addition to that video and screenshots right out of the box whether we are running locally or in the cloud we absolutely love that i can't tell you how many times we would run our selenium solution locally and because it took so long you know we'd set it up we'd run it we're like all right gonna go get some water go get some coffee use the restroom we'd come back and the test had already failed and we're like okay so i'm going to start this test over and re-run it again and then you know someone taps my shoulder and we're like oh yeah you know let's chat missed it again have to run it again um with cypress we just had the video dumped out so we could just click it watch the video and see exactly where the test failed we were able to wire up our results to slack which provided even faster feedback especially when we started doing our cloud executions and our overnight runs that made it super easy to come in in the morning click on our results and say okay where are we at for today where did all the dev teams deploy their stuff last night what do we need to start attacking right away um because of all this functionality that we got out of the box we were able to move back into areas or new parts of the site that were historically challenging for us to develop test automation for some devs even started participating and learning how to write tests in cyprus you know they started hearing rumblings they're like oh we heard you guys switch to a new tool like what are you using is it cypress yeah absolutely like here's what you need to do to install it here's our base repo that you can start digging into right away and learning um and the initial feedback was like light years ahead and way more positive than we ever got with selenium like with selenium we're like yeah we're using selenium sorry and now it was like yeah we're using cypress it's this new sexy thing like everyone's got to start using it it's interesting because you said the developers kind of found out and want to jump right because as a developer i feel there is yeah there's disconnect i don't want to throw my code and someone else to find box because i feel bad about front absolutely right i want to find them first and if i can run the test myself oh i'll do it of course so by the time it gets over the fence it actually looks good yeah there was a little bit of like a salesman ship thing that we did in the org for a while where we went around to the teams and we're like look what you can do with this like it has hot loading like as soon as you write a test you can just save it and it'll run itself and you can develop your code in your tests simultaneously and they were just like i know and we were like yeah we know like it's pretty freaking awesome it's kind of interesting because we're not paying you to save these things you're actually paying us i know i just like i know we talked about before but cyprus has been a huge step in the right direction for us at carvana and maybe this is me more as tyler personally like cyprus is awesome it's been a huge relief for us and all the things that it's been been allowing us to do at carmana there were still some things though that were a little bit difficult that we had to figure out you know some some had to happen with your support team other stuff we had to get with our devops team or with developers and figure out you know just unique stuff that we had to do at carvana we still need parallelization so when we started our cypress license and we started running tests um we only ran on one box because we didn't understand exactly like how to run on multiple boxes with just one command like we're like it's too easy like we need to research it we'll get into that um we needed to make sure we had a single platform for all of our teams to use so whatever we ran our tests on we wanted everyone to be able to access them at corvanna and we had to support multiple test environments at this point so between the time that we went from selenium to cypress we actually spun up another test environment our user acceptance testing environment so that way we could you know kind of sound bored or clarify a lot of our tests um in our test environment without having to go and like actually damage data in prod so we can keep that stuff pristine that would be bad yeah and it absolutely had to support multiple cypress projects and the video and screenshot capabilities were a must for wherever we ran these tests and there were other things too that we had to consider like we wanted to run different types of tests for for load testing functional api testing on these boxes as well and so we wanted to know from people around the industry how do you guys use parallelization with cypress because it took us quite a while to figure out a mixture and a magic setup that worked just right for us right so it's a great question you know go to slider.com enter carvana event code and pick an option i think many people run cyprus on a single cm machine hopefully it runs fast enough so they don't feel time pressure i'd say anything longer than five minutes you probably won't already like optimize it right it becomes a bottleneck i call it a coffee driven development where you fire it off and you go to get coffee and you come back exactly yep yeah that was a big concern for us too on a ci perspective so when we really established cyprus we started going to all the teams we're like hey you got to start writing cyprus tests to help us keep up with like the crazy amount of stuff that you're doing and a natural reaction especially from the leaders was like well how do i make sure that i don't slow down my builds yeah and we're like well that's a great question so what we have for you here is our you know our paralyzation option and that put a lot of people's worries and heartburn to rest when we finally did get our parallelization up and running excellent excellent i i'm glad that people are voting and i definitely see that majority of people can it's like untapped market right definitely a lot of room to grow perfect let's see how you folks have done it yeah so like i said it was a bit of a journey and i think that we're still you know figuring out the most optimized way to get there as our constraints have changed over time um but we'll share exactly where we are today so using like i said the dashboard to really hone in and see oh for example in this this regression test that we have we knew that we could go from one machine to four machines and shave the test time down by you know 57 when we started seeing these in our multiple projects those are usually the ones that we attacked first and prioritized when we wanted to do parallelization in terms of actually accomplishing that we used azure pipelines within azure devops formerly visual studio team services to build and maintain our cloud execution steps that's pretty typical similar to jenkins you know you set up your steps it runs your tests the trick that we had to figure out was like where do we send the tests how did they actually run like on the physical location so what we ended up coming up with using cypress's documentation and working with your support team was we have a set of 10 linux vms that we use they have all the basic requirements that we need to run cyprus in addition to some custom things that we have as well within that we also use slack integration so we can report our results and share failed tests faster more efficiently and then we followed cyprus's guides almost to the t for the azure stuff specifically and it actually came out to be something pretty simple so within our azure builds we have the option here to select how many agents we run across we had our devops team make the connection to point to those specific virtual machines and then for us all we had to do is add a bash command which is exactly what you see here um npx cypress run we choose the browser that we want um we we have a specific video flag so when we run locally on our our developer machines we keep video off because it actually taxes our cpus but when we run in the cloud we want it on so that way we can get that delayed feedback later we've got a custom group flag our record and our key settings that way we can actually apply it to the specific project that we want we use a spec flex so that way we can run specific spec files we kind of treat our spec files like test suites if you will so we have like a smoke spec for smoke sweet regression etc we could pick our environment from the config file that we followed from cypress's documentation as well so we can run it in dev test uat etc and then the parallel option which is obviously the most important piece for running in parallel and then the ci build id which we pass in an azure option which gives it a custom unique id that gets shared across all the agents and that's what actually stitches them together and sends all the tests up that for us was like the big kind of light bulb aha moment that's how they did it when our devops team asked we're like i don't know as long as you set this stuff up like it just works and we don't have to figure out like load balancing and all that fun stuff like just send it to those boxes and get it running it's kind of funny because the dashboard will figure out oh all these machines that pass the same build idea but for the same project they all want to split the tests right let's say you have 100 specs well i'll give each machine in the first pack and when it's done it will come back and i'll pull the next back from its cue and so if you look at dashboard and it says if you add two more machines you will speed it up you can just go and change number of agent to from five to seven and done nothing else will change but that's what we really want to accomplish and i think so far it's working yeah sometimes we describe it like like magic right because like you said it'll just pick which specs to run when and why um which for qa team i think is just i think it's phenomenal because before cyprus like we had to figure out exactly how to set up the connections how to set up the test how to spin up all the nodes um and with cypress once we realize the capabilities of how easy it was you know we just we just grabbed hard and ran yeah okay so in addition to the parallelization um we also take advantage of the analytics so that way we can make more powerful decisions at carmona because we're we're operating at such a large scale now going into a certain spot and you know making decisions without data can be costly sometimes so we really have started to take um advantage of the analytics especially as you as the cypress team have added more features recently um we've really started to latch on and bake them into part of our reporting our metrics and our kpis something we wanted to highlight a little bit which we thought was really neat was over the last three months we've increased our test suite size by about 41 and this arrow here will indicate when we actually added retries in when i think version 5.0 came out and how that affected our test duration time so you can see we added the code in on this date and within the next week our average test time skyrocketed pretty significantly up to an hour and a half which was obviously very bad but we haven't ran and retries with cypress since we started using it for almost a year so using the analytics dashboard to identify very quickly where you know we had problems creeping up on us we were able to go look at all the individual test cases that were running long that were flaky and able to shave them down and make efficiency gains very quickly and even though we've continued to add test cases i feel pretty good that we've kept our average run time this was a fluke day that we had with our vms we fixed that but other than that we keep it in a pretty steady 30 minute range even though we have almost 200 test cases and we want to try to keep it around there even as we add more spec files so like i said we were able to go to the additional analytics pages within cyprus we were able to see like why are these tests running for so long what are these outliers doing that are causing our tests to run um for extreme amounts of time so clicking on that bar graph gives you this readout we were able to see okay it's our financing purchase process test which are by far our longest running tests we were able to see not only is it taking you know at least on an average two minutes to run but when the retries were introduced it just stacked on top of itself and made it even longer so it flagged it that it was flaky too and we're like oh what's this like how do we take advantage of that and fix that as well seeing the failures too we were we were also noticing that some of our test cases were failing 100 of the time which i don't know without the analytics we would have caught as quickly we would have eventually saw it by reading our daily readouts but seeing this we were able to click on and see oh okay yeah someone fat fingered the element definition which caused it to retry even more exasperating a lot of the issues we were having so we're able to click into that and uh debug it and fix it very quickly you know darla i'm looking at this charts and now like i'm seeing it with like probably fresh perspective now i think everything like to a right of a median should be like warning red flags right here like just look at those right like that abnormal the things that are completely outside of what most of the tests are doing and just pay attention to that yeah yeah generally speaking like we'll start looking up at like the why are you failing more than one out of five times this one really caught our eye right away because you're like whoa 100 failure rate like what's going on there this test has never passed like how did it even get committed but you know once you break it down and you actually realize what it's doing it makes a lot more sense one thing we don't use as much yet just because we're still developing our strategy is like all the filters you have at the top filtering by branch we've used that a little bit because we've had some special cases where like the dev teams have asked us to like jump into specific settings so we'll switch to those branches and use that within the analytics the the duration of the data i use that one pretty frequently when i'm sharing it with other teams so i'll change it from 30 days to a year run group not as often bias committer sometimes we'll use the big ones for us are definitely branch um and whereas i think it's browser yeah that's the other big one we use pretty frequently so we can say you know what's the difference between chrome and edge and firefox are the big three there's there's one more thing like wireless filters sometimes management wants to see the overall historical directions of passing are we increasing number of tests if yes but how much absolutely are we running reliably and develop are we running reliably in your testing environment right or is it somehow somewhat worse right especially as you take longer view and you're asking yourself should i hire more developers or more qa right or pay for cyprus license renewal right and so on yeah yeah this one has become a pretty key figure when we come to like um you know what's what's the frequency because we can look at this versus like the commits that are being made by the devs like there should be some correlation like if we're adding new features there should be more tests being developed so we'll use that as another another good signal of like the test health as well all right so now i'm going to switch it over to eric he's going to talk about test retries which have opened up a whole new world for us at carmona like i said our long running and end tests have been historically painful but test free tries have given us a brand new a brand new like fresh breath into the space yep yes let's dig into test retries thanks tyler um like he was saying test retries we've been able to move into areas of our site previously avoided example our purchase process which is core functionality of our business it's always been very difficult to automate there's so many different steps of our purchase process that rely on many different internal and external services working together and changes are constantly being made and pushed to our testimony by multiple different engineering teams as you can see in this code here on the right just to create a purchase it takes approximately 50 lines of code so you can see where a lot of things can go wrong pretty easily um i know a lot of you might say why aren't you mocking or stubbing these calls that's because we have a requirement to complete an actual purchase hit all the downstream integrated services uh where he wouldn't quite get the feedback we wanted from mocking another great feature of retries is that we can retry an entire suite or we can retry individual tests you can see that this purchase process tests here we retry four times uh we typically set it a little higher for tests that tend to be a little more flaky for a typical ui test or for the entirety we would normally set retries to be around one retry per test and move on test retries and analytics has been a huge win for us we've been able to easily determine which tests are flaky we can see the overall flakiness the severity and breakdown of each flaky test uh right here on the dash right here on the dashboard as you see in the screenshot purchase process test it takes top spot is our flaky most flaky tests followed by prequalified and they all have medium severity so using all this data it's really helped us tremendously with development efforts determine understand why certain areas of the application are more flaky than others and i want to just kind of quickly explain to the users what is a flaky test when you enable test retries either globally or protest cyprus will record if that test failed and when was retried and then passed so all the says that with no changes in the same run sometimes past sometimes fail a mark is late that's it so you have to have enable test retry so we know that nothing else has changed when we retried it yeah yeah it's nice seeing these we can go into each test we can see that they've been flaky um you know they might be passing 100 you know day by day but we still want to go in and see you know why these tests are failing and this makes it so much easier to you know just pinpoint this test see what you know how many retries it's been doing and just go in there and see what the failures were and if you know anything could work because you know you could get into the aspect where stuff is actually failing one time and it could pass the next where you're actually could be like bypassing a bug so it's always good to go in there and you know have this data to look at and then let's get into debugging so as you know debugging in the past something that's very time consuming uh you're always running in circles trying to figure out why tests were breaking um you know just overall not a very fun process but when it comes to debugging tests with cyprus this flaky test dash has allowed us to quickly and efficiently debug failing and flaky tests so the test shown here again is another one that's involved with purchase process it tends to be pretty flaky and in the past it was very difficult to debug it's an area of the application it requires numerous data inputs that talk to many different services which determine the outcome of the tests that we're trying to do we've been able to narrow down that data and along with retries testing this area application has been made possible and despite the flakiness the test has a very high pass rate if you take a look at the screen shot on the right we can see the most common errors so you can see that line 65 panel with auto pay it's expected to find that element timed out and then line 6. yeah no worries there we go thanks all right yeah so we can see that line 65 66 our most common um errors um you know it it makes it easy to debug and fix this issue or to go and discuss with devs why this area the application is still flaky for us um i mean 20 times and eight times you can see that you know things up here we there to fix the test um or bring it to devs uh bring it to their attention and get this fixed in the airlift capacity and that's why it's so flaky many people say well test retry is just masked bugs but i disagree they in realistic system block you from actually finishing the build you and then what do you do well you just rerun the build again and all it passes and you you never like investigate further to be honest right but at least now you will continue with build because it actually passes right at the end of the day but still investigate it and maybe you know resolve it yeah exactly the purchase process like if you think about what you actually have to go through to buy a vehicle if you've ever done it like going to the dealership it's like it's a whole day event so to try and consolidate that like into an online format um it's a significant amount of steps that you have to go through which you try to make as fast and efficient as possible but from a testing perspective that's very difficult you know there's a lot of data in input and permutations that we have to take to like if you think we service the whole country you could have a customer buying in california who's using all cash who might have a trade-in who might want gap coverage um and then that car might have to ship like all the way across the country like multiply that by all the different data input that you have to deal with and flaky tests are just a part of our business so to have the ability to retry it and then like iron down or drill into exactly where that flaky data might be allows us to like you know knock that one out and move on to the next thing and still allow builds to pass and have devs feel confident that our code and our test code is working properly before we ship it because if dev is not he cannot do this right if we're not straight tries and the build fails and it fails in something you're not working on guess what you'll just stop writing end-to-end tests because you don't want to be you know doing this right so test retries is kind of compromise where you can still pass the bill if it passes not right with any errors if it passes and go back and say hey this panel if something's wrong is the team responsible for that can you please look at that and make sure that you can find the root cause it's a very controversial feature once we release it from the cypress team there was a little bit of pushback like no it kind of goes against our main thing but for n n tests it's been like a real big saving grace for us um because we've tried to follow like uh the cypress dictation like unitize as much as you can isolate as much as you can but we still had this gap where we just wanted to feel absolutely secure that all our downstream dependencies from a customer perspective were taken care of and retries really allowed us to to jump back into that space that we've been lacking in for a while right and i say if you have multiple environments and dev is least reliable when testing and staging and reproduction turn on test retries in dev right because it's flaking by default and then maybe bring it down right using cli parameters in the test environment so you still use test retries but not tan but let's say free and in staging maybe one retry and then you do a small test in production zero retries has to work right you can do it yep all right so conclusions finish this off yeah pass it back to tyler all right so um yeah we've covered a lot of stuff today i think the things that we definitely want to highlight have been our biggest wins since switching over to cyprus it was really hard to gather data from selenium because we didn't have the awesome dashboard analytics feature but our memory serves correct and looking from past logs we had a two plus hour execution time um taking the same amount of tests and actually multiplying that by you know 1.5 or 150 percent we've been able to shave down our test execution time by 72 percent which has allowed us to continue to scale and be as fast as the rest of the org we have significantly less failures even though a lot of the stuff we've shown you today seems like oh you know there's stuff all over we're running our test cases hundreds of times a day um and you know we have on average i'd say two to five failures and it's usually these flaky test cases that we've shared with you today so even though we still have some like before it was 30 40 50 failure rate now we've shaved it down to absolutely minimal true failures that need to be investigated um in that time since we've become more confident we've had retries reintroduced we've been able to jump into way more portions of the site we're up by 140 for the year and in the last i think it was three months that we showed we did 41 alone so we've had a lot of great growth because of that and because of all of the flexibility and ease of use of cyprus about half of our engineering teams at cypress or at caruana utilize cyprus in some way whether they're consuming our tests they're writing their own tests or they're collaborating across teams to use like a shared set of tests we've just had a huge uptick in people doing cyprus test automation at carvana which is you know one of our main goals is to grow in that quality engineering space get more people to think test testing minded and use the tools that we have to be to be more confident and ship better code very nice q a time well as you can see there are 50 questions 55 oh wait what questions are being added faster than test retries so uh tyler maybe um what question catches your eye so that we can pin it to the top of a board ew what do you need me to do ah move it around maybe maybe maybe this one right so i'll highlight the question right okay do you mind answering the first yeah so can you please share a little bit about best practices that you use to make it easy to maintain being scalable besides page objects um we've we've experimented with quite a bit actually trying to figure out like how do we get one team to be able to use another team's code and how do we collaborate across the board and just grow this repo that we have commands are probably one of the biggest things if you think about cypress specifically the cypress commands area has allowed us to write a lot of code across the board and just you know condense it into a single command we actually have multiple command files now so we had the first one that kind of comes with like the getting started example repo with cypress when you first work it but we've actually added more command files that are specific to sections of the site so it's kind of like a weird hybrid of page object model and cypress commands where like we have commands for our purchase process we have commands for our search page and we just try to consolidate all of those features and functionality to those specific command files to help spread it out so that way it's not like one ten thousand line command or cypress command file nice uh can i ask you what do you use for native mobile automation right yeah so we we would love to use cypress i'm just gonna throw that out there if we could but um in the meantime i think we're using a lot of espresso we've dabbled in appium as well a little bit but you know it's similar to selenium and we just we don't like that as much so i think our mobile dev team uses espresso and a couple other things that i'm honestly not that familiar with unfortunately they're you know we want to break into that space a little bit more using something similar to cypress or hopefully cyprus and in the long term this is something that i think you as heavy users can really answer right whatever pain points of cyprus that you would like us to address um retries was a big thing for for a long time but we've addressed that now right those have been added in which have been great um i think other things would be maybe like we've had problems with the docker implementations and i know we've worked with the support team trying like iron those out but certain parts of our site use desktop browser heavy dependencies that don't load into docker for you know reasons x y and z um so we've had to like make a lot of considerations and strategy changes because of that which you know really isn't that much of a detriment to cyprus but it is one of our big pain points that comes to mind as a recent um yeah i think you know overall we've had we've had a lot of good success with cyprus well if you have any pain points but you still want to address open github issue we'll be happy to address it um this is kind of interesting question right like how do you know when to write an entrant test or even if you need one yeah so back in the day with selenium we were a bit naive on like what tests to write and how so everything was essentially an end-to-end test now we take the approach more of like does this have to be an end-to-end test can we stop this can we knock this out like what are we really trying to get at here when determining what needs to be asserted at the end of that test so for our end to end tests the purchase process that we have is probably one of the few places that we actually end up still writing an end test and that has to do with those downstream dependencies that we actually can't call via the psi intercept command we're just stuck with having to actually go through the whole thing and assert from the ui at some point that something came back correctly and that has to do with the architecture of the system there's just like no way getting around it um i think for a lot of teams like when you first start out the end-to-end test is like the easy low hanging fruit that you start off with but as the the size of the repo grows the size of the app that your test against grows it becomes really difficult so we try to very quickly like pivot to let's just stub this let's just mock this let's isolate this test let's not chain our tests together and let's just really try to like adopt the cypress way of testing as much as possible and leave the end-to-end test to be very far and few in between gotcha well uh why not right you've see you've shown a command if i understand correctly your spec is really the name of a folder and then you have multiple spec files inside the folder right that's how you organize all your test scripts yeah so how do we organize our test scripts can't put all scripts into one folder um we've we've had good success with that so we have like our integration folder and then below that we actually break out a lot of our folders by team and then within that we have regression folders and smoke folders it can get a little bit tricky with doing the padding and the dynamic pathing in cyprus but going back to your guys documentation has made it really easy for us to figure out like okay like you can't use that type of path here you have to pull in that file this way has made it a lot easier for us to figure out exactly how to divide things up initially we just ported over everything the way that it was from selenium but then after looking at how cypress wants you to set things up we started breaking things out more by track and then by page and then by feature and then each feature essentially has a spec file per page and then per per team at carmana gotcha i'm going to put in a chat our real world app that we kind of put our design thought into and you can see how we organize the test scripts there um this is a good one uh do you use visual regression right you showed us functional tests yeah so i would love our team to jump into visual regression testing showing you guys our vehicle details page with that really cool spinner how it throws the whole vehicle around um cyprus can't do that with the functional testing like i can assert that that like the view is there but i can't actually click and drag across it without doing some really weird javascript stuff so visual regression testing is a place that we want to start looking into and researching because our site is so dependent upon being able to visually inspect vehicles and that presentation is so important to making a customer feel comfortable in the purchase um that you know it's it's top of mind for us to jump into that area and for our viewers we've done a bunch of webinars with open source tools ling and third party services that actually offer visual testing so just check it out all of them have a cypress plugin um this is a good one and maybe eric maybe you want to tell us how you debug test but only fail on ci like with frustrating ones but always work when you run cyprus open but somehow still manage to fail in ci um so for this yeah i mean so i mean we for this we like to go to the dashboard you know see why it's failing go log into the website make sure you know doing it manually we don't run into any issues and then also like look at what browser version we're running um and then try to duplicate that locally if it's sometimes it could be like a chrome versus electron uh issue that we run into that's like the most common thing we've seen um running in ci versus locally right right yeah you can definitely look at the failing test and see which browsers and which operating system it fails on yeah uh i want to suggest an outpost possible link we have a little bit of a documentation how to debug a flaky test that only fails in ci um it's difficult right there could be timing issues race conditions or browser issues and so on what can i say yes maybe we want to replace big end-to-end tests with smaller tests right but kind of do each part separately um so i'll on market um so tyler you said that you actually have multiple cypress projects right every team has pro their own project so in the end you have a bunch of them yeah so how do they organize them the teams have you know adopted cypress or at least some of them have started to um generally speaking we'll fork our repository and then we'll just kind of gut the stuff that they don't care about or they don't have to worry about and then we'll just place it like into a test folder or an existing test folder within their project or the repository um and then from there you know they already have access to like our configs which have like all of our back-end calls so they can create like an account for example they can start clicking around on some common use case um dom elements um i've not had the opportunity to do a cypress project or multiple projects within one repository actually that's very interesting um we've not thought of doing it that way one thing we started to think about is like how could we if possible like share common use cases without even actually having to have multiple cypress repo so like we started looking at npm a little bit like is there a package we could put out like internally to share some stuff around but the easiest thing for us right now is we fork our project we got what they don't need and then they get to keep all the good stuff like the base urls the configs and then just start up and running and scaffolding their tests out gotcha gotcha so you kind of not divide and conquer but instead like throw away and take it right yeah like decentralize it a little bit you know this is a question that's uh always on my mind right so do you use fixtures to load your data or eric said that you want to go through all the systems at the end of the day so maybe there is no pictures what do you want to take that one eric since you're you're the main user for our data yeah um so we do use fixtures and we also do create we do both actually so for like purchase process tests we we have fixture data that brings in um purchase data and then we also generate like a random user we have what we call star data users so we like hit an api call it brings in like a random person social name all that stuff so for each test basically we start with a fixture we run the random um generator to generate a random user generate a random email and then we kind of save that data to that fixture uh file and we use that fixture sheet throughout the whole test thank you which makes a lot easier yeah i think we have time for maybe two more questions um uh one uh i just wanna say that right now you cannot use cyprus for loaded performance testing because cyprus has its own overhead because it drops every event in your browser so it's not meant for performance or load testing you can certainly do load testing by just spinning up so many cypress boxes but where our tools are probably do it better um here's a good one right so you're probably doing a lot of end-to-end tests but you said that not everything is an end-to-end test uh do you write separate unit tests what's the balance right now yeah i think i so i wouldn't call them like traditional unit tests right they're not actually writing application code only on you know local instance and evaluating methods and functions and lines of code what we try to do is think of our tests that aren't end to end as unit tests so we try to like compartmentalize them and isolate them as much as possible you know using stubbing mocking that way we can just say i just want to test this one unit or one feature without any noise or any weird implications coming in from an unknown source um so we'll do we'll do some api testing but we don't actually do like traditional unit testing um with our cypress implementations today it's either a couple n tests a lot of unitized functional tests and api tests i gotcha thank you maybe one last question do you like this third one are you testing it against live web deployed or isolated environment do you yes so do you apparently you're testing a staging environment right a test environment do you run something against production by any chance uh not with cyprus we have a couple manual cases that we'll do there very very seldomly we do have a staging environment that shares some some prod information so there we don't do anything really heavy duty it's some light stuff we try to really follow like a shift left method like the lower environment we are the more regression we want to do and as the stages of the product move up through the cycle it's just like regression and then smoke and then staging and then it's out to the world um using live web deployed yeah generally speaking we try to isolate down to our tests in our uat environments and we try not to touch prod as much as possible just because we don't want to buy a car accidentally and then like you know have it shipped across the country and then like who's paying for it so we just we stay out of there completely yeah but you get to keep it right so that's true that's true that's true well excellent i think we ran out of time we will answer all the questions uh on slider so um can we go to the next slide please yeah absolutely and the next slide is thank you thank you to tyler and eric and to everyone who tuned in it was my pleasure thank you folks so much for preparing this webinar and sharing what you're doing with uh cyprus yeah absolutely we are so happy with cyprus and happy to share our story with that success um and you know get to learn from everyone in the industry on you know how they're using cypress and what they're doing uh to make their testing better thank you for the opportunity we appreciate it and reminder we'll send an email with this video link to the slides to everyone who registered for this webinar take care and happy testing bye

Channel: Cypress.io

Views: 1,970

Keywords:

Id: 6tMV28yaPJI

Length: 58min 1sec (3481 seconds)

Published: Wed Dec 02 2020