How PlanGrid (Autodesk) Achieved Consistency in Automation Across 20 Development Teams

hi everyone I'm Gleb akuto from cyprus and with me are two guests from planned grid Graham and Brandon and today we were having an excellent webinar we're going to discuss topics that have interest to many many large organizations in particular our guest today will tell how plain grid which is now part of Autodesk achieve consistency in automation across 20 development teams this is our longest title ever so it's a record Graham Brandon are you ready yeah excellent why don't we start by you guys introducing yourself sure sure go for it all right I'm Brendan drew I'm the tech lead for the automation platform team at Autodesk yeah Graham Harvey I'm the manager for that team excellent it's good to have you guys thank you for taking time the slides that you created available at Cypress the slides that come at quality in 1 million projects so why don't we start with you telling us about the plant grades and team organization and what the problem was sure yeah so this is like a quick kind of background about about plan grid so we are a software company solving problems in the construction space essentially allowing everybody involved in big construction projects from architects right through to kind of the boots on the ground building large or small scale projects just allowing them to kind of stay in sync keep their blueprints up-to-date I'd be able to annotate and keep track of any changes kind of in real time and make sure that no one misses information that they that they need on the job site so what the company established back in 2010 actually a Y Combinator graduate company and then acquired by Autodesk a much larger company this past year so we've primarily been composed of software across four platforms so that's like web Android iOS and Windows to provide kind of a consistent experience whether you work on like a Windows machine and a trailer or on like an iPad out in the field right now I think we're looking at about one hundred and twelve thousand monthly active users we're 50% of those users are made up by accessing it through the web app nice thank you for visiting to tag introduction and in this webinar you propose the following topics first you gonna talk more about organization especially with teams and your role in those teams and then we're gonna talk about the five problems you have solved right the problems are here I'm not going to repeat them because we'll get them in a second but for all our webinar our viewers will have a questions and answer session at the end of a webinar you can go and ask your questions right now add a slider and here's an event code side plan grid just go there where is a direct link if you want to use it direct link but once you ask a question you can see your question there you can upload other people's questions and we promise to get to every question at the end of a webinar in addition we'll send a post webinar survey and if you feel the survey and provide you address we promise to send you a Cyprus t-shirt and maybe a couple of stickers and I know it's a high value item okay so why don't we start why don't you tell us about your development a model and the testing yeah sounds good I think to best understand what we're gonna talk about it's important to understand the model that we work in so we just want to give a bit of a context to a little kind of picture of how we work let's see here so so this diagram kind of shows how our teams are structured we have kind of like larger groups and then teams within those groups so this there we go yeah so as part of this we have like a frameworks an infrastructure group and within that group we have a bunch of platform teams that handle like mobile platform web platform infrastructure related activities and then QA as a part of that and our automation platform team is one set of the framework and infrastructure teams so this is a team of six engineers who are primarily mandated to support provide improvements to any of our automation frameworks help with integrating all of this into the pipeline move towards better CI CD practices and this is technically across our web platform and all three of our mobile platforms and the kind of back-end services as well so it's kind of a pretty big ask and we get to work on a lot of interesting stuff as we kind of bounce between the different frameworks it's a really neat place to be because we kind of fully understand how quality fits into the whole system as far as web development where we're going to specifically talk about today we work kind of in a scrum ish model using Sprint's but you know not necessarily as strict as scrum might dictate each team kind of achieves varying degrees of C ICD we all use Jenkins and spinnaker we're all based in react AWS s3 using Postgres databases and a bunch of other technology is like launch Darkly to handler or future flagging as for the front-end apps we follow this app shell model where we have over ten independent github repositories which compile their front-end code down into like micro service applications and each repository is representative of a feature of the larger plan grid application each of these are independently built tested and deployed via their own pipelines and so to handle this we actually have shared libraries that help kind of standardize how teams can build and not have to have every team set up their own builds and pipelines and kind of like repeat across you know 20 plus teams so the Texas is probably a little small here but the point of this diagram is just to show how code kind of flows from our development right to our production so from the first pull request that's opened tests are run as they should be and then that app is actually deployed to what we call a review up so testers can go to exploratory testing with it we can point things at it whatever we need to kind of do to test that code before it lands anywhere kind of as to a wider audience once all of that passes deploys merges to our dev branch and then deploys to what we call our dev environment in this environment we still run smoke tests basically the battery of whatever we have available as we want to promote code through to test and master branches PRS are opened tests are run and anything failing is blocking obviously and then at each stage we deploy to a new environment finally ending up in our production environment and these red boxes just kind of show what the automation platform team is supporting which is kind of those test frameworks allowing people to test across any of these environments and pull request time as well grim can ask you a question really just to clarify your team is not writing the test for everyone you're writing the tools that allow every team to write the test and also to run the test yeah that's correct and we'll probably harp on this a bunch throughout the presentation this platform team is here to enable like speed efficiency all the things that are good in testing without us having to be responsible for the tests themselves but just to enable teams to be able to do good testing quickly because we know you know onboarding onto test tools and figuring how to write good proper tests can be time-consuming and we want to remove that barrier from teams perfect perfect thank you so we said we would talk about our journey for how we got to where we are and kind of that involves understanding where we've started so this all kind of started back in 2018 when the platform team was formed at that time our web testing was done in a selenium based framework with about 50 ish tests running they were largely written owned maintained by QA which was kind of a combination of some kind of co-located and some kind of scattered QA so not a lot of organization there necessarily mm-hm and I think the biggest problem that we realized was was kind of an impetus to this change was that the tests were not well understood by the larger dev team we were using Python behave with kind of a like a gherkin abstraction layer what happened was these tests were often skipped by developers when they just didn't had trouble debugging or didn't really understand and then we didn't have a culture of like don't skip the tests because skip it or not skipping tests kind of prevented people from being able to progress through their work so it kind of was where it put us at that point we decided we had to come up with some new grander vision of how we wanted to approach like a testing culture and how we were going to do that so there's one of us thinking about about what we wanted to achieve in the future is this Brandon's picture from the back so what we really wanted to do when we sat down and thought of what this was like how do we provide a testing framework that that I mean dare I say is like delightful to use right so will people right run and maintain tests and not feel like it's taking up too much of their time and that they can't do that that's really what the goal is here so we had some must-haves I think we kind of formulated these through knowing what was wrong with what we had at the time and the first big one was we wanted the test to be in JavaScript or a JavaScript based language this is the language that front-end developers are coding in and if we want dev teams to own their own tests it just makes sense that there are similar technologies to what they use every day I think it's reasonably obvious that if we ask people to switch between stacks and potentially ID es it's just gonna complicate it overhead that people are not going to want to tackle this has also have to run quickly so the framework has to enable this we want all of our tests to be blocking tests we don't believe in tests that run later we want this short feedback loop and to have that long running test just don't fit into that culture in any way the tests have to be debuggable did this kind of that shift in culture where if skipping tests like skipping failing tests is going to be the exception than not the rule then this has have to be easily debuggable or else like we are just getting in people's way yeah and it has integrate well with CI if we want to be a CI CP culture we can't have manual steps required you know from from merging code or attempting to merge code right through to shipping code so these were the contenders that we started with when we were vetting we know that there are more JavaScript based test frameworks and the three listed but this is kind of what we landed on fairly early so we wanted to vet those ones and how we did this was we started by just having engineers review the tools so put up a fake or a real issue and some tests to kind of show what that would look like and ask them to weigh in on it or just even do a code review or if they want pull that code and kind of manipulate it and play with it themselves so they can get a feel for what these different frameworks would feel like on a you know day to day work environment there was a very strong preference scene when we did this for the Cypress Runner and interesting like little tidbit of information is that I was kind of operating on this false assumption that developers always prefer CI tools to UI tools and so I didn't think that the UI was going to be that big of a draw but once people started seeing you know the Dom stayed snapshotting and how easy it was to debug and watching your test run in real time and be able to kind of revisit any section of that test like huge when that I couldn't have even perceived at the time interesting how long did the vetting process take right two or three weeks on it yeah amount of time between the two of us and then intermittent kind of investigation from different engineers mm-hmm it's interesting it's also like bottoms up right so engineers private tools come together and discuss them yeah for sure and I think like a bunch of the suggestions actually came in from like the front end developers who had worked in some of these frameworks before I think the other obvious you know benefit was that it felt very familiar to write tests in Cyprus as it did kind of for a front-end developer that was used to unit testing so you know using like coconut rice and then was all like familiar to them right and then you know there's the whole cross-browser thing which we shots fires yeah well I saw an email earlier today that said how cross-browser testing is supported and out there so we probably could have just wiped this one from the slate but we are talking about two years ago so we actually did a bit of research we dug for some data and based on the number of like p0 p1 issues coming in from the field there wasn't really any that were rooted in in browser cross browser issues so cross browser didn't feel like a very high priority on the list when compared to some of the other things we were looking for I understand which is what got us which what got us here ok so we've decided on Cyprus and we decide we're going to implement it across our development organization we ended up when we look back on this encountering about five major challenges to this and this is kind of what we what we're going to talk about the solutions for today that kind of the meat of the presentation is here so we had to make our tool the obvious choice we'll dive into what this means but really like we wanted people to use it and we wanted them to feel like it was the tool they should be using not the tool they were told to be using we also had to then encourage teams to replace the existing tests that they had and we had to kind of change this culture and break the old habits and show kind of the new smarter more efficient way of writing tests we really wanted to avoid problems with reinventing the wheel where teams do the same thing in different ways or or implement things that they didn't know other teams have already like problems that teams have already solved so solving problems multiple times just leads to to waste of time we wanted to reduce the maintenance burden so that we could unblock progress for like further innovations and testing if we're stuck maintaining how do we innovate and do better and then we really wanted to enable anyone to debug monitor and and really own these tests because at the end of the day like the people who are writing the code should be the ones who own the quality of the code and I guess I say writing but what I mean is like developing so that's the development teams as a whole and not advocating for necessarily one feature developer to also own all of the testing for that feature but if a team is going to ship a feature they should be empowered to to own the quality of that and that includes writing tests that includes fixing failing tests or updating tests that need updating and gentlemen judging from a background you pick for this slide these were completely insurmountable five problems I mean factors seem like it yeah slow to get started but then once once you kind of get some traction behind you and get someone besides yourselves saying yeah this is great you should use it it really kind of snowballs in well I'm glad but you actually solve these five problems because now you can just tell us of a solution we don't have to reinvent the wheel everyone can be just happy right we sure hope so let's go all right let's talk about them so we'll break this down so we had to make our tool the obvious choice the major reason for this is that we actually operate in a kind of a free market model in our development organization so teams can use whatever tools they they want to use they're empowered to do their own kind of build test deployment release however they want there's obviously some exceptions we we have suggested tools out of infrastructure we try not to mandate stuff unless we have to there's obviously you know the odd exception we're like mandating is required because we all need to deploy into s3 buckets and serve them up the same way but in general to achieve the goals that they want two teams should make decisions that they feel like they've chosen for their own problems this tends to result in tools being chosen based on some of the more senior or louder team members experience so if you've tested ingest versus mocha or Eve you sinan instead of knock and they're doing a the same work but they just if someone's comfortable in it they're more likely to advocate for none Jule there's an architect in our organization that uses this phrase a lot and actually really liked it so I stole it for this slide but it is kind of the culture that we wanted to to drive towards which is just making the the easy way to do things the right way to do things you know this just means that that there's less barrier to entry to do things the way we've established because because it's the easiest way to do it so to accomplish this we wanted to integrate with our shared react scripts library which is part of our create react fork that everyone uses when creating new repos and we wanted to integrate with our shared Jenkins build library so everyone builds the same way we may as well just be ingrained and that so that it's very easy yeah so if you're familiar with create react app it's it's made by Facebook and it's a tool you can use to initialize a new and a front-end application so you'll get something that you can just run NPM start from and it'll start an app for you that you can build off of so we've we forked that and added some kind of more specific things to our implementation and one of those things we've added to it was Cypress so when you use our Fork of create react up and create a new a new project you get Cypress in there and you can immediately start your app and run your Cypress tests and it comes with a little little dummy test so this is this kind of extracts the logic away from the individual repositories so we can make it run the same way from each application so a developer from application a can go in to application B and they know how to run the app how to test the app and approximately how the app works and how to develop in it this kind of stems from a culture where like sometimes you'll see Doc's about like how to run the tests like make sure you set the following environment variables and then NPM run this specific command if you're in this environment or this command in this environment and it can be very confusing yeah it can be yeah so we we have our sacra scripts inside of react scripts which does the heavy lifting for you instead of having to settle those environment variables so it can detect if you're in CI or if you run locally and then based on that it'll either open the GUI test runner or run headless Li in CI and it's able to run inside and outside of a docker container and in houses all of our logic there and similarly we have npm run cypress build which will start to build application before and run the cypress tests and this is generally used in CI and if i can add to this facebook create react app now allows you to specify a template for your application so you don't have before you can just specify your own template and we release the facial cypress templates with typescript support and without so yes yes so this is kind of what's that Cypress script would look like so it checks if you're in CI and if it does we run Cypress with the module API because we found we have a lot more control over how we're running it and what we do with the results after the fact right so we process the results and print out what we need to if it's not in CI then we will just open the runner forward so then the other thing we had to integrate with was our shared build library which was pre-existing and built from the web build sorry the web platform team so this is what a typical Jenkins file would look like on a freshly created app with great react tab all the logic to build read lint do your unit testing deploy an application everything comes kind of set in a default way inside of this run full build command so one of the things we added to that was running cypress in just a default way which is trying to guest run all your tests in electron browser but if you needed to do anything differently for example if you had some sort of part of your application that only worked in chrome you can specify like this example they've they've just chosen to run in Chrome browser and then you just pass that into that she'll run full build command unbelievable we spent so much time adding Firefox support and all your scripts have browser chrome set by default so what this looks like when you push your code up to github and it triggers a Jenkins build this is what your typical pipeline would look like with Cypress built into the pipeline and it blocks the pipeline so you know these tests are important we consider them just as valuable as you know tests so if you have made a change that's broken to Cypress test chances are you've also broken some functionality in your app so you're not going to have a passing build which means you're not going to be able to merge that that pull request so we chose to do this because we've we've worked in other environments where would be a schedule of run that that someone has to monitor so if you if you have a scheduled run you can get into a state where you've merged breaking changes that have broken tests but without the fix for the test so by blocking pieres on it we're naturally enforcing but the tests need to run fast because no one wants to have like a 45 minute build wait for every time you push up a commit and they need to be able to run regardless of the test environment because we run on dev and test and master branches so those have separate environments backing them you can't rely on static data as easily and the test fixes must be checked in with the code so if you've broken a test you'll have to commit a fix with your change all is one deliverable in your repository you have application code and Cypress tests right they go hand-in-hand yet they live together and we we had that concept pre-existing before Cypress but at that time we were a monolithic application it was just one repo with the one set of tests beside it and as we started breaking that apart into more of a micro service optional architecture we could see that that's not going to be sustainable and we had to come up with some kind of creative solutions to reduce duplication that we'll cover in a little bit here mm-hmm yeah yeah so was the first problem so once we were kind of integrated into those libraries now Cypress is by default but how do we go and encourage teams to replace what's existing and how do we go fix some of those old bad habits this is difficult because like teams had tests well they had some tests I guess and then like the tests worked I guess most of the time although slowly and steadily they were being skipped at kind of a higher rate and in these tests there was you know a lot of like setup in the UI they were generally longer running tests if we have 50 of them obviously they're gonna be longer running that if we have hundreds and so this is probably a lot of the factors that resulted in in kind of the lack of maintenance of these tests so so obviously we know there's a problem but also things are kind of working so how do you encourage people to go back and change so we really want to prove the value like why do we need to do this and what we did is we did say earlier we we're not responsible for writing tests on our team and while that is true we do believe in evangelizing and providing value to teams in whatever way we can and at this point the way we provided value was by actually porting some of the old tests over so in that case I guess we were writing tests but it's actually not a bad idea to to write tests in a framework you're proposing so that you know how easy or hard it is to actually write tests it's little and B testing where you wrote the B yes so then what we did is we started to run these new versions of the tests in CI in parallel to the existing tests and I actually wish I had a screenshot of what the pipeline looks like but I was so long ago I couldn't dig one up so we don't make the new tests blocking so that pipeline runs as it did but just this other job spins off and runs the Cypress tests and you can see the results for that build once we started to kind of have test coverage matched kind of kind of secretly kind of not so secretly we flipped which set of tests were blocking the build along so suddenly it's the Cyprus test that are the blocking ones and not the former selenium tests then we waited for a while and we didn't hear any complaining and things seem to be going rather smoothly we deleted the old tests people were pretty involved in this process but I like to think that if we had just done this kind of stealth I don't even know if people would have known that this transition happened other than they probably would get more passing tests out of the build and they might be thinking what happened because now our tests aren't blocking us as often so yeah this just allowed us to prove that the tests were faster more stable and easier to maintain because it wasn't really impacting people's lives and I hope that the builds got faster so while writing these tests we were set out to demonstrate some good testing patterns so previously the tests were more of a scenario that would cover you know an entire workflow but when you get several scenario tests you kind of end up with some duplication like every single test is logging in at the beginning every single test is creating a project or creating a document in a project and you really only need to test each of those things once so what we've done is write smaller or isolated target tests and we use the api's of our application to setup and teardown so you can you can see here we're only UI testing what we actually care about in this case and we're setting up our state via flipping the feature flag programmatically instead of relying on a static project or just being unable to test it if it's behind a feature flag and then our test is just checking that some things appear which means that feature is functional it's just kind of a basic smoke test I think like it's probably important to note like this when we start talking about like using API is for setup and teardown and only UI test what's actually under test I mean these are concepts that people talk about pretty frequently so I mean I don't think that advocating that this is a concept that you know we came up with obviously you can do this in many test frameworks and this is talked about frequently but it's the idea that you know we wanted to do this to demonstrate that like testing can be stable I think often people are just familiar with having worked in previous environments where the tests were long running and they set up in the UI just because it's by nature that's the way the things were and so they they start to believe that that's what testing is like anything beyond a unit test is going to be flaky it's going to be long running and so we really wanted to show that it doesn't have to be the case right and just as a reminder to our viewers if you disagree with graham and brandon don't send them hate mail send it to me yeah a good debate about this concept okay so you know once we've kind of proved the value and now we start to get some traction and teams are using this and what we don't want is for teams to have to solve the same problems as each other and kind of like do the same thing in inconsistent ways or relearn the same lessons or problems so as more teams are onboarding we start to recognize that everyone has to log into the application some way or another most of the tests that are being written need a project you can't do a whole lot in our application if you're not working in a project many of those tests need data in the project so documents and sheets or some annotations or some reports or whatever it is lots need that to be able to do anything and some of those tests actually need to manipulate feature flags so that they can turn unreleased features on for our test users or there's some features that might be behind like an entitlement like a paid entitlement and we have to be able to turn those on and off as well if everyone is doing these actions in the same way and that's contributing to our reduced maintenance that we're trying to achieve if if suddenly for some reason our login flow changes and every test has implemented login in a different way everyone has to go and learn what the new login flow looks like and how they have to modify theirs to to fit the new pattern the biggest visible benefit to this is that like if something major that people are reusing across multiple teams brakes it gets fixed quickly by whoever notices it first and then everyone reaps the benefits of that I think this is a motor for your entire team right you standardizing the wheel and when everyone just uses four wheel yeah yeah I mean it is it is the core concept of allowing people to move fast and and not worry about the details that don't affect them so getting people to to build on this concept we for a long time now followed the like the yagna principle at extreme programming so for anyone who may have not heard this term before the idea is don't add functionality until it's deemed necessary you can you can start to proactively think about what will be needed and how things will be used but you don't really know how things are used until people are using them and then you start to uncover what the next highest priority needs will be so we we tried not to get ahead of ourselves in predicting what people were going to need out of us and just build what was kind of being asked for at the time so in this case we really have started with creating a user and logging in with that user and the previous slide talks about all the other things people need like projects and annotations but that came when it was needed and not before mm-hmm so we did this by creating a shared library so we treated this like kind of an internal open-source piece of software so anyone in the company can contribute it to it and a really key aspect of it is having good documentation so this provides us with a single point of management for shared commands and plugins and it's an extension library it's not a wrapper Oh Cypress which seems to be outside of Cypress fairly common kind of model is wrapped like your selenium driver and put your custom stuff in it right so it's just published as a Ted fan package so you npm install and then it has a handy little connect script that'll add the imports necessary to connect in the the plug-in functionality and all the custom commands and we we did this because we have the separate repos for our front-end application it's yet the nice central point to manage everything and avoid the duplication between the different repos so one of the other concepts that we we did was the ability to generate data instead of relying on fixture data so in this in this you you reuse a user if if they exist in this fixture file which allows it's kind of like cashing a creative you created user to reduce the load of creating them for every single test you can get a user for a test run and then in CI we like the slate clean and we'll create a new user at the beginning of the test run and reuse that throughout the run but locally to keep the load down when you're developing tests it'll reuse the user unless you delete the file yourself I've interesting yeah so we originally didn't do this and we started noticing some issues as we're running like thousands of tests and creating thousands of users and yeah we saw some issues so they had to do some some caching and also run our tests like against production - which creates user so you have to be a little careful that you don't like deploy your production database you know to the water thankfully I think we noticed it in one of our development environments before production that was the next step yeah be careful about your smoke test make sure of a non-destructive yes exactly and so as part of this - like what we realized was some of the data set up is actually quite intense and takes quite a long time so you know you don't necessarily want to have a test where before the test starts its setting up your environment it's creating a project but may be uploading sheets you might have a test where you want a hundred sheets to test you know pagination and suddenly uploading a hundred sheets takes half an hour so how do we keep the test time down while still not rely on data being in a specific state so thankfully someone on the team actually on the infrastructure team thought about this problem from a larger testing perspective and actually developed a tool that we use alongside Cypress which is kind of asynchronously creating Poole of projects that can be used via a variety of template's and so we treat this as a service that we can make requests to to say we need a project that has a hundred sheets in it and that service can provide us one on demand and then just replenish that Poole has been nice it's a really neat it's a really neat concept to to allow us to I guess not generate data on the fly necessarily but we're not relying on data that may or may not actually be there mm-hmm so one of the other things we did is integrate with launch Darkly which is a feature flagging service that our application uses on the front end so we just wrapped this in a custom command this allows you to set a feature flag either on or off and run your tests with with that state the other thing we've done is for easy to maintain really good documentation we found huge benefits from switching a type script we originally JavaScript and we had to maintain separate docks alongside our custom commands and they would get out of sync and they just weren't as nice so from this we use the typescript function comments and we use a tool called type talk to it's kind of scraped those and generated into an HTML documentation page which the pipeline for our shared library will handle building and deploying for us so it's pretty hands-off you just have to write your your code comments and then the other bonus of this cypress is able to use those to generate the intellisense so when you type side dot you see not only the default cypress commands like visiting get you'll see our custom commands like set feature flag in this in the screenshot here and this has been there's been super helpful for people figuring out what we have available and how to write tests better versus having to look at some Docs that may or may not be up-to-date and ask us questions directly it's kind of self teaching this works really well for avoiding duplication to write because if people struggle to find a command they need like before looking for too long they're just going to write their own custom command so having it like easy to find means that people know the benef that other people have contributed as opposed to it just kind of being blind mm-hmm nice okay so now we have a whole bunch of people using our shared library and Cypress and our uptake is really good and it's and it's starting to become a success story but now you know how do we keep this maintenance burdened down especially for our team we would like to continue adding improvements to the framework and help people be more and more efficient and that's difficult to do if we have like up to 20 teams using our framework and there and they're looking for help debugging their their tests or understanding how to use the new functionality we're releasing so we want to enable an environment people can reuse stuff wherever possible and and and just kind of share the learnings and there's and this there's a few concepts that this ties into that we're talking about kind of in the previous points and the future points but some of this is like treat the test code like production code so that you have some standard of quality for what you're providing to other people especially as we talk about treating it as like an open source project part of that is we are encouraging people to contribute to it so how do we keep the quality bar up and one of that one of the ways to do that is to to really treat this code like production quality code so one of the things we did to reduce the maintenance burden is integrating with our same front-end API clients that the front-end app uses they're in a separate NPM package a separate library that the each front-end application imports and uses whichever client they need to make the request so we installed the same library and we wrap the client with a cypress custom command so you can just use the same same code that the front-end is using to make you request to the backend to set up your your state so the benefit of this is it stays in sync with the front-end we listed as a peer dependency so it uses the same exact installed version that the front of application is using so there's no discrepancy between different versions or anything so what this looks like in practice is in your one respects you'll import the one of the API clients from Web API clients you'll call into our relevant custom commands so Psyduck copy which is customer API here you pass it in a function and the the parameters and then it returns you whatever that function would have returned you and then behind the scenes what we do is slightly manipulate the client to inject in some additional logging so we get some nice output in the cypress console so you can see what their request in the response each look like so for testing the test code we're treating this the same as a production library there's a unit test this way we feel more comfortable with it being open source and anybody contributing to it because if you've added a feature you should be adding unit tests that prove that your feature works and the other thing we've done is use ember semantic versioning to kind of indicate what kind of changes are in there so whether it's a minor a major or a patch version change you can kind of infer if you'll need to do any adjustments and this is nice because we have the ability to rollback if we do introduce a bug Brendan can ask you across 20 teams or 20 projects we know if everyone keeps up to date with latest libraries or if they're falling behind like what's the upgrade story so if there's no real incentive to upgrade until there's an incentive to upgrade so if people aren't changing or adding anything to their test is they generally aren't going to upgrade but if they want to write a new test and they've asked us hey I want to flip feature flags from your tests and we could say oh you need version 3.2 that's when feature flagging got introduced oh okay you know we'll upgrade or if we have to make some sort of change to support something else that needs to be dispersed across all the front-end repos we once in a while we have to go in and bump the package for everybody up to whatever version that is mm-hmm yeah that's a great like area of improvement for us I think is figuring out how to release more production like and how to communicate like when we want people to upgrade we have done better with like migration Docs when necessary and stuff so we do take ownership of upgrading occasionally but it's on terms of main we do want to get that off high responsibility for sure if we add some feature maybe like better Logging's or something to help debug tests we'll send out like a message kind of across the dev organization and say hey we really assists new version it's got whatever x y&z feature if you upgrade today you'll get it out of the box you don't have to do anything else a little bit of a marketing email similar to what Cypress itself sends out nice okay so try and move a long time wise here so now that everyone is kind of using it and the maintenance burden is is down we really want to finalize this by making sure everybody who's a consumer can can do what's necessary to debug monitor and eventually just own their tests some of the discussions that come up that we've really tried to figure out how we can avoid or just like prevent from happening in the first place is it's you know getting messages to our team like my Cypress s are failing like can you take a look at it with me and like yes we are happy to go help but that's not very scalable so I would like you to know how you can get the information that we're just gonna go get anyways that's both us learning how to teach and mentor the tool better and also gives us areas of improvement in the tool that we could work on and this is all just an attempt to shift the mindset which we had at the start which was Oh a test is failing it's probably a flake to a first response should be like what change has caused this test to fail and we hope that the majority of the time that is like a production problem with the feature code and not with the Cypress code ideally the next set of bugs we would have would be with a break like a breaking change that we made to our Cypress framework and then the the least common we would like to see would be because of a flaky are in stable test so we've done a few things to help with this so we implemented retries using sakura spluttered retries plugin so there are some issues to come with using retries so they can hide poorly written tests if you have a test that he fails in the third attempt if you have five retries it's probably going to pass but in that kind of scenario your test Suites end up taking longer so to avoid this one of the things we did was only allow retries in CI so if you're doing local development you'll get the the failure kind of in your face while you're developing and it'll make you want to fix it but when you you go into CI or if it's just a build in you know davor test a flake is not likely to to fail that which is the real frustration for developers and then the other thing we did is allowed the teams to specify the number of retries they want so if they don't believe in retries they can totally opt out if they want ten of them they can have ten of them nice and since they own their own build if they retry 10 times it slows the pipeline down that's one of the other things we did is integrate with data dog this is a kind of metrics service that our friended application uses to report errors and whatnot so we already had it at our disposal so we added some hooks from cypress to collect test data and push it up to service we use this to kind of supplement to cypress dashboard so in this instance this is probably the graph gives the most which should it's just a crude way to show testability so you can see if tests are further up on the chart they're run more often and if they're further right they fail more often so if sometimes at the top right of the chart it's run a lot it fails a lot it's probably a big problem and while I collected this chart it was interesting to see I hovered over all the dots and they showed which tests they are and each one of those tests that's further right was increasingly longer the big bar at 0% fail on the left was a little short test and then it just kept getting longer and longer duration tests as it turn right which kind of confirms that shorter tests are generally more stable and when you say shorter versus longest test like roughly is it five seconds versus five minutes it's not so much duration as lines of code I would say okay so what number of commands yeah the more work the test is doing especially if it's to do with something like uploading it it's generally less stable gotcha right think that that far-right yellow dot is a test that in and of itself takes about a minute and a half really versus the ones on the left side are probably closer to five to 15 it's definitely correlated with duration as well but I think the lines of code is more of exact correlation so one of the other things we did is opt into the Cyprus dashboard so one of the things we wanted to do with this since it is easy to set up is keep it easy to set up with our kind of under the hood somewhat complex additions to Cyprus so what we did you can see here we handle collecting all the information that it needs for commit info the Jenkins environment info and whatnot behind the hood and the consumers don't have to worry about that they just have to add the record key and their project ID commit that and it'll automatically start pushing results the data I just want to mention this is not really record key no look at it actually record key don't try to go and upload all sorts of crazy data so what this looks like in practice you can see it's got all the relevant data at the top in one place instead of having to comb through Jenkins and it's a lot easier than looking at data dog or Jenkins to view the test results these screenshots of videos and whatnot and getting people to opt into the dashboard is a nice step towards one of our next objectives which is parallelization one of the next things we did is it sure that we have great logging if you look at this kind of example screenshot here this is what one of our commands used to look like when it would log to the the command runner and at a glance you don't really have any idea what this is doing but it has all kinds of info that if something went wrong you might be able to figure it out so we revamped this using the Cypress logger to hide all those logs and just output one nice clean step which is telling you this is creating a user and it's logging in and take all that relevant information and put it into the console so if you do need more information because something went wrong you can click on either one of those steps and they'll give you the information that whatever other wise just been polluting the log nice so what we also tried to do is match the style and the verbage of the default Cyprus commands so the top example is a unmodified command to come straight from Cyprus its side up yet you can see what it was called on what command was called what it returned and similarly with our side copy you can see what command was called what arguments were provided to it we added the requests in the response for debugging purposes and then you can see what that command returned as well creating one of the other things we did behind the scenes we load a lot of environment variables from different sources so when you initialize Cyprus and it launches the runner or or the test GUI you can see the the environments get printed out we hid them there because it's a lot of secret information but it'll print it all out there so you know if you're running against the development environment and you see something that you know is specific to the test environment something went wrong you kind of have a first place to look one of the other logging features we added is a handy extension library called Cypress log to output so what this does is click the browser console errors and output them to the cypress node console so if you're running in locally it's not really an issue because you have the browser console there and you can see if something goes wrong in the front-end but when you're running in CI these errors actually get hidden which is kind of unfortunate and adding this will output it to the Jake is kind of console ok so those were kind of the major challenges that we had to overcome and some of the implementations that we chose to try and to try and solve that and what does that mean for where we are at now it's been I guess roughly 2 years since this project was started every new front-end project comes with both cypress and our shared library out of the box and fully integrated into CI so you can like time to first test is just the amount of time it takes you to write a test and if you could you know put together a Cypress test in 30 seconds then you can run a build and you'll get that immediately we have over 20 repositories using this library now which means a lot more people contributing to it it's also a lot more signal for things we could do even better at which is just only going to encourage further growth there's over 20 shared custom commands in this library that multiple teams are using so we think that that is a lot of value because that's more than 20 commands that might have been rewritten 20 different times and I'm not gonna run the math on that but so we're up to about 900 or more tests per day this is an average because obviously with tests running on on pr's it's a result of how many times people push in a day but we're averaging about 900 and at that 900 we're at over 96% pass rate I really wanted to dig into this metric further but I didn't have the time yet so what this means is the 4% that are failing could very well be like a failing test because a feature was changed and the test wasn't updated so it could be failures pre merge it could be failures that we find during smoke runs or it could be flake we don't believe our flake is at 4% but it's one of the variables in this number mm-hm and when you look at all the tests across all 20 repos they're averaging about a four minute runtime for the entire test run for that application so that could be any number of tests within that test run but it's about a four minute average somewhere a little bit more somewhere a little bit less but obviously no one's going to be too mad about having their pipeline wait four minutes for automated test results right for end-to-end tests yes yeah I next step we're working on implementing parallelization right now it's a bit more complex than the average Cypress set up just with how we have everything in the back end but what we're trying to do is keep it just that the same as reporting the dashboard as hands-off as we can make it so all you'll specify is the number of machines you want to run in parallel mm-hm and one of the other things that's a little more distal the horizon is building a mock API server a lot of the instability causes we found are actually from our infrastructure and not from the tests themselves so we have the idea to use swagger or an open API file to generate a mock API server that you can point to from the the front-end so you don't have to rely on a unstable back-end which should increase our stability and from speed we hope hopefully hopefully so I mean just to summarize the the content that's kind of a whirlwind of information but I think when we when we reflected on kind of the core tenets of what we're trying to achieve with all of this it always comes back to unscalable maintenance really being one of the biggest problems that teams have that prevents them from doing really innovative testing and so with everything we do whenever I'm grooming the backlog for the team and we're thinking about what our next kind of goals and priorities are it's always going to come back to how can we make this framework better for everybody while still not being a maintenance burden on us and you kind of see that reflected in all the features that we've added there's some there's some really cool and flashy things there's some not so flashy things but they provide a lot of functionality but the end of the day it's how can people self-serve and how can we get the testing out of their way to give them what they need and not hold them up in an inconvenient way so if our our test framework is holding anybody up it better be because there's a problem that we want them to know about and you can be on that is painful for us so no it's our goal to fix that's a very good philosophy well thank you Brandon and Graham it was like my most favorite webinar so far right but I probably said about the next one you set a very high bar I personally enjoyed it a lot so I will make this slides public right after a webinar so if you go to Cyprus that slides calm at Cyprus - I will find them in a couple of minutes but Brandon Graham do you have five minutes to answer questions because we have a lot of questions but people have been submitting a little bit longer but yeah we just have time let's do it let me switch the window right so we have 23 questions we're just gonna spend maybe five minutes is there any question that kind of catches you re if they're uploaded why don't we start from most popular so how do you plan test coverage without getting too complex am i running so I think because the tests live alongside the front-end code and we encourage developers to write tests developers have the context of the unit tests and the end-to-end tests and because of that they know kind of where the gaps would be in coverage from the unit tests or the intent test and vice versa to make them complement each other instead of kind of duplicate each other yeah I should add like we do have a functional QA team that does exploratory testing and they sit there embedded in the different teams that they work with so what we really want in this model and we and we are getting there is the test coverage and planning should be a conversation between the the featured developers and the testers and and even the product owner if that's necessary but just like as a team decide on what the coverage is and then determine where that fits so how much of that can we do at the unit level what do we have to do at the cypress like end-to-end level is there anything that's not worth automating that the tester can then do in their exploratory cycles in this Ward's because during pull requests review you Reviva code and you look for corresponding tests right yes yeah gotcha so for this one so this person commonly hears they want their tests to run in less than 60 seconds so we had a kind of the average metric of about four minutes that we showed so this is kind of uncommon Leeloo for n 10 tests and it's a bit of a benefit of our app shell micro service infrastructure because we have our tests dispersed amongst 20 different repositories and only the tests that matter for that repository run so I don't know if this person is in that kind of environment but if they are they can take advantage of that and in in that case having her code your test code live with your front-end code is how you would take advantage of that if not you can look at parallelization like if we ran four different notes for those four minutes of tests we would be down to 60 seconds write more notes to get faster any pointers for API and UI tests it's definitely something we've talked about we don't do this currently part of part of this is I mentioned our philosophy of our front-end test code being in JavaScript because our front-end code is in JavaScript our back-end currently at least when we wear plan grid is python-based so we actually have a python-based like coded API test framework but I absolutely see the benefit of doing this we just haven't we haven't done it yet but we've definitely looked at it and like it's super easy so if you have the option to do that means you get all the benefits of what we've talked about as you improve logging and you improve environment variable handling it'll all be part of that so what I could see is a win for that is if you are testing the API you're commonly going to use the same API to do test setup for your UI test if you extract that into a custom command for like create a project in our case you could refer to that same custom command from your API test and your UI test to save some duplication I'll have to say but I've written a plugin called side - API but abstracts Sai requests we can do API testing and it outputs request response into Cypress iframe but it's empty so you can see we resolve but Graham and Brandon you cannot use it because it will just get so confusing you have side bi side - API perfectly rounded the problems loading data for smoke testing I think we we touched on that but just to reiterate we use like every test creates its own user and project and then any other additional data can be loaded in through api's or if it's going to be a time intensive data setup then we use our our projects big boy which is constantly creating projects for us so if you wanted to write a test kind of a s a front-end I don't want to use the term performance test but if you really wanted to run your test against a project of ours that had thousands of sheets that would totally be doable and and you don't run the risk of having to wait for that set up and we we use the cons if you modify a project we throw it away because it's an unknown State and if you haven't modified a project we actually allow it to be put back into the pool for another person to use as well excellent so let's take two more questions before we wrap up because we spend an hour is there anything that you think is important in this lists I can scroll slowly yeah I mean let's scrub I see handling resistance from developers about writing and owning tests I think we can quickly address that one in that I mean mostly we talk about our goal being like to make it easy and delightful so it'll be dependent on the resistance but I think we look at any resistance that we're met with as a useful piece of data and input into what we can do better as far as like what are your resistance what are your reasons of resistance to to using this tool and we would be open to hearing it because if there's something that's meeting your needs better perhaps we can take similar concepts mm-hmm and integrate that so that's kind of our approach to any resistance and we try to be really responsive to future requests so people are like happy that we don't say like we'll get to that in a year we're more likely to get to that in the next like couple of sprints thank you so much I have one more but I found that I think is interesting one oh and it's gone basically yeah it's updateui and me the question was how do you run just a subset of tests against production the smoke test how do you pick the smoke test how do you ensure they don't destroy production so it's it is an interesting challenge when it's across several repos so we have a dispatcher job and Jenkins that'll trigger a whole bunch of child jobs one for each of these front end repos and that it's up to the each team to decide what they did they consider a smoke test for their feature and then they just put smoke in the name of the spec okay the only rule we have for smoke testing is it absolutely cannot rely on static data it has to generate its own data because we want to be able to deploy a completely new environment with no data in it and run a set of smoke tests against it excellent it's a good strategy and I want to thank you folks so much you've done absolutely marvelous job for us coming up with content for this webinar and also telling us you know your story and language story and I really appreciate a thing this content is going to be super useful to a lot of teams a lot and again like I mean Brenda and I are very big proponents of using Cypress we really enjoy it so you know we're always active in the community and looking to help people who might be trying to get started or look for like creative ways to solve problems we're totally available to help it done absolutely everyone go go follow our presenters online on Twitter asking questions there if you want I will try to go through all the questions but people submit it on slider and answer them to the best of my knowledge please I'll make the slides public as soon as we're done with webinar and we'll send the webinar recording when it becomes available to everyone who register for this webinar again thank you so much I really appreciate that time and thank you from the bottom of my heart take care everyone bye

Channel: Cypress.io

Views: 2,392

Keywords: Cypressio, Testing, PlanGrid, Automation, Test Automation

Id: aaU41lsdX24

Length: 63min 22sec (3802 seconds)

Published: Wed Feb 12 2020