Continuous Integration with Cypress

hello everyone and welcome to another webcast by Cyprus data yo and today we're gonna cover running Cyprus on continuous integration server and it's not just me today we have an outside panelists justin james justin thank you so much for agreeing to this thank you it's pleasure to be here absolutely it's a pleasure to have you can you tell us a little bit about yourself absolutely I'm the founder of company called let your nerd be heard we're I'm hired by other nerds to transform them in the confident public speakers that are able to be seen as the go-to expert and make a massive difference in other people's lives excellent you can always find Justin he's active on Twitter at digital drummer Jay and I'm your host Gladbach moto VP of engineering at Cypress you can find me online as well we do have a lot of material today so I hope we can cover everything if we skip any slides due to time the slide deck is public already you can always consult it but in in overview we're gonna cover the difference between Cypress open and Cypress run commands we're gonna talk about how to actually correctly install and run Cypress on see I will have practical examples so justin will show how to have Hurons Cypress on teamcity and Travis and I'll talk about circle and at the end I'll describe common gouges and common problems that people might have running Cypress on CI and a few solutions during the event you might ask questions and we use slider slide that do to ask questions so all you have to do go to slider online and enter this code Sai CI and then you can ask questions and we can upvote or you can avoid the interesting question and we're gonna answer pretty much all of them if we cannot answer the questions live we will answer by email so few other questions will answer it sounds good Justin sounds fantastic excellent so let's jump right into it as you know very cypress open command and this is interactive mode that's how we call it interactive mode where when you run tests it shows the GUI and you have choice you can run a single speck or you can run all specs at once during interactive mode Cypress actually watching your test file so you know if you change something if you change your spec file and you click Save Cypress automatically reruns but that's but dumb snapshots are taking during each command so you can have a time travel in debugger where you can like like it shown here hover over each command and when Cypress shows the Dom how it looked at that particular or second but Cypress run is different if you do Cypress run it's not interactive mode we usually call the headless mode even if Cypress actually right launches electron but the main thing here is that instead of running all specs together Cypress opens and closes browser for each spec it tries to isolate each spec which might actually lead to some interesting behavior but you might not expect obviously there is no file watching because Cypress just executable files there is no way you can interactively change your files and rerun the test when you run tests on CI Burano don't snapshot because you cannot hover and go back but so we've trying to save some memory if a test fails in the headless mode Cypress will take a screenshot of a failure and usually vary the video but Cypress captures by default which it doesn't do in Cypress open I think in longest storms that see I should run tests on each commit so when you work locally you do changes you run cypress interactive mode but when you push code to CI your continuous integration server has to run all the tests in headless mode and justin has a very interesting quote Chaston do you want to take this one yeah so I always tell teams that I work with that running CI without tests it's like never changing the oil in your car it's just a matter of time before it blows up well Justin I think I love this quote does it apply to electric cars absolutely if you never charge it it's not gonna go anywhere that's a test kind of gonna do four so I see a lot of teams that and my team did it initially before we found Cypress - we had all of our continuous integration running for our UI but we didn't have any tests we'd actually run in there so ultimately we'd go to production and we'd find 1520 bugs that somehow we missed in testing and we have manual testers that are dedicated we've got Maule testing and user acceptance testing so it wasn't like we just went development right to prod but we still were having major issues once we got Cypress up and running we have now gone from 15 to 20 issues that we just magically find in production to having very few we've had one or two that have managed to escape and instantly write a test for those so they never escaped again and so and most of the time it's really kind of out there scenarios that somebody just happened in our QA team to go I wonder if I run it in this funky thing and paste all these funky characters and now what happens it's like okay that's not our no more users so we've reduced down all of those escapes in the production that are impacting our users our users are way hacker they were kind of annoyed at some of our releases they were a little bit buggy yeah I completely agree you know having actual human users find the bugs it's the most expensive thing you can do absolutely Justin the one that walk us for the next slide like how does it actually work in Prague yeah listen take a look at that but so from a team standpoint there's some responsibilities that we as developers need to follow the first one is to merge frequently so we don't want to have these long periods of time where we're working on something locally that never makes it back in to our source control repositories there's especially in a team environment you have multiple people working on things the longer you wait the harder it is to get everything merged together everything working all of your tests working so merge frequently every time you have something that works go ahead and commit that that and so it's at least up there and a lot of times I will actually merge functionality back even though my features not fully done if the functionality works and doesn't impact anybody else I'll go ahead and merge that in so it's there and the rest of the team can now get their code integrated in works really smoothly for that piece and one thing I'm always saying what if the master branch is always too playable at any point in time I can click the deploy button in our CI environment and we deploy straight from CI it's been three four years at least now since I've manually actually done a deploy to production all our deployments are done right in the CI I it starts with the bills that we're gonna talk about today and once we get a successful build and we're ready to go to production can we just click the Run button and off it goes and sends it out to wherever it needs to go whether it's as your AWS and virtual server wherever it's going but it's always deployable and this has saved us a couple times in my world we have some we call them faceless accounts or system-level accounts and that the password expires every 90 days and we had one system we hadn't done any development work on in about five months well for normally our process to redeploy is we have a CI build that updates all the passwords and then just restarts but normally needs all of our dependencies there all the artifacts in the build it had been so long our build server removed the artifact so we had to actually rebuild our master branch they it wasn't a very deployable state so we always keep them that way then from a committing standpoint only commit working and tested code it's really tempting sometimes you go on leaving for the day with my laptop let me just commit this in so that I have it here I don't lose it inevitably you forget sometimes that you hadn't finished a piece and you push that out then all sudden everything's broken and the minute it's broken it's really hard to then try to backtrack that and figure out what commit was at what feature was it so much easier to just make sure you went through all of the test Suites did all of the testing that you needed to do and then commit it that's why we do merges frequently so we have small blocks of code and then from a merging stand what only merge from the build is passing it's really tempting sometimes when you have longer builds one of our builds takes about 30 minutes to run because doing a ton of integration tests with api's and third-party systems well sometimes we make really small changes it's annoying waiting 30 minutes and it's really tempting to go in as an admin and turn back on the flag that says I can merge before the build finishes in the minute we do that we now put ourselves at risk of what happened if that really small change did actually have an impact and we've seen that happen a couple times and then you're scrambling because now you've merged code and in my road when you merge into master it automatically deploys it out to all of our non production environments so now if we've broken it QA can't do any testing our customers can't do any testing and we have to stop everything we're doing and go fix that yeah much easier just wait for the build then the last pieces don't leave for the day or for lunch right after you actually merge until the build passes it's really tempting to to go yeah I trust that this change is small well it shouldn't impact it or what happens when the build breaks and the rest of your team still working now they can't merge anything in and they're trying to figure out does anybody have that person's cell phone number can we get a hold of them can we get them back on line and if the answer's no your team's kind of stuck until the point of I can't merge anything until this bill passes I need that feature to get my stuff so just wait for the build the Paso plan that into your day that if you know it's gonna take you ten minutes give yourself like 12 to 15 minutes in case the machine slow and then once it passes go ahead and leave for the day definitely don't go you know you know camping until everything is good and ready exactly so that was excellent excellent philosophy now I will go to the next slide and I would like to talk about practical things like how do you set up Cypress to run you know how do you catch NPM how do you cache dependencies if you plan to run your tests in parallel and then we'll talk about actual examples so what do we need to actually run Cypress it's not much so if you're on Windows or Mac CI machine when Cypress should just walk out of a box if you run your tests on Linux on the other hand Cypress requires a couple of libraries that should be pre-installed you know like XP FB JDK and so on so we give this list in our documentation on continuous integration and the simplest thing to actually get your OS set up so you can run Cypress is to use one of our darker images so we have repository in darker hub account where we just push images we have Cypress based docker images that just include OS dependencies and a version of node you know 6a 12 whatever you want when you have Cypress browsers images that include Cypress base and a pre-installed chrome instance so you can run usgi test in Chrome browser finally we start building Cypress included images that are built on top of Cypress browsers but also include globally installed Cypress for example if you work not in NPM world but in a Python world and you don't want to mess with NPM you can just use Cypress included image the interesting thing with there is a command called Cypress verify and it can show you if everything is good for Cyprus to run if OS is ready or if there is an error it will show you an error and when you run Cyprus run through a very first time on your CI machine it will actually run Cyprus verify first and then save result if you still have problems running Cyprus binary no Cyprus - as a tool on Linux you can run Cyprus binary on its own Cyprus downloads itself and stores itself in a cache path so you can run Cyprus cache path to find where Cyprus gets installed and then you can run the binary by itself without NPM wrapper and that will tell you if any of oh s dependencies are still missing and that's what Cyprus verify actually does under the hood it runs Cyprus binary by itself with just a ping you know trying to see if it gets back the ping so if you run into problems you better ways to debug this so let's talk about how to actually install Cyprus on CI every CI server only really does three steps it has to check out the code using it check out it has to install NPM dependencies using NPM install or yarn Oh NPM CI and then it has to run Cypress run to actually run the test now the problem here is that during npm install NPM downloads a lot of dependencies to install them right it downloads install Cypress and Cypress then downloads a binary which is a large zip file and when and zips it and stores it in a cache path so that takes a while so what you really want is to cache and BM cache and also Cypress cache where the binary is installed usually on Linux there are in your user profile data and p.m. and that cache folders now if you want to save it cache after you do an NPM install you can actually find out where exact path but if you care something you have to restore cache after you get checkout but before you do in p.m. install so what you don't install and redownload and p.m. dependencies and cypress every time you run CI because it takes a long time so here's a typical CI configuration file first check out restoring cache a good practice is to store your cache with a key right and that key for example could be checksum of a package.json if you change package JSON you reinstall everything so you're not using out-of-date dependencies and you know Ranjan to where we're problems saving the cache after you run and PM CI this is very typical setup you running in PM install and the cypress run on a single machine on CI is great but once you start running many tests in parallel right for example if you plan to use parallel runs if you run in PM install and Cypress on on multiple CI machines at the same time it's unnecessary work because they're all we're gonna download the same files run the same in payment style they actually might run into problems trying to write in the same shared folder so what we suggest is that you do npm install in a single job which installs in a shared folder maybe there is a concept of workspaces on UCI and then that workspace will be passed to multiple run jobs but don't install anything but instead just run the test and so if you use Cypress ran record and you run in parallel mode when all these machines will load bounce-back files but will never install anything because on a single npm install' job has already installed the dependencies install cypress and prepared everything for run now Justin can you actually walk us through like some practical tips that you doing absolutely so from a CI process the first thing we need to do is build our code in a production mode but this means you're doing all your minification your concatenation of files any cache busting it's going to be the exact build that you would deploy out to production if your bill passes so this way you're testing the actual production code in my word I'm in the angular space and I know if I produce stuff in the development and mode versus production they have some stuff that they only check when you do a production build and we've seen times where we haven't done a prog build but first all of our tests passed then fell when we try to go to production because maybe we've had a variable in our HTML that we didn't have declaring the typescript some of that stuff you can get by in the dev environment you can't get by in the production mode so after we get our code built we need to start our web server typically in a CI environment you don't have any processes outside of the actual CI server running its a headless environment no one's logged in and so nothing's already running for us so we have to start up a web server then we need to wait for that web server to respond and that's typically doing some kind of a ping or something to the web server because it's got its own startup process we don't want to just start it immediately start running tests because then our tests will fail because the server is not up yet once the server's up then we run our Cypress test using the Cypress run and that we covered earlier and once our test is done we need to stop the web server because we're in that headless environment we don't want to leave anything running on the server whether our test passes or fails we always stop the web server afterwards and leave nothing behind that's actually running let's run through a couple things to actually install to make this work so first thing I like to use is an NPM package called NPM run all this allows me to run tasks in serial or in parallel and it allows me to do it in a cross-platform environment so in my space I'm on Windows and most of my team's on Windows but we have one developer on Mac and then our build server is also on Windows so if we try to do something like run command one then do the two and signs and run command two works amazing on Windows it doesn't work on Mac doesn't work on Linux MPM run all works on both of those for us I'll show you the command here so simple as doing MPX run - s so we say run in serial we're gonna run two commands lint : CI and build : prod even though I could run these in parallel the reason I run them in serial is for the lock file when you run them in parallel in inner mixes everything both commands are doing and if it happens to fail and I need to do troubleshooting there's no way to tell which log line went to which one of the commands and because they run quickly linting runs in 30 seconds building runs in a minute or so it's not like I'm adding a ton of extra time so by doing it in serial I get the log file so I can easily read it when something fails I need to do troubleshooting yeah then after we build mission we need to actually start up our server so Gleb actually wrote a great MPM package called start server and test with start server and test it goes ahead and starts up a server with the files that you tell it to use on the port you tell it and waits for a response it's a simple ascent MPX start server and test give it the command to run this case CI : start - server launch it on localhost 4200 and then once that response go ahead and run the command cycle and run and then once that's done whether this I run passes or succeeds it goes ahead and stops the web server automatically for us so I don't have to worry about any of that starting and stopping it doesn't all for me then in the angular space to start up our actual web server River though I don't actually want to use I use angular - HTTP server the primary thing that this server adds in over the HTTP server pack is that many people are familiar with is the deep linking so that I can't actually go to Brittany and not have to go to the homepage and then click through to get there and it comes like that out of a box start up the actual angular server say angular HTTP server give it what path so this is where our actual production build is located until it what port to actually start it on and then once this is done I actually have the web server up and running and I can actually run my test now on that that's a lot of groups but this is a lot of a lot of scripts this is what actually all those commands that that were in there like the cycle and run the size start server this is what my package JSON actually looks like and I have in the package JSON so any of the developers on the team can actually run these on their machine make sure everything passes and I can also move from CI server to CI server so if today I'm in Travis and tomorrow I'm in team city I don't have to redo all of my CI hey I just tell it npm run CI alphago's it knows all the commands to run so i have the run s command there run my linting my building and then the Sai run and the lintons built into angular here it's I know react has one built in almost every framework these days has a linting tool linting is great if you're not not doing yet today it stops you from having those developer discussions around stylistic preferences do you want four spaces two spaces how long do you want your lines to be all of those things that we love to argue over we can just set up rules and agree as a team and then now we can actually look at real code putting code reviews and stop worrying about all the stylistic stuff so I always run lint Inc in the angular space we have our build command we do ng build - - prod does all of our minification are cast busting are concatenating other files together there removes all of the development and debugging stuff out that you don't want in production and then we run the server command that we saw you tell it kick off our size server and then we do our seat why run to run our Cypress run in this case I'm just running it with no parameters you could run it with the - - record and - to us parallel well you can run it when we do the teamcity you'll see how to do it a different reporter in there so it reports out to the team cities that are in the format it and needs you can even turn off videos lose it and we'll talk about that a bit when we cover the Travis I example coming up so now we've got three examples we're gonna walk through of how do you get them set up and go through Travis CI team City and then circle CI for Travis CI it's a hosted continuous integration service for open source projects and private projects that are hosted on github that is the one caveat with Travis as its github only unlike many CI providers and services that can go against any source control repository Travis is strictly for github and they have two models for public repositories it's Travis CI org and it's free to run your builds so this is a great place to run all of your open source all of your demo builds run there for private repositories that one's a paid product and that's at Travis CI comm and there is one caveat to their paid side for open source projects they do not charge for those you just have to contact them get all of that that stuff set up but at least you get that for open source projects we're gonna walk through how do you get all of this actually set up so first thing when you go to Travis you're gonna log in with your github account and it's gonna set up the service hook and we do that by once we're logged in we see our list of repositories and we just have to click on the little toggle there to say which one we won't have to sir thankfully it doesn't look at every one of your repositories I know myself I have hundreds of repositories and I've got two builds actually running because most of them are documentation or samples or someone's repository I fork I don't want to run builds for those but I do in this case want to run builds for the second one that I have here the the ionic a zine website it so I just click that on and the minute I click that on it's gonna look yellow for file and we'll look at the travesty and we'll find on a little bit of how you configure that we a lot of settings we can set up for our actual build by default it's gonna build on every time you push to a branch and it's any branch master or any of your feature branches it's gonna build any time you've create a pull request just and you can turn off either one of these in most of my environments once I leave both of them on because you want to make sure you do it for poor request you want to make sure when you get it merged a master you rerun it just to make sure something that didn't happen in there so I never changed this setting it's the default one mm-hmm thing for this Auto cancellation so by default when a builds running and then you do another commit it queues up to commit next commit to run the bill well before that build kicks off if you queue up another commit you don't want it to run the out-of-date build and have to wait for that so Travis will Auto cancel that build for you so you only bills the latest ones in the queue nice little performance boost there for us then secrets you don't want to put secrets into your source code repository you know don't put passwords checked into some to a file that everyone can read in a public repository you're just asking for someone to log into your system and do things they shouldn't be doing but that's where environment variables come in so we can set up all of these environment variables and then tell our Travis script to use those environment variables and we can also tell it don't display those values in the build log by default it's gonna display the value in the build log but if this was a password I was setting up I just turn that off and all I know is that I used this environment config but I don't get to see the value in the build log a great way to keep all of your secrets out of your source code repository and we all think no one's watching my repository I've actually had friends verse one of the devs actually checked out in in their Amazon key and within five minutes they had thirty thousand dollars charged to their Amazon account some bots managed to grab that key that quickly yes there's not humans that are looking at its posture in essence don't check in that stuff use environment variables the other option that you can do that they don't turn out my default is scheduling a build so say that you want to run maybe once a month once a week automatically regardless if you have commits they have that ability to schedule those jobs when I'm in active development on a project I don't need to have the scheduled build there because it's we're doing commits we're doing merges builds are running all the time but once you're done with your development cycle if you're not going to touch her for a bit scheduled builds are fantastic to make sure is everything's still working thankfully I have those turned on for a couple of my demo projects my demo projects are for workshops that maybe I run once or twice a year I had this turned on and one of the cypress upgrades actually broke my build I didn't have a Linux package that I needed that it now needed if I had waited until I needed that at the very last minute I'd have been hunting that down now I had time to go actually fix that I knew once I got that working my build is ready to actually still do what it needs to do so great to schedule once you're not in an active development cycle so now that we've got all those settings in all the settings I showed were the default settings that Travis has for you so they had really intelligent settings out-of-the-box the next piece is how do I set up my actual build so I got the settings there but I haven't told it how do I build my project to do that I start create a dot Travis damo file and it's truly a llaman fire with all kinds of we start off here this case it's a node project so I want to tell it my language is no js' what this does for us is Travis is going to download a pre-configured container on their side that Ernie has node installed and set up and I tell in this case I want the LTS version of node I like to stay on the long-term supported versions of node but they're the most stable Verte versions and it's what I'm going to use in any of my production environments so by setting these up it'll download that container everything's already configured installed so I don't have to worry about trying to figure out how do I actually get node installed on a Linux machine because Travis run this case Travis is running on a Linux container for this build and I am definitely not a linux expert whatsoever I'm on a Windows machine so having all of this is fantastic for me and then I can also do a bunch of add-ons on top of that that Travis will install for me so Cypress doesn't need Chrome to be installed but typically a lot of my builds I'll have unit tests there's like karma and karma needs chrome to be installed and so I can run it in the headless browser so instead of having to figure that out I just tell it chrome stable and Travis is gonna automatically install and configure that for me and then we talked earlier about the ciphers packages in this case of this script I really need to actually install this one package in order for Cypress to actually work so Travis must have had all the other packages that gleb mentioned earlier installed for us if it didn't I would have just listed all of those packages here one on each line and then we set up our caching so the same caching that glove basically talked about earlier we want to cache NPM we want to cache r dot NPM r dot node underscore modules or - cache directory so this way we don't have to download all of our dependencies each time we don't have to download Cypress because it is a pretty lengthy download the very first time that it caches it it takes five to ten minutes sometimes depending on your internet connection and the speed of the server to download and unzip that you don't want to do that every time if your build takes 10-15 minutes to run they're gonna be really hesitant to run that thing lots of times especially a suggestion early was to run it on your local machine imagine that you had to download everything and didn't have that cache every time like you'd never run the build locally so caching on the server as well and then the next piece is what is our install commands how do we get our dependencies so instead of NPM install I use MP mci MP MCI strips out a lot of the things that you don't need for production kind of releases so it's just a performance improvement it'll still install your dev dependencies that it needs but it'll skip to all of the stuff that it doesn't feel like it needs for the builds that's way faster skips a lot of checks because one of the requirements for NPM CI is you have to check in your package lock file if you don't if your package lock doesn't match all of the ones that you've totaled to install on the package.json this command will actually fail and so it skips through a lot of figuring out what versions do you need to run it just uses the package lock for you and checks all of the shah's that come down to make sure everything looks good and then I still like to run the these ciphers verify hey I put cycle and verify is literally just running cypress verify the reason I have I put that in my package JSON and so that my team members don't have to remember it's Cypress verify myself I'm the only one on my team that really is a true Cypress expert everyone else knows how to create tests knows how to use it don't remember all the commands I'm not gonna dig into all of the parameters and things you can do they're looking for me as the architect to set that up so I try to give them all of those scripts in the package.json they don't have to worry about trying to remember it because everly you'll forget something in fact I actually had a funny incident recently were we didn't have that setup for some database stuff and I forgot a - I in our database script that said make this item potent so there's a bunch of if statements say if you've already run this don't do it again and thankfully I did not run it in production accidentally deleted like three tables out of the database because at one point we deleted him andrey added them and was like whoops i took down a whole environment because we didn't have this thing scripted out and i forgot one simple parameter so scripting them out is fantastic avoids that so we've got our install that talked about in the last piece is what script to run what is the commands to run because it's in our package.json I just say npm run CI it runs through that whole script so there's nothing really in here besides the setup that is Travis specific for those individual commands which makes it nice cuz I use the same scripts on my work environment where we're in the trap or in the team city space and you'll see that coming up shortly just look at the process that Travis actually goes through to run a build first thing we're gonna do is we're gonna add our Travis file and to get we're going to commit it and that's gonna trigger our service hook so no Travis is gonna is monitoring this whole time our git repository and waiting for those pushes once it sees it it goes ahead and it kicks off our test and all of our builds and it creates a fresh environment each time so we don't have anything left over it pulls in all of the casts that we told it to thing and the script doesn't have it in here that I should but one thing I do at the end of Travis is I always have it deploy someplace but because that's such a unique thing to to each person's environment I left that out of the generic script but for that example I would have actually have it push push out to github pages for me there's nice built-in commands for for that stuff so let's look at what does a build log actually look like here so we've got a few screenshots of an actual build running the first thing it does is it clones our github repo so it always gets a fresh version of that does our npm install well here sets up all of our cash lets us know what version of node did it an NPM did it actually pick and then it runs our NPM CIT to our installs and we start seeing it or NPM CI to run all of our commands we see it run the link command and here once that passes it's gonna go off and then start running our actual build prod and so this will take however long it happens to take and we can see all the cache busting in here with these crazy file names that spitting out and that is essentially what the bill blog actually looks like once and actually the somewhere our slide disappeared we would have seen a super that actually would have kicked off in here run through all of the tests and reported back did our test pass or did our tests fail for us and if it fails it fails the whole build well so that we can see it and it sends out notification so I get an email on this case every time I build actually fells but you could set it up to go to if you have a slack channel or teams they have lots of integrations where you could send the success and failure notifications you definitely want to make sure it's best practice that someone on your team if not the whole team has notifications set up for failed builds and then when it passes again I'm the only one on my team at work that actually gets the ferry notifications and I can't count the number of times that the developer had no clue that their build actually failed they were off working on the next thing I'm like like hey are you aware that the build or the deployment fail they're like no let me stop what I'm doing and go look at that yeah pretty soon they're like let's all turn on these notifications because what happens when I'm on vacation or doing a webinar right now if our build failed it's gonna sit there till we're done with this webinar it better not fail better not fail exactly so that was the Travis piece when we cover team city here and circle CI the process is fairly similar for all of these it's the set up that's a little bit different to how you actually tell it I want to run a build for this repository so in the case of team city it's a continuous integration environment out of the box that it has tons and tons of things we can do out of the box but it is not a hosted environment unlike Travis where it is hosted this one you actually have to install a team city server that does all of your orchestration and then you set up machines that have a team city agent or a service basically running on it that then monitor the configurations so this is great for someone that's like house worried you can't put stuff out in the cloud but Travis 100% it's in the cloud but if you need stuff to be on Prem team city's fantastic for on Prem I know from a cost perspective you can actually get three agents for free and then they start charging and there's some limitations like a hundred built build configurations takes a while to get to 100 build configurations even if you're doing development staging QA environment production like it's only four builds if I deployed each of those environments we're like eight builds so I get about ten full projects before bison does and I have to start looking at how much does it truly cost so it's a cost effective option for on-prem in order to successfully get your Builder report correctly actually need to install a reporter there so this will be a different way to actually report out the status of the Cypress test because Cyprus is sits on top of mocha they have a mocha team city reporter that will put out the exact log files that team city is going to look for in order to report back the status and when you actually use a reporter you need to also add mocha to your dependencies and these both our dev dependencies even though it doesn't list the dev dependency down there you don't need them in production they're strictly a dev dependency into run Cyprus with the reporter we say Cypress run - - reporter mocha - team City - reporter and it will do do its job for us I don't have to worry about anything else now how do we set up our configurations unlike Travis write a single file and it just monitored that team city has a you I've got to go through so once I've set up a project then I have to add a build configuration in and I have lots of options I can go from github it cloud visual studio team services from a repository URL or I can set it up all so it still starts with where am I going to monitor code changes and in this case I picked up on github I just put in any URL so I pick thee from URL option once I pick that one click the proceed button it does its thing cuz it's a public repository I don't have to do user name and password at all if it's a private repository you have to put in a username and password for that and once it gets my they call them VC so of version control control root once it has that setup I name my configuration it always tries to auto-detect steps and I have zero clue if it's ever successful because I've never seen it actually auto-detect like thankfully there's normally a cancel button so you don't have to wait for it I just click the configure build stops manually we're out in a few steps so the npm install command we saw earlier npm CI it's a command line script to run then we're gonna run in a second command and for cyprus verify npm run sy call and verify and then our last configuration step npm run this case CI : team city I chained I had to add the : team city because this project happens to have both the Travis configs and the team city ones and in there since it's a demo application she put in whatever command that you want in there and then you have to actually tell it where do I want to store the output of the build unlike Travis where Travis you don't get to store artifacts you have to actually deploy your artifacts or create an Amazon bucket or some online place to actually store your artifacts team city has that built in I just have to tell it what I want you to store so in this case I want the output of the distribution directory I want my both unit test coverage and if I've hooked up the Cypress code coverage I want to hook up both of those I won't need to store all the Cypress videos and the screenshots so if they do fail I can actually get to the videos I'll put the videos and screenshots as artifacts then you actually have to log in to the team city server and go find which directory I built it in and their director names are good names such you literally search in the logs words just stick them in the artifact path and it keeps it for a good amount of time and for that in once you have artifacts here as well when you create those deployment scripts that we're not talking about today you can tell it take that artifact in the out of the distribution directory and just deploy that forward or when you're ready so we store all of those as artifacts under the general settings then the process looks really similar to Travis we check stuff it and to github or in to get that since it can be any repository not just github the service hook kicks off with team city and it kicks off my test in my builds because I'm on Windows I'm not doing anything with docker it doesn't create a fresh environment to each time it'll pull down a fresh set of cut me it'll do my npm install is pulling the any cash that it needs but it's not like it's a brand new built it's a Windows virtual machine that's got whatever else it needs installed on there to run builds but at least it pulls everything fresh from like source code and everything else then it kicks off our test and from in output logging perspective because I'm using that Cypress reporter for that team city mocha reporter it actually lets team City know I have tests to run so just on the overview page it'll tell me how many tests do I actually have how many of them passed how many failed or didn't you so I can get a quick overview of that I could also click on that test tab that it was in the previous view and it will show me each individual test I can see how long it took to run I can add filters for anything that got skipped anything that passed anything that was successful I can download CSV if I want to do analysis of that and so all of that because I use that reporter without the reporter it doesn't report any of the test status at all you'd have to look through the log file which is horrible to try to look through the log file and pick off and then I can also click on build walked out and I can get the full bill log like I could in Travis and if I expand it each of these steps it would fully give me all of the details that it it went through the other piece that's in here that I didn't cover it all and Travis had it hidden a little bit in their logs is using nvm for my note installs so node version manager basically so that you can install multiple versions of node on a single machine which doesn't sound horrible important until you have multiple projects on your team that are on different node versions which is what my team has we got one project that's still on 8th and we've got one project on 10 well we get to use the same set of build servers in here just by installing NPM for Windows they have one for Mac as well I assume there's a potential for that in Linux mm-hmm fix it so that allows you to easily switch between the two and that's my first build step and almost all of my projects well no one's gonna walk us through Circle C I in what Circle CI does for us so thank you Justin and I have to save a circle sad does things a little bit differently yes you connect to your source of code like github and you set up a projects you want but then you don't actually write the configuration file oh you write it differently rather than like copy paste you know little commands like check out cache you in circle now can use a reusable pieces of configuration called orgs think of orbs as NPM modules but for circle say a configuration so for example we have written an oracle cypress - Oh / Cypress which you can import by name and version in your circle CI configuration file so in this case I'm importing it under name Cypress and then I can just say Cypress run and run is the name of a job was defined in Cypress by me offer her orb and that's it and that single command behind-the-scenes expands into git checkout and PM cash and PM install and Cypress run so a single two line file does all the things you have to do and this is the simplest run but even if you set up something more complicated like install on a single machine then build your sir your app or server and then run several machines and load balance and record on cypress dashboard they it looks like this because the cypress run is a job that has all those parameters and behind the scenes it expands into the full thing so really I love the way circle does configuration because the offer of a tool actually Rises the configuration pieces and I'm not going to go into any more details but we have done a full webinar we will show how you can use circle CI Cypress orb to actually run Cypress test jobs on circle that's pretty cool that you can do those reusable configurations it's pretty revolutionary yeah I know in team city I didn't didn't cover because it's a little bit of an advanced topic but you can take that built configuration and make it a template and then when you do build you just base it off of that and every time you update the template it automatically updates every build that's based off of that so that's how we truly run that's a long drawn-out process for setting those up that didn't really really have time to cover here but we we have that option but we don't have the versioning option right exactly you had the versioning option ours is if I update the template it automatically updates everything I can't just say at version 1 and I always leave it I have to say but I'm using latest at version 1 here but a circle style orbs use semantic versioning so you can specify exact version so there could be a lot more precise and immutable but again just look at our slides and webinar so we covered like basic workflows but now there's a question that everyone asks like how do I test across different environments imagine you building a local server and you run tests against local server right and then you may be deploying and you want to say my CI wants to run tests against local environments but also against a staging server why not run tests against local environment when staging and then once you deploy to master run things against production so how do you do it so one thing that comes up right away is how do you vary your configuration settings so think about Cypress Jason you're running tests again some kind of base URL on local it could be localhost 8080 but in staging environment you want to run your test against aging that your server that can grab a path you might not want a record video when you run things locally but you might want to record the video when you run tests against aging so the one thing you have to understand is that Cypress has two things that you can vary you can vary cypresses internal settings like configuration values like base URL video and you can also vary your own environment variables which you use inside the tests like username test accounts when anything but you need inside the test to enter or clearance so you can vary both of these things so first let's look how you very SAP with settings let's say you want to run local tests against localhost 88000 or 88 in this case you just put your localhost 8080 inside your site with Jason now when you run your test again staging environment you can pass - - config and then the name of a setting and a new value in your CLI command so in UCI file for example in this case you're gonna run the same tests using Cypress Jason plus overridden base URL you can also set that variable not through CLI but as environment variable it's really convenient to change environment rebel through your CI web interface for example Cypress and Rosco base URL will change the base URL over it overriding localhost well where usual settings that I vary is maybe base URL may be default command timeout maybe I want a shorter come out timeout locally but longer on CI server and I can pass both of them again fruit see a light or through environment variables common settings you probably want to change base URL default come on timeout maybe you want to change how long you wait for a page to load or ha know which house you blacklists and so on you can even change all the settings programmatically by changing the config and returning a new config in plugins file that gives you ultimate power and if you look at Cypress JSON environment variables CLI parameters and plug-in which one wins if you have multiple values it's according to this list what's coming soon but where work is done and where CPR is that you'll be able to pass a path to a different config so not just change individual variables but have a separate Cyprus Jason four different environments and just pass the whole config as a CLI argument that was config values Vettel Cyprus uses but you also use different environment variables for example you want to pass username and password but you will enter to login in your tests so you use Cypress that ends and then the name of environment variable to grab it from environment so how do you store it and how do you store for environment so here's my practice inside sighs Jason R is an object and that's why I will say user name is Joe tester and password and I will actually use a blank stream so what I know that mice test expects a password inside my test I can grab the name and I would say expect name to be a string and not be empty so if accidentally I delete user name I'll get a nice meaningful error and then I'll grab a password I will not use expect in this case because expect will put the value of an object in a test run a GUI so my password instead of being in a log file will be actually in a video and screenshot so instead I'll use a sort which is again comes with Cypress assert password this will just keep fro an error but will not in a showed in a screenshot and when I can use those values now the important thing here is that passwords are sensitive and username is not sensitive that's why I can check in username inside Cypress Jason I can pass and I mean variables using CLI argument when I use Cypress run it's - - and now I can do this - - and password equals but that's again a bad practice because as Justin said this will be reflected in my build logs so instead I prefer to pass environment variables through actual environment variables define the own CI GUI in that case they'll not be exposed in my CLI logs Cypress has this rule that if it finds an environment variable but starts with Cypress all uppercase underscore and it doesn't know it it's not base URL it's not default command timeout so it's unknown value it will assume it's the name of environment variable but it should just process and stick it in environment object luckily I have written a utility called as a so I can put all those environment variables and simulate how cio works and if I wanna run with my passwords just like when CI locally I can just say run as a and when the name of a section if you want to record your test runs on from CI and start on dashboard you need Cypress record key so again don't pass it through CLI argument because this will appear in vcli logs instead use Cypress record environment variable and in this case just say Cypress run - the record it's safer but one thing that you should watch out for is that if you allow your project to build for pull request means someone Forks your repo and then submits a pull request they might modify the code and just say echo your Cypress record key or any sensitive variable and in this case you don't want to allow them to run so if you mark some of your environment variables as sensitive do not pass secrets to bills from forepaugh request like it's just a safety rule so that's how you bury configuration settings that's how you pass environment variables but how do you actually change what you run so imagine you have specs you can put them in different subfolders including a prod folder where you put the tests with a safe to run in production on your CI whenever you want to test production you can just say Cypress run there's there's back and a wild card telling Cypress to only run the test with a safe to run in production you can also change how the test behaves depending on environment for example you can put environment node and put this variable from your plugins and store it inside Cypress and and then inside your test you can say if I'm not in development when I should actually test if my site has robots that text or do for opposite maybe if you're not running in development you wanna stop xhr call so that you get expected results which might not be the case in staging or in production it's just JavaScript afterworld we have a couple of solutions to problems with people have in on CI for example sometimes you record your tests and you get all unknown source information on a dashboard so there is no branch not offer no commit nothing so this happens for several reasons but Cypress uses git commands to actually get that information if you you know yes yeah I doesn't expose that git folder both commands will fail as a fallback Cypress will look at environment variables that most of C is set with that information like bitbucket branch a circle branch and so on if that's unavailable Cypress even will look at environment variables but start with commit info as a you know fourth fallback just to try to grab those values sometimes when you run builds inside docker containers on we say AWS yourself you create a container if it doesn't have those values so you'll have to pass on yourself and in that case apps will have the correct source information we often hear this well sometimes that my test passed locally when I do Cypress run but when I push from to CI they fail so here's what happens when you use Cypress run it uses the built-in version of electron we're working to AB on upgrading this version of electron but it still is older than current so the test running locally might be passing because the version of electron what we use or and shape is alled but you do get a video you can try running your Cypress tests on CI in edit mode in that case you will not get video which is a downer but it will behave more like Cypress interactive mode where in some of our hidden box might actually be another problem if that fails try running your tests inside Cyprus bra using browser Chrome flag again you will not get a video but it will spawn latest version of Chrome which is new and usually runs a lot more stable than old version of electron finally you know the thing might be just a race condition maybe when you run test locally your app works fast enough so you don't hit problems and when you run things on CI it's slightly slower and you catch problems so in that case you have to change your test maybe add more assertions maybe add more Network weights or wait for network requests to happen before the test continuous justin suggested this one all right so one of the things I first looked for every time one of my team members says the build keeps failing in CI but works locally I always ask him do you have your API running locally because we're mocking all of our API calls out for our tests and in everly 99% of the time they say yep but running locally I haven't turn it off and then the same test now fells locally because they've forgot it didn't catch in the cypress UI that they were forgetting to mock out one of the calls was so I always look for that one that one especially when you're mocking it's really easy to forget what I call or - if you're running the API locally I think that's true you if you said it mocking one where am i but not now it's easy to mess it up so you know change your test to modify its run behavior depending maybe on environment just like I've shown before yep if everything fails we have a couple of plugins that can give you more information on CI about what commands have ran and where it failed we even have a little plug in that if you run using browser chrome will give you all the chrome chrome console.log so maybe you'll see wet like elusive console error that actually blocks the application and causes problems this is written by zeg Bloomquist who is one of our engineers he is a person who has done come in a different config is working on upgrading electron and he's working on full network proxy so definitely a great guy to follow on Twitter and on github finally if everything else fails you can configure a CI to send the graphical output back to remote machine which should be your local machine and then you can see what happens when you run in edit mode it's kind of complex to set up we have a blog post which shows it but if it you know if you can you can see what's happening on CI the last issue I want to close this presentation before we take a couple of questions is that sometimes the video but Cypress takes NCI freezes or has gaps may be a related issues that it takes video it freezes them and browser crashes so we found that this is mostly due to CI machines with a ver water we CI machine the CIA agent is low on CPU and when it's low and CPU with video frames our first thing that the electron process starts dropping and that's why video freezes so hopefully by upgrading to more powerful CI machine you can avoid this so Justin I think we are almost at the end but before we take Q&A I think I want to thank you for being part of its webinar where people can follow you and what else but can they learn from you so that is a great great lead-in to a free masterclass I'm actually offering this week exclusive a masterclass actually for the cypress community as a thank you on my side hey we're going to talk about the ultimate presentation formula for nerds and we're gonna walk through how do you overcome fears of public speaking what are the key ingredients you need for effective storytelling honing your messages in designing slides that are rock-solid without being a powerpoint guru or a slide guru and then how do you deliver your talk without feeling anxious and stressed out basically all the techniques and strategies i've used to do well over 200 talks at conferences in the last six years there's you know sure all of those pieces with the cypress community on a master class when i'm gonna run this wednesday 11:00 a.m. um you want to definitely make sure you claim your spot for this it is class publicly normally this is a charged product and it's seater and limited to the first hundred people well some get in there you can learn all of it the stuff that i use for both my speaking business as well as when i get on the stage for that definitely register for those piece even if you can't make it live you can get the recording of it but sent to you i thank you and give back some love to see all of you on that masterclass wednesday we can all use more public speaking especially in the tech tech space it this done wonders from my career I've gotten promotions out of it I make about twenty five percent more in my income um just because I've done the public speaking and people are seeing me as the go-to expert and so I'm getting promotions that work I'm getting more responsibilities lots of stuff I was trying to get to that public speaking magically gave me and I didn't do anything different from my career tech side that I was doing before I just happen to be speaking now definitely having public speaking and Cyprus as your bonus skills will increase your salary and absolutely so we did talk about lots of things right Justin you know from running Cyprus and different size to how to vary it as per environment common problems and meanwhile people were asking questions on slider so before we go to questions I just want to say that this slide deck is available on Cypress that sliced calm already so you can grab it and use it as a reference we do have ducks for continuous integration online available and constantly being updated and if you want to follow Cyprus data you're on Twitter here's our handle and another thing is that all the people who register for this webinar will receive a video with this recording and also with a little survey and the cool thing about the survey is that if you fill it up and submit you will get one of those cool t-shirts like I'm wearing and so if you want to receive it you have to fill the survey so I think we at this point we can go to continue our questions and answers another thing we'll spend more than maybe six ten minutes on questions well I'll start with a one that's always on top of everyone's list is when is support for Firefox and ie 11 coming well it's soon for Firefox we have PR and a branch with conflict being updated and we have probably like 20 failing and to end test for Firefox so as soon as we can fix it will release Firefox support the good thing is that you'll be able to run Firefox on see I immediately so we answer this Oh I pressed the wrong button but oh this is interesting Justin do you have any examples of socket Leitao and signal or integration by any chance I do not actually have a example of socket I do or or signal are not used either of those yeah yeah I unfortunately now that different programming communities use a lot of you know end to end as with Cypress we discover that you know we get more examples right you name it Python blazer you name it I'm hoping that you can cover how to run Cypress tests against that built darker image using circle size as possible so Cypress org not really but we can talk about this offline we have a separate circle CI or presentation but we really meant orb to run with your no dependencies not using you know pre-installed Cypress only but definitely open an issue on Cyprus or repository is it possible to show Council on test failure in headless mode and I think by by this way users means browser dev tools console so not yet but soon it will do that because we are adding native messages that means will be a will be able to open their tools programmatically Justin have you ever have you ever run Cypress on Azure DevOps or Jenkins so I've not specifically run Cypress on Azure DevOps that's but I've run built on Azure DevOps before set up very similar to what we showed in the CI examples hooked it up into my github so every time I made it push it ran the test and then I just wired up to run the NPM run see I command and and to do the NPM installs and everything and it should just run for you the question that I would have to search for it as ur is how do you out put to test sometimes like Travis CI I didn't have to do anything in particular for it to output the test for me so some CI servers don't need a customer reporter like team city did that but the steps are very very similar now a DevOps hook it up to your repo run your npm install run your Cypress run so I can point people we have a repository called Cypress - example - kitchen sink where we actually hook it up to pretty much every CI Under the Sun and one of them is Asia DevOps so you can just look at our DevOps configuration and and see how we've done it we've been running it and problem out no problems the only issue of Asia DevOps and free you know accounts is that they do not allow you to catch dependencies so there is no NPM caching or Cypress caching meaning you have to download things for each build which is kind of downer but yeah definitely no problems running Cypress and Jenkins there is also a Jenkins example with problem with Jenkins and also it's super strength is that you have to configure Jenkins yourself so you can do it in so many ways and we are showing one of our ways in our kitchen sink example how do you do multiple test use but not run all of them on Cypress run all right so I think this this is a good question imagine that you have a bunch of spec files but inside you want to run just some tests right against production but not ours I think it's the same right it's just JavaScript so if you know which environment you're running you can have if else statement and let's around your tests test we've had to use best use and say if in production just you know return have a wise execute you know your suit so I think we've shown an example of this this is a good one Justin II wanna what's the best Prague project structure for continuous integration for me I keep my source for my site in one directory and my cypress test in its own directory outside of the source folder let it never intermingle my test with my actual source and that's what Cyprus does for basically anyway the first time you run Cyprus it goes ahead and creates all the examples and wires everything up creates the directory scaffolds it out but it puts it in a Cyprus directory I have followed that model well and then I have a source directory in SRC directory that has everything that's my website in it exile and I want to finish with answerable I think the best project structure is to keep your test Cypress in the same repository oh yes all right so every time you change the code you have to change the test will never go out of sync because the tests have to pass against that code don't keep your Cypress tests or any tests in a separate repository it's not gonna work yeah you're asking for the team we need if you keep them separate to not update the test excellent I think we ran out of time we kind of over over time limit for most people I have to say that all the questions that people have written I will answer by email so we're not you know going into the darkness and to be forgotten also of the slides and this video will be available pretty soon right as soon as we in the process of recording and we'll put it on our blog we'll send an email to registrants so the resources are all out now and will be available in the future and Justin I just want to thank you so much it's been a pleasure you know doing this webinar with you thank you it was fantastic and lots of fun well everyone have a great work week and see you later

Channel: Cypress.io

Views: 43,135

Keywords:

Id: saYovXS9Llk

Length: 75min 38sec (4538 seconds)

Published: Mon Sep 23 2019