Automated Form Submissions with Chrome Puppeteer & NodeJS

howdy folks Kevin here in this video we're gonna be looking at puppeteer what it is and how it works along with going through a brief example of how you can use it to automate form submission puppeteer is a nodejs library that allows you to control chrome or chromium whichever you prefer it'll run headless that means that it won't actually show the browser when it's running it'll just kind of run behind the scenes but you can run it full where you actually can see the browser that can be useful for debugging purposes we'll get into that in just a minute it can be used to create screenshots and PDFs of pages useful if you need to do some quick scraping of websites also for a form submission or UI testing we're gonna use it today for form submission you can also take advantage of its dev tools integration for performance issues if you wanna do performance testing and you can use it to test Chrome extensions which is actually something that I'd like to take more of a look into because I've been asked about that as far as webdriver IO goes if you can use it to test Chrome extensions but off the top of my mind I feel like puppeteer would work a lot better for testing that because it's so deeply integrated into Chrome now with all that said I'd like to get started on this videos topic which is to write a script which will do a form submission process for a website that I use on a fairly regular basis to record rainfall at my house now this website is called cocoa rose I believe I'm pronouncing that correctly I could be completely wrong but it's a nationwide maybe even looks like that goes into Canada as well so mostly North American wide network of individuals working to submit the rainfall at their houses and they use a pretty cool little rain gauge that's all scientific and stuff and if we zoom in to Texas which is where I'm located at the time being you can see these are all the reports that come in bear County which is my specific County that I live in has recently gotten a fair amount of rain last night some storms blew through and looks like anywhere from an half an inch to a full inch even out west we got four and a half inches which is a fair amount of rain they probably got a little bit of Flash floating there and then over here we also got some rain so I get too fascinated by this stuff but the whole purpose of this is just a track rainfall across a large area of land so we can get a better picture from a scientific perspective of how much rainfall has fallen how curtin conditions are all sorts of important data that meteorologists use and not just meteorologists but people who deal with concerns around drought around flooding all that kind of stuff it's really helpful to know from a broad perspective what actual rainfall is looking like and what impact that has for future predictions and things like that so what does this have to do with puppeteer well every day you need to submit a report of how much rainfall you got for the previous 24 hours and you do that through this data entry form so you'll go in you'll click and then you'll enter your username and password and then hit login you'll save your login if you want to and then you'll get a form that shows your daily precipitation report you can change when you report the time the date most importantly you want to set the amount of rain you received I'm also track snowfall and get pretty specific if you see any flooding or things like that but the main thing we want to track is this rainfall now going into this doesn't take too much time I've got it set up to where I can pretty easily click the bookmark come in and it takes me directly to this page if I need to change the time I can but then I say well let's see last night I got half an inch of rain so I hit submit and that saves my data for the day and I'm pretty much done all this is to say that I may not be saving a whole lot of time writing the script but it does make for a pretty good example of how you can use puppeteer for something a little bit more complex this just kind of keeps things pretty basic I don't have to jump through too many hoops to make things work but there are definitely some things that I need to work around that would apply for any sort of script that you write with puppeteer so we're going to go through that in just a minute but before we start writing our script we need to get puppeteer installed and running so I'm going to open a terminal window and I'm going to create a new directory I'll call it rein puppet just for fun then I'll go into that directory and run my npm init command this will create a package.json file that I can use for storing dependencies which I will now install my dependency which is puppeteer do make sure you get that spelling right there's been some security issues in NPM where if you misspell it you may download a malicious package they've done some work to kind of avoid that but it is definitely a concern to watch out for with a spelling with this install you can see it did download chromium which is used to run the scripts chromium is a chrome-like browser that can be run headless lee which is going to be important for keeping our script quick quick to run and everything like that okay we've now got puppeteer and chromium installed let's get started with writing our script so I'm gonna open up a sublime text editor and I'm gonna name my script it'll just be a JavaScript file I'll call it range a s I'm gonna need some help from the documentation on this so I'm gonna bring that side-by-side and I should probably open up the documentation there's our install and they have their API page which we'll go to but first I'm going to copy this example because it's a really good starting point for us so I've got the API here I'm gonna paste that example in now on this it goes to example.com and takes a screenshot of the page we actually don't want to go to that we want to go to our CoCoRaHS page so i'm gonna copy that over and it's not actually gonna be this URL but i'm gonna save it for later cuz i do want it later it's gonna be the login page so i need to go to the logout and then hit login again and that is the page I'm gonna want to go to there we go what I can do right now is I'm just going to go to that page and take a screenshot and close it and just make sure that puppeteer is all set up so I'll save jump back into my terminal and run the script by doing node range is and it's run and you don't see any really output because there's nothing to have outputted we are going to be able to see this screenshot so let's take a look at that so here's that package with my main script and if I open up this you can see that is the login page that I was hoping for so the first thing you need to do now is to enter text into this username which is my username and then my password won't worry about clicking save login because it's not going to be saved from time to time because I do believe the cookies are cleared every single time that you start up a new puppet to your session so it's not going to say it anyway but we are going to click login once we get to login we can enter the form but right now we just want to focus on entering text into this username and password so if I wander over to the API documentation again there are ways to get elements on the page if we go into this page class you'll notice the dollar sign selector double dollar sign which is pretty sure it's multiple elements you can use XPath as well and this is what I started off with is using this dollar sign selector but I actually learned that it's not specifically necessary for single-use items so if you're just going to use an element once you can actually just use that element directly through this page class so I want to type into my textbox while there is a page type command that takes the selector and text if you've used webdriver i/o this is actually pretty similar it follows basically the same pattern you would replace page with browser and browser type and instead of set value you use type but you pass in the selector which is the same and then the text you want to type so we're actually going to copy that over come into our page here and I need to set my username which is Kevin lamping and then I'm going to duplicate this for my password which is my password that's not actually my password but before I move on to typing in my actual password I need to update this selector to match the inputs that I actually want to type into so I'm going to come back to my page and I'm going to inspect okay I've got that set up now here is the ID that I'll use text username so I'll copy that come back over here use an ID selector via CSS and then I'm going to do the same thing for the password so I'll do they inspect again it also has an ID on it IDs are very useful for automation and I'll paste that in and then I'll need to click this login button so a the login button has an ID on it as well so now I can do page click ooh miss full page page click and I pass in the selector that I want to click and then let's go ahead and scroll down here and we're going or not scroll down and move that down and we're going to take a screenshot of the page after we hit the log in just to make sure that the input on this work save come back over to my terminal and run and the test ran and if I go back to my example it'll be a new example and it says username is required so it didn't actually type that in how strange that it didn't do that and the password seems off too that is really weird okay so I figured out what I did wrong and that is I need to await these items a weight is a keyword see this a sink and a weight page type actually returns a promise and that we need to wait on before it resolves because it has to actually go do the action so if we don't await it it will try and do the action and then it'll go immediately before waiting for the action to complete it'll go to meet me to the next one and try and do that and all that kind of stuff so that's why it didn't enter any text in here and that password is too short for what I wanted to type so instead I need to use this awake keyword the reason that I forgot about it is that webdriver IO kind of handles this for you it basically assumes any browser command you're gonna call you're gonna want to wait on so it has this neat little wrapper around that means that you don't have to type oh wait all the time but in puppeteer you do need to type oh wait all the time that's one of the reasons that actually like webdriver IO over puppeteer for more advanced browser automation is the fact that it kind of handles that for you and makes the development experience a lot simpler regardless I'll add that other weight in there and then I'll save and run again and now and I get a the error that I was expecting which is invalid username or password provided because this is not actually my password let me go and do a quick thing to have my correct password in there but not actually show you my password because I don't really want you don't know my password because then you'll say I've got 80 inches of rain last night and that's just incorrect okay so now I've updated this script to use a environmental variable where I hide my very secret password away from your prying eyes and now my example you see it's got the password in there and the login button is grayed out it hasn't actually taken me to the next page we're gonna use a little bit of a trick in puppeteer to allow us to go to that next page and I've got to find where it is so give me just one second okay it's this page that wait for navigation so basically what it does is it will create a promise that will resolve once the page has navigated so it's a neat little trick that I kind of like where we say wait until the page has navigated so this is gonna see what the current URL is and then it will resolve once the URL has changed and what's going to change it is this page click action so I already have that page click action here I just need to wrap it around this promise dot all promise not all if you're not familiar with it is a function that takes multiple promises and will resolve once all of those promises have either all resolved or one of them has been rejected so here we're gonna have two promises that we're waiting for we're waiting for the click action to complete and then we're waiting for this navigation to return so I'll copy this over and I can actually get rid of that await paste that in and now if I save and run my script again if I come back here now you see the screenshot has been updated to be this input form which is what I want it okay so now it comes time to enter text into this form and one thing that's important is I can't just type in here because what chromium or what puppeteer is going to do it's going to click on there and then it's going to start typing and you see that it adds 30 inches instead of 3 inches so what I really need to do is clear that input now there's a couple different ways you can do this the one that I found works really well is to triple click one two three that selects all the text and then no matter what I type it's going to delete the current text in there so to do that we're going to get a reference to the element that we want to click three times click it three times and then send the text in there so I'm going to get the ID of this input which is this really long complex thing then I'm going to use that dollar sign function that I mentioned to store the reference to this element so I'll say constant rain input is equal to page dot dollar sign and then that ID selector now I can do rain input dot click and I'm gonna pass in an object that's going to define some properties of this click property I care about is click count I'm going to set that click count to three so this is going to click on that input form three times which essentially will highlight any text in there and mean that when I start typing it's going to delete that text I did look for a clear command like a text input clear command I didn't find anything this was the best that I could find oh one thing that I forgot to do is add the await to my inputs or to my commands now I'm going to await my rain input dot tight and my amount I'll just put point three just to try it out then I'm gonna save this and run my command again and it's run and that's strange it didn't actually send the text into there no I have to figure out why well after trying it out a couple times I don't know maybe that screen shot was just out of date but I kept it the exact same and now it's working I really think that the screen shot just didn't refresh for some reason but if I run this again now it works just fine I actually don't need this URL I thought I did but it navigates to the page that I want to enter my report to so I think that's just an automatic navigation on the website side of things that's pretty nice if I didn't want to go to a different page I could just use this oh wait go to or oh wait up a shot or a wait page go to golly I can't get that right and pass in the URL that I want so the last thing I need to do is click this submit button and that's pretty much the same thing as that page dot click so I will copy that ID over and just do away keep getting it wrong page click and then I'll pass in the ID of that button and really I let's see if I hit submit it doesn't change the URL so I don't think I could use this wait for navigation it is gonna say that report already exists because I've already entered a report which is actually good for me cuz I don't want to save this I just want to use it for testing right now but I'll just go with page click and I will save this file and let's see if we get this error to show up with our type and we do get the error to show up and you see it's 0.3 so it has fully completed everything so one thing I could do at this point is add some more script to check if this error shows if an error does show I could have it log out if not if it's like a success message I could actually have it say this success message but I don't want to really get into that because it's more work than I really want to put into this script right now so instead I'm just gonna say console.log rainfall submitted yay and that way I just kind of know that it went through everything now there is one more thing I want to do to make this script a little bit easier to use and that is be able to send the amount that I want to input from the command line myself so I want to say node range a s plus point zero 3 or whatever that amount is right now you'd have to manually update this text and that would be annoying to have to do every single time you want to run it and could make it really unusable if you want to make this a little bit more flexible so what I'm gonna do is use the process Arg V variable that exists on all node commands and let me bring up the stack overflow page that kind of explains that so that you can see in a little bit clearer terms what I'm talking about so there's a quite a few different answers out there but this is the best one that I found really quickly basically it shows Arg V takes the arguments that are passed in when you run your script so 0 is equal to the first one which is node 1 the second which is the script that you want to run and then your arguments here go 2 3 4 so the one that we want is the one on the second index which is going to be the actual argument place so I'm going to copy this over come back to my script and say constant amount is equal to that value I also want to add a little bit here to say if you didn't enter in an amount it should give you an error and quit the script so it doesn't try to log in and all that kind of stuff so I'm gonna say if no amount or if that amount is not a float and now I'm not gonna use is not a number to parse through that then I'm going to do an error message so I'm not actually sure there's a difference between console log and console error but maybe and I'll say please enter an amount as an argument and really that should be please enter a numeric amount as an argument and then if it comes here I'm going to use process exit and use a non zero status code so a one in this case and this is gonna make node quit the entire process and not run anything else then I'll copy this amount and paste it down here save this file so now if I try to run without an amount it quits and says please enter a numeric amount or a non number it'll say that but if I come in here and I say to say I got two inches of rain which would be a fair amount it says rainfall submitted and then if I jump back over to my screenshot you see it says it's already existed but it does has have that two inches in there so that is the entirety of my script I did mention headless mode I think if I do this it should work headless is equal to false we should actually see it run pop up a browser there it goes you can see it loads up and tries to submit and it's done so this be useful for debugging is to be able to see that come through so that's if you want to run it outside of headless mode so it actually shows on the page but uh headless mode is preferred mainly because it's a bit quicker it doesn't have to go through all the process of starting up a visual browser it just all runs behind the scenes so that's puppeteer it's a really neat little technology especially useful for running quick test or automation scripts you could use it for all sorts of data entry mainly for things that don't have api's if this had an API it would actually be a lot quicker just to use the API to send that stuff they might actually have it they don't from what I've seen they don't advertise it so as a fallback I can just do this through a node script and it runs pretty darn quickly since a lot of my videos cover webdriver IO testing I can see the question being should I use this over webdriver IO or should I stick with webdriver IO and truthfully the the best answer is it depends it depends on what your purpose is it depends on how big of a script you're trying to write if you're trying to set up a quick test and you want to just keep things simple puppeteer is a great option because you don't have to worry about what test framework to use or setting up selenium or anything like that it's just all simple it's a quick install the test run really quick you're only testing in chrome things like that if you're looking to write a full set of test automation for a full website I'd probably go with webdriver IO one thing you don't have to type a wait all the time it handles navigation a little bit cleaner it's got a few more bits of functionality and the amount of integrations it has with tools like sauce labs or browser stack or applitools all sorts of great stuff actually I do think puppeteer does have an applet tools integration so I should take that back but they probably don't have sauce labs I don't really know I'm kind of just making things up check that up for yourself but I I still would recommend webdriver IO for larger scale test automation now running through a simple form entry I would definitely recommend puppeteer because it's just so simple and it's gonna run quickly you don't have to worry about setting up all that extra stuff so it just depends on what your needs are if you're gonna be testing Chrome extensions I'd probably recommend checking out puppeteer instead trying to use that first because it is so ingrained in the chrome ecosystem it's probably gonna have a lot better support then something like webdriver may have I actually haven't used webdriver for testing Chrome extensions and I haven't used puppeteer for testing Chrome extensions but my gut feelings telling me you would use puppeteer in this case so it's gonna wrap up this video if you've used puppeteer I'd love to hear about your experiences with it always interested in learning more about actual real world experiences because there's lots of examples out there but they don't always best represent how it's actually used in production cases I guess not really production but in real world cases they don't quite cover everything they're just too simple to go through that's why I kinda wanted to go through this one is because it does get into that waiting for navigation and having a clear text input that's not something that you might see in an example that just takes a screenshot of the page so yeah let me know in the comments if you have any other ideas for what you'd like me to cover I'm always happy to hear that in the comments as well and have a great one [Music]

Channel: Front-end Testing with Kevin

Views: 55,177

Keywords: webdriverio, selenium, testing, nodejs, automation, programming, javascript, coding, watchmecode, tutorial, webdriver, code, tests, chrome, chromium, puppeteer, node, forms

Id: 6IOrp8HgnJU

Length: 23min 21sec (1401 seconds)

Published: Wed Nov 14 2018