Dave Haeffner's Proven Method to Grading Selenium Tests

hey everyone thanks for joining today's event how to grade your selenium tests my name is Adi and I'm the events moderator today's speaker is Dave after as all of you know Dave is a test automation and selenium expert he's the writer of elemental selenium a1 a free once weekly selenium tips newsletter author of the selenium guidebook and the creator and maintainer of chemistry kit he is also a founder and co-organizer of the selenium hangout and has spoken numerous conferences and meetups about automated testing just before Dave kicks off a few important things to keep in mind feel free to ask questions during the webinar you can do that in the GoToWebinar control panel on your right we will try to answer as many questions as possible at the Q&A session at the end the webinar is recorded and the link to the recording will be emailed to you tomorrow for optimal viewing experience I start I strongly recommend you close off any unnecessary apps running in the background and if you encounter any technical issues please send me a private message in the chat room I am the applitools event organizer and I will try to take care of it as soon as possible so Dave the stage is yours Hey hi can hear me okay and my screen is it showing yeah it is great okay hi everybody thanks for joining and I'm really excited to talk about this topic because it is something that I've been thinking about for several years of this Amy and I'm sorry to cut you but know we can't see your screen for we can see your presentation for some reason okay how about no now we see your files we don't see yeah now we can see it okay weird um cool great let's try that okay so I'm excited to share this topic with everybody just because it's forced me to kind of quantify everything that I've been thinking about because I've been recently starting to put it into practice the idea of how do you take something that seems fairly subjective you know when you see it and putting it into practice in a repeatable way that can be applied to anyone's test code your test code if you use which between teams in your company's test code you switch between companies test code with selenium and so let's talk about tests and so I like to start by just saying let's ask a question if we were to ask each person here in a group setting what makes a bad test I think we would get kind of a smattering of answers and they might kind of range something like this we might see people who say tests that fail for no good reason that's a bad test it may be just because it's a false positive a transient failure those kinds of things changes in the application that we're testing application under test if there is a change and it forces us then to go and update multiple tests then potentially those tests are pretty brittle a might not withstand the test of time of course it depends on what kind of changes happen happen in the application that you're testing but if it's something that's not actually a fundamental rewrite of a feature and potentially just like the the page gets reoriented in terms of where elements are located then potentially that's something that would be a draw cause for thinking it a test is bad tests that are slow and inherently selenium has this bad rap and has over the years of tests being slow brittle and hard to maintain and slow tests can invariably lead you to think that a test is bad and of course the caveat to a slow test is it really depends on what you're comparing it to so if it's slow in regards to you have other selenium tests and they perform pretty quickly and this other test just seems to really be slow then that's that's definitely a cause for consideration that it might be a bad test tests that are unreliable that don't actually work consistently and create a lot of noise those are perfect candidates for bad tests because they can erode the trust that you and your team have in test suite that you're working on and then test it or of course hard to understand they're hard to maintain and again these are largely subjective know what when you see it things and so but if we were again if we have this kind of thought process this conversation I think this is a rough set of answers that people might come up with now if we take the flip side of this coin and say well what makes a good test let's talk about that I think a good test ideally should be written for some kind of existing test framework like a behavior driven development or a unit testing framework and each test that you write should test one thing and tests should be atomic and of course you want to make it so each test ideally can be run independently so there's no dependencies between tests so each test should be autonomous and then you want to make sure that anyone including yourself six months from now can understand what a test is doing and this goes back to just understandability which leads to things like helpful names and those kinds of things and also just not being clever and this is kind of an aside but anytime that you think at a solution in your test code either one you've written or one that you've found someone else has written if you think it's clever if you think it's doing something fancy then that's that's a cause to to think maybe this isn't a good test because clever typically leads to brittle which typically is too hard to understand and hard to maintain and so that's also just a hint that maybe there's some opportunity there to to clean up your tests and ideally you want tests to be grouped similarly so if you have a bunch of tests that exercise log in then potentially those would all want to live within one place within your within your test code and as opposed to cut kind of scattering them to the winds and I always put this here just as a reminder to people that through a good test framework you actually can execute disparate tests into dynamic groupings using metadata typically within Java and J unit then you have you have something called categories within the Ruby world you have tags and within c-sharp you have attributes and and an iPhone you have decorators and so the list goes on and on and then ideally your test should leverage some kind of centralized setup and teardown so you don't want to repeat the incantation to create a custom browser every time ideally this would be something that's centralized so that you can customize I configure it enhance it so that you can add in additional benefits like screenshots on failure and running it on a remote grid incorporating automated visual testing through a third party like applet tools and so on and so there's really a lot of benefits to this kind of architectural consideration and then the last one which is just kind of the most concise one here is they use page objects and I'll cover page objects in the next section but that's basically fundamentally this is what makes a good test this is a reasonably good and sound way to have a well-structured test and so again some of these most of these are actually somewhat subjective like they're not necessarily something that's easy to quantify and so to make this more concrete here's a rubric with some more rigor arounds way what you should look for in your tests and and I've started basically putting what I'm about to show you between tests page objects and locators in the practice within existing test Suites and it's really been an impressive helpful indicator of which tests need work and what things within them actually need to be improved and so the way to think about it is a lot like when you were in school you you know you're in a class and you're given an assignment a homework assignment or a test and when you receive that test back there is a grade attached to it now there are a number of questions you have to fill them out and then there's a grade it's either A through F or it's out of 100% and so each rubric I'm going to show you follows a very similar construct by the end of it each item that you're testing or each item that you're grading rather so in this case if you take a single test and grade it using this rubric you should end up with a grade and then you can convert this numerical grade into a letter grade if you so choose so basically every test that you want to review starts with 100 points and then you would deduct points as necessary based on the criteria in this rubric and so the first one follows this this pattern of there is there's a thing that we're going to look for and whether or not it belongs in a test and then there is a deduction or or not depending on the answer to whether or not belongs in the test and so the first one is selenium commands given that the goal of writing our tests and a good structure revolves around the idea of using page objects then all of our selenium commands should be taken out of our tests and put into page objects and so they should not belong in our tests and so if you have selenium commands in your tests then you really want to deduct points now we don't want to of course just lead to an automatic and imminent failure because you happen to not be using page objects in certain places but this is a helpful way to reduce the score enough to to give you an indication that there needs to be some improvement made so for each Slom command up to 3 you deduct 3 points each so if you have three commands in a test then you would say oh I need to deduct I need to deduct nine points and then locators follow the same construct and of course 99% of your Sounion commands use locators and so they should not belong in tests either same pattern applies for each one you see or a similar pattern rather for each one that you see you ended up two points up to a maximum of eight points and then a selenium setup and teardown so the central setup and teardown that I mentioned on the previous slide I this should not be in your test and if it is then wanted adduct 20 points and this is a heavy deduction because this is something that should really be considered to be taken quite seriously since it really does it is an architectural consideration that makes a huge advantage in terms of maintenance of your test suite over time the caveat here there's an exception to this there are times where you might not necessarily do too kind of constraint within your application you might not want to have your tests each test open its own browser or you might need to in an actual test class we use a browser instance between tests due to sums like I said some sort of constraint where it's a really expensive operation and there's something probably tied to test data typically in those circumstances where it's just a really expensive operation to launch a browser do all this preamble to get a user and get the machine the system into a certain state and so when that happens then potentially this this exception can be over or this rule could be overlooked because of that but for the most part under most circumstances now you should just have a central set up in teardown and then in terms of how you're waiting any sort of synchronization is going on in your in your testing shouldn't happen in your tests so hard-coded sleeves in general you should never use hard-coded sleeves but they could also not belong in your tests if you happen to be using them and if there if there are any in your tests then is minus five for each one up to four of them which would be a max of 20 and then implicit weight calls because selenium has two different weight functions and implicitly and explicitly it's again any weight functions shouldn't be in your test it should be in your page objects what although with the implicit wave function that's typically in the central setup and teardown so depending on how you use it and so if that happens to end up in your test then it's minus ten and explicit way it's also know and these to the minus ten and the minus three for implicit and explicit those it's just if you see if you see it just deduct that amount if there's more than one don't worry about it just just a flat deduction and then conditionals there should be no conditionals in your test and so you should go ahead and deduct points for each one you see and this gets back to cleverness like if there's or as I'll say a hint like if you end up seeing conditionals where I don't test should only run under certain conditions or if if the page is this then it should do this versus that like that kind of stuff a hint that that should belong somewhere else potentially an abstraction in your page objects or potentially you need to rethink how you're actually writing your test and so these things should be abstracted or rethought and put into the correct places and if they're not then you end up with these deductions and if you make these changes and follow this kind of rubric here then you should end up with tests that are better structured easier to understand easier to maintain and correct abstractions have happened in the correct places some other things page object calls to page objects those should belong in your tests and so there's no score deduction for that and then assertions I mean this is really the bread and butter of why we do a test every test should have an assertion and then with that there's of course no deductions I'm a point about assertions it's kind of the whole why we do this like we use selenium to drive the browser to mimic human action and then we get some information about the page and then we use our test framework to perform an assertion so I'm sort of check some sort of assumption about the page the state of the page and so ideally your test ends with an assertion and ideally you're testing a concept this goes back to a test being atomic it's testing one concept not necessarily a single assertion but that's generally a good rule to follow but testing one concept about the state of the page and so so those are the general tenants I think about when I think about what makes a good test in a quantifiable way and when I go to clients and consultant look at test code these are the things that I look at and these are the recommendations I make which end up typically leading to more robust more maintainable easier to read easier understand tests there's of course the the harder to automate things harder to quantify things like good test names and that kind of thing and that's a bit more subjective based on the is it using good domain logic and is it helpfully named and that's kind of something you really have to pick up through tacit experience unfortunately but these are the things these are the core tenets I think will get you 80% of the way there and so I'll leave this up here for just a moment and encourage anyone that if you have any questions to write them down and toss them into the questions input box on the go to webinar control panel and also I'll mention that these slides will be made available afterwards as well so if you happen to be getting a cramped hand by writing all these down or you're typing furiously on a keyboard that's great but if you happen to miss some of this in terms of transcribing the slides will be made available afterwards so I'm going to now switch gears and talk about page objects so let's talk about page objects and one resource is really helpful with in this selenium community is the on the selenium github page so if you go to github and you search for them selenium HQ you'll end up or just selenium you'll find the project page and then you'll end up finding the project there's a wiki for selenium and the wiki has some really helpful information that's usually pretty well up to date and one of those wiki pages is on page objects and it's um written from the perspective of one of the core maintainer x' one or two of them have waited on it since then but it's basically talking about what page objects should and should not be it's also written from the perspective of Java and so it's not really going to serve you if your language of choice isn't Java but that's one really good resource but a better one which kind of is better in the sense that it's probably more generalized is the quintessential blog post from Martin Fowler on his website about page objects and so this dates back to 2013 basically great job of articulating what a page object should and should not be and why it's important and then he of course links back to the wiki page on this learning project even throws a quote in there from from Simon Stewart about you might be doing it wrong if kind of a kind of situation and so the general gyah is that it's a pretty simple API you're basically creating a programmable interface for the page and and not specifically like the inner workings of the HTML so your test should be interacting with things like method names that have to do with the action you're trying to do then the page objects have to do with the actual interacting with the page and this protects you in shields your tests from from brittleness so if the application changes you can just go into a page object and and update it you know change the locator is update the behavior that kind of thing and then your tests just magically start working by virtue of that so that makes them obviously more maintainable and so page objects are you know kind of the tried-and-true like power law of do this do this pattern and things will be a bit happier for you when you do test automation with selenium but what a lot of people don't talk about is like what what really is a good page object what isn't some people have opinions on it some and it just it just kind of varies depending on who you talk to and so I have some core tenants I think are pretty straightforward for just a really good way to get started and make sure you're on the right path with page objects and so I like started thinking about like what makes a bad page object and so I've seen page objects that are gigantic absolutely absurdly huge hundreds if not thousands of lines of code and those you know we're basically created and then added on to and edit onto and eventually just not not much rigor put around like well if it ends up exceeding this like a few hundred lines maybe we should consider breaking it down and so people who inherit this just basically go up I'll just I guess add to it too and make it bigger and so if you see page objects are gigantic that's really not a good sign they will likely be very onerous to maintain and potentially brittle because they might contain lots of really oddball stuff that could be happening which can cause odd behavior in test code with actually really realizing it or if you change something somewhere maybe it ends up breaking something differently not the way other end of the page object it could just be really really challenging to maintain and there could be overly complicated logic unnecessarily okay that although I guess you should really strive to not have complicated logic when you can avoid it you should really a good code great code should be blindingly simple should be very straightforward in terms of what it's doing and so if you have something overly complex then it's maybe some a good room for you to pause and reflect and and then the other thing you you could end up with with a bad page object is it may not return enough information or it might return too much information back to the test be a leaky abstraction if that were the case where there's not enough information for the tests to perform its job or you provoke your returning exposing too much information from the page object which can lead to tight coupling to the page ultimately the actual application you're testing which we're trying to avoid we want to have that sweet spot of just the right amount of information being returned so if the test breaks due to the page changing we just have to update one thing really quickly and the page object and everything starts working again and then of course a bad sign would be if assertions are happening in the page object instead of in your test because your test is where the assertions should be happening not in your page objects so what makes a good page object so there's things like it contains state like on the simplest concept of what a page object is it's just it could just be a plain classically a simple class that contains state has locators these are the elements the HTML elements on the page that you want to interact with and this is these are the incantations that you'll use to tell selenium how to find it and then you want to contain behavior so you want to have methods to interact with the page so in an example of logging into a website you have a class it could just be called login page and it could have variables at the top of the class that denote the different the different locators that we want to interact with on the page so things like the user name input field the password input field the click the submit button and then we can have a method that's just going to actually log in it's going to input text into the password and user name field and then click Submit so that would be an example of containing state behavior and then we want to of course return some information about the page back to the test and so what what that would look like would be one of two options really or three options I guess technically so you'd either want to return something like text from the page or a boolean result based on a question like if you wanted to see if an element was displayed on the page you could ask that question of the page with selenium and it would return a boolean result or the more common approach is to actually return a new page object so an example of login if you actually log in it should take you to a secure account page and if that were the case then you could have another page object for the securacom page and then as the kind of closing act of the login page object it would create a new instance of the secure account page and return that and then the tests would then be able to interact with the behavior on the secure account page but the one thing you don't want to do is return a web element this would be a case of where we'd be providing too much information from the page object back to the test and so whenever you perform a find element with selenium it returns web element a web element or web elements depending on how you're actually doing the lookup and so that's something that the page object is aware of it needs to be aware of but that's not something your test ever needs to be aware of that's just anything that has to do with selenium should be abstracted up to the page objects and then one other thing that a good page object does is it verifies that the page is ready it verifies the state of the page as part of its initial initialization there's a couple different ways people can approach this you could either put it in like a class constructor or you can actually have a separate method well depending how you build your page your page objects and basically the gist here is that you want to make sure that the page that your test is about to interact with is loaded and the things on it that you want to interact with are there otherwise the test will try to proceed and it won't be ready and so the test will fail and it might not be immediately obvious why so you can just create kind of this this um checkpoint at the very beginning of when you create when you create the page object and so just like with the test rubric I've create one for the page objects and so each page object starts with 100 just like the tests and you deduct points is necessary and so things like lines of code so I picked a reasonably sized number for lines of code so if your if your page objects are greater than two hundred lines of code that shouldn't that should not be in your page object but if you happen to have inherited or created page objects that are this massive potential cabal of code then you want to deduct points and there's no cap to this which could be a real hurt for people who have like thousand or two thousand line page objects but you basically for every 50 lines of code that are greater than 200 deduct five points from your score and so that will add up in a hurry if you have very large files assertions know they shouldn't be in your page objects but there is a caveat to that there's an exception when you're actually verifying the page readiness in the constructor or however you're doing it you can use an assertion that's like a good poor man's way to check to make sure that an element is is correctly loaded make sure that the page is ready ideally you could do this differently you could actually raise an exception instead of using an assertion that's a much cleaner way that doesn't require you to actually import the actual test classes from your test framework for assertions but that's just one example of that's something you could do and that would actually be reasonably okay but other than that no you shouldn't do assertions and if you do then for each one that's obviously not verifying page date but actually maybe you're trying to assert that the you know something else about the page that should not exist in your page objects and so - five for each one there's a cap of - 20 for that and then we're going into the actual implications of the different weight strategies that you might be using in your tests for synchronization hard coded sleeves just should really never be used there's like a you know one in a one in a thousand maybe one in 500 situation where there's probably a good reason for hard coded sleeves because any time that you have no way to dynamically determine when an action is to be completed then that's a reasonable use case for our coded sleeves but for majority of what you're trying to do with selenium then you don't need hard code it's you really shouldn't so if you see one you should really flag it and put it you know find a better way to actually weight in your code and so for those for each one it's -5 with a cap of 20 and then the other two implicit weight and explicit weights so an implicit weight ideally you shouldn't use it and I'll explain kind of the rationale as I get further down here but the gist of it is implicit weight is set to 0 by default and so if you change it then you hopefully know that you're changing it but what ends up happening is you change it in it's kind of a global maximum my implicit weight means that when it's set every time a final action is cold with that instance of selenium that now has an implicit weight set it ends up trying retrying an action until it can be completed up to the amount of seconds that you've set so if you set an implicit weight - 15 every time a final iment action is going to be used it will the find out if it can't complete immediately it will retry up to 15 seconds and there's also some false positives and some race conditions stuff in there with it potentially that makes it not a great option because it's not as thorough as an explicit weight but the gist is if you avoid an implicit weight which is actually the recommendation from the committee's on the selenium project that in favor of explicit weights even though it's a little bit more work you'll end up with a much in a much sturdier position but there are people who use implicit weights and get benefit out of them so the general the easiest thing would be if you spot an implicit wait then really you want to deduct ten points and this will help help give you an indication of where to focus and if you're starting to rethink and and heed my advice and the recommendation from the selenium committers that explicit weights are the way to go then then you'll be in a better position because this will help you identify where those issues are or where the where the implicit way calls are because they will actually create potentially transient test failures for a number of different reasons but explicit weights are better and so of course you didn't know points and so for those of you that are not aware of the difference between an implicit and explicit weight I'll just briefly mention that so an implicit weight an explicit way it's fundamentally are very similar in that you set a timeout you call it you set a timeout and selenium will retry certain action a certain action and then in case of an implicit wait it's every find element in the case of explicit waits it's only when you call it it will retry for those actions until it can either do it successfully or hits the timeout and then return an exception implicit wait is the kind of thing where you should set it once and then forget it basically set it and forget it explicit waits are whenever you want to use it and you can change the value every time you use it what I see is a lot of anti-patterns around these of implicit wait which is the reason I I think it's a dangerous thing I think that people don't realize that and explicit they're basically people are taking an implicit wait and then using it and turning it like a dial and so it's they set it during test setup and then in a page object they change it and then they change it again later and this has potential ramifications for a test when it's in flight it can cause you know caused an element you think is waiting 15 seconds to actual only be waiting five seconds because a different command somewhere else in the page object could potentially have set it to a different value whereas with an explicit wait it you said it's explicit this is how long it's going to wait and actually you could theoretically combine implicit and explicit waits together and override the implicit wait with an explicit wait but don't do that mix them and if you see that you're mixing them in your test code really work diligently to try to fix that because there is actually a not so well documented bug where if you mix the two together you'll have some transient test failures depending on which browser OS combination and how you're run your tests but the general gist is just use explicit weights like that's the recommendation from the project that's the best way to go about it and implicit weight is a good way to get started but it's not a great way to kind of grow your testing practice so moving on Otto's along tangent moving on so verifying page ready state is something that you really want to do your page objects and and so something that we said here you really want to make sure that when you're verifying the page ready state you're verifying something relevant on the page the notes that the page is ready to be interacted with I see a lot of places where people have used URL lookups to say like well it's it's on this URL so I'm good to go or they look at the the title of the page like the actual title attribute of the page and these are not good indications that the page is loaded it's an indication that you may be ended up in the right place but it's not a good indication that the page has actually loaded to a point that it's ready to be used and so this could actually create false positives and cause your tests to derail anyway and you have kind of a false sense of security around your the way you're verifying the page ready state and so minus eight points if you're not verifying an element on the page so really you could perform a fine element to see if something is displayed on the page or something is clickable or something there's a bunch of different expected conditions that that you could you could use to wait for so basically it's a good it's a good candidate for using an explicit wait and waiting for a specific element and so so that's basically it so use an element look for it make sure it's something that can be interacted with and then if it is then the test can proceed if not then it should fail but non something like checking the URL of checking the page title then locators obviously since Salina commander should be in your page objects locators are the lifeblood of kind of what everything we're doing like without locators and selenium will won't know how to interact with page so those should end up in your page objects and there's no deductions here the only thing to mention with locators is that they should theoretically ideally they should be stored in a central place within the class they shouldn't be strewn about in line like you should put them in a variable at the top of the class so if it's Java and you're using just the simple Java class then you'd put it in a field variable at the top of the class this enables you to use that variable in multiple places in the class and then only how it gives you one place to update it if the for some reason the locator needs to change and so so that's the rubric for grading your page objects finding repeatable way to to make sure your you know your page objects are robust and following a good a good construct and so I'll leave this up here for just a brief moment and encourage anyone that has questions to put them into the question box okay so moving on let's go to the last topic which is um kind of a lion's share here because I think it requires a good amount of time to talk about and as locators and but before I do I do want to mention briefly the the tangent I went on about explicit versus implicit weights this is a link to a stack overflow post where somebody was asking a question about transient test failures in their selenium test code and Jim Evans a longtime committer of the project for those that don't know Jim is kind of the guy who doesn't require an introduction he's created ie driver he C sharp bindings like he's kind of a legend as far as I'm concerned within the same ecosystem but he answered very thoroughly a question as to why this person was experiencing transient test failures and had to do with the fact that they were mixing implicit and explicit weights in their test code and that it led to a undocumented bug that was really due to an architectural limitation and depending on the browser that you're testing with and whether or not you know just goes on and on there's just a lot of detail to it and so um I think that this just further bolsters the the position of the selenium committer is about about using explicit weights and so I encourage anyone that kind of is more curious about the position about implicit versus versus explicit ways to check out that post so without further ado let's talk about locators so within selenium there are seven different locator strategies and they're all listed here alphabetically and the way I typically think about locators is that they should really be unique descriptive and unlikely to change and so for simplicity I just think let's just roll a few of these out based on this criteria and think about starting with IDs and classes there's two reasons for that IDs and classes are the most performant the quickest way that selenium can interact with elements on the page it's the fastest lookup it's available and also if an application that you're testing was built to follow the HTML spec which more often than not is unfortunately not the case I've using classes theoretically should have helpful unique descriptive and unlikely the change names on them that you could use and so that your test would be very straightforward to write and when that is the case then it's great and things work really well and they are very fast but that's often not the case and when that happens you need to reach for CSS selectors or XPath now if you want to handle CSS and X you could write very brill very slow dangerous locators that end up being things that later on can break your tests inadvertently and so this is an unfortunate reality when you're trying to write automated tests for an application that wasn't built with testing in mind and you're kind of figuring out the best way to kind of Cluj something together and you get it working but then if you have to go back and maintain it because the page changes slightly and it causes it was a very brittle locator to begin with it's just kind of a nightmare and so so that's just something to consider CSS or XPath both are good when you don't have the happy path of IDs and classes and so there is actually this kind of contention within the within the selenium ecosystem which has to do with which is better CSS or XPath and for a long time and even still today there's this kind of assumed knowledge that XPath is potentially inferior and/or it depends on which which side of the argument you're on some people think XPath is far superior some people think CSS and so it's really interesting there's two reasons that this argument happens one is that people don't actually understand that both with CSS selectors and XPath you can accomplish almost entirely the same things there are some areas where XPath is more powerful but this they're typically for cases that you don't need and so I think that people just have their preference and have their experiences and and know what they know and so that's fine and so I generally encourage people to use which one they're more familiar with the the more recent or semi recent argument that people kept harkening back to was that XPath was slower you know basically is a performance benchmark that was cited they ended up being says at least four years old at this point where there was actual data at one point that reflected that XPath was considerably slower depending which browser you're using with selenium but selenium since gone through so many major upgrades and full version number at least being one of them from selenium RC to selenium webdriver and then is of course the fact that browsers constantly evolve as well and so the argument of which is faster is actually somewhat irrelevant and so to kind of help further the point I have gone back a couple years ago and I reran benchmarks between locator lookups with CSS and XPath between the different browsers at the time and then more recently within the last few months I've reran those benchmarks again and so I'll show you I'm going to share with you kind of the more recent benchmarks I've seen and so there's kind of a longer write-up that I have about this and some of the website that I'll add a link to this slide before I post them but basically I went through three iterations and the last one I ended up with had a lot of input from committers on the selenium project around what kind of an application would reflect a really good test for selenium and then what kind of process so all the all the benchmarks I've done is using a very large and deep nested Dom example and all of the I'm using an actual benchmarking library in Ruby and all of the locator this this application is running on my machine and all the locator lookups are are happening you know numerous times to kind of remove any sort of inconsistencies in terms of terms of load time so there's a whole there's like three different blog posts each about 1500 words that kind of go through the process that went through so again I'll link to that so you can have it if you're curious about what went into this but this is a very large and deep nested um it goes well below what you can see here it goes actually 50 divs wide there's a huge table at the bottom and there's different locators in them using and so the first two are horrible locators are gigantic they walk through the whole DOM and then these other ones are more reasonable but these first two as I mentioned are horrible you just keep going for days these are locators that you should never ever ever use and so for argument's sake though I'm using them so I can actually prove a point about the different performance benchmarks between browsers and so let's talk about Chrome so between Windows and OS X so you know it's a little bit of a performance difference generally speaking in terms of how long it takes things to load it could be partly because of the virtual machine that I'm running but largely I think it's simply showcasing the difference between we XPath and CSS selectors so on the left are the so basically it goes in order of the locators as it was shown in that little notepad document and so the first two are the the worst and then they get better the further to the right you go basically and so you can see that within chrome there is a least on Windows the there's a couple of locator lookups that are a little bit worse but in the relation to what basically the time scale showing second so at most on Windows it's taking up to two and a half seconds for a locator to be found and so compared so compared to just over a second so only one of those locators actually is a two to one everything else is reasonably consistent again these are some pretty rough locators and some of them are less rough and then on an OS X there's a more of a pronounced difference in terms of the execution times but again the time scale is actually much smaller here where it's it's actually sub-second so 700 milliseconds for the slowest of the locators versus almost 400 milliseconds for the slowest so it's it's roughly it's close enough it's basically within your test code you're not going to notice too much of a difference and again at one detail I did not mention this is actually not for a single locator lookup this is actually the median of a hundred different lookups so actually the individual locator lookup if you're to performance in a test would actually be probably less than like less than a millisecond so I basically ran this benchmark data numerous times so that the numbers would actually be just easier to kind of represent here so but but generally speaking things are a bit more tuned basically the moral story here is in chrome CSS selectors are what's tuned for this other more perform at the next path but not to the point of the data that used to be reflected which was XPath used to be something on the order of it would take minutes to accomplish a lookup where whereas with CSS lectures it would be very trivial and so really it is a trivial comparison but it's slightly faster in Chrome for CSS lectures whereas for Firefox it's even more obvious that it's being tuned for CSS selectors XPath obviously is it's almost two-to-one across the board for on for Windows verse N and on on OS X or Firefox it's even more obvious it's like a four to one in terms of the latency so CSS selectors are clearly faster when it comes to Firefox but again it's not to a point where it's so slow that it's something that you should be concerned about and then the outliers here Internet Explorer 11 and for the one of the worst locator lookups is of course the most poorly performant ones it's almost it's almost 20 seconds to load but other than that CSS is actually a bit slower internet explorer X pass a little bit faster whereas for Safari it's kind of a motorsport here CSS is a bit faster so generally speaking with exception of Internet Explorer CSS selectors are actually faster but not to a point where you should consider XPath to be completely trashed and throw it away but then again it goes back to one of the two points I mentioned earlier which is CSM go with what you know and what your team is kind of fluent and when it with regards to the locators I generally recommend encourage people to use CSS selectors because it gives you a common language to have a conversation with the front-end developers so the people that are building the application that you're potentially trying to do tests for ideally they're involved in the test writing process but if they're not and you have to have a conversation with them to find a locator they're probably going to give you a CSS selector they're not going to give you an XPath and if you are both having that same conversation in the same language then that's going to be a much more efficient conversation but hopefully this data has reflected that it's you know it's up for grabs but the browsers are typically tuning for CSS selectors over XPath so that leads to the locator rubric so the best way to think about this is that instead of kind of starting at 100 and then deducting points you should look at each locator that it gets a grade every locator gets a grade and so and so I'll start from kind of the worst offenders up to what the gold standard is for locators and over time I think that this list will become more comprehensive but I think these are the broad strokes that really capture what would make a good locator and help identify which locators are not that great which mainly I think from everything I've been talking about today follows one general strategy one specific thing and that's how do you write your tests so that they will be reliable over time so if the application slightly changes in terms of how things are laid out and the test won't break because locators are brittle you know or it's easy to maintain so really it's just something like you create you create a test and page objects and you want them to last you want them to work and be reliable and give you helpful information so these locators fit into that strategy and so this is basically a hopeful way for you to get kind of an information radiator about which locators are good which locator to bad and if you're using any of these within your test code it'll help point you to which ones need attention and so dynamical locators just d-minus so I'm what I mean by that are things like user account specific information or things are tied to page render so if you're thinking about like if it's a a lot of data on the page it's pulled from an account so maybe it's just like if you're purchasing if you're if you're doing some kind of walk through a financial account and you want to update information about purchase history or something it would list potentially IDs on the page a common pattern is IDs on the page that are have a number in them and so this is typically either tied to something like a user's ID in the system or it's actually just a collection of numbers that just change every time of page loads so either it's tied specifically to the user that you're testing with and so if you use a different if you take this exact same workflow and use a different user account it would have a different ID because that ID would be changed to reflect the idea of the new user or you have something like a bunch of elements on the page they're just random numbers that are getting added for the IDs and so every time we refresh the page it would change so these would be obviously horrible locators because they would create very brittle tests that would not be reliable over time and so so that just gets a d-minus anytime you you basically the easy way would be if it's not if there is an ID and it contains numbers then it likely is it an dynamic locator I see very it's very seldom where I see IVs that contain numbers that are not dynamic so things that are tied to page layout like really tightly tied like the horrible locators I showed you for doing the benchmark data those just get a D those are absolutely guaranteed to at some point in the reasonable future cause your test to fail for no good reason and so a simple way to think about this would be XPath or CSS selectors if they had contained three or more slashes for XPath or greater than symbols for CSS selectors then they are likely very tightly coupled to the page and so those are very brittle and so you should be able to basically take a glance if you see more than three slashes oh that's a be more than three greater than symbols oh that's a B text is the next one text on the page guys with in selenium you can actually do a lookup on the text of a link or a partial link lookup and anytime that you're dealing with looking up and verifying even copy on the page is another another case which kind of speaks more to your assertions but purely on locators anytime you're doing a text look at text based locator like link text it's a see and so that's basically it's very straightforward to see it it's very I mean there's times where you might need it though but even still there ideally should be an ID or some other way to interact with an element rather than just using the link text because copy could so easily be changed by somebody that's not a developer potentially you know if it's something like I go to the CMS and it's like the link text is controlled through the WYSIWYG editor like the page could change your test will break and there's just out-of-band of any development work efforts and it's just noise it's just unnecessary changes that have to happen in your test code and so when a test fails it's just it's just a false positive so let's try to avoid those great those with a C and then we get kind of into the more reasonable locators so ones that require reasonable traversal not tightly tied to page layout so things that are reasonable with their traversal of CSS an XPath get a B and so I don't have a specific concrete example but basically the way I think about it is if you have a if you have something on the page you want to interact with and there is a parent and a child is within if it's just a specific like widget on a page basically if you can actually scoped to the top of that widget and then scope from there into a child element within it so if the child element doesn't have a locator this unique that you could use then this would be a reasonable way to do that you can scope to actually I do have a reasonably okay example so I have there's a login form I was testing recently where noodle there's a submit button but there's no helpful ID or class on it but it's part of the login form and so when there's an ID on the login form so I could actually traverse from the IV of the login form into the button and so that way I know that I'm only going to interact with the button that's part of the login form and so that's an example of reasonable traversal and then using semantic name is pretty reliable typically at some I'd say it's potentially be plus worthy and so in put labels on on forms is a classic example of this where there's like a name attribute name equals username for login field example you know that kind of information generally doesn't change for performs that kind of semantic information denotes the actual domain information that and the behavior that supposed to happen who's supposed to interact with it and so those are generally pretty good reliable options to use and then the gold standard would be a semantic ID and so a for that and those are of course ones that are unique descriptive and unlikely to change so IVs that don't contain numbers basically that denote something helpful and informative about the thing that you're trying to interact with so for in going back to a login form example if the username input field has an idea of user name then you could just do ID equals username and you're off to the races um and so if you you know or if the or if there was a yeah so basically that's it like an ID that just has some sort of helpful name about what it is you're trying to accomplish and so there are some front of developers that feel like this kind of can clutter up the mark-up of a page but if if you don't have them then you really can't do any sort of test information and so that's where a conversation really needs to happen once you can kind of get mark and identify all the locators that you're using apply this rubric see what see where your tests stack up in terms of locators that you're using and then have a conversation with the front-end developers about these are the kinds of things that we need to make this page more testable and then once you do not you can redo this this grading rubric and then see how you've improved so so again I'll just leave this up here for a brief minute before I move on okay so there's one thing I didn't mention which was ideally as I mentioned this would be unique locator should be unique and so all of them really should be and if not then you should deduct two full letter grades because so in the in the case when I described reasonable traversal if you didn't do reasonable traversal to interact with a button that's part of a form and you just said I want to just grab this button it's the only one on the page if a second but if actually there were two buttons on the page then slowly would find the first one and interact with it but it wouldn't be unique there would be two and if the order on the page changed where this second button instead got moved up in the hierarchy to above where the button you're interacting with was then your test would fail non-unique locators have the have a great potential for causing brittleness in your tests and they can be inadvertent because it's just like the second line item here tied to page layout something is tightly tied to page layout when the page order changes causes your test to break same thing happens for non non unique locators which can happen regardless of which locator that you're using in terms of this grading scheme so you could have a look at everything it's pretty good but it could also be non unique and so then you'd really want to strive to maybe make it a little bit more unique so that M so that it you know it could be more reliable over time and so this is kind of the final gotcha as far as locators are concerned so what is your testing GPA so the way I think about it is once you've gone through with your grading rubrics for each year your tests your page objects in your locators then you can kind of tally them up for each page object each test and then kind of come up with a cumulative grade whereas also when you drill into it you can see what each test in each page object in each locator has scored but the way I do it is I weight it based on tests page objects and locators so tests in this in a cumulative GPA would be 20% of the grade point average and then page objects would be 30% and locators would be 50% because they're you know without locators you have nothing and they move the needle the most in terms of whether or not your tests are reliable um introduced in terms of other interacting with the page so so that's how once you tally things up and go through the rubric on your own this this will help you in terms of how to actually put that all together into a final grade and so and so I'll just leave before we start taking questions there's a couple of things I wanted to mention and one of them is there is a slime comp coming up we just had one in June in Bangalore India and we're doing a second one this year and it's coming up really soon it's kind of up in November and yeah we just published the all of these sessions we've accepted all speakers and we started putting together the program and we're really really excited and really proud of the program that we put together for this year for this fall and so tickets are selling out quick and so if you're interested in attending then be sure to check out slime Kampf that code at UK there's still a couple of workshop sessions that are available some of them sold out but there's um believe the getting started and the committers workshops are still open and the and the program can be viewed on the website in terms of who's been accepted and also what talks they're giving and and so right now the general tickets are still there's still some general tickets available but once they sell out then the price goes up because we're ending up into the last the coma last chance tickets so the final block of tickets so if you want to if you were thinking about attending or interests in checking this attitude and be sure to move quick to buy your ticket before the price goes up and then the other thing I want to mention is that Apple tools the gracious organizers of this webinar they have a an ongoing kind of recurring webinar that they do it's a 45-minute session and basically the gist is that it shows you kind of the gist of automated visual testing and how to get it incorporated into your existing test practice and so you could take your existing selenium tests and incorporate automated visual testing which is something I do in my tests and I think it's it's honestly it's interesting to see the industry having evolved over the last couple of years automated visual testing is something that I think everybody should be paying attention to and I think most people are starting to pay attention to it quite seriously and now Appleton's Eyes has I think the best the best tool out there for automate visual testing there's some great open source tools out there too but apples eyes hands down is an impressive offering and so for 45-minute session you can learn how to basically become a master of automated visual testing and there's so many powerful benefits that it just takes your your existing slowing tests and and helps you automate something that most people just don't even know that they can automate which is basically checking the visual rendering of a page using computer vision and machine learning and just all this really cool stuff and so basically I always tell people that the future is here and for a few lines of code you too can be in the future and so check out this webinar there's a bunch of different timeslots go to this link to find it and so you can sign up pick us a lot and then it's worth checking out so and so that's it so we now will start taking some questions ready yeah I think we are actually painfully out of time but if people you know whoever can stay with us for a bit longer we'll enjoy some answers to some of the questions we have a lot of them and I'll start with the first one I'm going to merge two questions here we got a lot of questions about conditionals so when and where should these conditionals be put and the second one was if conditionals are not a good practice how would you use a model test for example log in for various environments where only the URL is different okay okay and so so conditionals ideally all the all the logic that you end up with in your tests should be in your page objects so if you end up with conditionals in your tests it's worth seeing if they can be moved from your test into your page objects and so that ended and so that's really the kind of this big the reasonable place for it and logic that has to do with different interactions on the page um but typically I I wager that if you put enough thought into it you could actually find a way to write code that doesn't require conditionals but that's that might be a separate topic entirely but but for argument's sake the simplest thing would be move it out of your test and in your page objects but if you're dealing with things like the URL is constantly changing basically if you want to run the same test on different environments and the URL is changing then I would there's a couple different ways to handle that and one way would be to make it so you can pass in a you can make it so there's a base URL for your test code so that it all leverages this this base URL which would be just the environment you're using and then your page objects would just use a URL path and so at runtime you can specify a different base URL and have the test run on a different environment if you wanted to just have a test actually run through our set of tests run through and do the same thing but on different environments just have one shot do it then you can do what's called a parameterised test and have a collection of URLs and then pass those into your page object potentially and so there's a couple different ways to scratch that itch but that would be how I would structure it I would start with is there a way to basically put the burden somewhere else either through the actual test harness that I'm writing or parameterizing the test okay next question is it okay to have UI tests testing the same thing as API tests UI test testing the same thing as API tests so um I think that if you can test if the business logic exists in it not in the UI like if a business if the business logic is in the API then I would say test through the API and and selenium isn't the best tool for the job but I would I would still I would wager that you'd want a couple of tests through the front end so that you could at least make sure the broad-strokes have an actual user stepping through the system exist but to get as granular as testing a specific thing that you could already test to the API I would say that's not the best strategy API tests are orders of magnitude faster than selenium and the end API HTTP libraries are the right tool for the job selenium is really the best tool for the job one there's a user specific behavior you want to test your tho through a browser so if you can test this elsewhere I would encourage people to do that okay the next question is actually from from one of our customers when doing visual tests with applitools eyes should applicants eyes be integrated in the page object level or in the test level OS interesting question um I think you could do you could do either I mean it really depends on how you structure things typically i am i think when you're just getting started you could very easily just do it in the test level and then as you start to have things be more robust you could move things into the page objects because there's specific check points where you would want to take do a check performance call the check window so you take a snapshot of page which you would then compare to a baseline image a snapshot that was previously taken for the same page at a different time and then it would compare those two together there's times where you definitely want to have tight control about when those check windows are happening and if and and if the flow is happening for the user in the page object then you really want to start moving things up into the page objects okay another question is what is the ideal score range well it's just like when you're in school right it's like what's the what's the right grade for you I think that I mean it's there's no perfect grade other than getting an A across everything that's I mean those are the gold standards but the reality is that we're you're constantly your hands are tied because the application you're testing isn't built for testability and so really this is meant as as a as a helpful information radiator for you to figure out where where you need to improve both in terms of the tests and tests could you have as well as what led to the decisions as to why things are like they are and if it points you to well the application is just hard right test for then that leads you to a conversation with the developers to make it more testable which then improves the test code so if you know ideally it would be you know the higher the grade the better and anything honestly anything like anything below a B is really kind of potentially just a brittle hard to maintain you know or to on a slippery slope to be coming that way so so I would say a mere site is high and just you know strive to to hit as high a score as you can but it's probably not possible to get a perfect score unless you're building something brand new from scratch and that all with stuff in mind okay just there are a lot of questions so I'm just browsing through them next question how do we score page objects I mean just like I showed with the rubric you take a page object and then you you you start with 100 and then you deduct based on the things that I list as infractions essentially and then you fight you end up with with a score for that page object and then you proceed and you could do it again and so you just take that rubric and apply it to each page object that you have okay and I think and I think the last one that we're going to have because we are as I said painfully out of time can we have multiple assertions in a single test this is a question that comes up a lot and I think that um I mean the classic answer is always it depends but I think that the reason behind why that question is typically asked is we just went through all this expensive time to to get the browser set up to get to get to the correct place with the correct user and there's multiple things we want to check and so I think if you're testing something for testing multiple things potentially it's tough to say that because what can happen is that you could end up with you could test say there's three assertions and the first one passes the second one fails the third one never gets run so that's an issue where you end up create you know you don't know if that last thing was checked or not and what if that's the most crucial thing and so there's there's two different ways to think about it you could potentially find a way to write a single assertion that checks three things and and and so for example if you have three things on the like three if you're testing for broken images which is the example I tend to show people um if you know if you could find which images on the page are broken and store the URLs to those images or the names of those images in a collection you could compare that collection to an empty collection and then when when you run a single assertion to compare the two it would say oh these three images were broken like as I expected nothing I got these three whereas you could just say I want to test these three images so that's a much cleaner way to do it a little more sophisticated and probably might not work for every case the other option would be to write separate tests for each assertion that you want to do and run them in parallel which arguably would be the same amount of performance but has more overhead in terms of setup and all of that so so the short answer is yes and and see where it gets you and and just if there's if each if each thing you're asserting is of reasonable importance like they're you know now one thing is more important than the other two then I would say that's probably okay but if there's something that's absolutely crucial then it probably should be on its own test okay Dave we are we are over time already I think almost 15 minutes over on overtime so I think we're going to cut off at this point I just wanted to mention again that the recording will be sent tomorrow to everyone and we're going to leave the current slide up for another minute for everyone that still hasn't copied the information and won't or wants the links so they thank you so much for this in-depth session and thank you everyone for joining us today and I hope to see with our next event yeah thanks everybody

Channel: Applitools: Visual AI Powered Test Automation

Views: 2,753

Rating: 5 out of 5

Keywords: Selenium, Selenium Testing, Test Automation, Dave Haeffner, Software Testing, Page Objects, Implicit and/or Explicit Waits, Test Code Quality, Agile Testing, Webinar, Applitools, UI Testing, Visual AI, Automated UI Testing, Functional UI Testing, Visual Testing, Application Visual Management

Id: IjGDxXCaDnE

Length: 73min 31sec (4411 seconds)

Published: Thu Sep 29 2016