Excel VBA Introduction Part 57.4 - Finding Web Elements in Selenium

welcome to this weizel vba tutorial in this video we'll cover how to find web elements in selenium basic the aim here is to create a reference for all the strategies you can use in selenium to locate an element on a page we'll start with a simple example to find a single element using its unique id and then do the same thing using the element's name we'll then perform a search which returns multiple elements using the element type and explain how you can loop through the collection of elements you return we'll show you how you can narrow the scope of that type of search by specifying a parent or a container element and then look at how to return multiple elements based on the class assigned to the elements before finding a list of links on the page based on the text that's actually displayed on the page itself in the second half of the video we'll provide an overview of using css selectors and the xpath language these are two very powerful techniques to create dynamic flexible selectors but they do require a bit of effort to learn so this won't be a comprehensive guide but it will be enough to give you a flavor for how to write these more complex selectors so there's quite a lot to do in this one let's get started okay to get started i've created a new excel workbook and all i've done so far is saved as a macro enabled file just a quick reminder that if you want to follow along with video you'll need to make sure that you have selenium basic installed along with web driver for whichever web browser you'd like to use if you haven't done that already we do have a video which explains how to get that set up and it explains how to install the driver for google chrome which is the web browser i'm going to use for this video while i'm in chrome i've also navigated to the main page we're going to be using for the examples for this video which is the main page of wikipedia so while i'm here i'm going to copy the url for the main page of wikipedia to the clipboard by clicking into the address bar and pressing ctrl and c now that i've done that i'm going to head back to excel and then i'm going to open up the visual basic editor from the developer tab in the ribbon the first thing i need to do in here is set a reference to the selenium type library so i can do that by heading to the tools menu and choosing references i can type in the letter s to jump straight into the s section of the list scroll down a little bit and find the reference for the selenium type library once i've checked the box next to that i can click ok then i can insert a new module into the project to begin with i'm going to declare a variable at the module level which will hold a reference to my chrome driver the reason i'm declaring it at the module level is so that it doesn't immediately close down the instance of chrome at the end of the subroutine when the variable goes out of scope so i'm going to declare a private variable called cd as selenium.chrome driver i'm then going to create a new subroutine called finding elements and the first thing i'll do in here is set a reference to the um to the chrome driver so i'm going to say set cd equals new selenium dot chrome driver from there i can start the application so i can say cd.start and then i'd like to navigate to the home page or the main page of wikipedia so i'm going to say cd.get and then in some double quotes i'm just going to paste in the url that i've just copied so having done that i'm going to run that subroutine just to make sure that everything works it'll open up a new instance of chrome navigate to that page so that's looking pretty good so i'll close that down and then we're ready to get started with finding some elements on the page for the first example i'd like to try to find the input box at the top of the wikipedia page so that i can type in something to search for in order to do that i need to have some way to identify that element and depending on how the web page has been constructed i could have several choices for how i can do that one neat way to find out what options you have for identifying an element is to inspect it in the web browser you're using so in google chrome for example i can right click on that element and choose to inspect it once i've done that a panel opens on the right hand side of the page which lists out all of the elements on the page and it highlights the one that i've just selected in this pale blue color so this tells me everything there is to know about this particular element it tells me first of all inside the open angle brackets here what type of element it is this tag name indicates the type of thing so this is an input element inside there is a list of attributes so every element could have a list of attributes which describe it in more detail so here i've got a name attribute whose value is search and an id element whose value is search input and all of these values could be used to identify that particular thing on the page but which one's the best one to use is there is there a difference um does selenium prefer one over the other and it certainly does there is a an order of preference for selenium to locate elements if you're interested in reading a bit of the background about this i'll post a link to this page in the description below the video but here are the element selection strategies for selenium there are multiple ways of doing this of course and there's a little section just below that about tips on using selectors and the first paragraph of that says that if ids are available then those are the preferred method so luckily for us the search input box does indeed have an id element or an id attribute i should say it's called search input so i'm just going to double click on search input copy that to the clipboard and then head back to the visual basic editor so that we can locate that element using its id i'll start with a quick comment just to remind myself what i'm trying to do so i'll say find by id and then i'll declare a variable which will hold a reference to the element i'm trying to return so i'll say dim search input as selenium.web element now the variable isn't really necessary here but it is helpful if you'd like to do multiple things to the same element once you've found it so just to demonstrate how that works i can then say set search input equals and then cd dot find element you'll see there are lots of find element methods most of which correspond to one of the search strategies listed in the selenium documentation so um there'll be a ver an option find by method relating to each of these different options in this list here the one we're going to use if i head back to the visual basic editor very obviously is the one called find element by id i can then open up some round brackets after that and you'll see there's a single compulsory parameter there the id as a string i can in a set of double quotes paste in what i copied from the inspect panel from chrome from earlier on close a double quotes close around brackets and that will get a reference to that specific element once we've found an element we can then manipulate it by applying methods and properties to it so on the next line i can say search input dot and then there's a variety of things i could do i'm going to use the send keys method here to send some keystrokes directly to that search box so i'm going to send the keys let's let's go for selenium why not once i've done that i can run the subroutine and we should see that the new instance of chrome opens up navigates to wikipedia and then types in selenium to that text box now we can click on the search button to navigate to the results page and again in order to do that we need to know how to identify that search button so if we head back to my open instance of chrome and then find the search button i can right click on that and choose inspect so we'll find again that there are various ways i could identify this element so it's got an input tag it's got a name property or a name attribute i should say and also an id attribute so if there is an id attribute that's what we want to use let's double click on the id text search button copy that to the clipboard and then we can head back to the vb editor and create a new variable to hold a reference to that button so let's say dim search button as selenium.web element and then once i've typed in some text into the search input i can say set search button equals cd dot find element by id and then in some double quotes round brackets and double quotes i should say i can paste in the id that i've just copied close the double quotes and close the round brackets once i've captured a reference to that element i can then do something to it so i can say search button dot and this time all i'm going to do is apply the click method to it so having done all of that i can run the subroutine again and it will fire up wikipedia type in selenium into the search box and then click the search button to go to the first matching entry an alternative way to find an element by id is to use something called a by object now obviously the pi object isn't required in this case we've already successfully found our elements by id by using this specific method however sometimes the by object is required for certain techniques in selenium and this represents a nice easy way to introduce you to the basic concept so i'm going to declare a new variable in this section i'm going to say dim find by as new selenium dot by so by is the name of the class i'm using the new keyword here to automatically create a new instance of the by class whenever one is required so i don't have to explicitly say set find by equals new by just to demonstrate how this works i'm going to copy and paste a little bit of code i'm going to copy and paste the set search input equals cd dot paste that on the next line and then comment out the previous line clicking just after the dot i can then type in or press control and space bar or ctrl and j to list the intellisense and then i'm going to use the find element method so as well as having a specific find element by particular technique there's a generic find element method and if i open up a set of round brackets you can see that the first compulsory parameter here says buy as buy so it requires a reference to a by object so i'm going to pass in my find by object and then type in a full stop after that and then you'll see that in the list here there's a list of the different find strategies so i can find my class by id by name etc in this example i want to find by id and then i need to pass in the name of the id or the string for the id so again i can just copy and paste this from the previous line and then close an extra set of round brackets for the find element method i can do exactly the same thing for the search button so in fact let me just copy and paste this entire line this is probably the quickest way to do it paste it in below when i'm searching for my search button i can then comment out the previous line do a quick swap of variable names by copying and pasting and then the id likewise copy and paste so i can use the find by object multiple times in the same same code same subroutine the end result is identical so if i were to run that subroutine at this point i'll end up with a new instance of chrome and it will type in selenium and then click the go button to find the first matching entry so there's no real difference in in terms of the way it works the end result is the same but as i say the find buy or the buy class is necessary in certain situations one example of when it is necessary to use a buy object is when you want to check if an element exists before doing something to it at the moment our code assumes that the search input and search button ids will definitely exist on the page we've landed on what if that wasn't the case we can quickly test this by modifying one of the ids that we're passing in so i'll say search input zzz and then if i run the subroutine we'll find that although it lands on the wikipedia main page nothing gets typed into the input box and if i click back on the background of my vb editor we'll find we've generated a runtime error saying that it couldn't find an element with that id because it's a runtime error we could write some error handling code we could write an on error statement and then redirect the code to a different location to deal with that but there's a slightly easier way to do this i think in selenium if i click the end button to stop running that subroutine i'm going to add a couple of blank lines above my line which tries to capture a reference to the search input and then i'm going to say if not cd dot is element present what a fantastic name for the method so the id behind the is elements present as its name suggests is you can check if it exists before attempting to do something with it you'll see that if i open up the round brackets the first parameter which is required i must pass in a reference to a by object so what i can do here in fact is just copy and paste what i've put on this line type in then at the end a couple of blank lines and say end if i'll throw up a quick message box to say could not find search box or search input box or something along those lines and then exit from the subroutine to avoid the runtime error when we reach this line okay so having done that i can run the subroutine again and this time although of course the procedure won't work i haven't found the search input box i don't get presented with a runtime error i get a slightly more user-friendly message explaining why the thing hasn't worked i can use the same idea to check if the search button is present let me just reset the names or the ids that i'm checking for here so that it will actually find the search input and then i can simply copy this entire if statement and place it just before we attempt to get a reference to the search button so i can paste that in there and then modify their name i'm testing for here so i can go for search button and then let's again just add some extra text to the end of this and then change my message so i'll change this so that it says could not find search button all right so i can run the subroutine again and we'll find this time it lands on the wikipedia main page and it does indeed type in selenium into the input box but it doesn't click the button because once again that element has not been found so i can click ok and then just reset all of this back to the actual name i'm looking for and then all of that will finally work again when i run it one more time we land on the page type in selenium click the button and end up on the first matching entry so we've seen a couple of techniques to reference an element by its unique id and as we saw in the selenium documentation earlier if the id is present then that's the preferred method for locating the element on the page but that also suggests that the id might not always be present and indeed the id attribute is not required you'll often find yourself working with elements which don't have an id so we do need to consider some of these other techniques for the next example i'm going to look at the name attribute and the name attribute is usually something assigned to a form control on a web page if i head back to wikipedia and have a look at both the search input box and the search button in the inspect panel you can see that both the search input box and the search button are nested inside a form tag each of the input box and button have a name attribute so the input box is called search and the button is called go and these name attributes are usually used by the back end code to reference the data that these elements contain but we can use those in selenium as well so i'm going to head back to my vb editor and i'm just going to scroll to the section which tries to capture a reference to my search input i'll do this from scratch i'll comment out any existing lines which reference the search input and then say set search input equals cd dot find element by name so there's a dedicated method for finding an element by name of course i can then open up some round brackets and you'll see there's a single required parameter name as string so i can type in the name of the element called search close a double quotes and then close around brackets i could do the same thing using the buy object as well so for the search button let's head down a little bit i'm just going to copy this entire line and then comment the existing one out before pasting in the copy so we've done the find by dot id what i'm going to do here is say find by dot and then use the name method instead now the name method i need to pass in the the name rather than the id of course so the name of that button or the value of the name attribute of that button was called go so there's a find element by name and there's a find by dot name method having changed both of those things if i've simply run the subroutine again we'll see the exact same end results but we know that this one would have worked even if those elements did not have a unique id for the next example i'd like to find all the elements of one particular type specifically i'm interested in writing out the names of each section title on the main page if i were to right click on one of those section titles and choose to inspect it i'll find that in the elements list i've actually selected a span element now it's quite likely i'm going to encounter lots of span elements on the page some of which won't necessarily be related to the individual section titles so instead we're going to rely on the parent tag which is an h2 or header 2 element and we'll find all of the h2 elements before accessing the text nested inside them now you'll notice that this h2 element has a unique id so we could individually reference all the h2 elements by their unique ids but that gets a bit tedious especially when i don't know how many there actually are so what we'll do instead is use the tag name to refer to all the h2 elements and then loop through them so referring back to the documentation we'll see that we can indeed refer to elements by tag name so to do this let's head back to the visual basic editor and i'll start by just commenting out all of the code that we used to find the search input and search button all the way down to search button dot click so i can highlight all that code and then just comment it out by clicking the comment block button if you don't have that button available you'll find it on the edit toolbar so you can right click next to any existing menu at the top of the screen and choose to display the edit toolbar at the bottom there then i'll give myself a couple of blank lines and then add a comment find elements by tag and then i'll declare a variable which can hold multiple web elements at the same time so i'll call this dim h2 headers as selenium dot web elements so we saw a web element earlier this one's going to allow us to store multiple web elements from there i can set h2 headers to be equal to cd dot and then previously we use we were using a find element method which of course as the name suggests returns a singular element if we scroll past that list we'll find the find elements methods they work in fundamentally the same way as the find element but they return multiple objects so i'll scroll down to find elements by tag insert that and then the tag name is nice and simple it's h2 one way to check if i've found any h2 elements is to test the count property of the collection so i can say something like debug.print h2 headers dot count and then if i were to simply run the subroutine at that point i'll find that it fires up the main page of wikipedia and then when that's finished if i just close down that instance of chrome it should tell me how many elements i've actually returned so i've found nine sections with an h2 header tag we can use the count property of a web elements collection to avoid having to test if an element is present before continuing with our code unlike the previous examples we've looked at when finding a singular element which causes a runtime error if that element can't be found with the web elements collection if the elements can't be found it simply returns a collection with a count of zero so if i enter a non-existent tag h2z and then run the subroutine we'll find that the code runs it loads the webpage but i don't get any runtime errors when the when the code is running it simply returns a count of zero so what we could do here is check if h2 headers.count is equal to zero then we can print a message box or show a message box which says something like no h2 headers found and then exit from the subroutine before we add the end if statement so if i were to clear the contents of the immediate window again i'll leave this set as h2z so we're not going to find any elements of that particular type but when it runs we see a warning message a message box popping up telling us that nothing has been found and we won't proceed unnecessarily with the rest of this subroutine next i'd like to list out some useful information about each of the elements we've found one way we can do that is by referencing an element based on its index number in the collection so just before we do that let's reset the tag name we're looking for back to h2 so that we do actually find something and then below the if statement we can say something like debug.print then we can refer to the h2 headers collection and open up some round brackets where we can pass in the id number or the index number of the item we want to reference the indexing of the web elements collection is based from one so the number one represents the first element so i can then close the round brackets and type in a full stop to access the individual methods and properties of that specific element i'll refer to the text method to access the inner text of that element and then if i were to run the subroutine at that point it should write out the name of the first h2 section that it finds on the page which should be from today's featured article and that is indeed what it's found slightly more usefully we can loop through this collection by counting through the index numbers let's add an extra variable to help with that i'll declare a variable at the top of this section called n as long and then just above this debug.print statement i'm going to say for n equals 1 to the count of items in the h2 headers collection so 2h2 headers.count ok what i can then do is indent this debug.print statement and say next n and then rather than always referencing id number one i can pass in a reference to the n variable in fact i'd like to also include the value of the n variable so i'm going to concatenate this i'm going to say n and then a bit of literal text before i concatenate it with the actual inner text of that particular section i'll comment out or sorry i'll delete the the code from the immediate window at that point and then run the subroutine again and this time we should see more than one thing being printed out into the immediate window so it has written out it's written out all nine although it appears that there is no inner text for the ninth element but that's okay i don't mind about that too much the point was to show you how to loop through the collection an alternative way to loop through the collection of elements would be to use a for each loop in order to make that work we need a different variable so let's head back up to the top of this little section and say dim h2 header as a selenium dot web element so back to the singular web element rather than the plural i'm then just going to get rid of the n variable and i'll get rid of the for next loop as well just to keep things neat and tidy and i'll say for each if i can spell each h2 header in h2 headers inside here i can then say debug.print h2 header dot text so i can refer to the same methods and properties just by referring to the variable name now and then i can say next h2 header for the for each syntax it really is a case of personal preference i think whether you use a for loop or a for each loop if i just clear the contents of the immediate window one thing we do lose from this is the the number of the element we're looping over although we could have included that by just retaining the n variable and manually incrementing it inside this loop but anyway if i were to run that subroutine one more time we'll see we get the list of h2 header text written out into the immediate window again just without the number in front of it so finding elements by tag can be quite useful when you don't care about referencing an element by its unique id but it can also be quite dangerous particularly when there are lots of elements of the same type on the same page for the next example i'd like to try to return a list of significant events which happened on this day so if i scroll down the sample page in wikipedia there's a section called on this day and there's a bullet pointed list showing significant events which happened in history on the same day that we're currently on so january 19th if i were to right click somewhere in one of those bullet pointed items and choose to inspect it i'll find that i've selected something called an l i element a list item element so what happens if i try to find all the li elements on the same page and write out something useful about each one well let's find out let's head back to the visual basic editor i'm just going to comment out all of the code we've just written there for looping through the h2 headers and again i'll use my comment block button for that and then just below that a couple of blank lines and a new comment which says something like narrowing the scope which is what this section is all about so i'll declare a couple of variables to be able to loop through all the list items in the on this day section so i'll say dim i'll abbreviate it to otd list items as a selenium pokemon selenium dot web elements and then dim otd list item as selenium dot web element i can then say set otd list items equals cd dot find elements by tag scroll down a bit there we go find elements by tag and then in some double quotes and some round brackets l i closer double quotes and close around brackets before i attempt to loop over these things it's probably worthwhile discovering how many i've found so let's say debug dot print otd list items dot count i'll just clear the contents of the immediate window first and then run this subroutine again to find out how many list items there are so if we look back at the results a 218 a few more than i really want considering they're only about four or five in the list in the on this day section so we need to think about how we can limit the range of tags that we found with the li tag name one of the incredibly useful thing about all the find element and find elements methods in selenium is that as well as being able to apply them to the web driver you can also apply them to other web elements so let's have a quick look back at my wikipedia page and if i look at this list item where it's situated in the page we can see that it's nested inside a ul an unordered list element which is natural but it's also nested inside a div tag with a unique id of mp-otd while i'm here i'm going to double-click on that id and copy it to the clipboard i can then head back to the visual basic editor and i'm going to declare a new variable in this section i'll call it dim otd div as a selenium dot web element and then i'm going to set a reference to that to that object i'm going to say set otd div equals cd dot find element by id and then in some double quotes and some round brackets or round brackets and double quotes i'll paste in mp otd close the double quotes and then close around brackets now that i've done that i can apply my find elements by tag li method not to the entire web driver the entire page i can reference otd div so having done that now let's see the difference in terms of the number of list items we've returned if we run the subroutine again when we look back at the immediate window we've limited the list to just 14 this time 14 is certainly much better than 218 but it's still a few more than i was expecting if i compare this with the sample wikipedia page so i can only see five bullet points in this list and indeed if i look over at the elements panel there are only five li items listed there the problem of course is that there's more than one unordered list there's more than one ul tag in this same div tag so for example if i look at the list of deaths and births if i right click on one of those and choose to inspect them there's another unordered list nested somewhere within the same on this day div and there's another one here that lists out the different anniversaries if i right click on one of those and inspect it there's yet another ul tag with multiple li tags within it so basically we need to limit the scope of this search even more i want to only return the collection of li items which are nested inside the first ul element inside the otd div okay so to make that work we can break this down um probably a little bit more than we really need to but let's just go the whole hog this time let's say dim otd lists so this is going to return all of the ul elements inside the on this day div so i'll call this as selenium dot web elements then i'm going to declare another variable to say otd list one as selenium dot web element so this will hold a reference to just the first ul element inside the div tag so after we've captured a reference to the mp-otd div we can say set otd lists equals otd div dot find elements by tag so we're going to find all of the ul elements here so we can elements by tag round brackets double quotes ul close double quotes close round brackets we can then say set otd list 1 equals otd lists and then open up some round brackets and enter the index number of the element we're interested in which of course is number one is the first list finally for our list items we can change the the starting point from otd div to otd list one so find all the li items in otd list one having done that we can run this subroutine one more time and when it finally loads the page we can have a look back and indeed this time we do indeed return just the five elements now this was a somewhat extravagant way to achieve what we're trying to do here we can make this code a lot more concise at the slight expense of making it a little more difficult to debug but rather than writing out four separate statements to get to the list items we want i'm just going to comment out those four lines there and instead we're going to say set if i can spell set otd list items i know that my starting point is to find the uniquely id'd element mpd-otd so i'm just going to copy and paste that rather than write it out again but then rather than having a separate instruction to search within that tag i can just extend this line by adding another full stop and then finding all the ul elements in there so i can again copy and paste something from one of my previous lines so find all the ul elements in the mpd otd div it isn't all of the ul elements that i want it's just the first one so i can shortcut to the first ul element within that collection of elements by just writing the index number in round brackets at the end and then i can extend that even further by typing in another full stop and then i can find the elements by tag li so again i can copy and paste that from my previous line okay can i show the entire thing in one single screen width just about so that means that most of these variables now are redundant as well all i really need is the otd list items i'll keep the otd list item there as well because i'll need that in just a moment but i'm going to comment out to the other variables there as well so having written it out in four separate lines we can also do the exact same thing in one single instruction and if i just run the subroutine at this point we should find that the number of elements returned is once again the number five and that is exactly what we get finally in this section we could just loop through the collection of list items we've returned to write out their inner text so let's just say something simple like for each otd list item in otd list items we can add a debug.print statement to say debug.print otd list item dot text and then next otd list item and then if i just clear the contents of the immediate window at that point and then run the subroutine again we can actually write out something useful about each of the list items that we've discovered okay we can see that's appeared in there in the background already and we should have five lines one for each of the items in that bullet pointed list for the next example i'd like to select elements based on their class and to demonstrate this we'll head back to the example of listing out the section titles from the wikipedia main page if i scroll back up to the top and then right click on the first section title from today's featured article and then choose to inspect it i'll see that in the elements list i've highlighted a span element now earlier i avoided selecting the span elements i said that it's likely there's lots of span elements on the page but not all of them will be related to a section title you can see the first attribute of this selected span element is the class attribute and it has a value of mw-headline you can see that there's another span element immediately above this one which does not have the same class assigned to it so it's quite likely that this particular class is assigned only to those span elements which represent the section titles so let's double click to select and then control c to copy that class name and then we can head back to the visual basic editor i'm just going to comment out all the code about narrowing the scope from the previous example so let's highlight that and comment it all out and then add a new couple of blank lines below with a new comment find by class we can then declare a couple of variables i'll say dim headlines as selenium dot web elements and then dim headline as selenium dot web element from there we can say set headlines equals cd dot and then i want to use one of the find elements methods and i'm going to say find elements by class then in some around brackets and some double quotes i can paste in the class name that i've just copied close the double quotes and close around brackets and that will find all of the elements with that particular class assigned to them i can then write a quick loop or i could count the number of elements i've returned but i'm fairly confident that if i say for each headline in headlines i can debug.print headline.txt and then say next headline and then if i run that subroutine we should find that we print out the section titles just as we did earlier on by finding the h2 elements it's probably worthwhile just quickly mentioning one thing that the find elements by class method can't do classes are quite flexible things in web design you can assign a class to just one single element on a page or to multiple elements on the same page which is what we've just taken advantage of here you can assign the same class to elements of different types so in this example it doesn't actually matter whether all the headline objects are span elements or a mix and match of different element types you can also assign multiple classes to the same element and that is sadly one thing that this particular method can't handle if i switch back to wikipedia and just have a look at the the the td tag or the td element that sits just above the span we've selected here so td is a table data cell it's got two classes assigned to it main page bg followed by a space and then mp-bordered if i tried to copy those two class names and then tried to return a reference to those using the find elements by class method sadly when i run the subroutine at this point that won't work i'll find eventually when the page loads that in the background i've got a runtime error informing me that i cannot use compound class names now there is a way to use compound class names but sadly it's not using the find elements by class method so you'll see an example of that a little bit later on so i'm just going to click the undo button to go back to the working example and then clear the contents of the immediate window before we move on to the next example for the next example i'd like to return a reference to a link based on the text used to display that link on the page so if i head back to my wikipedia example and scroll down to the on this day section there's a couple of links in this section based on the current date so january 19 there and another one down towards the bottom of that section now i don't need to inspect these elements the whole point of this particular technique is that you can use the text that's displayed on the page to find that element so i'm going to head back to the visual basic editor and then i'll comment out the section which we used to find by class then give myself a couple of blank lines below and a new comment find by link text now as we saw it's possible that there's multiple elements with the same link text so i'm going to declare a variable called otd links as selenium.web elements as there might be more than one of those things i can then set otd links to be equal to cd dot find elements by link text and all i have to do then is type in the text that's displayed on the link so i've just discovered that that was january 19 so i can enter that as a string close the double quotes in the round brackets and then we could print out something useful about the first link that we find with that text so we could say debug.print otd links open some round brackets and refer to the first element in that collection and then we could print out let's print out the attribute which shows the uh the the reference it points to the url that it points to so there's an attribute method and in there we can pass in a reference to any attribute that that element might have so i'm going to use the href element to refer to the actual url that it links to if you're not sure that that is the correct element by oh sorry the correct attribute by the way you could inspect the element in the browser and you'll find that there is indeed an href attribute which gives you the actual url okay so having done that i can head back to my visual basic editor and then run that subroutine and we should print out in a moment if i just navigate back to my vb editor and look in the immediate window there we go there's the url printed out we could make this technique a little more dynamic by calculating the link text that we're looking for if we assume that there is always a link on the main page of wikipedia which shows this particular format for the current date so the month name followed by the number of the day of the month we could calculate that based on the current date so let's just get rid of what's in that set of parentheses there and we're going to replace that with a reference to the format function we can type in format open and close and round brackets and then i want to format the current date so inside those round brackets i can refer to the date property or the date function to return the current date and then i can apply a string to format the date so i'll type in four m's to format the the months as the full month name and then a space followed by two d's to format the number of the day of the month i can then close the double quotes and then if i were to run that subroutine again i should find essentially i get the same thing printed out at the bottom of the page in the immediate window indeed i do so at this point maybe rather than just printing that out we could follow that link so i could copy that otd links one and then simply click on the element to follow the link so one more time if i try to run that one we'll land on the wikipedia homepage and then jump to the page dedicated to the current date you can also find links based on part of the text that's displayed on them so again heading back to the wikipedia example the on this day section contains multiple links with the word january in them so i could head back to the visual basic editor and then let's just copy and paste this set otd links and then comment out the original version i'm then going to modify the method used here so i'm going to click on the find elements by link text and press ctrl and j on the keyboard to redisplay the intellisense i can then highlight find elements by partial link text and hit the tab key to have that replace the original method now i don't want to format today's date with the month and the day so i'm just going to take away the day component and format the date to show the full month name now what i'm also going to do is loop through the collection of links that i've returned so let's have another variable dim otd link as selenium.web element and then i'm just going to comment out the click method and then wrap around the debug.print statement for each loop to say for each otd link in otd links and we'll write out the url or the href attribute of each link that we find so i'll say next otd link and then just modify this code here to say otd link dot attribute href okay so having done that we can run that subroutine and this time we should find that we write out multiple hrefs for each of the links containing the word january so we've used a variety of selenium basic methods up to this point to find our web elements using specific techniques if we just refer back to the selenium documentation we've looked at referring to elements by id we've looked at using the name we've looked at using the tag the class link text and partial link text the problem with having done this is that apart from the very first one using the id none of these techniques are actually the recommended way of finding elements don't get me wrong though they're really handy things to have these specific methods built into the selenium basic library they fit into the vba syntax we essentially just pass in a string of text to a method to find what it is we're looking for but they're not fantastically dynamic or powerful the recommended technique going back to the selenium documentation again is that after the the ids if those aren't available something called a css selector is the preferred method and there is indeed a method in the selenium basic library to find elements by css now the css essentially combines all the techniques we've used so far into far more dynamic and powerful ways of finding things on a page the problem with it is it's a horrendously complicated thing to learn and i'm not going to show you absolutely everything you can do with css selectors partly because i don't know everything you can do yet however i can point you to a useful reference page on w3schools and here's a table of examples of css selectors you can see why i'm not going to be able to cover absolutely everything they can do in one single video it probably lends itself to a separate playlist which i may well get around to creating at some point but basically what i'd like to do is show you how to replicate the techniques we've looked at so far but just using the one single method in selenium basic find element by css so i'm going to head back to the vb editor i'm just going to comment out that last subroutine on that last little section that we created and then head back up to the top of this subroutine right back to where we started finding elements by id so the first thing we'll do is use a css selector to find the element whose id is search input if i just head back to the w3 schools reference page one of the first examples on here shows you how to reference an element by its id so if you precede it with a hash symbol or a pound symbol i believe you call it in the states then you can write out the id of the element and one nice thing about chrome i suspect other browsers do the same thing is that i can actually copy the unique id of an element as a css selector in the inspect panel if i head back to wikipedia and then scroll back up to the top of this page i can right click on my search input box and choose to inspect it and then when i found that element in the elements list i can right click on it again and then choose to copy and then copy selector so having done that i'm going to head back to the vb editor i'm just going to uncomment everything from where i've declared my search input box all the way down to the end of that little section where we say search button dot click i'll just uncomment all that code then i've got a little section here which is setting my search input i'm just going to comment out the last example i did where i'm finding the element by name and instead i'm going to say set search input if i can spell that correctly equals cd dot find element by css if i then open up the round brackets i can type in my css selector as a string in vba so in some double quotes i'm now just going to paste in the copied selector from chrome so i'll press ctrl v to paste that in and you can see it simply copied the unique id of that element preceded with a hash symbol or a pound symbol however you'd like to refer to it i can then close a double quotes and close around brackets and that will essentially do the exact same thing as find element by id the by class also supports using css selectors so let's just head back to wikipedia and i'm going to find my submit button in the elements panel i can right click on it and choose to copy and then copy selector again i can then go back to my visual basic editor scroll down to the section where i'm setting my search button and then i can comment that out and then i can say on the next line set search button again failing to spell it correctly set search button equals cd dot find element and then open up some round brackets and refer to my find by object so i can say find by dot and then refer to the css method in there i can open up some round brackets and some double quotes paste in what i've just copied from chrome which again is just the same the id of the object with the hash or pound symbol in front of it and then close the double quotes close around brackets for css and then close around brackets for find element as well okay so having done all that again we could replace the uh the the um find by dot id in the is element present methods as well by using the css selector but just to give that one a quick check to make sure it runs we can run the subroutine and we'll load up the main page type in selenium click the go button and end up on the first matching page now if these two elements didn't each have their own unique id but they did have a name attribute we can still use a css selector to reference them it's a little trickier than referencing them by id because there's no shortcut to the name attribute like there is for the id attribute but again using this reference page on w3schools we can work out how to do this if we scroll down a little bit we can see that we can check the value of any attribute by writing it out in a set of square brackets the attribute name and then the value we're testing for and we don't even just have to test whether it's equal to something we can find attributes containing or starting with or ending with etc so lots of examples here on this page to work out how to reference these various attribute values to make this work for our particular example i can head back to the visual basic editor and i'm going to copy and paste the line which sets the or finds the element by its id and then i'll comment that copied line out and then just change the search string here for the css selector to a set of square brackets then the name of the attribute which is called name and then the equals operator and the value i'm searching for so as we saw earlier on the name of that attribute or the name of that particular object element is called search if it's possible that there are spaces in the value of your attribute it's worthwhile wrapping up the search string in a set of single quotes and then having done that i can do something similar for the for the search button so again the name of that was go so i can copy and paste my search button set statement comment out the copied line and then replace the search string with a set of square brackets the name of the attribute which is name equals and then the name of the element which was go so once again having done all that we can run the subroutine and the end result is once again exactly the same we type in selenium click the go button and land on the first page matching that search phrase the next thing we did was try to find all the elements of a particular type using the find elements by tag method and of course we can use the tags using css selectors as well in fact it's probably the easiest thing to do with a css selector let's just head back up and comment out all the code for the find by id or by name method again so let's uh let's highlight all that and comment it all out and then i'm going to uncomment the first little part of find elements by tag so the two variables all the way down to the loop which loops through the h2 headers so i can uncomment that to use a css selector for this is incredibly simple in fact i'm just going to copy and paste this line set h2 headers then i can paste it in just below i'll comment out the original version of that and then replace the find elements by tag method by highlighting it and pressing ctrl and j and instead use find elements by css and that's all i need to change in fact when you want to reference a particular tag then it really is just the name of the tag and passed into the the css search string and again you can find that out on the uh the w3 schools page to find a particular tag somewhere here or an element um that's you simply write in the element itself so having done that back to the vb code this should loop through all the h2 headers providing there are some to find and then if i simply run that subroutine we will end up eventually with a list of h2 header text printed out in the immediate window even with a basic example such as this there are more interesting things we can do with the css selectors compared to the built-in methods such as find elements by tag so for example let's say i wanted to find multiple different types of elements in the same search string again referring back to the w3 schools reference there's a little item here that shows you can write a comma separated list of element names to find all the elements of any type that you list so back into the visual basic editor let's say i wanted to find all the h1 headers all the h2 headers and all the h3 headers these go all the way up to h6 by the way i appreciate that my variable name is no longer a very good name for this but i don't want to change it at this point so i'll just accept that that's not the best variable name and then in the debug.print statement perhaps it's worthwhile listing out what particular type of tag this has so i can say h2 header dot tag sorry tag name there we go and then comma separate that from the actual text so at this point i could run the subroutine again and this time we should print out a little more than just those nine h2 headers and there we go so we can see we've got some h3 headers quite a lot of those some without any text to display our familiar h2 headers and then an h1 header as well without any text to display css selectors start to become incredibly useful when you want to narrow the scope of the elements you're looking for so let's just comment out the find elements by tag section and then scroll down a little bit to the narrowing the scope section and then i'd like to uncomment all of this code in here so below that comment uncomment all of that and i'd also like to uncomment the extra variables and the extra set statements that i commented out earlier on and then i'm actually going to comment out that single statement that sets the list otd list items variable in one single step okay so i just like to modify these four set statements to use the css syntax instead so this first example is finding an element by id so i'm going to change that to find element by css and we already know that to find it by an id we can just insert the pound symbol or the hash symbol in front of the id finding elements by tag that's nice and straightforward we can say find elements by css and then we don't need to change anything there that will just find all of the ul elements inside that div tag which we've already captured i don't need to change this i'm going to leave this one exactly as it is and then again for the final one to find elements by tag i can say find elements by css and then just pass in the type of element as a string so at this point this will list out all of the list items in the on this day section in the first unordered list in that div so let's just run that one to prove that that one actually works at this stage and there we go so we can find those five individual items listed out in the same way we had earlier on so that's the setup but the interesting thing here is in how we can combine these different techniques into a single css selector to get straight to the otd list items that we're interested in but the great thing about css selectors is we can do this in a much more succinct way than what we've done here by chaining together a sequence of find element by id find elements by tag find elements by tag again so to make this work let's comment out these four separate set statements again and then we can add a new set statement below this one that we commented out earlier we'll say set otd otd list items equals and i'm going to start by just copying and pasting this cd.find element by css so i'm going to copy and paste that and then i'm going to change the method to find elements because we want to return all the list items so at this point we can modify this search string i'm going to say after the unique id for the div element i'm going to type in a space and then just type in the name of the tag that i'm interested in l i so that's going to return any li element inside the div element regardless of where it's located so this is going to return more than we want of course if i run that subroutine at this point it's going to return eventually there we go way more than we really wanted in fact so you can see that it's got every single list item all 14 of them from inside that div so that's not quite there but we're getting a bit closer what i could do now is insert in between the otd and the list the ul element so it's the the unordered list now if i just do that by typing in a space that's going to return any li element in any ul element inside the div tag so again if i were to run that one at this point i will find that i get way more than i want i'll end up with all 14 elements again all 14 list items so that's really no different to what we've just had previously the key to this if i just clear out the contents of the immediate window again is to use the to use various symbols to indicate the relationship between the different elements so i'm going to use a closed angle bracket symbol between these different elements so that indicates that the unique id here is the parent of an unordered list which is the parent of a list item now as it turns out if i switch back to wikipedia and i find any of the list items inside the on this day section i'll find that the only ul element that is the at the next level down from the div tag is the first one in the entire sections the one with the five list item elements i want all of the other ul elements are nested inside a further element so here in this case it's inside a div tag and then i suspect a little further down this one is inside another div tag which is again actually is nested to yet another level inside that div tag so the only unordered list which is at the level i've specified here a direct descendant or a direct child of this tag is the one we want so if i were to run that one we'll find that we end up with just the five individual list items that i initially wanted i guess the next obvious question would be what if there was more than one ul element that was a direct descendant of that div tag and the answer would be i'd get more than the five list items i really wanted in the output of this but we can use css selectors to handle that as well so let's say i wanted to get just the first ul element that was directly nested inside this particular div tag if i switch back to the w3schools reference page again i can scroll down the page to find a bunch of different modifiers we can place on these different elements so for example i could use the first of type modifier to find the first ul element inside that div tag in fact let me just copy and paste that from here and then i can paste that at the end of the ul element it won't make any difference of course in this particular example as there is only one ul element that's a direct descendant of that tag but just in case of future proofing you wanted to make sure that was always the case if i run this one again it's going to return the exact same five five items from that list if it wasn't the first element i was interested in but maybe the second or any other number then again there's a modifier we can use for that if i switch back to the selector's reference page there's something called an nth of type so i could do the exact same thing again here by using the nth of type modifier so i can just paste that on to the end there and then excuse me i need that colon to separate that then i can change the number two to the number one clear the contents of the immediate window and then once again run that to end up with the exact same list of five list items next let's have a look at how to use classes with css selectors i'm just going to comment out all of them narrowing the scope code and then scroll down to the find by class section this isn't particularly interesting or at least the starting point isn't particularly interesting so we've got the find elements by class let's just change that to find elements by css and to find a class we simply need to precede the class with a full stop so full stop mw-headline if i then run that subroutine we should find the list of section titles as we described earlier on from wikipedia and there they all are just as we saw from the previous example you can also modify the class selector to work with a specific type of element so in this example if i typed in span before the class and the full stop the end results will be the same of course because the mw-headline class is only applied to span elements in the wikipedia main page but just to demonstrate that you could limit this to a particular element type you can write them out like so one other useful thing is that the css selector allows you to reference multiple classes in the same element so again if i head back to wikipedia and find an example of that let's right click on one of my one of my section titles and choose to inspect it we'll see that somewhere just above that is a td element with two classes assigned to it main page bg and mp-bordered now we saw earlier on that this didn't work with our basic example with the find elements by class but if i copy this to the clipboard now and then head back to the visual basic editor and then just paste that in as my css selector i need to separate these with full stops so full stop main page bg and then full stop mp dash bordered with no spaces in between i appreciate that the uh the variable names no longer make any sense but just to demonstrate a point if i clear the contents of the immediate window and then i run that subroutine i'm going to end up with quite a large amount of text here as you can see but um it's basically printed out all the text from each of those uh those bordered sections or those table cell tags td tags okay so there we go the final thing we did with specific selenium basic methods was to find links using the text displayed on the page and this is sadly where the css selectors break down a little bit we don't have any access to the text part of an element on a page using css selectors for this particular example we can kind of get around that by referring to one of the attributes of the links we're looking for just to demonstrate that i'm going to comment out the headlines section there and then head back over to wikipedia and have a look at the links we were interested in so these were the uh the links with the current date displayed so i can right click on one of those and choose to inspect it so we can see that this link has a title attribute and we know that we can interrogate that the attributes of an element using css selectors we did it for the name attribute earlier on so let's have a look at how we can make that work if i uncomment the section of code that finds my otd links let's just uncomment all of that apart from the the click part there let's just uncomment that and then what i'd like to do is use the css selector to find elements whose title attribute equals january 19. so i'm just going to comment out this otd links or set otd links statement and replace it with a new one which says set otd links equals cd dot find elements sorry try that again find elements by css and then in a set of round brackets i then want to have a set of double quotes and then i could use some square brackets to contain the search phrase i'm looking for as we did earlier on for the name attribute so if i just scroll back up to the top to remind you of how that one works we have the name attribute and then equals then the name itself it's also possible using css selectors to limit the search to a particular type of element so for this one i'd like to limit it to only the link elements so i'm going to look specifically for elements whose tag is a to make that work back in the vb editor i can type in the name of the element in front of the search phrase and then have the square brackets to refer to the the attribute which is title and then the search string i'm looking for which in this case would be january 19. again i've written out the actual search phrase in a set of single quotes and it's important here because there is a space between the word january and the number 19. okay so having done that i can run the subroutine again and this one should list out all of the links which have the search text although the display text of january 19. we can of course make this more dynamic again using the format function in vba we need to be a little more careful about how we do that this time so we've got to insert the results of the format function in place of that explicit string in our search phrase so let me just start by taking away the january 19 and then i'm going to replace that with two double quote characters then position the cursor back inside those two double quotes and then type in a space and an ampersand and another space and another ampersand and another space so what we're going to do is concatenate the various parts so the first part of the string will be a open square bracket title equals followed by a single quote and the second part will be the result of the format function so i can just copy and paste that from a previous line and then the final part will be the concatenation of the single quote and the closed square bracket okay so same thing again just a quick little check that that runs and we'll end up with the same two links printed out in the immediate window for january 19. we can perform more interesting string comparisons by changing the operator after the attribute name so let's say for example i want to find all the links or all the a elements whose title attribute contains the word january i can modify my format function to return just the name of the month or name of the current month which will be january in this case of course and then to find out which operator i need to use i can have a look back on w3school's reference page and find that i could use an asterisk equals to find elements which contain a string so if i just head back to my vb code add an asterisk in front of the equals operator when i run the subroutine this time i'll return multiple links any of which could contain the word january so there we go i've got an extra one down at the bottom there and the next one up at the top as well if i wanted to limit it to those that begin with january then again i can just modify the operator that i'm using to compare the strings so here i can find the begins with which is an a correct symbol in front of the equals operator so i can change the asterisk to a correct clear the contents of the immediate window and then run the subroutine again and i'll find links whose title begins with january this time rather than just contains the word january anywhere in it okay the good news i'm sure you'll be pleased to hear is that there is only one more technique for finding elements that we're going to cover in this video the slightly bad news however is it is probably the most complicated one to learn i mean in fact before i actually even start showing you how this technique works it's worthwhile briefly having a look at the page on the selenium documentation website so the technique we're going to use is the x path method for finding elements and there is a little section in this paragraph here about tips on using selectors that um it works as well as css selectors but it's complicated and difficult to debug and although they're quite flexible they're not often performance tested by browser vendors so they can be quite slow so always bear these in mind xpath is not the recommended method if you've got ids use those if you don't have ids use css selectors and only then should you consider using xpath selectors if you can't do what you need for reference material here's how complicated it is um the world wide web consortium uh maintain the documentation for the specification of this language at the xpath language so you'll find all the previous historical articles on this page here's the current specification i'm not even going to attempt to read that or point out various parts of it i'll leave that entirely in your hands if you want a slightly more succinct tutorial for how to use xpath then google w3schools has a nice basic tutorial to get you used to the basics so i'd recommend having a quick look there again i'll stick these links in the video description so you have them to hand anyway just to go back to our example i'm going to start by commenting out this links section at the bottom and then going all the way back up to the top to the section where we're finding things by id or by name and once again i'm going to uncomment everything down to the search button dot click method and then look at how we can replace our existing techniques using the x path method instead next i'm just going to comment out the set search input line once again and replace it with a new one which will use the x path to locate the element so i'll say set search input equals cd dot find element by x path i can then open up some round brackets and i'll see that again i can pass in the x path as a string so i'll just open and close a set of double quotes and then close around brackets before i start looking at how i can find the x path of the element i want a neat way to start actually is rather than reading the documentation just head back to your web browser find the page you're interested in and inspect the element you want to select so i'm going to right click on the search input box and choose to inspect it earlier i showed you can copy the css selector by right clicking on the element in the elements panel and then choosing copy copy selector for xpath i've got two options i've got the full xpath or just xpath and just to show you what the full xpath is this is essentially like a set of directions for how to reach that specific node in the structure of the individual tags in this document so i'm going to choose copy full xpath and then head back to the vb editor and paste it in inside these double quotes you ready for this there we go so that's basically a set of directions going from the root of the page which is contained in an html tag into the body tag into the fifth div tag into the first div tag inside there into the second div tag inside there into the only div tag apparently in there into the form tag into the div tag into the first input tag fantastic it's pretty precise isn't it i would definitely end up where i needed to be if i followed those directions precisely however you can probably tell that's pretty fragile and prone to errors if any part of the structure of that main page changed then my directions don't lead me to where i need to go just to demonstrate that it does in fact work let's just get to the point i'm just going to comment out the bit that clicks the search button so we don't actually perform the search just to prove that it does indeed type in the the text selenium into that text box if i run the subroutine we'll load a new instance of chrome and it does indeed type selenium into that search box a much better approach however rather than using the full actual path to that element because it is so fragile is we can find it using its unique id so i'm just going to delete that part and then head back to my open wikipedia page right click on this input element again in the elements panel and choose copy and then choose copy x path this time i can then head back to the vb editor and paste that in inside the double quotes and that's a much more succinct way of doing things so we're essentially this asterisk here represents any type of element so it could be a div it could be a span it could be an h2 and then in the square brackets is the search phrase to find that element so it's a lot like the search phrases for the css selectors it's for this specific attribute and if it is equal to this bit of text now if i click away from that annoyingly it's going to leave me with uh with a syntax error and that's because the search phrase has been entered in a set of double quotes and as we know in vba we already use double quotes to contain strings so i can't put a double quote inside double quotes it's not interpreted correctly i've got a couple of options for changing this i can simply double up my double quotes if they're to be contained within another set of double quotes or slightly more readable i think use single quotes instead so once again we're just going to find that single search input and then we'll type in selenium into that search input text box so i'll run the subroutine and find once again we get the exact same end results finding an element by using its name attribute is basically the same syntax using the xpath technique so if i scroll back downwards a little bit and then we can uncomment all the code about the search button and then we can replace this find by dot css technique let's just comment that out in fact let's just copy and paste some of it at least everything up to the find by dot and then we can paste that in and then find by x path i can then open up some round brackets and open and close some double quotes and then close the round brackets a couple of times and then i can just paste in or or write out the x path selector for that element again a nice way to to find these out to get used to the syntax is to find the element in the web browser in the elements panel right click on it and we can choose to copy copy xpath i can then go back to the vb editor and then paste that in now of course in preference it's going to use the unique id for that element as there is one but i can modify this to refer to a different attribute so where it says at id i can change that to say at name and then i can change the oh i'll change the double quotes to single quotes as well and inside the single quotes i'll type in the name of the object which is go so having done that we can run the subroutine again and once again we're getting bored of seeing this page by now i'm sure we are we'll end up on the first match for selenium in wikipedia finding all the elements of a particular type is almost always the easiest thing to do and the xpath technique is no different to that let's just comment out the find by stuff let's comment out the search input and search button all the way down to search button dot click and then let's quickly uncomment the find elements by tag so let's see everything from the h2 headers down to the little for each loop to write out their names okay so we've got an example here already which is setting the h2 headers using the uh h1 h2 and h3 but let's just comment that out and replace that line by saying set h2 headers equals cd dot find elements by xpath the last one in the list for find elements and then open and close some round brackets open and close some double quotes inside there and then to find all the elements of a particular type i can type in two forward slashes to go from the root of the document and find all the nodes or all the elements with an h2 tag okay so having done that i can run that subroutine and it's going to write out all of the h2 elements into the immediate window so there we go h2 and then all of the actual text of those elements we can also find elements of multiple different types using the xpath technique just by separating the different search strings with a vertical bar so for example i could type in forward slash forward slash h1 and then a vertical bar to separate that from the h2 elements and then another vertical bar after that followed by forward slash forward slash h3 so it essentially acts as the or operator so if the tag is either h1 or it's h2 or it's h3 if i run the subroutine at that point getting bored of seeing these results by now maybe we should have picked different pages for the examples but here we go there's the examples of h3 titles and h2 titles and h1 titles some of which don't have any text to display next let's look at how to narrow the scope using an xpath search string i'll start by commenting out this section of finding elements by tag and then head down to the next section narrowing the scope and then just uncomment all the code in here down to the end of the for each loop i'll also just comment out these three variables for the otd div lists and list one i shan't need those i'm not going to bother showing you the individual techniques to get to these elements in fact we've already seen them how to get access to an element by its unique id and an element by its tag name so what i'm going to do is jump straight to the section which shows you how to combine the techniques into a single search string i'll start by copying part of this line and then comment out that version of the line give myself a blank line or two and then paste in what i've copied and then find elements by xpath with a couple of round brackets and a couple of double quotes okay a good place to start with this is to get the unique id of the uh the div tag with this id mp-otd so we could head back to our wikipedia sample page find that particular div tag so i can right click on the on this day and choose to inspect it and then i've got my div tag whose id is mpotd right click copy copy xpath and then back into the vb editor and paste that part in perfect next i want to find any unordered list any ul element that's a direct child of that specifically id'd element so i can type in a forward slash followed by ul and then i want to find any list item element that's a direct child of that ul element so i can type in a forward slash and then say li now i know that i can get away with this because as we saw earlier on there's only one ul element that's a direct descendant of that div tag but we can specify a unique id number or a unique uh index number for the element we're interested in in a set of square brackets after the ul element we can type in the number one to refer to the first unordered list in that uh that section now again i get the syntax error because of the double quotes that have annoyingly been copied so the double quotes that are wrapped around the id name i'm just going to replace those with single quotes and then that would be fine all right so i can run that subroutine again and we should find a list of the five li elements in the first unordered list within that specific div tag okay let's look at class names next i'm just going to comment out this section of code we've just been working with and then head down to the find by class section this one's not particularly interesting again it's it's very much just referring to an attribute of a particular type of element so i'll uncomment the find by class code and let's comment out the set headlines statement and replace that with a set headlines equals cd dot find elements by find elements by xpath and then in some round brackets and some double quotes we can enter the x path that we need so if we're going from the root of the document we can type in forward slash forward slash and then we can type in an asterisk to represent any type of element we could be more specific of course if we're looking for h2 elements then we can say h2 and then put in the search string in a set of square brackets so again i'm going to back to the idea of finding the span elements which have the the mp mw-headline class so a quick reminder of that if i head back to here and find oh there's one there already a span class mw dash headline in fact i'm just going to copy that from the from the elements panel okay so uh this one i'm going to go not with h2 i can either go with a an asterisk to represent any element or span if it's a specific span element i'm interested in and then i can say at class equals and in a set of single quotes paste in the class name that i've just copied so having done that i can run that subroutine and that will write out the names of the individual section titles from these spams just as we've seen several times already we can also use multiple class names with the same xpath search phrase so if an element has more than one class assigned to it we can let me just copy and paste this from the previous line rather than mw-headline we can paste that in and then just take away these full stop characters and replace them or replace the one that separates the two with a space so main page be g space mp dash bordered i also want to make sure that i'm not exclusively looking for span elements so i can either replace that with i believe it was a td element which had that uh that those classes assigned to them or just for um convenience i can use an asterisk to represent any kind of element if i then just clear out the contents of the immediate window and then run that subroutine again once again we'll write out all the text from the elements with those two classes assigned to them a little bit more than we really wanted again but there we go okay so the last thing we did with the previous techniques was to find links based on the text displayed on the page as it turns out the way the find elements by link text or by partial link text methods work behind the scenes is to actually construct an x path selection string so if you have a quick check on the uh the selenium documentation website it tells you that these two techniques use xpath selectors internally so we should definitely be able to construct our own version of that it's certainly possible so let's just have a quick look if we head back to the visual basic editor i'm going to comment out the find by class section and then let's uncomment the find by link text section i won't bother with the links click uh section there though or statement let you get this out all right so i'm going to comment out the set otd links statement and then we can just copy and paste part of that to get started so we can paste that in and say find elements by x path there we go in some round brackets and some double quotes we can then enter our x path string so this time we want to find all the a elements whose text contains january 19 let's say let's just fix it to a specific bit of text to begin with so i want to go from the root of the document forward slash forward slash i want to find all the a elements and specifically those which match a certain condition so in a set of square brackets i can refer to a function in xpath so i'm going to use the contains function which has a couple of parameters so in a set of round brackets the first parameter is what i'm looking in and that is the text of the element that i'm looking at so i'm going to refer to the text function open and close and round brackets to return the text of the a element followed by a comma and then the specific string i'm looking for so in some single quotes i can type in january 19. and that's essentially what happens when we use the find elements by partial link text method so having done that let's just give that one a quick whirl if we run the subroutine we should print out two links which have january 19 as their display text and indeed that's exactly what we get so this is one situation where using xpath selectors is more useful than css selectors because you can get at the text of the element you're looking at there's a whole bunch of different functions available in the xpath specification just as a quick example of that is the the documentation for version one actually of the xpath language but there's a section for string functions so we've just had a quick look at the contains function but this also starts with and various other ones as well i can't claim to be any sort of expert in using them but there's obviously a lot more power using xpath when you get into this sort of expression but i think we've pretty much covered all the basics for ways you can find elements in selenium we know now how to select stuff by id by name by tag by class by link text and partial link text and how to construct our own useful and dynamic css selectors and a little bit about xpath selectors as well i really hope you found that useful i may well reference some parts of this video in later sections of this playlist so you haven't escaped it yet i'm afraid well done for sticking with it if you're still with us thanks so much for watching see you next time

Channel: WiseOwlTutorials

Views: 11,142

Rating: 4.9652176 out of 5

Keywords: vba, visual basic for applications, excel, selenium, web scraping, scrape website, screen scraping, selenium basic, findelement, findelements, element, webelement, webelements, id, name, tag, class, linktext, partiallinktext, css selector, xpath, wise owl

Id: lr7CFZEI2YA

Length: 93min 0sec (5580 seconds)

Published: Wed Jan 20 2021