How to Scrape Infinite Scroll Sites with Power Automate Desktop

doing infinite scrolling with power automate desktop is straightforward here I have a page where I need to scroll because I need to extract all these reviews here but if I scroll down more reviews will appear so I'll create a solution that first Scrolls all the way to the button and then extract all these reviews let's do this together first I'll refresh the page so we're back at the start I put a u l in the course description down below so you can do this with me that will be a URL to this page let's go to Power toate desktop first of all I want to create a browser instance I'll do that with a launch new I'll pick Edge because I opened the page in Edge so I'll drag it in but fine you can do it in Chrome or Firefox it will be just as a good solution I could open up the URL but since I already have the browser tap open I'll just click this drop down attach to running instance we will choose by title because then I can click this tab title drop down here and The Alchemist will appear this is by the way a Copenhagen restaurant we will produce a variable called browser which we can refer to later in this flow whenever we need to automate this tab which we will I'll click save then the next step let's just open up the browser I want to make sure that I'm in this sidebar here and not in the map once the page loads or for example here how can I do that I will just like a human do I will click here because then I can send an in Click and where should I click I will find an element that's always there and not having a link so for example I could not click Peter because Peter could be Anders next time or I will not click this t- more sensors because that's a link so the state of the sidebar changes but this reviews are verified I might could click this because Google will rarely change this so it'll be very stable and it will always be there whether we have The Alchemist Noma or another D restaurant or Foreign Restaurant so let's do that click and then we can start the solution to make a click on any element I'll find a click link on web page and drag it in you can see now we work in the browser instance which is fine click this drop down now we will create a UI element let me drag it over here this UI element picker opens now you can see I can pick all of these different UI elements we agreed that we wanted to pick the reviews aren verified so here I will press control on my keyboard once this is red click with the mouse we have created our first UI element we'll click save whenever we create a UI element we always want to inspect it so I go up to the right corner here click it and here you can see it first of all the name diff reviews aren't verified that's a great name we will not change that but let's double click it here is the address of this UI element what we want to look for is that we don't want to to be it unique that could be weird looking ideas that will only work with The Alchemist and not the next company so first of all that is the child element that's called diff with an attribute called text reviews unverified this will be the same over and over but the parent here or the great parent that will be diff ID equals this could change this ID and then this select would not work so let's remove this and see if we can replace it with something else more stable to find the first element I go over to this element Tre and then I find the first element that is ticked that will be this five so if I click that you can find the ID here please do it with me you will learn so much more then I untick it now this might not be unique enough we cannot find something un unique here for this reviews unen verified but if I go one back for example to the four I can see there's an ID equal content container I believe that this ID will always be there and again it's just believe because practice make perfect I created a lot of these selectors so if I click this you can now see that we added a div element with an attribute at a value called ID and content container now we clicked then we can do our infinite scrolling what will we do what I want to do here is that before I do anything I want to count the number of reviews so that will be 1 two three probably around eight once that is counted so that will be eight then I will send an end click because an end click will scroll to the bottom and load more then I will do another count that could be 18 then I will say if these two numbers eight and 18 in this in this occasion is not the same then I will do it once more I will count the number of reviews send an end click count the number of reviews again after the end click are these two not equal to each other uh no that will be different again because I scroll down more appeared then I will repeat it once I'm at the button these two numbers will be the same because I cannot load more reviews and then my flow should stop and continue to the next state which will be web extraction let's create that I'll refresh the page up here so now we have it at start again it will make our lives a little bit easier that's always also how the page looks once we load our flow so first of all if we want to do something over and over as long as a certain condition is meth we can find a loop condition and drag it in a loop condition just does exactly that it runs over and over as long as a certain condition is meth here it will be the count of reviews before the unclick if that is not equal to the count of RWS after the end click then it will run over and over we don't have the variables here so we will wait a bit filling it in I'll click save it's fine it will produce an error we will fill it in in a little while first let's go up to actions here then find and extract data from web page here so now we're going to extract this these names that could be eight to activate our page I go to a browser this live web H will open I'll move it over here now we want to find the first one so I'll just extract a name because there's always a name to these reviews so find something unique I will right click on this Peter pako diff extract element value take the Peter you can see it over here in the extraction preview then I will go to the next one in my case it's Thomas I will right click extract element value picked out now you can see it in the preview it extracts a eight names that will be the first eight reviews and if I let me just make sure I click over here I send an end click now you can see more loads if I click refresh here you can see our web scraping works because now we have 18 and I can do it click over here say end more can do another refresh now we have 28 so we can do the scrolling this works but to make sure let's just refresh it again now we at the beginning but we this works I can click finish I also want to give it a better name than data for web page so here I can say amount of reviews before end click always name your variables by the values they represent then it will be much more easy for you to U find what I want to do here is to click save now we want to send the end click to do so I simply just find a sent keys and drag it in after the extract data from web page we will click the insert special keys and in MK here you will find the end click click it here you can see you could also have written curly brackets end but it's easier to find in these drop downs I believe I'll click save then we will repeat this and since it's the same elements in the structured data I could just click this contrl + C to copy it click the end that's because I need to click the action where I want my new action to be pasted in so I'll say contrl V now we have a copy of this it will be the same extraction but I just need to change before to after I'll double click to open it go down here in the variables produced and find an F so now I do these two things then I just need to fix my Loop condition double click to open it so here I will say as long as these two aren't equal to each other then I'll try once more I will have the amount of reviews before end click not equal to click down here click the X say after and click I'll click save two things that we can fix that will be the Run delay because we're doing some end clicks so I can change this run delay to one this is just whenever we are here in the editor there will be a little delay between these actions that is nice because then when we develop there will be a much longer delay of 0.1 second um it will not happen in production so we will just fix it here one other things is that when we start this these two variables amount of reviews before end click and amount of reviews after end click they'll be equal to each other because they will have no value and then we reach this Loop condition then this condition is this true well no they're not different from each other then it will just go down here to the end and end it so let's give for example this beforeand click a value so we know that it will at least Loop once so here I will have a set variable and let us click it click thex amount of reviews beforeand click and let's give it just the value zero and click save it will get overridden as soon as we get here because now we will run it they're different from each other so once this is done we know that we are at the button let's just create the last part so here I'll find and extract data from web page and that will be after the loop condition that is very important now I can take my previews out so here I'll will activate this again then I just need to pick what data do I want well I could use the Peter so again I will right click extract element value and take the Peter I'll also have this one name here so I'll have the Thomas extract boom power toate desktop takes these names I also wanted the reviews I know I could click more here that will be the next part of the exercise so for now I could either click span or div just take the in one I will take span right click extract element value and we have the first part of these reviews from all of them this will of course works because even um if we have 470 reviews which we have today then I click finish and let's just instead of a variable let's open up an Excel spreadsheet click save now let's go back to the page put it in the initial condition we can start this and please I can click here if you like these kind of videos please give it a thumbs up because that will help me and my channel so much I can make many more videos for you of course only do it if you like it now you can see that we are scrolling down we are scrolling down all the way to the button we are we will have 470 rows in our Excel book in a little while so I think I will go up pick some coffee I'll come back fast forward the video a few moments later that's it our automation completed first of all it opened up this Excel and by the way no more coffee so I had to grab a glass of water first of all we can see our extractions here we have a lot of extractions we have 465 uh written reviews some of them were not uh reviews that's because the old ones didn't carry any reviews so that looks fine I can close my Excel or choose to save it then I can also go here and can see that I scrolled all the way to the button that's how you do infinite scrolling in power automate desktop your next video is right here go click it you will learn so much more power toate desktop see you

Channel: Anders Jensen

Views: 2,593

Keywords: Power Automate Desktop, automate infinite scrolling, web scraping tutorial, infinite scrolling websites, automate web scraping, Power Automate web scraping, hands-on Power Automate tutorial, step-by-step web scraping, beginner web scraping guide, Microsoft Power Automate, web automation, data scraping, automate browsing, web scraping techniques, scraping infinite scroll sites, Anders Jensen

Id: CYz6itPwdTU

Length: 13min 58sec (838 seconds)

Published: Mon May 20 2024