How to scrape INFINITE scrolling pages using Python and Selenium (2 Methods)

[Music] hello welcome to my channel and in this video i'm going to show you how you can scrape pages that have infinite scrolling using python and selenium let's get started so for this i'm going to assume that you have selenium installed if you don't have a video on that that i'm gonna link there we go there are actually two methods to do this one using javascript and the other one using keys so let's try to scrape elon musk to the page let's copy this and start importing selenium oh from selenium import webdriver then let's also import time and let's create our driver oh my selenium is stored here there we go and let's get this page there we go now the first method is using javascript so what we can do is like this execute script which is window dot 0 from the beginning of the page on the top of the page to the bottom so which is document.body scroll height there we go so this will scroll down to the page so let's try this out let's say sleep here so that the page has time to charge let's test it out there we go it charged and scrolled okay this works now we're gonna make this work for infinite pages so what we're gonna do is say previous height equals this but we're not gonna scroll we just need a document height so we're gonna say this there we go and let's do a loop here so now we're gonna execute the scrolling driver that execute the same thing window scroll to zero that document i there we go uh then we're gonna sleep so that it has time to scroll we should be enough then we're gonna get the new height of the page because once you scroll the the height will change because the page is longer so we have now the new height which will be the same as this here there we go and then let's say we we reach the last page it doesn't score anymore we have to compare if the new height which is the basic the new height of the page once you scroll is equal to the previous height which was the basically the max height of the page before you scroll if that happens we break because we're at the end of the page i'm going to show you once i execute let's say we scrolled the now here is the previous height height once we scroll the new height will be here let's say we're at the end here if the last height was here and the new height is also here then we stop okay let's also say previous height equal new height we can do this completely and let's execute this okay okay let's scroll and it continues scrolling there you go this works i don't know the limit on twitter for the scroll so this might take a while yeah let's keep it at that then you have a second method which is using keys so we don't need all of these here and you have to import from selenium that it was a web driver common keys if i remember correctly yes there we go and then what you're going to do is you have to select an item of something on the page so let's say an element basically let's say we select driver dot find element by tag and let's say we take the whole body then we can use element dot send keys and we're gonna use keys page down this will basically uh scroll the page also using keys oh it has the same result so let's do the same loop here while true here and let's add a little sleep here okay try this what did we do wrong here find element i tag oh it was maybe tag name yeah it was tag name i think it should work now okay it works it's almost the same as the previous one okay there you go these are the two way two ways well that i know of whether you can uh scroll pages infinite pages anyway if you have enjoyed this video please leave a like and comment and subscribe if you want more videos like this you bye [Music] you

Channel: Tech Path

Views: 5,036

Rating: 4.9396987 out of 5

Keywords: How to scrape infinite scrolling pages python, How to scrape infinite scrolling pages selenium, How to scrape infinite scrolling pages using selenium and python, Scrape infinite scrolling pages with selenium, Scrape infinite scrolling pages with selenium and python, How to scroll pages in Selenium and python, How to scroll pages in selenium, How to scroll pages with selenium, Python Web Scraping selenium, Selenium web scraping, Scrolling pages in selenium, Python web scraping

Id: qhJ_gMB772U

Length: 6min 54sec (414 seconds)

Published: Tue Jan 26 2021