How to scrape Google search suggestion using puppeteer?

all right guys in this video i'm going to show you how you can automatically pass the keyword suggested by google using puppeteer so let's say you are typing for some keyword and you can see that google automatically suggests other keywords based on the primary keyword and if we look at the network response whenever you type something here google sends a request to its server and a response is received and this response is not in the form of json so it's not easy to pass this directly from the response that we receive for this request so we will be following other approach in the popular we can easily intercept the response and whenever we get the response and we will pass all the text inside this list elements so let's go to that code editor and make this happen so in the script i already imported the pivotal library and in the iffy function we are creating a browser with the option headless false so this will launch a physical browser next we are creating a new page using browser.new page api and we are navigating to google.com so very first thing that we need to do is we need to type in the input area and the selector for that is if you use the inspect element we can say the selector for this element is name equal to q that will not change whenever you visit this page so let's say weight page dot type and first argument to type is the selector so we will say star so this will match any element and we will type name equal to q next you need to type the text so text that we are going to type is puppy tier get and we are going to type this slowly as a normal human being would do with a delay of 500 milliseconds so once the user is typing we will be receiving the response from the google server so we need to now pass that response as we have seen google make a search query based on this url this portion of the query remains same each and every time so we'll make use of this portion to intercept the network request so whenever we receive the response for this particular type of network request we will pass the data so we will say page dot on response and within this response function we are going to create a sync function that will handle the response and in this async function we are going to check if response dot url dot includes now it should include this particular keyword so we will wrap that in single quotes so if it includes that then inside this if condition we are going to say con suggestions next we need to use page dot evaluate method here and within this page dot evaluate we are going to pass anonymous function and within this anonymous function we need to pass all the list items as we can see here if we inspect individual list item we can see this is the unordered list having a role of list box and within this list we have individual list items however this list item contain other elements like you can see search icons here and there is a some other hidden elements here so we need to filter out that so to get that text content only so we need to further go down in this particular div and select this child development so our selector will be something like ul having a role of list box and within this list box we are going to target the li element and within the alloy element we will be targeting this particular element having a class of spl1 so let's construct that so in the base dot evaluate method we are going to say array dot from we are using edit dot from here because document.queryselector returns the nodes that are not written however when you convert that to array it will be easy for us to iterate over them so we will say document.queryselectorall and in this we are going to pass the selector so we will say ul and ul should be having a role of this box and once the ul is selected we are going to select the individual list item and within that list item we are going to select the text content that is having the class of spl1 so once we have the list item we need to iterate over them and get the text content so in the array.form we can pass the second argument which will iterate over each and every element that it finds so we will say element dot text content all right so we have all the suggestions now however uh the suggestions could be duplicate so while storing the suggestions we need to make sure they are unique so to create the unique suggestions we are going to use set data structure so we will say unique equal to new set and we will be pushing each and every suggestion to this set so if we come back to this if condition we will say suggestions or for each remember suggestions are individual list items that are having the text so we will be iterating over them and pushing them to the unique set so we will say for each we want to add it to unique list uniques.add s so what this will do is it will iterate over each and every suggestion and put them in the set so once the operation is complete you will be having the set of unique keyword items that were suggested by google so here we will say print all unique keywords and i am going to console.log them however based on your requirement you can push them into database or store them in text file that's up to you so let's try out our script we will open a terminal and type node index.js this will open the browser type puppeteer space get as you can see on the terminal now we have complete list of the keyword that was suggested by google and that's how you can parse the network response using pupiter and fetch the keyword suggestions from google if you enjoyed this video please smash that like button and subscribe to my channel make sure you turn on the bell notifications so you get latest updates whenever i post a new video see you in next one

Channel: Gulshan Saini

Views: 409

Keywords: scrape google search suggestions, scrape google autocomplete, scrape google search results puppeteer, scrape google search results, scrape google autocomplete using puppeteer, youtube autocomplete keyword scraper, puppeteer parse network response, puppeteer intercept network response, google suggest keyword scraper, puppeteer filter network request, puppeteer automation tutorials, puppeteer tutorials, gulshan saini

Id: 94yQ2g8QyYc

Length: 6min 42sec (402 seconds)

Published: Sat Jan 30 2021