Web Scraping With ChatGPT Code Interpreter: Mind-Blowing !!

the old way of web scraping looks like this the new way of web scraping looks like this it's so much easier so much faster and you don't need much of technical knowledge to do this I'm going to show you three different examples that all use a unique approach all using chat EBT code interpreter or now Advanced Data analysis so let me show you step number one go to the website you want to scrape I'm going to show you three different examples today but let's begin with amazon.com and I've searched headphones here now imagine you can just get the entire name the price of every single item here very simply well the first thing you need to do is click on control s and click on save as type and HTML only now just go to step number two which is to click on the Advanced Data analysis which was code interpreter before and then we're going to drag the HTML file in here now you could simply say scrape this website and put it into CSV but you will get all of these different things that you may not want so that's where we need a special prompt scrape this website into a CSV file where we only need the following items let's go back to Amazon again and now what we need to do is hover over the title of the item then right click and click on inspect in the inspect tab you will see that this is the span that we actually want to copy so we're just going to click on that copy and paste that right in here then we're going to go over the price click inspect once again and you'll see the class right here where you copy and paste that in as well now you can see that the problem was it only has 17 and it's actually 99 so we need more information so we're going to copy both of these so you can see that we need to move up the folders and here is the actual one that we want so we're actually going to copy this element and paste that right in here and lastly end it with do this for all items if there are missing elements make this null data we don't have this problem here where there's always a title and a price but sometimes they pop up a little Carousel item or sometimes the price is missing kind of like what you see on this page so what it actually would do is copy this price here and put it on top of this and we don't want that now let's send the message there are different ways to do this that I'm going to show you in a minute but let's click on the download CSV file and we just scraped all of the names and the price of the items pretty cool right and now a word from our sponsors if you love chat gbt but not as a Chrome extension what are you doing let me show you one of the best ones it's called Hicks Ai and you just add it to Chrome for free in Google Docs just write slash slash now you can ask anything from brainstorming to outline to social media post just click on enter I can click on done continue writing shorter longer and you just did it inside of docs or go to Google and search 10 lessons from Warren Buffett and you will see that it will answer in the sidebar as well so you don't have to go through all these different links to get the answer you want all with relevant sources even if you're on Twitter you can just write double slash and ask AI anything select the tone encouraging maybe and generate click the link in the description to get instant access thank you so much to Hicks for sponsoring this section of the video the second way of doing this is for example go to Twitter and let's go to lists and I'm going to choose one of my lists of my favorite AI people and let's start scrolling down so we load more of these tweets and yeah stop when you're ready now if we do Ctrl s on this save the file and then try to open that file again you will see hmm this page doesn't exist try searching for something else which means we don't have all the data we want and this will happen on most JavaScript websites that has a dynamic page that you know keeps refreshing but there is a workaround search for single file Chrome extension and download this Chrome extension now just click on the plugin icon now it's going to take a little bit of time to get all the content and just like that if we click on this one you can see that we got all the data we need so let's drag that file into chat gbt it's a little bit bigger 20 megabytes let's use the problem put all these tweets into a CSV file with the following items for every tweet and once again let's hover over the name right click copy element write name and paste that in here and once again hover over the Tweet inspect right click copy element just write name and paste that element let's write tweet text back again hover over it inspect and let's get the likes as well over over the likes inspect and copy and paste now let's end it if any of the elements are blank make it an old data and let's send the message let's check the result download the CSV file and it seems like it didn't work out so well so let's give it more instructions just modifying the initial prompt let's check it again all of the tweets scrape right there this is more impressive than seeing Barbie and Oppenheimer at the same time but there is one more example let's go for example to TechCrunch or any news website you like I'm gonna search for AI and here we go all the Articles they wrote for the last 24 hours or so control s drag that file in and there's a little trick here once again hover over in Spec copy element and here you can see that we get the entire link of the article as well as the title so we're gonna write Link in the top then we're going to split this paragraph in half and go title but that wasn't the trick Also let's get the date that it was published inspect copy again added as date published now what's cool with code interpreter is that you can do multiple pages so now let's go back to TechCrunch let's click next and let's save that file with a separate name add it into chat GPT again click on the next page different name upload that one I don't even know how many files we can add four five six okay that's enough send message download the CSV file and they're all here the date links and everything if this helped you give it a like comment down below also to let me know what you think also if you want to know more ways to automate your work go in the link in the description to my patreon thank you so much for watching I'll see you in the next one peace

Channel: AI Andy

Views: 28,947

Keywords: web scraping, web scraping with code interpteter, chatgpt web scraping, web scraping amazon product, web scraping amazon, web scraping tutorial, chatgpt code interpreter, chatgpt code interpreter plugin, chatgpt 4, data engineer, portfolio project, computer science, data nerd, data scientist, chatgpt, code interpreter chatgpt, code interpreter chatgpt, code interpreter chatgpt data analysis, code interpreter web scraping

Id: wlY7K_ktAHw

Length: 6min 0sec (360 seconds)

Published: Sat Sep 09 2023