Easy Web Scraping in Python using Pandas for Data Science

have you ever wanted to scrape data from websites in Python there are packages available such as URL Lib and beautifulsoup but wouldn't it be better if there's a much simpler way so in this video I'm going to show you how you can use pandas read HTML function to scrape data from websites so without further ado let's get started so the first thing that you want to do is fire up your web browser and head over to the github of the data professor and then you want to click on the code repository click on the Python repository and then scroll down find pandas read HTML for web scraping click on that and then right-click on the raw link safely ass and then save it into your computer okay and a second way is to fire up your Google collab click on open notebook click on the github tab type in data professor scroll down and pandas read HTML okay and because I have that I will open up my local version and so for this tutorial I will be clearing out the output and so if you want to follow along please do so so I'll click on the Edit on the menu bar and then clear all outputs okay and so only the input cell will be shown so that we can do this together but if you don't have access to a interactive version you can follow along using the github and so the first thing that we want to do is check out the website that we are going to scrape our data from and say that we want data from the Year 2019 so the data will be coming from the Basketball Reference comm and so this is the NBA player stats for the season of 2018 to 2019 so let's have a look what's in the table so you will be seeing all of the players okay and notice that the header is shown right here with various fields such as the rank the player name the position the age the team okay and the number of games played and etc and notice that the header will repeat itself every 20 players you see that there's 1 through 20 here and then the header is repeated and then you have 21 through 40 and then the same thing repeat every 20 players and so we're going to have to delete the subsequent header but we're going to keep only the first header okay and then we're gonna do that inside the tubular notebook okay so head back to the Jupiter notebook okay so we have two methods of doing this we can either use the URL directly as in meta 2 or we could do it programmatically as in method 1 where we break it up into the building blocks and so the building block will contain the URL component and the year so the URL component will be this line here and then the year will be replaced by the open and closing braces okay and we're going to use the format function to do that which will replace the open and closing braces with the year because the argument is year right here so we're combining it using the string which we define here which is the URL and this URL string contains the braces opening and closing and then we're going to use the format function and as argument we're going to use the year and the years 2019 and so let's run this okay and then here we get the URL and so the benefit of this first approach is that you could do it programmatically if you want to whip script several years for example 2015 to 2019 and you're gonna make a list of the year and then you're gonna make a for loop so let's run that and then you're gonna get five URL with the years that are changing here okay so you could try this out that's your homework and let me know how it goes alright so we have already defined the URL and now let's use pandas to read in the web page so first thing is to load in the pandas package so import pandas SPD and then we're gonna define a data frame variable and inside it we will be assigning the contents from the function PD read HTML and in S argument there's two part the first is the URL and then the URL will be the string which contains the URL to the website and Heather equals to zero so that we will be able to define that the first role is the Heather okay so let's run this and we see that the data is loaded into this data frame so you see that it easily loads the content from the HTML web page without using any URL Lib okay and then we're gonna do the beautification of this data frame directly using pandas so no additional libraries that are needed just plain pandas to do this so you just read in the HTML and then you're gonna beautify it using pandas okay let's continue so let's see how many tables are in the web page and so using the length function we see that there is only one table and so this is really straightforward in cases where web pages contain multiple tables you will have to determine which table do you want okay and so here we're going to select the first table and so we're gonna type in DF open bracket 0 closing bracket and so if you want to select a second table if there is a second table then we're going to type in DF open bracket 1 closing bracket okay and do the same if there is a particular table that you want to be selected in cases where there are multiple tables on the webpage so because this webpage has only one table we will use 0 which represents the first table and so it looks pretty neat it looks really nice here and notice that there are some missing values and notice that some players have multiple occurrences because they have been a part of different teams in the same year okay and so we're gonna assign the first table into the DF 2019 variable okay and so let's do some data cleaning so before I mentioned to you that the table header will repeat every 20 players so let's remove that let's remove the second and subsequent header roles because there will be more than 10 so what we need to do is define the f 2019 and in the bracket we're gonna use as argument D F 2019 dot H so we're selecting the first column age and whenever and whenever the first column is selected we're going to look for the string age because for the H column there is only numerical values and whenever we see the word age we will remove the entire row okay and so the entire row will also contain the table header okay let's do that so you see that there are all of the subsequent table header selected for this entire data frame and so let's have a look at the link how many Heather are there and there are a total of 26 Heather okay and then we're gonna use the D F 2019 drop function so the dot drop function will allow us to drop all of these roles from the data frame okay okay and now let's have a look at the dimension of the table again and now we have 708 comma 30 so let's look at the before 2019 dot shape and so before we have 734 rolls and 30 columns and so here we have 708 rolls and 30 columns okay so 26 are indeed removed and so let's do a quick exploit or a data analysis so in this tutorial we're gonna use the C board and so here we're gonna display a simple histogram using the dist plot function and the variable we will be using the points the F dot pts okay and KDE will be false because we want to retain the original frequency otherwise if KDE is true it will be the probability that will be shown here okay so I can show you if it's true then it won't be the frequency it'll be probability here so we'll set it to false because we want to have the actual frequency the count number that for each bar how many players have that much points okay so between zero and one so one point how many players there are about 40-something players having one point okay and number of players having 35 points or more will be less than 10 so we see that the majority have points between 0 and 20 and very few have points greater than 20 okay and so let's say that we want to change the bar line color because right now it's transparent so we're gonna use the dictionary function here inside the hist underscore kws so we're gonna define the edge color of the bars to be black and we're gonna make it about two which is the size of the line width okay right here and let's say that we want to change the fill color to another color so we're gonna use the color option okay so you could use two helps code so the hex code is the hash tag followed by the six alphanumerical characters and so congratulations you have successfully used pandas read HTML function to script data from websites and so to read in the web page content is simply using the read HTML function and so the subsequent process will be using pandas to pre-process the data by removing the redundant table header or looking for missing values and etc okay so as always the best way to learn data science is to do data science so please feel free to modify this notebook to script data from other websites and upload it to your github so that you can grow your data science portfolio okay so enjoy the journey thank you for watching please like subscribe and share and I'll see you in the next one but in the meantime please check out these videos

Channel: Data Professor

Views: 39,364

Rating: 4.9230771 out of 5

Keywords: pd.read_html, read_html, web scraping, webscraping, webscrape, web scrape, web scraping in pandas, pandas web scraping, pandas webscraping, parsing, web parsing, pandas, dataframe, data frame, google colab, jupyter notebook, jupyter, Data Professor, DataProfessor, data, bigdata, big data, data science, data mining, data science tutorial, data science project, AI, python, python programming, python data science, data science python, python data science project, scrape data, scraping data

Id: ooj84UP3r6M

Length: 9min 49sec (589 seconds)

Published: Mon Apr 13 2020