Welcome to our data analysis with Python tutorial.
My name is Santiago and I will be your instructor. This is a joint initiative between Free Code
Camp and remoter. In this tutorial, we'll explore the capabilities of Python on the
entire PI Data stack to perform data analysis, we'll learn how to read data from multiple
sources such as databases, CSV and Excel files, how to clean and transform it by applying
statistical functions and how to create beautiful visualizations will show you all the important
tools of the PI Data stack pandas, matplotlib, Seabourn and many others. This tutorial is
going to be useful both for Python beginners that want to learn how to manage data with
Python, and also traditional data analysts coming from Excel tableau, etc. You learn
how programming can power up your day to day analysis. So let's get started. Welcome to our data analysis with Python tutorial
My name is Santiago and I am an instructor@remoter.com an online Data Science Academy. This tutorial
is a result of a joint effort by remoter and Free Code Camp, and it's totally free. It
includes slides, Jupyter, notebooks and coding exercises. Let me tell you a little bit more
about remoter were an online hands on Data Science Academy. We specialize in data science,
including data analysis, programming and machine learning. We have a complete course catalog
and we're adding more content every month. If you're interested in learning data science
or data analysis, check us out. As part of this joint effort between Free Code Camp and
remoter you can get a 10% discount in your first month by using the following discount
coupon. Let's quickly review the contents of this tutorial. In the description of this
video, we have included direct links to each section, so you can jump between them. This
is the first section and we are going to discuss one is data analysis. We'll also talk about
data analysis with Python and why programming tools like Python SQL and pandas are important.
In the following section will show you a real example of data analysis using Python. So
you can see the power of it will not explain the tools in detail. It's just a quick demonstration
for you to understand what this tutorial is about. The following sections will be the
ones explaining each tool in detail, there are two more sections that I want to especially
point out. The first one is section number three Jupiter tutorial. This is not mandatory,
and you can skip it if you already know how to use Jupyter notebooks. Also the last section
Python in under 10 minutes. This is just a recap of Python. If you're coming from other
languages, you might want to take this first. If that's the case, again, you can use the
links in the video description to jump straight to it. All right now let's define what is
data analysis. I think the Wikipedia article summarizes perfectly the process of inspecting,
cleansing, transforming and modeling data with the goal of discovering useful information,
you forming conclusions and support decision making. Let's analyze this definition piece
by piece. The first part of the process of data analysis is usually tedious. It starts
by gathering the data and cleaning it and transforming it for further analysis. This
is where Python and the PI Data Tools Excel. We're going to be using pandas to read, clean
and transform our data. Modeling data means adapting real life scenarios to information
systems using inferential statistics to see if any pattern or model arise. For this we're
going to be using the statistical analysis features panelists and visualizations for
matplotlib and Seabourn. Once we have processed the data and created models out of it, we'll
try to drive conclusions from it finding interesting patterns or anomalies that might arise. The
word information here is key. We're trying to transform data into information. Our data
might be a huge list of all the purchases made in Walmart in the last year, the information
will be something like pop tarts sell better on Tuesdays. This is the final objective data
analysis we need to provide evidence of our findings, create a readable reports and dashboards
and aid other departments with the information we've gathered. Multiple actors will use your
analysis, marketing sales, accounting executives, etc. They might need to see a different view
of the same information. They might all need different reports or level of detail what
tools are available today for data analysis. We've broken these down into two main categories,
our managed tools, our close products, tools you can buy and start using right out of the
box. Excel is a good example. Tableau and luchar are probably the most popular ones
for data analysis. In the other extreme, we have what we call programming languages or
we Call them open tools. These are not sold by an individual vendor, but they are a combination
of languages open source libraries and products. Python R and Giulia are the most popular ones
in this category. Let's explore the advantages and disadvantages of them. The main advantage
of close tools like Tableau or Excel is that they are generally easy to learn. There is
a company writing documentation providing support and driving the creation of the product.
The biggest disadvantage is that the scope of the tool is limited, you can cross the
boundaries of it. In contrast, using Python and the universe of PI Data Tools gives you
amazing flexibility. Do you need to read data from a closed API using secret key authentication
for example, you can do it? Do you need to consume data directly from AWS kinases, you
can do it a programming language is the most powerful tool you can learn. Another important
advantage is a general scope of a programming language. What happens if Tableau for example,
goes out of business. Or if you just get bored from it and feel like your career is taught
you need a career change? learning how to process data, using a programming language
gives you freedom? The main disadvantage of a programming language is that it's not as
simple to learn as with a tool, you need to learn the basics of coding first, and it takes
time. Why are we choosing Python to do data analysis? Python is the best programming language
to learn to code. It's simple, intuitive, and unreadable. It includes 1000s of libraries
to do virtually anything from cryptography to IoT. Python is free and open source. That
means that there are 1000s of PI's very smart people seeing the internals of the language
under libraries. from Google to Bank of America, major institutions rely on Python every day,
which means that it's very hard for it to go away. Finally, Python has a great open
source spirit. The community is amazing, the documentation, so exhaustive, and there are
a lot of free tutorials around checkout for conferences in your area, it's very likely
that there is a local group of Python developers in your city. We couldn't be talking about
data analysis without mentioning r r is also a great programming language. We prefer Python
because it's easier to get started and more general in the libraries and tools it includes.
R has a huge library of statistical functions. And if you're in a highly technical discipline,
you should check it out. Let's quickly review the data analysis process. The process starts
by getting the data where is your data coming from? Usually it's in your own database, but
it could also come from files stored in a different format, or a web API. Once you've
collected the data, you will need to clean it. If the source of the data is your own
database, then it's probably in writing shape. If you're using more extreme sources like
web scraping, then the process will be more tedious. With your data clean, you'll now
need to rearrange and reshape the data for better analysis, transforming fields merging
tables, combining data from multiple sources, etc. The objective of this process to get
the data ready for the next step. The process of analysis involves extracting patterns from
the data that is now clean and in shape. Capturing trends or anomalies. statistical analysis
will be fundamental in this process. Finally, it's time to do something with data analysis.
If this was a data science project, we could be ready to implement machine learning models.
If we focus strictly on data analysis, we'll probably need to build reports communicate
our results, and support decision making. Let's finish by saying that in real life,
this process isn't so linear, we're usually jumping back and forth between the step and
it looks more like a cycle than a straight line. What is the difference between data
analysis and data science? The boundaries between data analysis and data science are
not very clear. The main differences are that data scientists usually have more programming
and math skills, they can then apply these skills in machine learning on ETL processes.
The analysts on the other hand, have a better communication skills creating better reports
with stronger storytelling abilities. By the way, these Weiler chart you're seeing right
here is available in the notes in case you want to check out the source code. Let's explore
the Python and PI Data ecosystem, all the tools and libraries that we will be using.
The most important libraries that we will be using are pandas for data analysis, and
matplotlib and Seabourn for visualizations. But the ecosystem is large and there are many
useful libraries for specific use cases. How do Python data analysts think if you're coming
from a traditional data analysis place using tools like Excel and Tableau you're probably
used to have a constant visual reference of your data. All these tools are point on Click.
This works great for a small amount of data. But it's less useful when the amount of records
grow. It's just impossible for humans to visually reference too much data, and the processing
gets incredibly slow. In contrast, when we work with Python, we don't have a constant
visual reference of the data we're working with. We know it's there. We know how it looks
like. We know the main statistical properties of it, but we're not constantly looking at
it. These allows us to work with millions of records incredibly fast. This also means
you can move your data analysis processes from one computer to the other, and for example,
to the cloud without much overhead. And finally, why would you like to add Python to your data
analysis skills aside from the advantages of freedom and power theories, another important
reason, according to PayScale, data analysts that no Python and SQL are better paid than
the ones that don't know how to use programming tools. So that's it. Let's get started in
our following section will show you a real world example of data analysis with Python,
we want you to see right away what you will be able to do after this tutorial. We're gonna start this tutorial by working
with a real example of data analysis and data processing with Python, we're not going to
get into the details yet, the following sections will explain what each one of the tools does,
and what is the best way to apply them combining and the details of them. In general, this
is just for you to have a quick on high level reference of our day to day processes, data
analysts, data managers, data scientist using Python. So the first data set that we're going
to use is a CSV file that has this form, you can find it right here, under the data directory,
the data we're going to be used is this, I have just transformed it into a spreadsheet.
So we can pretty much look at it from a more visual perspective. But remember, as we said
in the introduction, as data analysts are not constantly looking at the data, right,
we don't have a constant visual reference, we are more driven by the understanding of
the data right in the back of our head, and we understand how what the data looks like,
what's the shape of it. And that's what it's conducting our analysis. So the first thing
we're going to do is we're going to read it this CSV into Python, and you can see how
simple it is just one line of code gets us the CSV read into byte, then we're going to
give a quick reference. And this is what the data frame that we have created looks like
data frame is a special word is a special data structure, we use independent tool. And
again, we're going to see that in detail in the pan this part of this tutorial. The data
frame is pretty much the CSV representation, but it has a few more enforced things like
for example, each column has a strict data type. And we will not be able to change it
to tetra, it's a better way to conduct our analysis, the shape of our data frame tells
us how many rows and how many columns we have. So you can imagine that with these amount
of rows, it's not so simple to again, follow a visual representation of it's like, it's
pretty much infants crawling, in this point 100,000 rows. But the way we work is by immediately
after we load our data we have we want to find some sort of reference in the shape and
the the properties of the data we're working with. And for that we're going to do first
an info to quickly understand the columns we're working with. In this case, we have
date, which is a date time field, we have day, month year on that are just complimentary
to date, we have the customer age, which is uninjured, which makes sense right? age group,
you can say it's right here. It's age group youth, customer gender, we have an idea again,
of the of the entire data set, we know the columns we have, but we also know how large
it is. And we don't care what's in between, we will be cleaning it probably, but we don't
need to actually start looking row per row, right just with our very limited eyes, we
have a better understanding of the structure of our data in this way. And we're going one
step further, we will also have a better understanding of the statistical properties of this data
frame with a describe method. For all those numeric fields, I can have an idea of the statistical properties of those. So
for example, I know that the average age of these data set is 35 years old. I also know
that the maximum age in this case if these Or is the sales data is 87 years old, I know
the minimum is 17 years old. And again, I can start building right if my understanding
of this that physical properties of it. So in this case, the median of my age is very
close to the mean. So this is telling me, all is telling me something, and the same
thing is going to happen for each one of the columns that we are using. For example, we have a negative profit here,
and we have very large values here are these correct, is maybe there's a mistake, again,
it's by having a quick statistical view of our data, we're going to be driving the process
of analysis without the need of constantly looking at all the rows that we have. It's
a, it's a more general holistic overview. So we're gonna start with unit cost, let's,
let's see what it looks like. And we're going to do a describe only if you need coast, which
is pretty much what we had right here. In the previous in this line, what we did was
for the entire data frame for the entire data, in this case, we're just focusing in the unit
coast, cost, sorry, column, the mean, the median, all fields, we know already pretty
much from this, and we're gonna quickly plot them, we're going to use these tools to visualize
them. And it's the same tool, it's paying this that it's using on top, right? It's using
matplotlib. So the visualization is created with matplotlib. But we're doing it directly
from pandas. And again, don't worry, this is all explained in pandas lessons. So this
is unit costs, right is what this is the box, but we have just created, we have the whiskers
that mean that shows us the the first and third quartile, the median. And then we see
all the outliers that we have right here. So we see that our product study is around
$500 is considered to be an outlier. And the same thing if we do a density plot, right.
So this is what it looks like. We're going to draw two more charts, right, in which we're
going to pretty much point out the mean and the median, right in the distribution charts.
And we're going to do a quick histogram of the costs of our products. Moving forward,
we're going to talk about age groups with the age of a customer. And at any moment,
we can always do something like sales sort here to give a quick reference, we know that
the the age of the customer is expressed in actual years old they were but also they have
been categorized with three, four, actually four age groups, seniors, youth, young adults
and adults, right. So they we have given categories were creative, right to better understand
these groups, and we do that with values. Value counts, we can quickly get a pie chart
out of it, or we could get a bar chart out of it. As you can see, right here, we're doing
an analysis of our data, we see that adults right here are the largest group in our for
our data at least. So moving forward, what about a correlation analysis? What is a correlation
between some of our properties, we will probably have high correlation for example, between
profit and unit cost, for example, or order quantity, that's kind of expected, but that's
all something that we can do right here. This is matrix right of correlation showing in
red high correlation. So order quantity, and unit cost or where is profit right here. Profit
is right here. So we see high correlation with unit with cost with profit. Now with
profit, actually, it's the opposite blue is high correlation, I'm sorry, the diagonal,
which is blue, is correlation is equals one. So high correlation is blue. And we see that
profit has huge correlate has a lot of correlation, positive correlation with unit cost and unit
price. And negative correlation is with dark red. So we again can have a quick idea. Let's
see, for example, here profit, it has negative correlation with order quantity, which is
interesting, right? It's we wouldn't dig deeper into that, of course, the profit has a high
correlation positive with revenue, right? And again, it's just a quick correlation analysis.
We can also do a quick scatterplot to analyze the customer age and the revenue right to
see if there is any, any correlation there. Right? And the same thing for revenue and
profit. This is obvious, right? We can we can quickly draw a diagonal here, right. So
there is a lot Linear depth and dependency between these variables. So a form a few more
box plots, in this case, understanding the profit per age group, right, so we can see
how the profit will be, will change depending of the customer's age, and a few more box
plots. And we're creating these these grid of year customer age, unit costs, etc, for
multiple things. So moving forward, something that we can quickly do when we're working
with Python, especially within this is Drew shape or data or derive it from other columns,
right. So this is pretty common in Excel, we can create these revenue per age column,
if you're here in Google spreadsheets, you're going to do something like revenue, per age,
and you're going to do something like equals, right? Equals revenue, divided, I don't remember
if this correct formula we're using, but just for, for you to have a reference. And we're
going to pretty much extend this whole thing. There we go, Oh, well is processing, and I
have 100,000 rows. So you can see how slow it is, I let's compare that just to the way
Python works, I'm gonna execute this thing. It was instant, you know, extremely fast.
And it was all calculated seems that we have the same results as expected. same results
as expected. And we can quickly plot both the in a density plot and in a histogram,
as you can see, right there, now that revenue parade is going to be relevant. In any case,
it's just to show you the capabilities of what we can do. Let's annual analyze, well,
we're gonna create a new column, which is calculated cost is the total, the total orders
the total, the quantity of the order, times the cost, right, extremely simple formula,
very fast process. And we're gonna get right here, how many rows had a different value
than what was provided by cost? So what we're doing right here is like, we're quickly checking
if the cost provided by the data set, at some point doesn't align with the actual cost we
are calculating. So is there any mistakes that were made by the I don't know the original
system, or people doing a data entry, if these new column is different from cost, we want
to know about that. And that doesn't happen. So again, quick, quick, regression plot. In
this case, it's very obvious that there is some linear dependency between calculate cost
and profit. So more formulas, in this case costs part cost plus profit. So we're going
to adding a little bit more, there is no difference with the revenue and the calculated revenue
that we are having. So that all makes sense, we're going to do a quick histogram of the
revenue. We can, for example, on 3%, to all the prices that we are using, we need to increase
prices. How are you going to do that? Well, it's very simple with Python, we're just going
to do increase everything by point 03. And now all the prices have changed. What else we're going to be able to do quick
filtering, let's get all the sales from the state of Kentucky right. So these are all
the sales from the state of Kentucky, we can get only the average of the sales by these
age group on only revenue, right. So these, all these filtering options, and extremely
simple to get with Python. In this case, we say, give me all the sales from these age
group, and also from this country, right, and we're gonna get the average revenue from
these groups that we are selecting. And again, to modify the data, we can make just a few
quick modifications, like in this case, we're going to say, all the sales from country right
to revenue, we're going to increase it by 1.1. I don't know why, which is doing it arbitrarily.
It's just for me to show you how it works. So far, so good. Again, we've done a couple
things, you don't need to know about the details, we will actually go through that in the NumPy
independence sections in this tutorial. So just for you to have a quick reference of
it. There are exercises associated with these given lectures. So if you want to pause right
now and get into the exercises, that's going to be very helpful. We're going to move forward
now with the second lecture in which we will be using a database this Akila database and
we're going Be erasing data, instead of from a CSV file, as we did before, we're going
to read data now from a database. Reading data from a SQL database is as simple as it
is from an Excel file or a CSV file, as we were doing with our previous example. And
once you've read the data, that's we're going to do now the process is the same. So what
we have right here is a query a SQL query, if you don't know about SQL, you can check
our courses or other courses online. Basically, we're pulling the data from the database.
This is one of the advantages of Python, it's not, there are connectors for pretty much
every database provided out there, Oracle, Postgres, MySQL, SQL Server, etc. In this
particular example, we're going to be using MySQL. So once you construct the query, and
you pull the data from the database, then the process is the same, we have just converted
these outside data into a data frame that we can use with our Python skills. The first
step, as usual, is to check the shape information description of our data of our data frame.
In this case, we want to, again understand the structure of it. So we want to know how
many rows we have 16,000, we want to know a little bit more about our rows, we want
to know about a little bit more about our columns, and how many rows how many records
we have for each one of them and the type of each one of these columns. And we also
want to have a better statistical understanding of our data. So we do a quick describe, and
we have more details about it. If we want to focus in individual columns, right, we
can just do that by in this case, we're gonna focus in film rental rate, right, pretty much
how much you pay to rent a film. Um, we're gonna see the kind of distribution we have,
we can call it distribution, it's pretty much a categorical field in this case, but basically,
the rentals are divided into three main categories are prices, zero 99 299 499. So that's these
box plot these pretty much perfect, never seen in real life plot box plot gives you
those prices. And move forward, we can also check very quickly a categorical analysis,
understanding the distribution of rentals between cities, so we have two cities. And
it's pretty much even as you can see right here, creating new columns and reshaping the
data for further analysis, etc, is relatively simple. In this case, we're going to analyze
their return in rentals, right, which, which films are going to be more profitable for
the company div, dividing the rental rate, how much we charge, divided by the cost, how
much it costs us to acquire the film. So in this case, we can see the distribution of
that, right. So most rentals are here in the beginning. And then we have more profitable
rentals, were making up to 60% above the rental. And we can quickly analyze the mean and the
median fit right to have a quick idea of all that. Finally, selection and indexing, if you want
to start focusing, if you want to go into data, right, you want to zoom in, you want
to have a better understanding. So you start filtering, in this case, we can filter by
customer, but if you want to do it per city, if you want to do it per state, if you want
to do it per film, per price category, etc. It's very simple to filter to filter and zooming,
which is one particular characteristic of your data. So you can perform a more detailed
analysis. So in this case, we have all the the films are rented by the customer last
name, Hanson, which doesn't mean it's the same person. But again, it's very simple to
filter dot. And here, we can do we can very quickly see which ones are the price, the
film's sorry, that have the highest replacement cost, right. So basically, what we're doing
is we're going to isolate those films that have the highest replacement cost. And also
we can see right here just for you to have an idea, all the films that are in the category
PG or pG 13. It's very simple to to filter that data. So this is the process we usually
follow. we imported the data, we reshape it somehow create columns, there is an important
process of cleaning up or not highlighting this part of the tutorial, we're going to
talk about it in the tutorial itself. There's the process of cleaning, then reshaping creating
new columns, combining data and creating visualizations. This is the process, right? We're following
here with our Python skills, but it's a tone more to odd as you might imagine, from creating
reports to running machine learning processes, creating linear regressions, etc. For now,
this is just a quick understanding of the process. We follow. Now starting now we're
gonna move forward with more details of each one of the individual tools we're going to
talk about. We're going to talk about Jupyter notebooks. We're going to talk about NumPy.
We're going to talk about pandas, we're going to talk about mapa, lib, seaborne, etc. Starting
now, right? The first thing we're going to see is, what is this whole thing that I've
been using this Jupyter Notebook, I want you to now too, if you want, if you if you don't
have experience with it, I want you to have an idea of how it works. And then we're going
to move forward the individual tools, NumPy, pandas, etc. Remember, there are exercises
also associated with this particular lecture. So you can always go back again, and work
with them. Once you get more a better understanding of the tools we are using. Before we jump into the actual data analysis
course, and we start talking about Python, pandas, all the tools, we're going to use
import files, read data from databases, etc, I want to show you the environment that we
work with. It's our primary environment, it's the tool that we use 99% of the time on its
Jupyter Notebook, there are going to be different terms here, I'm going to be referring to it
as Jupyter Notebook. But as you are going to see in this, in this part of the of our
tutorial, you can see that Jupiter is actually a whole ecosystem of tools. And it's a very
interesting project. Jupiter is a free and open source, again, ecosystem of multiple
tools. And primarily, we're gonna talk about first, what is a Jupyter Notebook. What you're
seeing right here, and you're gonna see live in a second, I can actually show it to you
is this thing we're going to use. And we are also going to talk about Jupiter lab. Okay,
which is the evolution of the regular Jupyter Notebook. So, I think this could be familiar
to you already. Usually the questions in the question is, what's the difference between
Jupyter Notebook and Jupiter lab? Well, the difference is that Jupiter lab is just a nicer
interface on top of Jupyter notebooks. It's not just the plain notebook. This is a notebook,
but I'm scrolling right now. It's also the addition of tree view, it's an addition of
get tools, as an addition of command to lead and multiple other things. You can open some
files with a nice preview in it, etc. So, Jupiter lab Jupyter Notebook, they are similar
Jupiter lab easy, again, the evolution of a Jupyter Notebook. And that's what we're
using. Again, Jupiter is a free and open source project. So anybody can install it, anybody
can download it, it's very simple to get it set up in your local computer. In this case,
we're using something we call notebooks AI, it's a project that provides Jupiter environment
for free in the cloud. So you don't need to install things locally, you don't need to
put things in sync in your own hard drive, right you That means you don't need to buck
it up, for example, because it's just a service, it's all worked in the cloud. So said that,
I want to tell you that we have compiled a very quick list of everything, we're going
to talk in this part of the tutorial, in this list of two, it's just a thread of with multiple,
multiple hints of how to use Jupyter notebooks. So after the video after the course, if you
forget some of these concepts, you can always go back to this to it, it's a quick reference
for you to have. So let's get started. Why do we use a Jupyter Notebook? Because it's
an interactive real time environment to produce our or to to explore our data and to do our
data analysis. It's a tool you're gonna fire commands, and it will immediately respond
with something back. It's a very interactive tool, when we're working with data analysis,
and this is mainly main difference with some other tools like for example, Excel, tableau,
etc, is that we are not constantly looking at the data, there is no visual reference,
like for example you have in Excel, right? So in Excel, you're constantly looking at
the data, you have it in front of you, there are 100,000 cells and you can stroll and see
them. The problem is that that's not scalable, right? It's like nobody can work with 100,000
rows in their, in their, in their mind, we will always forget something. So the way we
work with Python indeed, analysis is by always having a reference of how our data looks like
but always at the back of our head and we're not constantly looking at it. We're like this
person from the matrix, you know, the, the the commander of the matrix that commands
people to get get in and out. We're basically telling people telling people that basically
asking data, right asking questions to the data, and having a picture in our mind of
how that's going to work, we're not constantly looking at it, we're just having a reference,
or in our in the back of our heads of what our data looks like. So that's why this tool
is very useful. This tool is useful Also, if you're just training your Python skills,
and or their permanent language skills, because what you're gonna see is it's just a regular
Python interpreter. In this case, I can execute some code, that's two one times, actually
one plus three, there we go. And the result is four. Right. So this is a Python is a fully
featured Python interpreter. The good thing is that again, it's going to respond to us
pretty much immediately I create a command and I immediately get a response, I can do
something a print here, hello world. And I immediately get a response, I can do Hello,
world, times, times three. Again, it's a again, a Python interpreter, a fully feature Python
interpreter, but it's not being accessed from a terminal, which you can write this is the
good thing about Jupiter lab to have a terminal, I can do Python, right. And I can do two,
time three, and I get an answer back. But this is not convenient to work with our data,
we need something a little bit more interactive, we can also mix with documents, that's going
to be the advantage of a Jupyter. notebook. So what what's the way we work with Jupyter
notebooks, there are a few concepts, very important concepts that we are going to follow
a Jupyter Notebook is just a sequence of multiple cells, okay, everything is a cell. And as
you can see, when I click on these cells, even if even if it doesn't look like being
a cell, it is, you will see that these blue thing right here, right is pretty much following
me because I'm clicking on the cell, and I'm selecting that particular cell. Everything
happens within a cell, if I want to execute some code I can do, again, one plus five,
and to get a result or a result back, right, that's, that's how it works. So I'm creating
a cell, I'm deleting a cell, I create another cell again. So it's everything happens with
a cell, and I'm going to tell you how to add the cells, how to remove them how to execute
code, etc. The interesting thing about a cell is that it can either be Python code, or any
other programming language you're using in this case is a Python data analysis course.
It can be Python code, as we're we were doing before one plus three, this is Python code,
or it can be what we call markdown, okay, which is a formatting format, right? To create text, that will be
a render with sort of HTML ID at the output. So in this case, this is what the source code
of the markdown looks like in markdown, any line that starts with this part, it's going
to be a title, in this case, it's going to be the largest, the biggest title you can
have is just one pod, and then you keep adding to reviews the size in this case, level three
title. And then you can have for example, this is a quote this is bold, this is it Alex,
this is a link, right? So let me actually, I could copy the cell and open the source
code. There we go. So this is a link right issue, issue is created or it's rendered as
a link. So markdown, what is is that is a text formatting tool, right or protocol, we
could say that in this case, we just specify us we have some some rules to use in our in our text, and
markdown knows how to interpret them and format right or return a formatted document after
them. So for example, here, we have green divider, which is a picture and we know it's
a picture because it starts with an exclamation marks. And that's that what you're saying
right here. So again, a cell can be either Python code, or it can be markdown. markdown
is an entire thing on its own. You can get any tutorial online free, it's it's fairly
simple to get started with. And it's also very important because when you're formatting
your reports, right, when you're creating your reports, you want them to look pretty,
you can use markdown for not and what we're going to see later So you can export these
notebooks and they will generate PDFs, right. So this whole thing can be a PDF or an or
an HTML page. So after you're done with your data analysis, you can hand over to whoever
asked for the analysis, a PDF report, which is pretty neat. So moving forward, again,
any cell is going to be either markdown, or it's going to be code right here. So these
ones code, and you can switch the modes, you can say, this LS code, or actually, let's
make it markdown. So right now, if it's a code, it doesn't doesn't matter, or just,
it's not executing anything, because the cell is interpret as markdown. So now, I'm switched
back to code. And now it works. Again, I said, Sure. It can also be raw, but to be honest,
we don't use raw very often. So again, you have this this general cell type, this cell
we're using, what type is it? Is it code is it markdown, you can switch it with these
with the selector right here. So a few more things that I have to tell you right away,
so you can start internalizing them, and it's gonna take some time to get used to it. But
once you get used to it, you're gonna move very fast in your data analysis with Python
Jupyter notebooks. The first thing is, as you're seeing right here, every cell has been
given an execution number. So any, the cells will be moved, right, they will be moving
around, you will be moving them around. But you will always know which one executed before
another one. And that's because every execution, you run will be assigned an execution number.
In this case, this is the seventh time I have executed code. If I execute code again, for
example, I don't know, two times two, this is the eighth time that I've executed code.
And if I move this thing, right here, if you're reading this thing, top down, you will not
be full, right? You will understand this thing. It was moved, the cell was moved, the structure
of the notebook changed. But these thing was executed after this other cell, right? xact.
And this is seven. So the execution order is always preserved. So that's an important
thing. Something else that you're seeing me change the structure, and do things with the
notebook without using any menu. And that's because I know how to use keyboard commands
keyword shortcuts to run most of these commands. So for example, how can I add a new cell I
have these is a markdown cell. This is a code cell, if I need a cell before these one, what's
what's that command that I'm going to issue in order to create the cell, in this case,
the command is going to be the letter A, I just type A, and there is a new cell creative.
How can I delete the cell, it can be two times that the key two times the D key. And again,
this is all these reference with built. So for example, right here, whereas hit at some
point, you can. Here, you can type, you can press A to create
a new cell, you can press B to create a new cell, what we call below. So let me put something
here, this is a reference. And I'm going to put here the letter B and it's going to create
a cell B below the currently selected one. So the selection here is here in the blue,
I hit let me delete this one, I hit B. And again, it's going to create a cell below the
previously selected one, if I hit a, it's going to create a cell above that previously
created one. So these, these are the mnemonics of the creation. Something else and it's very
important why when I'm in this cell, and I hit the letter, a leader, literally it just
hits the letter A in my keyword, no control, no command, just a, it creates a new cell,
and it doesn't type A inside the document, right? So right here, if I type A, it's adding
an actual a character in the cell. Why didn't that happen before. And you're going to notice
that when I change, when I'm going to call a mode in a second, you're going to see that
the content of the cell is grayed out, show what now when I when I press on the letter
A it actually creates to sell and it's not adding content to the sell itself. If I go
back again to the other mode, and I'm going to give you a better explanation in a second.
If I type anything, in this case, a it's actually appended to the text within it. So this is
my interaction to sell modes and this is very important. The Jupyter Notebook is a mode
base editor, right? So there are multiple editors are, for example, vim or VI, vi, those
are mode based editors, which basically, the behavior of your work will change depending
on the mode that it's currently activated. So for example, in this case, I am in addition
mode, because any character that I type will be appended to the cell, A, B, C, D, etc.
If I switch out of editing mode to what we're gonna call command mode, I switch out of that
mode. Now the cell is grayed out, and any key that I hit, it's gonna do something different
associated with that key. So A is going to create a new cell above, B is going to create
a new cell below, Double D is going to delete this cell, right. So that's, that's the important
part of Mo. That's one of the most important parts in order to understand how to work with
Jupyter notebooks, the mode that you're currently working with, and there are only two modes,
so it's fairly simple. This is command mode. And we recognize command mode, because this
cell is grayed out. When we get into edit mode, there is a regular prompt, as you're
saying before, the number one on the cell is actually subjects of addition. So that's
the way we can realize that, how are you going to switch from modes, in this case, I'm in
editing mode, if I'm using my mouse just pointing, I can click outside, I'm gonna get out of
the edit mode into command mode. If I point inside and going back again, to the Edit Mode,
but let me tell you something right away and then say, we don't like to use our mouse,
we don't like to point and click, because that's very slow. We like to use our keyboard,
we move very fast with our keyboard. So how are you going to switch from, from editing
mode back to command mode, that's going to be with the Escape key to go from editing
to command, edit as Escape key, it's going to switch out of editing, but when mode. And
if you actually want to make modifications to the cell, basically, you want to get into
edit mode, you're going to hit the return key, that's going to get you into edit mode,
again. So we have tackle multiple things are writing, again, we said in Jupyter notebooks,
we're going to use Python code very quickly to interact with our data, we need a real
time, you know, I'm asking a you're answering type of editor. That's what the Jupyter Notebook
is. The Jupyter Notebook has these two modes, edit and, and command mode. And then the cells
which is pretty much everything is the most important, it's a fundamental part of the
notebook, the cell is going to have two types can be either code, or it's going to be markdown,
right. And now I'm going to start showing you more features. And I'm going to show you,
I'm going to show you the most important commands. And of course, how the what the keyboard shortcuts
for those commands are, so you can move freely. And and and work with Jupyter Notebooks in
the most efficient way. So let's get started. First of all, for for from the most important
commands is moving right. So navigating, it's very simple to navigate, just use your arrow
keys, up and down, up and down. And you're going to move around in your notebook. If
you wanted to switch the type, right going from markdown to code, etc, you can switch
use these drop down or you can press the specific key is to switch to either markdown or Python.
So for my markdown, you're gonna switch sorry, hit the M key, that's going to make it markdown.
For Python, you're going to hit the Y key, that's going to make it Python code. So M
and y are going to switch you back and forth. Keep an eye on the selector you're going to
hit y m y m is going to switch it from code to markdown. What else how can you execute code once you
are within your typing code and you want to execute it, there are two types of executions
you can run. The first one is going to keep the selection the currently selected an active
cell is going to stay the same place you are and that's going to be my by keeping press
the Ctrl key and hitting return that's going to run decode on the cell there the prompt
or the current selected cell will remain being the same. So I'm running this thing a couple
of times already on this selection or the currently highlighted cell stays the same,
I can change that by using shift return. So I keep the shift key pressed. And I hit return
and is going to execute the code. But it will immediately switch the prompt or the currently
selected cell to the following one. And that's useful when you have multiple cells, you want
to execute one after the other. So you can keep hitting shift, return, return, return
return, and it keeps you moving right from top to bottom. Alright, so Ctrl return or
shift return to change the execution is the same is just what's going to happen with the
currently selected cell. We already saw how to create cells with the A key, we create
a cell above with B key we create a cell below. To delete a cell, you're going to hit the
D key, the D key two times one after the other very quickly, dd is going to delete these
the cell. What happens if you made a mistake, and you want to undo the previously issued
commands? Well, the mnemonic here is going to be Ctrl Z, you know the mnemonic, it's
not the command, it's going to be Ctrl Z, you only need to press the Z key, you know,
you don't need Ctrl Z, and it's gonna undo whatever you did in your previous command.
Alright, so a B, D deletion, and then Z to undo the all the commands were saying they
all have a correspondence in this toolbar or in this command palette. So for example,
right here, I could run this code by pressing these play button right here you see it, the
execution is changing. There are multiple ones and you can search them if you don't
remember right here. And the neat thing about it is that you actually have the shortcuts
to issue the same command. So let's say you don't remember how execute and stay stay in
the same cell, or move whatever you can search for run. And you can see what's the name,
and what's the actual command that you have, right there, right. So you can, at least for
your first ad or a month working with Jupyter notebooks, you will usually need to go back
to these commands, and try to remember the the quick shortcuts. And with time and practice,
those will just come naturally. So moving forward, what else, we have a few other commands,
in this case, we have something to cut and paste the cell somewhere else, just cut and
paste, that's going to be x to cut it, or you can also use the scissors here, x to cut
it. And to paste it, you can use this button or actually these buttons sorry, or you can
just press the V key V is going to paste it wherever you're currently standing it. So
I'm going to cut it, I'm going to remove it from here, and I'm going to paste it below
there. Or you can also copy it. So instead of cutting it, you can press the C key just
going to cut, sorry, copy. And then you can actually say where you want to paste it. In
this case, we have duplicated the same cell. And it looks something interesting here, the
execution count remains the same. So again, there is like this unique identifier for your
executions, which means that you know, when and where something was executed. Moving forward,
we're going to use some code here, we're going to import some tools, you can see some characteristics
or advantages of Jupyter notebooks and why we use it so often compared to, for example,
the regular Python terminal. One very important thing is visualizations,
we as data analyst, we're constantly getting data on expressing it through images, or animated
animations, right. But most commonly, images. The main library we use in Python is model
live. And model lib is a first class citizen in Jupyter notebooks, which means that you
can just run the figures from matplotlib. And they will just show up directly in your
notebook without the need of doing anything. Crazy. So can you imagine showing these these
beautiful picture in this terminal? That's that's very hard, of course. So again, that's
one of the main advantages of a Jupyter Notebook. Moving forward, what we're going to do is
we're going to first we're going to get some data from a public API. So there is these
crypto watch service, which basically has crypto information, Bitcoin, ether, etc. And
you can check the docs, we can actually open them. It's gonna give you market data Tesla.
You can check the docs and How you can get in this case it's BTC Bitcoin to euro, sexual
see if we can change it to USD USD price. There we go. So this is the current price
of bitcoin results, surprise, etc. And we're actually going to do markets do we have crack
and BTC USD, let's do, let's actually issue the same query we're going to use which is
open high, low, close Oh h LC. And don't worry, this looks ugly. But this is actually what
we're using. There's a list of results write for all different candles, we call them, we
get the idea of the open price, close price, high price and low price. So we're going to
issue those, we're going to issue these requests to the internet to these API, the crypto the
crypto watch API, so you can get information about bacon to do some analysis, I say they
can, you can actually get it from ether for for ether for author different types of crypto
or currencies. So the function we're defining is get history, get historic price, it's a
very simple function that uses pandas is one of the most important tools, we're going to
be using this course. And the requests library, which is also very famous library for Python.
And what we're going to do here is we're going to get Bitcoin on ether prize for an entire
week. Right. So from ferreted that the second February sorry, February 25, up to today,
right? So depending on when I'm shooting this video, and we're gonna get a quick reference
of the prices open, high, low, close. So in this case, we have four information per hour.
Okay, so this is something you can actually change in the in the, in the request you're
making to the API, you can reuse the candles eyes. In this case, we're keeping it per hour.
So we have by the hour information about Bitcoin, in this particular market, which is bitstamp.
Here, we have these day these day, and these are right, when I'm in the morning, open,
close, highest price and lowest price, and also the volume that was operated within this
time period. And we're gonna immediately plot the price. So we see that in these time, which
I think is an entire day, we the price dropped, it's actually a few days, like an entire week,
the price dropped from $9,600 below, right 9000. So it was a pretty significant drop.
Let's see ether highperformance. We have here all the records, and how it moved. So this
is what I tell you that when you're doing data analysis with programming tool like Python
rar, you're not constantly looking at the data. So what I'm showing you right here are
the first five records, we actually have. Let's do that. We actually have 169. Records,
okay, 169 Records. And this is per hour. So if we do 169 hours divided by 24 hours, we
have seven days, right? So we have seven days of data 169 Records, and then we have a little
bit more information keeps this to go. I'm gonna get to that in a second. But basically,
this is one I tell you 169 Records, to be honest, something you could be saying in a
spreadsheet. But I want you to get the concept here. We're not just looking at our data,
we have it in our brain, we know what did it we know what shape it has. We know how
many records it had, we know information standard deviation, what's the mean of that? Right?
So close price was the standard deviation, right? What's the the average, the mean, the
median, right? So we have information about our data. It's sitting behind, you know, in
our brain, but we're not looking at it. And that's because with a very simple example,
with only 169 Records, but in real life, we're dealing with millions of records, so it's
impossible to see it. Have you ever tried is crawling in an Excel spreadsheet through
millions of records. It's crazy. It's not possible. It's just unusable. So that's again,
the way we work with data analysis in Python and R and other tools. We don't constantly
keep an eye on the data. We know the shape of it. And we just have these quick references
like show me the first five records. I mean, the last five records, show me this chunk
here down there, but that's it. So again, these are the visualizations we're creating
on Jupyter notebooks. Again, it's just very simple to get the plot done right there. We're
going to also see in Jupyter notebooks, a few other pretty neat things. The first one
is that we can use another library, which is called bokeem. And the difference is that
boakye will have charts that are interactive. So I'm moving it right here, it has JavaScript.
And it's interactive, you look back again, to what we had here. This is a static chart,
it's just a PNG, you can actually export it as a PNG, there is nothing you can do with
it. With bokeem, it's actually a dynamic, dynamically generated interactive charts.
So I can, I can zoom in piece of data, right, I can move it around, I can just do whatever
I want with it. I can refresh and reset it to whatever it was. And it's a dynamically
generated chart. The difference is, if you're working with data, dynamically in your analysis,
sort of in your exploration, then boek is a planning tool because you can zoom in, right,
so what's going on here, let's, let's look at these things. If we're working on a mean,
reverting strategy, for example, we see a high volume, we see a low volume, the mean
is going to be here. So we see some mean reversion in there. It's very interesting. If you need
to, for example, export a PDF, export a huge HTML file, then static images are going to
be probably better. So that's the difference between them. To be honest, model lib is a
lot more popular than bogey, we use model live a lot more because it's we actually have
a few other tools like seaborne that make it very easy to access and use it. What else
Jupyter Notebooks work very well with some Excel, Excel files with all the file formats
csvs, XML, Excel files, etc. And that's also the the availability of Jupiter lab. So Jupiter
lab can immediately interpret and opens his v files can open with some extensions, XLS
files, XML files, JSON files has a very nice editor and tree view for Jason. So the Jupiter
lab environment combined with Python Jupyter Notebooks will give you a good idea of Jupiter
in general. So in this case, we have just saved I'm not going to execute these you can
try it out. But you can execute and run what we have just done and export this crypto file
us an Excel spreadsheet. So you can just click on here and you can basically download it,
you're going to open it and see what has There we go. So let me reduce the size of this thing. There
we go. So you can see that we have just exported to spread two sheets, in this case, Bitcoin
on ether, right? With the data that we had in our previous notebook, right. So that's
all again, the combination of Jupiter, the combination of Python and the combination
of Jupiter lab, which are tools just work very well together. So we're gonna keep moving
forward, in this video, this tutorial, I'm talking about more data analysis, in general,
we're going to talk about Python, we're going to do a quick review of Python. Maybe when
we when I was running these commands, you felt you felt a little bit lost what I was
doing with it. So we're gonna do a quick review of Python and all that. And of course, we're
gonna get directly deep into data analysis with pandas with some other tools, I want
to tell you something before we finish this chapter. And it's not, it's very important
for you to get familiar with data analysis, with sorry, with Jupyter notebooks, because
you're going to spend a ton of time with it. And it's a very, very valuable skill that
you can get if you get proficient, comfortable with Jupyter notebooks, you know, like creating
cells, deleting cells, cutting, pasting, moving things around, etc. For you to generate reports
Jupyter notebooks are going to be excellent. So keep an eye on it. Keep practicing, it's
the only way to learn it to the to the analysis. Keep practicing it, keep open the command
Polat. So you can always want if you forgot, how can it caught a cell? Well, there is here
it is command x, right? It's gonna just tell you upfront, keep an eye on it, keep working
with it and practicing it. And once you get familiar with Jupyter notebooks, you're going
to move very, very fast. Remember, they have these nice list of compiled commands and reference
you can always access if you need extra help. And we're going to keep moving forward now
with more data analysis. Now it's time to talk about NumPy, one of
the most important libraries in the Python ecosystem for data processing. In general,
it's the one that got pretty much everything started. And if you trace back NumPy, it,
it's a very old developed library. 20 years, maybe it's it's an extremely popular library
and important library, I'm not gonna say popular. And I'm going to explain why in just a second.
But it's a very, very important library in the Python ecosystem for data processing.
NumPy is a library that will lead you it's a numeric competing library, it's just to
process numbers to calculate things with numbers. And that's it. So NumPy has a very limited
scope, we could say, and this is an on purpose, a very simple library, when you look at it,
and when you look at the API, which is very consistent, by the way, why is NumPy so important?
Well, in Python, numeric processing, and just pure Python processing numbers is very slow.
Okay, Python is not slow as itself compared to other programming languages. But when you
go down, right to very deep levels of performance, when you are processing large amounts of data,
right, and you need to squeeze, even, you know, that tiny bite at the end of your pipeline,
you need to squeeze every flow up from your CPU, then Python is not the right tool for
non Python as as a pure python programming language. NumPy is actually solving that NumPy
is a very efficient numeric processing library that sits on top of Python, and gives you
the same API as you're going to work with with just writing Python code, as you're saying
here. But low level, it's going to be using high performance, numeric computations and,
and arrays of numbers and representations, etc. That's it. That's it for pi NumPy. It's
extremely simple from from an API perspective, but it's extremely powerful. Why did I say
that? It's not so popular. But yes, it's so important. Well, because in reality, we don't
usually employ NumPy directly, you will not see yourself using NumPy. So often, but you
will be using other tools in Python, like for example, pandas, and matplotlib. And they
are all working on top of NumPy. They're all relying on relying on NumPy for their numeric
processing. So that's why NumPy is so important. So the for, at least for this part of the
tutorial NumPy. I'm going to divide it into pieces. The first one is going to be a very
detail, low level explanation of how NumPy works, why we need to use NumPy. And what
are the differences between different bite sizes for numbers, we're going to talk about
integers. But this is going to apply for decimals and data types also. And why you need a very
low level, optimize to us number. Now you can, you can skip this part, you're going
to find in the description of this tutorial, the precise moment in time. So you can just
skip and go directly to the second part, which is when we actually start using NumPy. And
I show you how to create arrays, how to make computations, etc. So for now, we're going
to divide it in two parts, we're going to start first with the low level explanation
which you can escape if you want, because it's not going to be crucial, you can easily
use NumPy. Without it. We have found that for some of our students, it's it's important
to understand the low level basics of it, especially if you didn't have a computer science
background, it can help you get you know, raise your right your level of understanding
of computers, and how to make your computations more efficient. But don't worry if you if
you don't want to go through that now it's fine. You can skip this part and come back
later or any other at any other moment. You don't need the ease to use NumPy seriously,
you don't need it. It's going to be beneficial, but you don't absolutely lead so you can just
skip and come later. So with that said, let's actually go into into a deep understanding and explanation of how computers
store integers, numbers in memory and what are bytes bits etc. In order to understand
why NumPy is so important. We have to go back again to the basics. What are numbers, how
they are represented in computers, etc. As you might know already a computer can only
process ones and zeros bits, it can't process numbers or just decimal numbers to be more
correct, sorry, it only can process ones and zeros. A computer is just always storing and
processing ones and zeros. It's a binary machine. Your memory is the central place around the
random access memory in your computer is the the central place where your computer is storing
the data that it's actively processing, right. So you have, for example, a hard drive, which
stores long term data. But the computer can process data directly from your hard drive.
Before doing that, it has to load it into your ram into your random access memory again,
usually, right a computer is going to have what eight gigabytes 1632 doesn't matter.
Let's say you have eight gigabytes of memory, that at some point is going to translate to
number of bits that your computer can store. So if you follow, if you follow these we have
right here, you can see the total number of bits available in a regular computer with
eight gigabytes of memory. Why is this important? Because again, the objective of these of these
tutorial is the objective of this bar, at least is to explain how you can squeeze out
of every single bit you can in your computer, right? How can you make it more efficient?
For your numeric processing, both in storage? use less memory for the same data? And also
how to make it faster, right for your calculations. So in terms of physical storage, or actually
memory storage, right? How can we make it? How can we optimize to use the least amount
of memory for this given problem? That's the objective of optimizing it, we need to understand
how numbers decimals or sorry, integers into decimal numeric system are represented in
binary, right. So these table right here shows you the first nine numbers, 01234, etc. and
their binary representation. In your computer. Let's say you want to store the age of user
age of a user, which is 32. You can't store 32 in here, because your computer again doesn't
know about decimals, it only knows about binary. To do that, you will need to find the correct
representation in ones and zeros of 3030. All right, sorry, two, which is not this one,
to be honest, I'm just making it up as we go. But again, you need to know the correct
binary representation of these number in norther. To store that data, how can you know that?
Well, there is this whole binary arithmetic, right? There's a whole part of math dedicated
to binary doesn't matter for now. But I'm going to just drive the intuition of it so
you can have a better understanding. And if you're interested, you can dig deeper later.
So basically, any decimal number needs to be stored in a binary format, which of course
only steaks ones and zeros. And what we usually do is just we keep increasing zeros and ones
in positions, right. So in this case, we have the number zero, the number one, that's fine.
Once we need to store the number two, winning now to increase the number, the position right
here we need to increase, right, so we need to go from two to one zero, we'd go to the
number three, it's one one, and then we need to go to number four, we need to increase
positions again, because we only have two symbols, zero and one. So as you're seeing
right here, up to these level, we need only one position. Up to this level, we need two
positions. This level, we need three positions. And these levels going to need four positions.
And you'll see how the size of each of these is increasing. And it has a an explanation behind it that we're going
to see in a second. So the question is how many decimal numbers you can store with n
bytes and bits, sorry, bits. So let's say we have n bits. And let's say n is equals
to three. That means that you only have three positions, right three bits, how many total
decimal numbers, you can store with it? Well we can store 000 we can store zero, we can
store 100 we can start stores are you one zero, right? So in this size, we can store
up to here, we can store up to seven numbers 111 is equals to seven was, once we've filled
all the positions, right, we've reached the limit, right? The largest number, the largest
binary for this amount of symbols or positions. That's the number seven. So these means that
with three numbers, you can start from zero from zero, here, zero up to one, one. In total,
you can store eight decimal numbers, here you have eight decimal numbers 012345678,
total decimal numbers from zero to seven. The equation if you want behind this is as follows.
If you have n equals three, and it's, in order to know how many decimal numbers you can store
with those bits, it's two to the power of n, in this case, is total a bit. So if we
go back into our drawings, we said that with three bits, we can store up to eight decimal
numbers. And again, the equation is two to the power of n is going to give you how many
decimal numbers you need. You can always do the opposite process using logarithm and get
how many bits you're going to need to create to store a given decimal number. I'm, I'm
not going to get into that. So we don't complicate it. But again, the math behind it is extremely
simple. So now, moving forward, we're going to delete this whole thing. Moving forward.
Why is this important? When you're working with your data, when you're doing your data
analysis, you know what, what data you're what type of data, you're working with their
own numbers, but numbers only usually have a connotation behind, right? So let's say
that you have here it's a table of people, and you have the total net worth of the person.
And also you have the age of the person. The age is a value that will range between what
zero, right? Just born to, I don't know, 120, we can say I don't know,
what's the maximum age registered right now, the oldest human being but zero to 120, it
seems, seems reasonable. In your other column net worth for this person, the range is it's
completely difference. We can go from something like $0 up to, I don't know $60 billion, I
think Mark Zuckerberg or Jeff Bezos or one of those. So we go from zero to 62 billions
in this case, if there are dollars, what happened if this is a highly devaluated currency? Would
we have to go to trillions, right? So these two even though they're just plain numbers,
and we can say they are integers, even though these are pulling numbers, they have an integers,
they have a different connotation, and they will need different requirements in terms
of storage size, right? So if we say that nh goes from zero to 120, we don't need so
many. So many bits to store it in memory, right? We can do the math, actually, how many
bits Do we need in order to store 120 100? And what do we say 120. Right? Well, if you
do the math, you will see that two to the power two to the power of seven is 128. So
if you have if you have seven bits here, seven bits, you're going to store from zero, up
to 1111111, which is actually 127. Okay, these number, all ones, seven ones in binary is
equals to 127. in decimal, in total, we can store 128 numbers 00 matters, up to 127. So
that means that for our column right to column, age, here age, we only we can use the size
of the men We need to use is going to be seven bits per user, or costumer or person, whatever.
What about these number right here, if we have to go up to a couple billions? Well,
in that case, the numbers a little bit more complicated, we're going to need, for example,
we can say 64, or 3232. It's actually 64, probably, but with 32 bits, right, you can
store up to from zero up to these volume. So again, I don't know about the currency
we're using or anything, so we can assume. But here, we need 32 bits in order to store that. And now you can
do the math, how many how much memory space you need, in order to process this data? How
many records Do you have, if you have only 1000 Records, that's not significant. You
can use whatever, you can use 64 bits here to store the age, and you're not going to
have a problem. But what happens if you have more what happens? What happens if you have
the entire population of the earth, you have 7 billion records here 7 billion records,
then every bit that you're saving in these columns is going to be important? Because
he's going to take a ton of data. And of course, you have a ton more columns, right? What happens
if you are processing trillions of records from financial transactions, right, you want
to be very, you want to be very efficient and optimize every single bit, you can't.
And that means again, selecting the correct number of have a bit per the columns you're
currently processing. So so far, so good, again, when there's 10, that the the number
in decimal that we need to store has a correspondence with emits, right? eight bits is one byte.
And the more we can optimize that, the less memory we're going to use for our obligations.
Where does NumPy come in place? Why are we talking about data in these NumPy lessons?
Well, they're right. The idea is that NumPy is a library that will lead you has a very
advanced numeric processing, in order to let you select the number of bits you want to
take for an integer. Even more, let's say you for forget about NumPy, you want to process
this thing with pure Python. So you x equals five, for example, working with Python, you
create a number, we're storing age as a five, how many bytes? How many bits? Do you think
the simple variable takes in memory? How many? Well, in reality, even though we think it
should be around, what, three, three bits, eight, let's say to be simple, too simplistic.
In reality, for Python, this is going to take around 20 bytes. Okay, so we are wasting a
ton of memory in order to store this number. And why is that? Well, because Python is a
high level, object oriented programming language. The reasoning behind it is that Python is
simple to write, write simple to also read and, and, and code on top of it. But again,
in order to create that simplicity, in its syrup, all the numbers in objects, which have
all these attributes, that if you know, advanced Python, you're going to recognize that are
not necessary. So these is taking a ton of memory. And a regular, very simple number
in Python ends up consuming 100 times more memory than what it should be consumed. And
this one NumPy comes in place in NumPy, you can create numbers that are for example, you
can control the size, in terms of bits, you can say I want to create a number that has
only eight bits. And that's it, that you're going to create a one byte integer, and you're
very precise and how much memory it takes, you can create a number that it's actually
need a little bit more space, we're going to do NP int, and we can hear us a talkie,
you're going to get auto completion 6016 bit or eight or 32 or 64, right. So we can actually
be a lot more precise in the number of bits that we need. And this is extremely important
for again, our high level processing. On top of that, NumPy is our array processing library
at NumPy is 99%, about processing a race constantly processing erase the data structures we have
in Python, the built in data structures we have in Python, for example, the list dictionary,
they are not optimized for high level computing. So if you have a list of numbers in Python,
let's say you have, I don't know, l equals 3224, right, you have three
numbers in your list. In Python there, it's not guaranteed that the least they'll the
list is gonna contain all the numbers, three to four in contiguous positions is gonna,
it might put them in separate positions in memory. On top of that, you can't rely on
advanced CPU directives and instructions for processing matrix matrices, sorry, because
Python, again, is wrapping these things in objects. So there is no access to these high
performance, low level instructions with NumPy, that changes because when you create an array
NumPy, you say, I want to create an array of three numbers, and they are all into eight,
then imposition forget about this is not these are not bytes I am, I'm using these drawing
as a general representation of memory. So in that case, in NumPy, when you create these
three element, int, eight array, it's going to create those three elements in contiguous
positions in memory, three to four, and they are only going to take that amount of memory
the police said they were going to take and on top of that, we can rely on a bunch of
very efficient low level instructions from your CPU for matrix matrix calculation, this
is something that it's a little bit more advanced. And it's something that has exploded in the
past 10 years CPUs with more with richer instruction sets, and the same thing for GPUs, you might
have heard, especially with machine learning and all that we need, we need fast array processing,
when we are storing features and weights and all that's a topic for for different story.
But again, the idea is we need right a ton of week, sorry, we can use all these important
and very efficient, low level directives from our CPU, which makes our computations a lot
faster. So again, as a recap, you don't need to know all these to work with NumPy. That's
the first thing. Second, you don't need to get extremely, extremely conscious about all
the numbers you use. At the beginning, you're just going to use NumPy as it is, and you're
going to use just the default types that it picks in 38 cents or in 32. In 64, that's
fine. But then, with when you get into bottlenecks,
when you're working with with larger amount of with more amount of data, then you might
need to get into the details of that size of the integers that you're using. And this
all applies to float. So I'm just using integers because it's simpler. But it's all applies
to floats. So again, NumPy, the main advantage is that it's it has built in very fast and
I raised kit, take advantage of CPU instructions for matrices and arrays and all that. And
it also has a very efficient representations of numbers, right are not the regular objects
of Python. Again, recap, you don't need a list. If you want to get into more details,
I recommend you to get a little bit more understanding about binary arithmetic, and how numbers are
uncomputable architecture, how numbers are stored in memory, etc, especially for floats
and all that's a completely different representations. So with that said, we're going to see now
how we actually use NumPy without worrying so much about the low level details. And that's
the beauty of NumPy. So we have already done our low level explanation of binary arithmetic,
why unknown vice important and all that if you skipped it, that's perfectly fine, you
will not need it. The reasoning was to include was that if you're in this tutorial, you're
probably looking for fast and efficient options to process large volumes of data. And that's
when all those things come into play. So let's without further ado, let's just get started
and start using NumPy as a library. So again, as I told you, a NumPy is a very simple library
for array, processing and numeric powers. To sing, it has a few objects, numbers, floats,
integer floats, arrays, and that's it. And it's very simple, but it's extremely powerful.
So, in NumPy, we're going to create these arrays, which look a lot like Python lists,
but there are going to be significant differences. The first one is, of course, performance.
If you go to the previous part, when we were discussing the binary representation of an
array of numbers, in Python and NumPy, you're going to see the difference between them.
So in this case, we're creating two arrays. And you will see right that the creation is
extremely simple. The only thing that changes we need to add this NP dot array, and then
we're passing in this case, a list of numbers. This is something we will usually be reading
from external sources. Now, how can you access individual elements of a NumPy array is this
works in the same way as with a Python list, you can say give me the first element, give
me the second element. And it's zero index, like, again, in a Python list. Slicing works
the same way. So in this case, up a zero to something, a one up to three rights, just
getting low level, right, on high level of the index, negative indexing, and steps, they
all work in the same way as with a Python list. So if you know how to use a Python list,
you will know how to use a NumPy array. There is one new thing right here so differently
from a Python list. And it's what it's called multi indexing. Let's say you have a, an array
this case B, and you need to extract three elements out of it, you need the element of
the first position, third position and last position, you can just type B of zero, B,
A to B at minus one, or, and this works, this also works for a list. Or you can use again,
mod the indexing, which is from B, I want to select the elements in zero to n minus
one first element, third element on last element, right, so you pass an int, another list containing
the indices of the elements that you want to select. And in this case, the important
part is the result. It's another NumPy array, it's not just individual elements, you're
creating another NumPy array, which again, if you're processing, it's gonna be a lot
faster. So arrays have types associated. And this is related to what we were speaking before.
As a NumPy array is a continuous is continuously assigning memory, the NumPy library needs
to know what's the type of the object you're storing, you can't just or you know, anything,
a string a number within it, because it will not be able to provide performance and optimizations for
arrays or non consistence insights. So for example, when we create these arrays only
had injures by default NumPy selected in 64, is because of the platform, it's a 64 bit
platform, you can tune this, and you can select us, we're going to see other sizes in a second,
when we created the array B that contain decimals or floats, it assign a different type, which
is float 64. Again, the default type is always six, at least in this platform that is 64
bits, it's going to be float 64 and integer 64. You can always change that you can say
Actually, I want these, even though these are all integers, I want you to create them using a float type, or as we saw
in our previous video, we can say it should be actually type integer x. So smaller integers,
for performance, for performance for better performance. Alright. So moving forward, we
were also going to see a few other types like for example, strings on the regular objects.
But as you're going to see this, there is no point of storing these things in NumPy
NumPy, stores numbers date Booleans, but not a regular individual objects as we're seeing
right here. There is a way to store strings, it's perfectly valid and it has its own time.
Its own type sorry, and it's related to the Unicode representation memory etc. But again,
NumPy is usually used for numeric processing. So the idea of NumPy arrays is we can create
multi dimensional arrays we can create the what we had created before. This is a one
dimensional array right? Just one dimension, you can create matrices, which in this case
are two dimensional, we have two rows and three columns. And NumPy has a ton of attributes
and functions to work with multi dimensional arrays. So the first thing we're going to
see is the shape of an array, which is two rows by three columns, how many dimensions
it has, it has one vertical and one horizontal, we have two dimensions. And what's the total
size of the array in this case, the total size is six, the total number of elements
we have, let's go one dimension. Further, let's create a three dimensional object, a
three dimensional array, which is basically a cube. In this case, for B, we have that
the shape is two by two by three, the number of dimensions is three, and the size is a
total count of elements. 12, you always have to be careful when you're creating these multi
multi dimensional arrays. If the dimension dimensions don't much, like in this case,
right here, where we have this second list that only has one less than bits in it, then
the dimensions will not match. And it will just tape you they'll use sorry, that the
array is of type objects. And the shape is only two only has two elements, these one
element, and there's another element. So in this case, we've done we've done it wrong,
basically. And you have to be careful when creating these these objects by hand. So how
can you index and slice matrices? We've done it for a one dimensional array. So we were
selecting elements, individual elements, give me the first element, give me the second element
cetera? How can we do it with a matrix with a matrix, what we're going to do is going
to be very similar to what we did before. The difference is that now we have to account
for multiple dimensions when I do give me a at one, is it the column add one, or is
it the row at one? Well, as you can see, it's the row. So this is going to be right here.
012. Right. And there is also another dimension, right? So this is 012. In terms of index,
index positions for our slicing. So here, how can you get the first element, the first
element of this second? rope. In that case, you're going to first select the first row,
the sorry, the second row, and then select the first element. And that's what you get
number four. But there is a better way, which is by using the multi dimensional selection
of NumPy. In this case, you're going to say from this matrix, I want to select and here
you're going to pass a in this case, you're going to pass dimension one dimension to dimension
three, dimension four, etc, right. And these are selectors for each one of those dimensions
that you're passing. In this case, we say, for a row level one, the element, the position
one second element, and for a column level, we want the first element in it. And it's
the same thing as we did before. The advantage of this index and keeping in mind and remaining
it is that it will also let you add slicing, right, so you'd say I want to select every
thing from dimension one, which is rows. So in this case, you say from zero up to two
is these two ones, the two is not included upper limit the same as as Python. And then you can also pass other other dimensions,
you say, I want to select every row, that's fine. But then I want to select from column
level, I only want to select the elements up to two. So these two and these two, and
the two, right, so 124578. These all works as intuitive as it gets. Remember this syntax
is the important that you need to keep in mind. Moving forward for modification, you
can say I want to assign these new array to this entire row, right? So if the dimensions
match, that is going to work now 10 is equals it's added to the second row, or you can just
use what we call usually an expand operation. We're just going to say for row number two,
I want to assign the number 99 and NumPy is going to take care of expanding it into this
corresponding array, given the number of dimensions that you have So so far that selection, it's
simple, we're going to see also is that NumPy has a huge advantage of containing a ton of
operations you can perform on top of your arrays and matrices, your multi dimensional
arrays in general. So the first one is the all the summers basic methods we have. So
given an array, all these methods are already built in the sum, the mean average, right,
standard deviation, variance, etc. And that also works for matrices. So in this case,
we can get the sum the mean standard deviation, or we can do it per axis. So this is very
useful, we can get the, the here, let's compare these two, there we go, we can get the some
of these, what is this first column, the second column or the third column, we can get it
the first row, second row and the third row. So it's either this dimension, this dimension
one, or it's a vertical dimension, which is x equals one, right? So per row per column.
Or, if you have more dimensions, you can just keep increasing the number of this answers.
And that's just going to work as expected. Broadcasting vectorized operations, this is
a fundamental topic that we're going to talk about. And it's going to be extremely related
to Boolean arrays. And these are a few new things that you have to keep in mind with
working with NumPy. And now we're going to talk about vectorized operations and broadcasting,
which can be a counterintuitive topic at the beginning, but then you're going to understand
how much sense it makes. It's one of the fundamental pieces of NumPy. We've seen how NumPy works
in a very general way we saw the multi dimensional arrays and all those advantages. But you might
be thinking, I mean, I don't need another library just to complete the summer domain.
When I show you the vectorized operations and broadcasting part, this is going to make
a little bit more sense of why NumPy is so important. So to get started, we're going
to have these array, which is a right, that's just very simple array vectorize vectorized
operations are operations performed between both arrays and arrays and arrays and scalars,
like in this case right here, which are extremely fast, they're optimized to be extremely fast.
In this case, what we're going to do is we're going to sum the entire array plus 10. And
what it means we're going to see an example of what happens without with Python. But what it means is that let me show you
the results, that each one of the elements within the array will be applied the same
operation. So usually, that's the concept of vectorizing an operation you have the number
and then this operation is applied to each one of the elements in here are actually in
these other one, right, so here and here and here. And here to result in these new array,
the operation is expressed at an array level, right, we say a plus 10. That's it. But then
again, internally, this is broadcast said to each one of the individual elements within
the array. And this gives me how a plus 10? Well, a times 10, for example, which also
in this case is we're playing the times 10 operations to each one of the elements in
the array, resulting in a new array with the result of that operation. And these resulting
in a new array is very important, because as we're going to see, NumPy is an immutable
first library, it will not any operation, you performing an array will not modify it,
but it will return a new array, if we check the status of a, you're going to see that
the elements are the same, it has never changed, we are creating a new array and returning
it. There are ways to override these behavior if you want. And this they all these operations
were performing these way always have the interface of plus equals minus equals times
equals etc, which will indeed modify their rights. In this case, we're making a broadcasting
operation, adding 100 to each one of the elements in this array. And now this operation was
immutable. A was modified and did it hasn't returned a new operation. If you remember
from your pure Python skills write the correspondence of vectorized operations are list comprehensions,
in which you're expressing an operation for each one of the elements in your collection.
Right. So that's a list comprehension. It's a it's pretty similar to what we're doing
with NumPy. The main difference is that this is all optimized and extreme. It's extremely
fast. So, the operations are these vectorized operations are reduced broadcasting doesn't
need to be only between arrays and scalars can only be between arrays and arrays. So
in this case, we have a and we have B and showing you right here. And we can do something
like a plus b. And what you're saying is that if there is a correspondence, right, so zero
plus 10, one plus 10, two plus 10, right? Let me, let me do it in this way. 110 210
and 310. There we go. And that's the result that we get right here. So these for these
to work, you of course, need the arrays to be online and to have the same shape. But when that does work, then the operation
is extremely fast in memory. And it's aligned, it's a vectorized operations with seen so
far. Why is this topic of vectorize operations so important? Well, because of the following,
which is bull in a race. And this is a very, very, very important thing. If you don't completely
get it now, I asked you please, to go and check the exercises we have for this lesson,
because we're gonna use it a ton. And we're gonna, we're gonna see that in pan, this,
the same syntax, the same primitives of Boolean arrays, a play apply, and we're going to use
the same things. So why are Boolean arrays similar to vectorize? operations? Well, all
these operations we've had performed here are just arithmetic operations, mathematical
operations, plus something times something, etc. If you look at the operators that you
have in your programming language, it's it's not only mathematical operators, like plus
or minus, or times, you also have Boolean operators. And the question now is going to
be what happens when you apply Boolean operations, when you apply Boolean operators to it. So
given our right, we had, what ways we had to select different numbers. For example,
in this case, we need the first and last element, we do zero and minus one. That's, that's the
way we saw with NumPy. We also saw the traditional Python one, right, so we can say a zero, and
also want to get a minus one. So this is the first, the first way of selecting these elements,
we know there's a second way with multi index selection. And there is a third way and this
is new, which is with Boolean arrays right here. So in this case, we're gonna say I want
to select the elements in this order, right? And you're gonna pass either true or false
if you want to actually select the element or not, right, so if you have four elements,
you have to pass four Boolean values, saying, I want to select this element, I don't want
to select these ones. I mean, I don't want to like this element. And I do want to select
this element right here. So I want the first one, and the last one, and the result will
be the same 030303. So so far, it's it's nothing terribly new, right? So this is new, but it's
not extremely complicated. We are showing you a brand new way of selecting data, you
can select regular Python multi index, or a Boolean array. Now, you might be thinking,
well, I manually write true false false, true, true false, for I don't know how many records
you have a million records, this is not scalable, right, you will not say to write all the strong
forces. But this is actually very important, because these arrays are the ones that are
the result of broadcasting Boolean operations. So we saw again, regular arithmetic operation
like this, but we also have it for Boolean operations. So we what happens if we ask a
greater than or equals to the number two, right, and array A is this right here is 0123,
then the result is false for zero, false for one, because they are not greater or equal
to do true for number two, of course, and two untrue for number three. So all the individual
elements that match this condition will have true and false. In other cases, this is the
power of Boolean arrays, we will be able now to combine these operations. So now we can
do a greater than or equals to two, right that a equals A being greater than or equals
to the number two. The advantage of this is just filtering, we're
filtering No, no numeric arrays very quickly with a very familiar syntax a greater than
equals to and we just provide that as the index of the operation. It's pretty much What
is happening right here? We're saying use these Boolean array. It's a willing list,
right? is a Python list with Boolean, to filter or sorry to select elements based on that.
But the question is, how do we construct that list of Boolean? Well, in this case, we have
constructed it by including a predicate by including a condition that needs to be matched.
The result, again, is filtering. It's a query method, you're looking, looking up some data,
you're saying, Give me all the elements that match this condition. So you can say, for
example, these values can be of course calculated, you can say, give me all the elements that
are greater than the mean. Or you can actually provide other Boolean appraiser operators
like for example, all the elements that are not greater than the mean. So that means they're
less or equals and the mean, or you can also include all their Boolean operators like or,
or, and so or n and in NumPy, are expressed with a pipe or an ampersand ampersand. Because
we can't use just the regular or and then in Python, we can, but it's a good choice,
they've selected this. So again, this is the concept of Boolean arrays, we are going to
construct these arrays that artist Boolean representations or Booleans, based on conditions,
right, so we have this matrix, and we're gonna say I want to select these one, and these
one end is one, etc. So in that case, this is the result right here. This is the result
of that. And we can generate a dynamic Boolean array, we never manually type all these right,
we don't sit and say true, false false through etc. We just Run Query filtering option, a
Boolean operation, which results in a Boolean array. And now we can use it as filtering.
So again, the idea here is that the operations we saw in broadcasting before, a timestamp
are also defined for Boolean operators. Boolean operators return Boolean, a race, which can
be used in filtering, that's the idea of all of it. And you can even combine these operations,
you can say, A equals zero, or a equals one, a less or equal to two. And it's also divisible
by zero, you can combine all these queries. So now it looks a lot more powerful than when
we were doing before. So moving forward, talking about linear algebra very quickly. And this
is we're approaching the end of the NumPy lesson. The part the important part of of
linear algebra is that NumPy already contains all the most important operations for it already
optimized with low level semantics, it's going to be extremely fast, adult product cross
products, and all that transposing majors is all that works as expected. And again,
these might be very important, specially, for example, machine learning, etc. It's it's
extremely important. And finally, to wrap up what we saw in our, in our binary explanation
at the beginning, what you might have escaped is the difference in sizes between NumPy and
Python, the differences in terms of performance between them. So in Python, a regular number,
this is just a regular engine in Python, that total size is 28 bytes in order and just let
this thing for a second. The total number of bytes, not bits bytes that you need in
Python to store a simple number, as the number one is 28 unit 28 bytes to store just the
number one is extremely, super space consuming, right? It's not very
efficient, larger numbers will even take more bytes to store them. What's the size of the
integers? Well, we've seen it we have, for example, we can create integers with eight
bytes. We can create integers with one byte right which were something like here we have
np.int eight will already know how many bytes has only one byte, right, but you can have
control of how many bytes or bits write your numbers will take. And you can see here the
difference between the size of an integer in Python which is extremely large 28 byte
on NumPy and also the difference in performance. Let's say for example, we want here you have
the ultimate difference in size of lists, which is also significant. But I want to focus
on performance, we have two elements two, we have one list that has the first 1000 numbers,
I will have a NumPy array that has the first 1000 numbers, we're going to perform the same
operation in both of them. Let's use the Python one. First, we're going to do the Python one
first. In this case, we're, we're squaring all the elements in the list, okay, the elements
A squared, and then we're summing all the operations might so we express it at saying,
create a new list, x times x, sorry, squared, 4x, nl, and then some everything, how much
time it takes 321 microseconds, we're gonna do the same thing with NumPy, we're gonna
say NP dot sum, a square. And you're gonna see that it's a lot faster in the NumPy perspective,
then the Python perspective. And these are all very, very tiny, tiny operations with
small numbers. What happens if we add more numbers, let's add two more numbers here.
That's odd. Two more numbers here. And we're going to do the same two operations. So as
you see here, that that the units have even changed, we're still in the microsecond layer
here with NumPy, we've gone to the millisecond layer in Python. So as the size of your objects
increase, NumPy will prove to be extremely fast compared to Python. So there are a few
other functions you can see here, for example, extracting normal, random numbers, etc. I'm
going to live let these for you to look, if you're interested in them, I remember you
have the exercises, which can help you solidify all the concepts we discussed. And we're going
to move forward now to work with pandas, we're going to see also visualizations are gonna
keep moving forward this data analysis with Python tutorial. Now, it's finally time to talk about pandas
is the most important library that we use for data analysis in our day to day basis
with Python. It's a library that will aid in the entire process of your data analysis
project, you're going to start getting the data, step one, getting the data from multiple
sources, like databases, Excel files, CSV, files, etc. That's all gonna get into pandas,
you're going to be processing the data, right? So you're going to be combining merging, doing
different types of analysis, you're going to be visualizing the data, right, so a bar
chart, you're going to be visualizing the data with pandas, and you're going to be creating
reports, you're going to be also doing simple statistical analysis, you're going to be doing
machine learning close to it, with the help of other libraries, but everything from the
platform that provides the pandas library, it's, again, one of the most important libraries
in in in the data analysis data science ecosystem with Python. pandas has recently released
the version 1.0. So we are talking about a very mature library. It's been around for
a long time now. And again, it's the primary library that we use in Python for data analysis
and data science. So I'm going to do a quick introduction to the data structures of pandas
house, and we're gonna understand how they work. So you can start building right the
phone, we're gonna start building the foundations, I need you to be very familiar with the way
the data structures from pandas are processed. And then we're going to move into other things
like reading files, grouping data, etc. So to get things started, we're going to talk
about the first data structure to pandas house, which is this series. In reality, pandas has
two main data structures that it uses all the time, and it's the series under the data
frame. The data frame is the one you will probably be more familiar with. It looks just
like an Excel table. But we're gonna start first with a series. Okay, so just stay with
me here. We're going to talk about a series for a second. In this case, we have important
pandas, and we have also imported NumPy. As, as you might imagine, as I told you before,
in the NumPy part of this tutorial, we're saying NumPy is fundamental for data analysis
because every other library pandas, matplotlib, they all sit on top of NumPy and you can see
it right here. We're gonna be using some features from NumPy within this lesson, too. So these
is a series in pandas, what you see right here, it's The concept of a series is this
ordered sequence of elements right? Or indexed right with they are all indexed by a given
index, of course. And you might think that this looks a lot like a Python list, right?
So in this case, we're storing the population of countries, right in millions of inhabitants.
In this case, it's jevelin. g7. pub is because we're getting the population of the Group
of Seven, you can console the Wikipedia page. But basically, we are storing population in
here in this series. And again, it looks a lot like a list, but we're gonna find a ton
of differences in here. So the first one is that the the series has an associated data
type. And this is something we saw in NumPy, when a NumPy array couldn't hold different
types of objects, we were all we were only having one type of object. In this case, it's
float 64. So all the numbers of the series will be of type float 64, the underlying data
structure, the 10, this is using to store these objects is a NumPy array. So a second
difference we see very quickly is that zeros can have a name, right. So now when we display
the series, we see that it has a name. Now it might not make a ton of sense. But once
this series is part of a data frame in the form of a column, then the name is going to
make a lot more sense. So moving forward, again, we saw that A has a type. And again,
this is because the backed the data is backed by a NumPy array that you can always consult,
you can check values of a series. And you're going to get the array that it's backing up
that pandas series, right, so you can see that it's a NumPy array. Once you have these series, we were just consulting
here, design pop, you can in you can select elements as you good in a regular list, right?
So for example, give me the first element, give me the second element, the last element,
etc. And that's because a series inherently has an index, similar to list a list when
you create a list in Python, right? So if I create L equals a, b, and see, but there
is something wrong here missing, quote, this list, we don't say it right. But the idea
is that there is an index here, zero, this is one, and this is two, right? In the pendous
series, this is a lot more explicit, each element has an associated value within it.
And you might think that is pretty much the same thing. They're all they're both the list
on the series, there are both sequences, they're ordered sequences of elements. But we're going
to see that there is a fundamental difference, and is that we can arbitrarily change the
index of a series. So by default, when we created it, we didn't assign any indices.
So by default, it was a range index from zero up to n minus one elements. But you can actually
arbitrarily again, say, what is the index of your series. And in this case, these data
structure these series has now these indices that we're seeing right here. Why is this
important? Because now we're going to be referring to these values, not by a sequential position,
but by a name, but by a label by the index, which has a meaningful name for us humans.
Okay. So now, these thing looks a little bit more like a dictionary we could say, than
a list, we started thinking that a series was similar to list but now, we can think
that a series is limit similar to a dictionary. But wait, don't get me wrong here. The series
has a fundamental trait, and it's that it's still ordered something that didn't happen
with. With dictionaries, dictionaries in Python, are not ordered, actually, in python 3.7.
They're ordered, but we shouldn't be thinking that they are ordered their unordered data
structures. In this case, a series is in the order. So it has both those advantages. It's
ordered candidates always before friends, that's as we decided to create it, but also
it has names or labels or keys associated with the values as a dictionary. So this is
creating the series from scratch. Right? All these methods, you can see you can create
a series bypassing the index it doesn't have To be a two step process where you first created
the series, and then add the index, in this case, you can do everything at once. And the
indexing is now going to be done by those indices, right. So those labels that make
up the index will be used to index specific data. So g7 pop, we see has these countries
with these population. And now, before the index, we were saying, I want to get what's
the population of Canada, and then we had to remember, what was the position of Canada,
oh, it's the first help countries, we have to do g7, pop zero. With the index, now we
can just consult what's the population of Canada, what's the population of Japan. And
as you can see, the syntax is the same as with a Python dictionary, it's just pretty
much same, you pass the key and is going to get the value. So again, summary, the advantage
of a series is it's it's a ordered sequence of elements, backed by a NumPy array, very
efficient very fast. But it also has an index that can take any labels we pass,
so it's going to make it a lot better for indexing, you can steal when you have a series,
you can still get the elements by the sequential ordering. After all, it's a sequential data
structure, and doesn't matter if you have in an index, you can still say, Hey, I know
we have on the index. But if you want to get the last element, or the first element or
the second element, you're going to do that by using the attributes, I look at it and
say to this series from this series, I'm going to ilok locate by sequential position, these
element the element in position zero or the last element. And that still works as expected
series also support multiple indices as we saw with NumPy. So in this case, we can get
two elements out of two, three n elements, you can pass multiple indices. And the same
thing happens with more with sequential multi index series also support range or selection
or slices. But there is a fundamental difference here, this is very important here attention,
there's a fundamental difference with Python, and it's not in Python, the upper limit of
a slice is not returned. So from our list that we created before, if I do l, up to number
two, I don't get the index See, right, so this is zero. This is one, this is two, two
is not included in our pandas series, the upper limit is indeed included. So if when
you asked from Canada up to Italy, Italy is in the result. Okay, so this is something
to consider when using index selection in pandas, I think this is still valid, it's
very, I understand the reasoning behind it's just different from Python. So, you should
remember, Boolean arrays, which was a topic we discussed in our previous lesson of NumPy.
Boolean arrays is still a thing in pandas, the difference is we instead of saying Boolean
arrays, we should say Boolean series right, the idea is that we will be able to perform
operations on top of series. So for example, right here we have mathematical operations
on top of series in this case, we have the zero D seven pop, which as I told you the
beginning is in millions of inhabitants. If we want to get the series of interest units,
we will need to do Jessamine pop times 1 million and there we go now is in terms of units these
operations right these vectorized operations the bras these broadcasting operations can
also be performed with Boolean operands. So instead of a multiplication, a summation and
subtraction, etc. We can add we can use a Boolean operators. So in this case, we get
asked what are the countries that have more than 770
million inhabitants we will receive receive their assault is a bull in aerates, Nebraska,
right? Well, let's hear it you know, but basically, it's the same concept of with us with a NumPy
Boolean array. Canada, friends, they do not have more than 70 million inhabitants in Germany
does have seven more than 70 million inhabitants here. 80 on the same for Japan, so Japan Here
is the same on the same for the US, the US also have past more than 70 million inhabitants.
So again, the Boolean array or Boolean series in this case, works in the same way, as with
NumPy. And selection also applies. So I can now select, I can say, give me from these
series g7 pop, all the countries that have more than 70 million inhabitants, the value
is more than 70. So now, again, we are building filtering, we're building a query language
if you want on top of pandas, we're selecting data based on this condition. Remember, when
if you ever have trouble remember all these, the idea is that you can always track down
the way this index is being built. In this case, we are it's not that the selection knows
anything, these first election knows anything about how to select countries with more than
70 these operation was performed first, which resulted in these series. And now this series
will be indexed by these array, this Boolean array. And the result is as you can see it,
and again, these operations can be run with calculator methods, and all the operators
we saw in our previous lesson, which was not, which was or this irregular pipe, or, and
amberson, which is the and all these can be applied in any order you want. So if we read
this thing, which is complicated in purpose, it's worth saying give me all the elements
that are above the mean, minus two standard deviations or below the mean, actually, above
the mean, and here was below the mean, or if this isn't correct, but it doesn't matter.
It's just an OR operation between two ends of the it's actually, it's above the mean,
minus the standard deviation. So we are applying this operation or right, that operation we
have before so they're not the or, and the and they all work with Boolean selection as
well. The operations we saw from a mathematical perspective mean in in statistical operations,
we saw a NumPy. Some mean, average standard deviation, we're actually using standard deviation
before, they're all still relevant in this case, but also you can use traditional NumPy
functions with our pandas series, because again, a panda's series is internally backed
by a NumPy array. So this is all the same, as you can see, here
is an example that it's a little bit more clear, we're getting all the countries that
have more than 80 million inhabitants, and all the countries have less than 200 million
inhabitants. So it has to be above 80. But it also has to be below 200. Okay, or in this
case, we say either above 80, or below 40, or below 40. Right. So that's with the OR
operator or the NOT operator. Modifying series is relatively simple. Whenever you have a
value, you can just assign it all together. In this case, we're saying Canada is now 40.5.
I don't know why we just wanted to do it. This is by index, you can also do it by sequential
positions. So in this case, we're going to say the last country should have 500 now.
So we're going to see a right here, the last country has 500 now, or you can also modify
elements based now bool and selection. So you can say all the countries that have less
than 70 million inhabitants, all these from our previous query, all these will now be
99.9. So as you can see, it has changed all these countries. So this the assignment works
by direct indexing, or also works by Boolean indexing. And this is going to be extremely
important when we are cleaning data. So let's move forward and start talking about data
frames now before you have exercises in for series, and also for data frames, so I recommend
you to check them out. So talking about data frames, this is what a data frame is going
to look like. It's pretty much the same thing. us an Excel table. So this was our series
and this is going to be our data frame. It's a table. So it looks a lot like an Excel spreadsheet.
And actually, it's very common to create pandas data frames out of CSV files, which are tables
basically. And I'm going to create it we created with these data frame object I created. There
you go, these are data frame. And as you can see, right, it has columns that we have assigned.
In this case, we were designing the columns, and we have rows of values right below each
one of these columns. Why? What's the similarity with with series, and it's not a data frame
column will be basically a series. So we can think a data frame is a combination of multiple
series one per column, we're going to assign an index to the data frame the same way that
we did with our series. So in this case, this is our data frame. Sorry, right here. This
is our data frame that has the index, right? And it has the columns as we had before, what
columns Do we have, what's the index of the data frame, these are all attributes that
you can consult, there are a couple of very interesting methods from data frames that
we use all the time. The first one is the info method. That's going to give you quick
quick information about the structure of your data frame. Right. So it's going to tell you
what columns you have population GDP surface area, HDI continent. And it's also going to
tell you the types and how many no values you have, it's actually telling you how many
non null values you have. But we use these when we're cleaning data to quickly then define
those columns and have missing values, we can check for the size of the data frame,
we can check for the shape. And this is similar to a matrix right, a two dimensional array
in NumPy is pretty much a data frame. And also similar to info the voice again, to check
a summary of the structure of the data frame, we can also use this cribe, which is going
to give you a summary of the statistics of the data frame. And in this case, what we
see is that for each numeric column, only those columns are numeric continent is not
here, for example, this is continent so you can see the type is object is a string, basically,
all the numeric columns, we're going to have summary statistics for them. So for example,
for population, how many elements we have, what's the mean, right? What's the average
Romney, what's the standard deviation, the minimum, the maximum, and in between a couple
of percentiles 25th 50th and 75th percentiles. So this is quick summary statistics. And we
do this a lot. So keep in mind, his crime method is very popular. As you could see, in the in the info method,
the columns have associated types, okay, so this is very important. They continent is
an object that means that it's basically a string HDI is a float and surface area is
an integer. And that's because NumPy, pandas is automatically with through NumPy, is automatically
recognizing the correct type to assign to each one of the columns. This is similar to
what we saw with a series in which the series contain natural datatype, a series was part
of a given data type. So that's something you cannot change. And in this case, checking
value counts, you can have a quick reference of the types of your series. So moving forward,
how will we we will be selecting data from series Well, there are a couple of methods.
And this might be a little bit confusing. So what I'm going to do is I'm going to skip
and just going to give you a quick reference first, and then you can read if you want through
the process we follow here, given a data frame, and this is just two quick rules, given a
data frame, you're going to select by index using the lock attributes. So the lock attribute
is will let you select individual rows. So for example, when I get Canada and that's
the value of Canada, when I lock attribute will let you select similar to the series,
the row by sequential position. So let's say We want to select the last row. In this case,
it's the United States of America. So again, look lets you select a select rows by route
by index, give me the row under this index, I log will let you select rows by sequential
position, give me the last row, the first row, the second row, etc. And finally, without
using lock without using a lock, just by saying the f up something, you are selecting that
column, give me the entire give me a V and tire column population right here, the entire
column population. So what you're seeing here, first, first of all, this is a quick reference
dot dot Lok will give you an element by index, I look we'll give you an element by position,
I wrote by position and just doing df, on some things gonna give you the element, the
column sorry that you are passing. So it's like, both look on I look, look and I look
work in a horizontal ladder, give me this, while bf art, whatever works in in a vertical
montanus, which is getting you a given row. But something more interesting here is that
all the results, these one and these one and these one, they're all series, what are being
returned our series. So that's what we saw before. And the way it works is first, if
we focus in this last example, we're going to see that it's pretty standard, just these
series right here was is a one return I remember it has a type and everything. So that's, that's
fine. If If we ask for a row, like in this case, we can get for example, here easily.
There you go. The result is also series. But what you can see here is that this thing is
kind of transposed in a way dot here was the volume of this year is population is here,
and GDP is here and surface area is here HDI on continent on here you have volleys. So
it's it's again, it's it's being transposed, right from vertical to horizontal, in our
regular series manner on the index of this series is extracted as the name that the column hot. So in this case, the
name right there is the value of the index that it had. So you can read more about it
right here. But I just want you to remember these rules don't lock you select by index
dot I lock you select by sequential position, the F at something you go by column, there
are times when these might not apply. So or not want to apply, there will be some issues.
So for example, if your rows if your index is numeric, you might have issues with these
form or dot form, just respecting these three. For now, it's gonna get you any element you
want to get either by row or column. So from what we've seen, the oldest slicing also works
as expected. So we can get, for example easily, or we can get friends up too easily. So the
upper limit is included. But again, it's built look and we select by indices from France
to Italy, we can also do the second dimension similar to the way we worked with NumPy, we
can do second dimension here. And we can get all the countries that are from France, or
to Italy, including Italy, but only the population column or population and GDP. So here you
can see the second dimension being applied at the concept of of multiple dimensions in
selection being applied also to famous for ilok. It works in the same way that in that
then multi index and the slicing. So we get for example, from one to three right in sequential
positions. In this case, the upper limit is not included. So that's another difference
from what we have. And we can also do multi dimensions we can say give me the countries
from one to three and the column should be 0123 should be the third column, the fourth
column, the column under index three which is HDI, so that also works as expected. And
again, recommended, always use Look, I like to select rows and just use the naked data
frame to select columns as we saw before. Now moving forward, conditional selection
Boolean arrays will series, whatever you want to call it. This also works for data frames.
And it's very important, it's a way to filter data, it's a way for us to consult the data
when the when, when it so in this case, what we have is, we want to select all those countries,
which the population is greater than 70. Okay, so all the countries that have more than 70
million habitants, similar to what we were we did with a series, but in this case, we
want to do it with a data frame. So what you're going to see here is that we're going to construct
a Boolean series as we did in our previous video, right? So every country with more than
70, false false, true false. And we're going to inject that result, that Boolean series
in a dot lock selection, give me all the countries which match here than that the true value
in it. And remember, just this is kind of mnemonics are a way to remember, the way pandas
knows how to filter things is by matching this index, right from the resulting series.
With these index of the resulting data frame. These are two different objects, they are
completely different objects, but their index much. So here, Japan, March, Germany March,
so here, Germany, on Japan, they are the same, and that's why that thing is working us expect
that they This is just the first dimension, which is give me these rows, you can also
on the second dimension, saying give me these column, or these columns, right. So that's
steel, that's the awards us desire. So what about dropping stuff, you can say, whenever
you have data from you can say give me just these pieces, or you can say drop the others,
right, it's just pretty much the same. Dropping is very simple, you can drop by index, drop
this value drop Canada altogether, period, or drop these indices can in Japan, or you
can also drop columns, drop population, and HDI as columns. These ways also have a more
advanced usage, which is with access similar to NumPy. I don't recommend them so much,
but you can still use them and see them here. So all the operations we've seen, so far,
they're all working. The most important part here is the broadcasting operation that we're
going to do between series. So we're going to create a new series crisis. And I'm gonna show you what it looks like. So we have here
crisis. And we're going to perform a broadcasting operation between between these, I'm going
to show you what this thing looks like first, between that two, these data frame on the
crisis. And the result will be that we will subtract, I don't know what's this number
1 million, subtract 1 million for each volume in here. And we're gonna subtract 0.3 HDI
for each one of those. So what you can see here is again, this alignment between columns
and indices, the GDP here is matched with these GDP and the HDI is much with these HDI.
So there are two different objects, two independent objects, these series and these data frame
here. But when we combine them with an operation like this, the the columns in this case are
aligned GDP, and HDI and they work together. So you're gonna subtract these value in all
these column, let me remove this, you can subtract these values in all this column for
all these values, I'm going to subtract these value here in these column for all these values.
That's the way it's going to work. So moving forward, what about modifying data frames?
Now I wanna I want to show you something. And that's when we were dropping stuff before.
We were not actually modifying the data frame. So here we did df dot drop Canada, but df
still has Canada in it. And that's because similar to what happened with NumPy these
operations are all immutable. They are not changing the underlying data frame. We are
storing. We are storing we're creating new data frames that store the result of the given
operation. So in this case, you have to drop Canada. The result is that the these new data
frame but the underlying That iframe is not changed. That's because again, they are immutable
operations. 99.9 operations in pandas, it are immutable, there are ways to change it,
there are ways to make the changes permanent. But for now, I want you just to think that
everything is immutable. Whenever you want to perform an operation, it's going to create
a new series. If you want to keep track of this, you will just need to do something like
df two equals that, or even df equals, you know, just to modify the current data frame.
Again, there will be a way to not do that. But we're going to save in a sec. So modifying
series more explicitly, that affrontare modifying data frame more explicitly, how can you create
a new column? Well, very simple. Assign a column, I said, let's say in this, this column
right here, it says similar to say, here, language. Oh, it's just read only. But if
I say language equals, and I can just write whatever I want. In this case, what we've
done is that the language, let me show you what Lynx had, in this case, was a tiny series,
it didn't have elements for all the indices in the data frames, but that doesn't matter.
pandas will match all the indices of our chill exist. And it will live like the rest. This
na n is what we use for a blank. It's another number from NumPy. We're going to talk more
about it when we start doing cleaning data. Data cleaning, sorry. So again, links France,
Germany, Italy, you can see the volleys are all up there. What happens if you want to
change a value the language series already exist, you want to change it or column or
read exist, you want to change it. So in this case, we're going to say df language equals
English. So we're going to change it all together, df now will be affected, and all the values
of language will be English. How can you relate How can you realize when there is an operation
that is changing the underlying data from the underlying series or than the line NumPy
array, it's usually when you have an equal symbol, remember, NumPy, we saw something
plus equals, in this case, whenever you have a plus and equals symbol is you're modifying
the underlying data frame. So for example, check this out, the Rename
function or method of a data frame will let you pass columns and indices to rename. So
in this case, we want to change the United States to USA, the EU, United Kingdom to UK
and Argentina to AR, Argentina doesn't exist in this data frame. But that doesn't cause
a problem. And that's why we want to show you, the US, UK were modified correctly, and
HDI was modified correctly. And a PC which doesn't exist, didn't cause any problems.
Now, why am I showing you this because remember, these operations are immutable. If I check
what's the state of the data frame, we see that the original data frame has not been
changed HDI a steel HDI, it doesn't matter if we renamed it before, it's still the same
data from the same thing for days, indices, all these operations are immutable. A few
more examples of modifying data just for you to look at. And something that is very common
for us is creating columns that are combinations of other columns. So again, this is read only,
but you can you can imagine, that I could do is hear something like for example, GDP
per capita, right? If I go here, and I do GDP per capita, GDP, p per capita, per capita,
and here I say is equals to the GDP, this column divided by this column, right? So I
do something like B, B three, actually, C three, C three, divided by b three, right.
And then we would extend the values all the way along here. In pen this, we could do something
very similar. We can do just any column, we can just perform operations, broadcasting
operations between them, in this case, GDP by population. And we can assign that series
which is a result right there. So it's a series we are going to assign that series to a new
column. So GDP per capita Now, there you go is now a column of our data for. Again, all
these broadcasting operations are extremely fast, they are backed by their NumPy array,
and they result in a series. So very quick statistical information, a few methods, right
to do summary statistics. We saw them with this crime method. But minimum maximums mean,
median, all that works as expected. Something that I want you to note here, if possible,
is that with pandas, we have, I'm going to change colors here, we're going to use red.
With pandas, you have this concept of a data frame, right data frame that has multiple
columns, multiple rows. And these operations are resulting operations are resulting in
just one series. So in pandas, you have your data frame, and you have your series. And
we could say we have individual numbers. And it's like always, the data frame is always
resorting back to this, it's like some operations will just return a series. And the series
can be used in a data frame, right. So in this case, these resulted in a series, but
then we merely use the series to set the value of a column. Right. So that's why understanding
series is so important. So there are a few more assignment exercises for you here. So
you can check them out and complete them if it's going to make a little bit more sense
once you're working with it. Finally, I want to give you a very quick introduction
to reading the external data on plotting. And to do that, we're going to use a few methods
that are very popular in there, maybe we can look them up very quickly here, we can say
read CSV, use the read CSV function from pandas. So these function, read CSV. And as we have
read CSV, we actually have a few others read sequel, read Excel, read XML, there are multiple
adjacent or multiple ones, read HTML will be able to automatically parse an HTML page
and read it. So a few functions like these like, what we're going to do with these read
CSV, right here is the structure of it. A few of these functions will let us import
data from an external source into our pain this workflow. So in this case, what we're
going to read is these BTC market prize volumes, so it's right here, if I open the CSV, this
is what it looks like. It's the date of the price taken a read and devalue the bread,
the timestamp, and the value the timestamp of the value no decide the price of bitcoin
2017. Now it's close to $9,000, I think. But just note inside, but again, this is a CSV,
and this is a CSV that we're going to be writing. To do that, again, we're going to use this
method read CSV, the method will automatically parse the CSV, as expected. And there you
go. And the process now will be for us to start tuning it to get to the right point.
So I'm going to show you a few customization SP customizations, we can do with the receipt,
read CSV function. So the first one, and sorry, let me tell you first, we have a ton of attributes
here. So we have a ton of customization to do with read CSV, you will not remember all
this, you will not remember everything out of the top of your head. So don't worry, you
can always go back again to the documentation and just practice, it's going to come naturally.
So the first thing, the first row of the CSV was considered to be the column names. So
in this case, this fine lesson have a column name, let's say I add it, I'm going to do
timestamp, timestamp price, you're going to save it, I'm going to rearrange the file and
re re read it. There you go. So by default, pandas is assuming that the first line of
the CSV is the rd columns. I'm going to go back into what it was. Right, and I'm gonna
show you again, that's the assumption that pandas is doing. We're gonna Of course, of
course, change that assumption, because in this case, our CSV file does not have column
names. So we're going to just say Heather equals none. And this is when we start seeing
the attributes that we're going to use from the read CSV function, read CSV. When I do
hether equals none for us going to be known. That means don't infer don't read a header.
Don't try to infer a header, a header from the CSV file. And the columns are zero and
one. So now I'm going to change the columns. And I say, actually to be time something prize.
And now what I'm going to do is show you the first rows. So you're saying here that I have
these df dot head method that I'm doing. That's because this is a significantly large file.
So we're going to say not not that long, but at least it doesn't fit in my screen. What's
the shape of the day CSV or the data frame? It has 365 rows, and we have two columns.
So we can do df the info, for example, to have a little bit more reference about we
have 365 values, there are no no values, and price is actually float, that Tamsin is an
object and we're going to fix that in a second. I'm sorry, that the F that head on the F dot
tail, are the methods we used to get either the first and files or the end row sorry,
or the last n rows, which are five rows, by default, you can change that and say, Show
me the last three rows, for example, that's something you can do. And again, the types
so the types is the timestamp in this case, the timestamp column was not properly parsed
as a date, he was parsed as an object as a string, which we don't want. So we're going
to use the function PD dot today time, something we're gonna explore in more detail in the
reading in the cleaning data cleaning course. Part sorry, if it weren't tutorial, we're
gonna use it today time function to turn these column D f, the timestamp into an actual date.
And now we're going to say, the F that timestamp equals to this function resulting, and now
everything looks as expected, there is one more change that we want to do, we want to
set the index of the data frame to be the timestamp, because by doing so, we can quickly
access price information led me see what was the price of bitcoin in 2000 1709 29. And
I make a mistake here, I forgot to do the LLC. There you go. So we have the value of
Bitcoin. On these particular date, forgot, look, remember that to get value from a particular
row, you have to do dot lock. There we go. So we are getting Dodd's particular value.
Because we've made a timestamp the index, we get artists value directly from the index.
So what happens if you want to turn this thing into an automated script, for example, when
I run this process, every day at 5am, whatever we can, we want to read the CSV, strip the
columns, rename them turn into timestamps, etc. This is what we've done so far. Read
the CSV without a header, create the columns, turn it into a daytime timestamp into a daytime
and assign it to the index. And that's the result again, well, actually, the read CSV,
oh, sorry, the read CSV method is so powerful that it will let us do all these actions in
just one call of the read CSV method, we there are parameters that will let you customize
the behavior to achieve the same results that we did with four lines of code right here.
So in this case, we're gonna say, read this CSV, don't assign a header, that's something
we do already or don't don't infer our header from the first line. These are the column
names. So we don't need an extra line, we can just say these are the columns names.
Oh, and by the way, the first column is going to be the index of the data frame. Oh, and
also part of the date. They've the index, it's a date, so part of the date, and we have
the same result as before. So now I'm going to pro try and same thing. There we go. So
you can see it's work. So very quickly pan this plotting. Alright, so we're going to
be doing here is I want to show you very quickly, I don't know what's this thing is as a vertical
scrolling. I want to show you very quickly that you can create plots with Hannah's interest
a breeze. It's so simple to create a block. So in this case, what we're going to be doing
is, given a data frame, you can always invoke the plot method. And the plot method, what
it's doing, it's using the map plot live library, something that you can check if you want in
the docs. But for now, it's not necessary with these, we're going to be more than enough.
What it's doing is just using, again, the regular plug library, as you can see dimopoulos
Library, which is part of the standard PI Data stack. And again, for us to access using
pandas is extremely simple, just df dot plot, you're done, you can set the plot as you want,
we're gonna see more details of matplotlib. So don't worry too much about that later.
So there is a more challenging example here that I can just run very quickly, you can
inspect the process we follow to fix the data. But this is what we have, there we go. And what you can see right here is the difference
between the Bitcoin and ether in this period of time right here, and they are both loaded
in the same chart. And that's because this is the resulting data frame, we have Bitcoin
on one side, and we have ether on the other side on we are plotting it right here, we're
creating one plot with all of it. And we are noticing these empty value right here. So
what we can do is we can go from December 1 up to January the first these period, so
we can select that period, is in that lock. And we can just go ahead and plot it again.
And this is what you see right here, the gap that we're seeing. So again, this was the
introduction to pindus. We have a real life example of pandas following up. Also we have
a little bit of data, more data cleaning on reading all the interesting files and sources
of data for in getting more data into the pipeline, right. So the idea is going to be
showing you how you can import data from Excel from SQL and then do the actual processing
and analysis. Now it's time to talk about data cleaning,
we have arrived to that point in our tutorial, in which we have pulled the data, I've shown
you how to manipulate it with pandas, the beginning at least the introduction to data
manipulation with pandas, and now it's time to properly fix it. For the sake of brevity,
we are skipping a few parts of the process of data cleaning, especially you're going
to find it in this first notebook that we talked about basics, conceptual, missing data
with Python with NumPy. And we're going to miss a few other things. But I'm just going
to mention them. pretty generic, pretty general form. And then you can of course dig deeper,
you can check our courses if you want to know more about it. Usually when we talk about
data cleaning, where it's in from a more conceptual level, we're going to talk about a four step
process. The first step is usually finding missing data, which is the simplest problem
to identify from a data set when something is missing. So you have car sales data. And
there is a car that has no name right? Or there is a card has no price, right? So there
is an number missing or there is a category missing and there's a string missing. And
of course, each one of those is going to have a different meaning how to solve how to fix
data set that is missing data, it can be very simple. If you can just for example, drop
the record, if you can fill the value, right. So for example, the prices fill in these missing,
you can fill it with the average value of the sales data or something like that. Or
it can be very complicated if the volume is important if you can't move forward until
you actually find that missing volume. And it can involve something like picking up the
phone calling your ETL team asking what's going on that the data is missing. Or even
if you're buying the data, you have to call the vendor, ask them why their ID if you've
you're paying for that and there is data mentioning etc. So it can be a very political process.
It depends what's your use case. But again, from a technical perspective, identifying
missing data and fixing it is going to be extremely simple. Once you have fixed the
missing values, then you start looking for the data is assuming the data is not clean
yet in this process of data cleaning. The second step is when there are invalid values.
So you have for example, column that is price and there is a string
within it right here. We're expecting only numbers and there are strings in it. So then
It's not going to be complicated to identify, it's not going to be too complicated to fix
it. But again, we're increasing the complexity until a deeann of these data cleaning process,
we're gonna reach problems that have to do with the domain of the day you're looking
right. So for example, you have a column that is customer age, and there is a value that
is 170. Right? So that is not an invalid value, it's a perfectly valid integer. The problem
is that given the domain, right, but speaking about customer age, is highly unlikely that
a customer is 170 years old, right? So in that case, the vowel is completely valid,
there is no missing data, there is no invalid values, etc, is just about the domain. And
this is when things get very complicated, because in this case, that example of age
is something that resonates with all of us, we know about age of humans. But if you're
working in a domain, if you're working as a data analyst, in a domain that you don't
know much about, right, then you might not be able to judge if a value is invalid or
not. If I am working in a biology lab, and I have something like white cells count per
milliliter of blood, I don't know what's what it's a good value, or what's an invalid value,
right. So it's, it's something you need to know the domain. So that's usually the the
most complicated part of data cleaning, when you reach the limit of everything is valid,
everything checks out. And now I need to make sure that these value is valid for these domain
that we're working. So again, this is the spectrum that we're going to be revisiting
today. So to get things started, the way pin this works with no values is is it has four
functions, which actually there are synonyms, it's going to be it's going to be relatively
simple, just trust me on that. There are a few things first, everything that pandas does
in the process of missing values, is related to the way NumPy works. So again, we're skipping
it, you can go to that notebook, check it out by yourself. But it's extremely simple.
NumPy has these objects and n not a number to identify a missing value or no value in
Python world to have the non value. But again, in pandas and NumPy, we're going to use na
n none on there, or in this case, at the beginning, we have these two functions is no n is na,
which are complete synonyms, we're going to find also is no and no we have it isn't a
and they're also complete synonyms. So no n na for pan, this is the same. You can use
the one you prefer. Sadly, I like is na because it's the way I learned it. I think for my
students I usually recommend is no, because it feels more correct. And it feels more self
explanatory. So you can use the one you prefer, if you can use is no, I think that's going
to be better. If you get used to ease in a then you're going to be on my side, just do
whatever you prefer. So again, it's no one's gonna say true or false, depending if the
value is no or none, right? And of course not No, it's going to be or not na is going
to be the opposite. So not na have not a number is false, and not an A of three is true. If
you get to this first notebook, you're going to set all the false e values on the true
fi values in detail in terms of Python, anything that is not empty or non etc is going to be
considered to be truthy. So anything you pass here that again, is not an empty string or
a no is going to be considered a true fi value. So it's no not no or is it a and none an A,
they both work also with entire series or entire data frames, right? So it's not just
for one of Valley you can pass an entire series. And the result back is going to be if the
series is if the series what values in this series are either no or not no, depending
what's the question you're asking either is null or not null. So in this case, we say
which one is of the series are no, this is not, no this is not No, this is no so this
is only true. And the opposite for the following method we are applying are actually function.
And again the same thing works with not entire entire data frame. So something we
do usually is if you look in to not know unknown, a few hacks that we usually apply are the
count on actually this be the sum of all the no values or not no value. So we have this
entire series, we can say how many not null values we have. And if we sum those, not no
values. In this case, we're going to get a result out which is the entire the entire
summary Have the nod no bounds we have asked, and the same thing is gonna happen if we say
is no. So if I do here is no, some, we're gonna get how many novels we have? And it's
pretty much the opposite of this question is no. And the way it works is in Python bullions
are pretty much integers, they're ones and zeros. So every true Val is going to count
as one and every four is going to count as zero. So if you ask for the sum of a Boolean
series, you're going to get out the result of the number of truths that are available
in that series right. So, in this case, we have to know values we ask how many knows
value we have is know that some we get two out, you can use these tricks to filter the
data with a series. So in this case, we can say give me all the values that are not known.
Right? Just not know. Also, something interesting is that both for data frames are for series.
The not not no is no isn't a not an A methods also, sorry, functions also work as methods.
So in this case, we can say instead of PV dot know, we can say s.is, no load s, that
is no. So now, it gets a little bit more, a little bit simpler. But if the final objective
of these core as equals alzarri, s selecting only the boundaries are no no, was to drop
the null values, then there is a simpler form, which is dropping, okay, so in this case,
we can say s dot drop in a, and we're basically invoking the same thing that is happening
here, we're missing we're just excluding sorry, all the missing values in the series or the
data frame, because this also works for data frames. So what's the one, one important thing
to remember here is that all these methods are immutable, we are not actually changing
or modifying the original series, the underlying series is not being modified, there is a new
series that is returned. So if I invoke s, again, this thing has is not modifying their
series, you're creating a new series, and that's the one that hasn't, that doesn't have
the missing values. Everything we've said also works for data frames. So right here,
with these on a frame, we can say how many, right? The first thing usually is to start
with an info method, right? So we have info, and we see that there are in total, four entries,
four rows, we can also do a shape, if we need more information about the structure of our
data frame. So there are four rows, four entries in our index, column A has only two no no
values. So that means there are two values that are actually no no, sorry, no, there
is column B that has three nought non null values. So that means that one value must
be known, and that's for column B, again, so usually info gets you very close to understand
the structure of your data frame and how many values there are missing. The same thing happens
with some, we can just do df.is, null isn't a and then some, we're gonna get a quick reference
of how many null values we have in that given data frame. Drop in a works in the same way,
but there is a significant difference. The way drop in a works in a data frame by default
is by dropping any row that has at least one, no value. So these row has no value dropped,
these row has no value dropped, these row has two new values dropped, this is the only
one that it's not being dropped, right. So it's very harsh in that respect, you can change
that to make it to the column only, only keep the column that has no no values, and that's
by switching the axis equals to one. And there is also a way to select a subset or thresholds.
So only delete rows that have less than three valid values. For example, in that case, you're
going to use something like the strategy of the drop in a you're gonna say, drop the columns,
the rows, sorry, are the columns because it is also works for columns that have all the
values and no, or drop. The This is the default behavior, drop all the rows that have any
value in an NA or specify a threshold, which you mean by basically saying, I need this
amount of valid values in order to keep the rope it's the way it works. Now, which ones
to drop is which wants to keep based on the fresco. So once you have identified it No
values, it's extremely simple to clean them to sorry, fix them. So the first method we're
going to see is fill in a within a particular value, we're going to say from this series,
I want you to fill the blanks or fill the missing values with or fill the anaise. fill
them with numbers zero in this case. So these two are numbers zero, or, of course, you can
use any statistical method you want. In this case, we can use the main. Remember, this
is not altering the series, the original series is still the same, we're not changing it,
it's creating new series because all these methods are immutable. The following method is or this the following
way This method works is by passing a method which is for field or backward fields, these
are the possibilities. And basically the way it works is it's overflowing, all the values
top down, at least in Fairfield, right starting here, it's dropping this value here, dropping
this volley here. And dropping now three here, as this thing is a nun, it gets replaced.
So this thing is three now, which gets throw up here. And now this thing is three again.
So that's what we have right there. And of course, backward fields works in the other
way, starts with four and moves, it moves it here and then moves here, etc. You have
to be careful when using these. Because if you have no no values at the beginning or
the end, then you're gonna end up again, with no values because there is nothing to fifth
forward, right, this is the first volley you have India. And all we've seen also works
for Donna friend. So both boggler fail for field or both in terms of rows for feeling,
right, so we have, we have these, these data sets. So we do for field row base is going
to be one to here too. And then five. So that's going to be for field x is one, if you use
for field x zero, then it's a vertical filling, right? It's going to go here, one 130 30.
So that's for the column, that is y here, one 130 30. So it's either for filling in,
in, sorry, this direction for failing, or it's going to be in this direction, depending
on the axes that you are passing. And actually, let me we're going to put the correct forms
with axes equals zero, it's going to be columns, it's going to be visit direction with axes
equals one, it's going to be row based. So it's this direction, right? So we had a no
volley here, that got fail in this way. Okay, moving forward, we what else we have, we have
here, checking for values. And we've pretty much seen this already, you can use the is
know, the sum method to get how many values you have. And there is also any an old, which
will give you very quick. These are usually called Boolean tests, you can say ask if there
are any values are valid, or all the values are valid is just to build more complicated
queries. So so far, so good. So the process we said was at the beginning, we were fixing
missing data, missing values, there is nothing in there. We have read a data frame, where's
our data frame right here? We have read our data frame from CSV from a database, and the
value is missing. No, there is a hole in it. So we have quickly identified it with isn't
a or is no, we were able to drop the ones we didn't want to keep dropping a or we were
able to fill the volume we wanted to fill fill a name that was simple isn't a drop in
a fill in a what happens when you're cleaning data that actually has a value, so there is
no nothing missing. But those warnings are invalid. So for example,
here, the sex column is a categorical column that only accepts an on f. d on question mark,
those are invalid they are, it's very simple to see an invalid value here because it's
completely out of the scope. The same thing as we have, for example, question mark in
the age column where we have we have a string in the age column, it's very simple to identify
that, how we're going to clean those. Let's start with sex first, because it's simpler
in this case, the first check we can do is with either unique or with volley counts,
I'm going to use value counts. We've seen this method before. It's a quick summary of
all the unique values you have. And in this case, volley counts also gives you a total
count for those values. How can you fix them? Well, there is a replace method which is extremely
intuitive. You can just replace in this case, we're changing all of these two F's and The
End two M's, and it can work in multiple columns. For those volleys, that again, we said were
more complicated to fix, like, in this case, we know age, in this case, is 290. And we
know because we know the domain, that 290 as an invalid age for a human. So we will
need usually in those cases, we're going to need more complicated fixing, and it will
involve more programming, that's the reality, you have to be better coding. In this case,
we know that these volley is invalid, because it's probably an extra zero. So all these
values, you're pulling a CSV with ages, and there are a total of 180 290 32 320, for example,
invalid values out of 100, right in the 100 places. And that's because there were typos
when they were creating the ages. So how are you going to fix that? Well, in this case,
it involves a little bit more programming, we're dividing everything by 10. So also, something that may be useful is dealing
with duplicates. And we need to first define what's going to be a duplicate value. So this
is, this is usually a little bit more political, if you want, you have to define what's going
to be a duplicate. In this case, we have a series that contains ambassadors, and each,
their master is the index, the country of the ambassador is going to be the value, right?
This is usually the important part. The rating here says the word conducting a party, and
we want to invite one Ambassador per country, we don't want to repeat ambassadors, ambassadors.
So in this case, what's going to happen is that these two in our humanize at least, we
can click clearly and quickly see that these two belong to the same country. And these
three belong to the same country. But here again, we have to define which ones are the
duplicate, if you want, and which ones are not duplicate. So for example, maybe we can
say the first one is duplicate, or we can say the last one is duplicate. So this is
the first one not duplicate, or actually can say this, the last one is one, and when I
bite, it's not to duplicate. So we're going to have political rules if you want for each
one of those. So let's see the duplicated method and the way it works by default. By
default, duplicated method is going to return true for duplicate for all the it's it, I'm
going to invert it, it's going to not treat it as a duplicate as the first instance that
it says. So the method is actually walking top down right now saying, Do I have friends?
No, I don't have friends. I'm going to keep it here. Because it's the first time I see
friends. Do I have the UK? No, I don't have the UK, it's just gonna keep it here. Then
it sees the UK again realizes the UK is already there, too. It's already present. So this
one is going to be considered a duplicate. Italy is here, it's fine. The first occurrence
of Germany, it's fine wrightstown, Germany, but then it says Germany two more times. And
it realizes that Germany was there. So those are now duplicates, right. So the way it works
by default, we can change that and change it to last to the last element is not considered
to be duplicate, and the other two are considered to be duplicate. And the same thing here.
Kim, here is the one consider duplicate. So it's either top down or bottom up depending
the way the parameter you're passing, it's either keep default or keep last, or you can
be a little bit more harsh on say everything duplicate it is actually to be needs to be
considered duplicate. So these two are duplicates, and these three are all duplicates, as you
can see, right there. Similar to the duplicated method, which pretty much tells you which
values are duplicated, it's it helps you identify them, you also have the drop duplicates, and
in this case, what this method is going to do is basically the same thing as before,
but dropping all the values are checked for true, right if the method is if the value
is missing, it's gonna just drop it. And the same rules apply default, last and false.
For subsets in this case, we have Ace, we have multiple, we have multiple players in
the data frame. But what happens is that these player Colby is present three times for humanize
we see Kobe three times. What is going to happen here is that the The way we're going
to think about duplicates is by understanding the correct subset that we should check. In
this case, Coby plain as sn SG is duplicated two times but COBie, playing us in SF could
be considered a different player if you want, because maybe it's a different season, or
it's a different, a different position they played. So in that case, we need to pass What's
this subset that we are going to consider duplicate, only check for the column name,
or check for the column name on or not check for the column name, which is the default
is going to check the entire data frame. And when that happens, then these two are considered
to be duplicate. So these one is a duplicate with this rule, if we put keep last, sorry,
keep false, both are going to be considered duplicate. So this second occurrence is the
duplicate one. And the last one is a completely different row, because the the value in position
is different. That's the way it works here. Moving forward with more cleaning of values,
we're going to talk about string handling. And this is a very neat feature of panelists,
that special types of columns will have special attributes. So given the column type, so df
info, which is an object, which is a string, right, in pandas, that all the strings columns
are going to have these special attribute which is str, all the daytime columns, something
we're not going to cover, but you need to know, all the daytime columns have a.dt, Math
attribute, all the categorical columns don't have a.ca t cat attributes. And those attributes,
str DT cart, they have a special methods associated the domain of that column. All the methods
associated with string are of course, we're string handling, or the methods associated
with DT r for data handling. So in this case, we're going to review all not all very good
subset of the string methods we can apply. And something interesting is that all these
methods have a very good have a lot of relevance. And they're related to the ones in pure Python.
So if you have a pure Python string, there's a split method. There is a contains method
or I don't know if there was a contain actual, it's actually, I think it's the in operator,
but there is a strip, and there is a replace, right, so most of the methods under the str
attribute in pandas have, have an analogy in the standard library of
string handling with Python. So starting at the beginning, this data we have, I'm going
to delete this this data we have, what we are going to do is split the values right
by an underscore. So in this case, that's what we have, we have split all the volleys
with that underscore, and we're going to use the special attribute is expand, expand sorry,
equals true. And what it's going to do, it's going to create a data frame out of that.
So we create a data frame with 70 columns. And this is what we have now. So we can keep
applying methods. So for example, contains or content contains, regular or contains with
regular expressions rights for you to see the power of it, we can just strip replace,
and we can do even regular expressions with replacing so we could fix something like this
question mark in a string, we could fix it with regular expressions if you know how to
handle them. And finally, something that is going to be very helpful when you're doing
data cleaning, is looking at the data from a visualization perspective. data cleaning
has a ton to do with statistical understanding of your data to when a volume is considered
an outlier. For example, it might be invalid, and you want to claim it. So but that's a
lot more about statistics. And this case, I want to show you very quickly, the mottled
leave library, I've been promising for some some time now, the mapa lib library. So far,
we've accessed it directly from pandas, from pandas, or we're doing a data frame dot plot.
It's these library mapper lib is the one backing all those methods and we're going to see how
to use it directly. Now. The model live library has two important API's we're gonna call him
one is the one that I don't prefer, which is the global API, but it's the most common
one. It's the one you're gonna find around the global API. And the second one is the
object oriented API. So it's around here. And usually there are there are ways it's
just two different ways of doing the same thing. Okay. The global API is an API that
it's in part inspired in MATLAB. It's been around for a long time on sadly Most of the
answers you find in Stack Overflow tutorials and books will be using these global API.
The way the word the one I prefer the most. And I'm gonna explain you why in a second.
It's going to be the object oriented API. But I want to show you both. So you have a
reference. If you follow me in this feeling of preferring the object oriented API, you
will always have to translate global to Opie. Why is it considered a global API? Well, we
have imported matplotlib.pi plot as PLT. So we haven't imported the whole module, the
whole Python module, depending how much you know about Python programming is going to
make sense or not. We have important the whole module. And now what we're doing is we're
invoking PLT dot figure. And finally, and then we're going to do a title. And then finally,
we're planning two things. We're plotting x, our plotting x squared and minus x squared.
And why is this global because we're invoking functions that are at module level. And there
is an object, the final plot, that it's being modified by these very generalistic and global
courts, right. So by by doing these call right here, I'm modifying the final result of the
plot. Let me show you a more complicated example. So you see the problems with the global API. If you look at these line, if you could delete
everything, let's actually delete everything. What is this line doing which plot is affecting,
you do not know, there is no object oriented way of saying in this second plot the plot
on the right or the figure on the right, or actually the sub plot on the right, I want
you to plot this thing, you're just saying it to the entire module. And depending the
order that you set it, is where it's going to land, that particular figure where it's
going to land in which plot, it's going to lend. Again, it's a global API. So we start
saying, I'm going to create a figure, trust me from So from now on, I'm going to start
drawing on it, there's going to be the title. And hey, by the way, it's going to have one
row, it's going to have two columns. And I'm gonna start drawing in the first plot these
one right here, these one right here on the left, okay. So now I have kind of activated
if you want that plot, it's active. So now I'm going to start drawing on it. So every
action that happens after this line is going to be affecting these blocks, these blocks,
right. So then I plot x and x square, I plot this vertical line, I put a legend, I set
labels, etc. And at some point, I just stop and say, Hey, now I want to switch the plot,
I want to now start plotting. Sorry, I want to start plotting here in this second one,
because I have just changed that the first line these one. Oh, sorry, the way it works
is by saying the first row, second column, but second plot. So now I want to start plotting
in here, every successive line will affect that line. And again, you can see that understanding
a code, given the order that the order in the sequence of lines is very hard. If you
have to debug a report that has a plot that takes 100 lines, then you have to keep in
your brain, what's happening top down, a different approach is going to be the object oriented
approach, in which we're creating a figure. And we're creating axes. So in this case,
we have in this case, we have right here, one entire figure in red. And we have in here,
purple, we have two axes. So these axes one, and this is access to so we have two axes.
We're going to create those using an object oriented approach. And we're going to keep
references to them. So we're going to say later, to these blocks to these artists, sorry,
I want to plot something. And that will be very explicit, it's going to be an object
oriented way. So the first thing is creating the figure on DCE. The axis in this case,
we have just one axis, that's it, but you can have more and then you say in this axis,
I want to plug this thing in this axis, I want to pull up that thing, etc. When you
have multiple axes, so I could show you. I'm going to go back again to that in a second.
But In this case in which we have four axes, right, so we create one figure. And it has
four axes, we do it with this subplots, method saying and rows and columns. Now we say to
the axes number one, I want to put this thing to axis number two, I don't want to put that
thing, right. So it's 1234. And now it's a lot more explicit, it's not depending on the
order, I could change this order, that doesn't matter. They're that the results are gonna be the
same oxes number four has yellow, regardless of the position that we're following. So the
map will live. And now that we have clear out the differences in both API's, maple leaf
has this very simple plot function, or method, depending on sugar enter global, that we'll
plot something you specify. In this case, we're passing all the values in x and all
the values in y. And in this case, we're passing a given line style, this can change with these
type of syntax, you're saying, I'm plotting this thing in X, I'm blowing this thing in
y second parameter and why. And I want you to use a straight line, it's a straight line,
yes, with this marker, the dot and in green. So this is if you are very familiar with it.
If you're very familiar with my bullet you can use to send links in other games, you
can just say line style market marker, sorry, color specific keyword arguments for each
one of those. So do we only have line plots in APA live? No, of course not. We have a
huge variety of plots. And by the way, there is another one here, if you want to see more
events are grids, you can create these grids and put different things in it. And again,
not only land plots, one good example is a nice scatterplot. So basically, we're plotting
X and Y correlation. And there is also our value, our color map, right. So given the
volume, there is going to be a change in color. So these kind of lets you plot three to four
dimensions of your data, the volume x, the volume, y, the size of the bubble, and the
color of the bubble. So where you're pretty much encoding four dimensions in just one
figure, right. So in this case, we're just using two different scatter plots, there's
more information here, we can also block histograms, that we've very quickly seen that with pandas
with pandas is, is very simple with just plot type histogram, current histogram hist, actually,
you can look it up in our previous lessons. So just go back into the index in the video.
And the histogram is extremely simple just takes the valleys you're plotting and how
many bends you want, or some more advanced arguments here, like the alpha level, etc.
But it's simple. And similar to the histogram, you can also create kernel density estimator
diagrams, which is very similar to distance to simulate if you want a continuous distribution.
You can combine these plots if you want, in this case, we are creating the plots were
plotting a histogram. And they were plotting the lines and they were plotting our changing
limits. But that's pretty much it. And you can also create bar plots, right? So in this
case, we have PLT dot bar, or here we have two bars are stacked, right? That's the different
way to look at it. And finally, check in outliers. You can always plot histograms or box plots,
right? So box plots are also a nice feature to have in here. So this was all with data
cleaning, we're gonna keep moving forward this tutorial, I want to mention one more
thing here. And it's there are notes here for kind of a task that you can follow with
data cleaning, which where we are identifying where indentifying missing values in given
positions with is known as an A. And right here, we're looking into more detail about
some statistical properties of the data, in case we need to clean it. Okay, so this is
little bit more events. And it's it's related to the concept of cleaning data given the
domain. So the statistical analysis can tell you that this value is an outlier. For this
distribution, the value might be valid. So for example, a human being is 90 years old.
That's, that's valid, that's a valid age. But if you're analyzing data about high school
students, and a human that it's not a year soul, it's going to be completely invalid
or it's going to be an outlier in that distribution. And you can treat it as such You valid valid
and clean it out, remove it, for example. So that's, that's deal a little bit more with
the whole statistical analysis you can follow here, it's a little bit more advanced for
the scenario. So let's move forward with the rest of the videos. Now it's time to get into more advanced features
of pandas to import external data. So we've seen already in our real life example, the
way we can import data from CSV files, and from SQL databases, right, we had actually
those two lessons, the objective of these part of the tutorial is to show you how you
can improve or get into more advanced use cases of importing data. So we're going to
start for example, with csvs, and text files. And again, you've seen it already. But here,
we're gonna give it an extra twist. So I'm going to show you more advanced features.
And for special use cases, txt files, CSV files, is, conceptually speaking, a CSV file
is a text file, it's just human readable text, right? That it's encoding information. The
idea for CSV file is that it's tabular. Right? So it's a plain text file that contains tabular
data in it, and it's separated. csv stands for comma separated, but it can be separated
can be anything, we can see more examples later. But basically, the idea is that it's
a text file that it's tabular into in a tabular format. So though, both CSV files and text
files will be read with the same method. So to get things started, I want to show you
the basic way we import will read data from, from from external sources using Python without
even starting yet, with pandas. So you don't need to know this, it's usually it's usually
productive if you want for data scientists or data analysts to understand a little bit
more how fire reading and writing works in computers, because there are multiple, multiple
concepts align, here, they evolved, operating systems processes your language, right, it's
not same thing to read a file with our or with Python or with another language. So there
are multiple concepts here. And even though pandas in this case can make it simple, very
simple to read and write data, you can get a little bit of a more advanced use case,
if you know the internals of again, both the operating system processes on your language.
So this the way we read data with a reader file, sorry, using pure Python, we use a function
open. And in this case, we're using a context manager, just a security feature, again, related
to to the advanced usage of reading and writing files. But it creates a file pointer, right.
And with a file pointer, you can then use the very simple API x point post. But they
but that pointer, which is something like red line, red lines, read a number of bytes
or characters, or you can just even trade FP as an iterator, just do a four line in
FP. But basically, we're going to do something like this, we'll start reading data from top
to bottom, just a month to, I don't know, we hit I've given in this case, we're doing
it just for a couple of lines. What else we can, it gets very difficult when you're reading
text files to process them, because it's usually hard to parse the structure of the file. So
it's not the same thing to have a funnel that is separated by comma separated by colons
separated by pipes, spaces, etc. So you're gonna see that once you want to get a little
bit more, I don't know a little bit more with an advanced usage, right, or a little bit
more fancy your calculations and and the way you parse the data, it's gonna, it's gonna
get harder. So that's why we're going to use pandas, or I'm going to show you in a second,
this is the module that is part of, of Python. So this is the file that we're going to be
reading. It's the XM review file, and I'm going to open it. And even though it doesn't
look like a CSV, it isn't either CSV. The difference is that here the separator is the
greater sign, it's not the comma, it's a greater sign. That's going to be what marks the elimination
between different fields in our CSV file. So we're gonna use the CSV module. And the way right here to parse the data using
that module is by passing a special delegator, right? So that's gonna be the type of work
you might need to do when you're parsing the data. It's not the same thing to have that
limiter dates a greater sign. It's not the same thing to have numbers for example, that
are enclosed in quotes. All those things right will change the way you work on all days is
going to be abstracted away by the pandas module. So to get things started, again, with
pandas, at least, pandas has multiple read underscore something methods that will work
for different sources, right. So we saw already have read sequel we've seen read CSV, there's
also a read HTML to directly parse information from a table, it's literally you can just
you pass a website's going to read information from a table, or read Jason read more advanced
formats like pocket, or Stata, etc. And, again, each file format will usually have a correspondence
in pandas, it's, I've never had the chance to rewrite my own stuff. To be honest, the
same thing is going to happen for something like Excel, which might need external modules,
it's not directly provided by pandas, but by installing those modules, you can easily
incorporate Excel files in your day to day work. So the read CSV file methods already
has a ton of parameters. So this day, the main characteristic of all these rate something
methods, given the amount of possibilities you're going to have with these files, there
exist a ton of different ways to customize the method invocation. Alright, so again,
CSV files, we saw, there are multiple things happen. csv is a passage that have a header
don't have a header, different delimiters different and closing of strings or numbers,
multiple things, blank lines, etc, multiple things are going to happen. And that's all
you're able to customize all that with the read CSV method. So this is the reference
of all the attributes you can pass to it, usually something
that I do, and I do this very often, and I use pandas a lot, and I still do something
like read CSV, and I get the documentation right here, to look into the, the parameters
that I think I need to pass to my particular use case. So keep an eye always in the docs,
because it's impossible to remember all the parameters in the CSV. So in this case, what
we're gonna do is something very interesting is we're gonna parse a CSV file, but it's
not located in this computer, it's not locally available in the computer. The CSV file is
these one right here, which actually is the source, if I get the raw version is this thing.
So this is CSV file, what I could do here is download the file, right, so just do File,
Save, get the CSV file on my computer uploaded here, right, so just copy and paste here,
drag and drop it here. But actually pain this has this nice characteristic that it will
read a CSV that it's either locally as we did with BTC market price, or you can also
do it remotely, it's automatically going to download the content of those files. And it's
going to provide, it's going to save it in memory for further usage. So there's a very
neat feature. And again, this is the the CSV file that we are using. And again, the same
thing, if it's a local file, it works in the same way. So a few features you've seen already,
in this case, we can do Heather known, if you don't want to treat the first row as a
header. Or what about missing values, we can treat some of these values like a question
mark, or like an exclamation mark, or dash etc. us not a number, not a value, right,
so it's a missing value. And now any of these values we have passed, will be transformed
into another number for easier and easier process cleaning, we can pass names, which
is going to be basically the column names for each one. And we can also specify column
types, as you can see, right there. So now the types are going to be float. And object.
We've done this already in one of our lessons, we are parsing the time and there you go.
So putting all together, we get to these advanced forms of reading csvs where we're passing
column names were passing types, were asking to read dates, were passing no values, Heather's
etc. So this is a pretty common thing we are doing. So what about XM review, if we try
parsing this thing, we get this very ugly format. In this case, they put the parameter
to specify the what we used to call delimiter in CSV is now set from separator so the separator,
it's going to be the greatest sign and that just works as it needs. So, a few more examples
you can check on here the most important part is following right, the documentation to find
those particular use cases that you are having so for example, some Like skip blank lines,
or whenever there are like empty rows at the beginning, right. So if you have empty rows
at the beginning is something you can also say skip rows. So you don't need to parse
those out, it's not going to break, etc. So that is all part of the read CSV file. And
to finalize these part, at least csvs, I'm going to tell you something that applies to
pretty much every other data format. As you have a read something method, there's going
to be a to something method, it's basically the process of writing. So you can do read
CSV, or you can do to CSV. So these CSV that we imported from the external source and the
remote source, we can just do to CSV and it's going to store it locally. Alright, and there
are multiple options also to pass the CSV delimiter, or actually the separator, if you
want to include a header if you want include an index, etc. They're pretty much the same
as the other one. But the idea is that for every read something method, there's gonna
exist a to something method that it's basically the process of writing. So let's move forward
with a few more data formats. And interesting, we're gonna get to read directly HTML pages
in just a couple of minutes. And now it's time to read data from databases. We have
already done that in our real example with Panis part of the tutorial. But I want to
show you a little bit more details details for you understand how data is being processed
in case, this is a common scenario for me importing data from databases. So the libraries
you will need first thing, depending on what database engine, you're using Postgres, MySQL,
Oracle, etc, you will need to install different libraries. But the API's, once you have installed,
those libraries are going to be the same. There's actually p Ep from Python that actually
defines the interface for databases, libraries, unpin, this can work with pretty much any
any database of these SQL common database that comply with that interface. In this example,
we're going to use SQL lite because the database right here, there's nothing, no server to
connect, etc, is extremely simple to get started. And the example we're going to use, or the
danavas example we're going to use is actually different one from our previous video is reading
in the previous one, we were using circular, in this case, we're going to be using chinuch,
which is smaller both in structure and in size. So it's going to be a little bit simpler.
So to get things going here, the same thing that we did with our previous part, that was
how to read data from files, I show you how to actually read data using Python. So forget
about pandas for a second, I told you, if we go back again, to the beginning of time,
there was no pain this, this was the way we were writing, finance, open FP, FP, the red
lines, etc. So I now want to show you what predates to pin this, what was the default
way to read data before paying this, which is with the regular again, interface from
Python. So the way it works is we're gonna import SQL lite three, we're gonna create
a connection. And now with this connection, we have these common interface that again,
it's common for pretty much any other database that you're used to. And the default behavior
is we're going to create a cursor. And we're going to execute queries using that cursor.
In this case, we're going to execute a regular Select star from employees limit, Fox will
want to have five, five records out of the table employees, once you have executed a
query, it's like they're waiting, you can do a fetch all to get all the results of that
query. And here are all these results. As you are noticing this is the result is a list
of tables. So it's not extremely useful. Now, if you combine it with pain, this you can
just create a data frame out of that info. And we're close. It's not perfect, but we're
close. So let me show you now before we were gonna close it Kurt Dickerson on the connection.
Let me show you now how we work with pandas. With pandas we have as we have a read CSV
method, we also have a read, see as read SQL method, and in this case, what this method
is going to receive is the first parameter is going to be the query that we're passing
and the second parameter is going to be the connection. That's the object the connection
object to actually issue the connection by panelists. So it gets a simple as writing
the query. And now everything has been imported into a data frame, including column names
and all that if you want to get a little bit fancier, you can either specify the index
column, there's going to be use, of course as a index, and also what types to parse for
a specific column. So now we have pretty much all the work down. So we're going from something
very manual as processing things with a coarser etc, which might also be as low to using pain
this to do Actually imported data from the database. There is actually a caveat here
that I'm going to tell you is kind of a very deep detail of the way pandas works, and is
that the read SQL method is actually a shell for two other methods, read SQL query and
read SQL table. Alright, so right SQL table on read SQL query, when you're using read
SQL, it's actually kind of forward in the work to either query or table, or an SQL query
is the default behavior, what we've done so far, so in this case, it's just going to issue
a query and the connection is going to read it for you. In contrast rate SQL table is
can I read an entire table, you just pass a name, and it's going to automatically give
you all the information for it. So in this case, all the column names, etc. So it's a
lot simpler to read an entire table, the only thing to keep in mind is that to use this
method, you need to install these libraries, SQL alchemy, and the connections generated
from it. So in this case, we create an engine on we create a connection objects, and now
we can pass an actual auction object sorry for pandas to do it. So again, it's pretty
much the same, if you find yourself doing Red Star from this table, Red Star from that
table, it's a lot easier just to write SQL table, and that's going to do it just advance.
As we saw that read CSV files hard to CSV, sorry, read CSV method had a to CSV method,
the same thing happens with read SQL, there is a read SQL and the results are to SQL,
what's what it's going to let you do is get the from the database and write it down into
a database table directly. So it's going to also receive the connection, right? So to
SQL, it's gonna receive what he will name of these data frame, what table name is going
to be, and a connection object. Now something to keep in mind is that to SQL has an important
parameter, which is what happens if the table already exists, that in the default way, it's
going to fail, just going to throw an error when you are trying to save data to a table.
And this makes sense, because as data analysts were usually reading data and processing it,
we're not so much writing it. So we want to meet make sure that it's not by mistake. But
if you do actually want to write data, you can just change this parameter if exists something
like replace or append. Usually, we're writing to intermediate intermediary table tables,
again, you can choose either to replace the whole concept of the table, be careful here,
or to append, write, just write it a dn of the current table. So that's just for to see.
So this was the way to read data from databases, of course, we're not touching on anything
like SQL and all that, that it's a lot more advanced, it's just for you, if you already
know SQL, if you're already working with databases, you can pretty much copy and paste what we're
doing here. And you're gonna, you're gonna get your data import into Python. So let's
move forward to read some HTML files. And now very quickly, I'm going to show you how
to read tables or data frames directly from HTML web pages. To be honest, this is a simple
method is going to be just read HTML, but it depends a lot on the structure of the web
page. So if it's not well structured, or the tables are not correctly created, you're going
to have issues and you will have to do a ton of data cleaning. In my experience, whenever
I try to parse a table from a well structure site like Wikipedia, or some stats site, it
usually works very well. And it's a very quick way of hacking. You know, whenever you have
questions, you know, like, I don't know, I need to know the GDP of countries. Instead
of looking for a GDP data set, you can just go to Wikipedia page, there is usually a table
there, you can directly parse it and you are done. So again, it's it's a relatively simple
way to get some data for quick hacking and exploration. The way it's going to work is
we have these HTML creative. It's just for testing purposes. To get started, usually,
of course, you will try to read something from a live website. So you're going to pass
the URL to the read HTML method. And the read HTML method will download the content of the
page and parse it. Let's suppose we have the the content already the HTML, and this is
what it looks like. This is a exactly the same HTML we have on top, I'm just displaying
it here in a book. And what we're gonna do is we're gonna invoke the method, read the
HTML. And the read HTML method is going to parse the entire HTML and look for multiple
tables, not just one site will potentially have multiple tables, even if you don't see
them. The is a common way to structure things in HTML to use tables. That's why it's going
to pause multiple tables. In this case, we stored them all in a DFS, multiple player
like multiple data frames. And we see that there is only one. So in that case, we're
just going to get the first data frame. And it has correctly parsed what we had before
just working in the same way. The same is going to happen with for example,
things for headings and all that if the table doesn't have a header, it's gonna automatically
right understand.in that case. So that's pretty much as we know it already. In this case,
what you're going to see is what I told you before about data cleaning process that these
table does not have a header like the previous one that has a T head to head attribute, in
this case, a header is just another row. So that's why read HTML is going to have issues
and you have to provide a little bit extra information. So let's see another more realistic
example. And we're going to parse data directly from a website, let me tell you here, just
just for educational purposes, you always need to understand if you have if the data
is public, so you can actually parse it. Again, for Wikipedia, at least what I do, the content
is created comments, so you can get a hand on it. There. What we want to show you here
is that a very complicated table that has multiple headers, etc. So that's why we're
using this example. So we're gonna get the URL, and we're gonna directly do NBA tables.
Equals read HTML, the only table in this page is this one, the large one. So that works.
And now we're gonna get NBA is going to be that and we see that the all the players in
this case have been parsed. What about something else, let's actually open this page right
here to Wikipedia, for the Simpsons. And here, we will probably find several tables. See,
we have one right here, this one. So I'm going to import it. We have 27 tables, again, you
don't see it. You don't see them, sorry, but they are there. And the most important one
is the one we care is these one right here. So the problem you're gonna have with this
table is that each using both columns, pans and rows pans. So in this case, this column
here is pans for one to three columns. And these row here stands for 123, at least three
rows. So that column spans results in these very ugly data frame, and you will need a
little bit of extra cleaning. So that probably you're going to find with HTML tables that
usually there are things that are not well formatted for machines that are formatted
for humans. So for example, in this case, we have this header repeated, when you parse
this data, you're going to find that every 20 rows, there is going to be header row,
and you will have to clinic every for in this case, to enter rows, you will need to drop
it you will do something like df the drop, let's see, actually, if we can see it haven't
tried this, but let's just do it like that head, and you're going to find 25 Records
now. So here, record 22, we find that, Heather, so what we're going to do is you will need
to do something like df the drop df dot drop, range 22 starting in 22, up to the F the shape,
zero, right, these many rows plus one plus one and every 20 rows, I don't know this is
going to work, just run it. Hope didn't it didn't even work. It didn't compile. Oh, this
is NBA actually. There you go. So maybe it works, you can check it. But what
I'm going to say is, again, there is some cleaning to do because HTML pages are optimized
for humans, not for machines. So usually, it's going to take a little bit more time.
The good news is that there is usually a service associated that you can consult. So for example,
there is a Wikipedia API that you can use instead of a page. But again, sometimes just
easier to pull the data directly from Wikipedia. So that's it. You can also write data to CSV
or of course or HTML. That's pretty much the standard. As we've said, this is up all we
had for the read data portion. And we're gonna move forward now with a few other methods,
especially what we call data wrangling. We're going to do a little bit of grouping and keep
moving forward with our tutorial. We have decided kind of last minute to our final source
of external data that it's going to be an Excel file. It's just a common Excel files,
you know it, because we imagine that you might come from an Excel backgrounds, you can just
export the data you have in your Excel files, Excel spreadsheets, and load them into Jupyter
Notebook and start working with them with him this so you can try things out and kind
of draw the pearls in between Excel and what you do with pandas and Python. So the first
thing is, an Excel file is not a text file. So if you try getting the content of it, it's
not a text file, it's not so simple to parse it. So that's why it's gonna require external
tools that they already installed in notebooks AI, there might be a student's holding goal
up, but it depends on your computer, how you're going to install it. So just keep in mind
that there might be issues when importing data from Excel, if they if there is low compatibility
between the library you're using another spreadsheet version you're using. But without those without
getting into those details, there is read Excel method, which pretty much takes care
of everything for you has different parameters, like the finding the the sheet that you're
reading from, of course, the path, etc. So we're going to start reading these file, which
is products file that has three sheets, products, descriptions, and merchants. And it's actually
something we use in an Excel file to sorry, in our data analysis, from Excel to pandas
course, to show how to merge data and all that. And from this file, what we're gonna
do is just read Excel. And what you're gonna see is that it reads the first sheet of the
Excel file, I mean, a data frame is just corresponds to one sheet only, right? And the first one
is product. So that's what we are writing. There are different behaviors for it, you
can change the way you parse, Heather's etc, you notoriety defining and specific index,
that's pretty much everything we have seen. So far, it's selecting specific shifts is
simple, just pause the sheet name, and you can share the rate story either products,
merchants, whatever is available in the current Excel file. There is another format or a new
specific class that it's a little bit more advanced. But it's the Excel file class. So
it's not, as we were doing here, right, Excel directly is going to read thought Excel file
into a data frame, but you're going to instantiate this Excel file class, with the parameter
being the file name. And now these files gonna have just a reference of everything you have.
In this case, we can do for example, sheet names, it's going to tell you how product
descriptions merchants, there's a little bit more explanatory data analysis. So let's say
you can't use Excel to actually see the contents of the Excel file, this is going to be helpful,
you're going to first parse the Excel file, get the sheet names, and a little bit more
of an understanding of it. And now we can say from these files we have previously parsed
right here or instantiated, we can parse the product, the product sheet, and that's going
to get you that that frame. And the same thing is going to happen with all the parameters
weekend pass, they are the same as read Excel. Finally, you can see that the results are
to excel file. And it works pretty much the same way as to CSV, and decide if you pass
an index or not. And also you can define if you're going to pass a sheet name or not,
are just going to be the default one. So as you can see, getting your data into a from
an Excel file into a CSV, data frames array is extremely simple. There are more customizations
to do, let's say all your file is shifted array, either rows or columns, you can change
that with Star row or column that's going to work, too. So that's pretty much the only
thing we need. If your writing process is a little bit more complicated. Like for example,
you want to write specific sheets in our multi sheets. Excel file, you can use what we call
an Excel right and it's also part of fantasy, you instantiate the rider, and then you can
start the ride process saying which shades you want to ride with each one of those, that
friend. So again, reading and writing data from on to Excel files is relatively simple.
It all depends on the libraries are installed. It depends on on what libraries you have in
your current environment, if it's windows or if it's a Linux slash slash mark, the documentation
of PD dot read Excel might have more details for the given platform
that you have. So let's see if it names per document, if it's not here, it's gonna be
in the pandas documentation, but there might be a requirement
For each one of the platforms, that pan This is supported. So just check it out, check
for your own for your own platform if you're in Windows, Mac Linux, how to get those libraries
installed. So in case you're just getting started with
Python, and you might come from another language, the objective of this quick section is to
show you Python. Ideally, in under 10 minutes, I think it's going to take a little bit more.
But there's a very, very, very quick reference of Python, again, just the high level features
of the language, how to use it, how to code functions, how to import modules, variables,
data types, collections, etc. You can just scroll through this notebook, if you want
to take less time, I will be providing an explanation on top of all the topics, but
there's a very good reference of the entire language. So to get things started, Python
is an old language period. It has card, it has caught more attention in the past five
to 10 years. But it's a very old language. It's even older than Java. It's up here in
1990s. And it was created by this person good by Guido van Rossum. And it's an important
actor in our ecosystem he is used to be I think he still the one deciding discussions
etc, when it comes to defining features of the language, etc. Python is a high level
interpreted dynamic language. And this means a tone actually, if we read these entire sentence,
interpreted high level, general purpose, this is basically high level programming language,
it's object oriented. And it also includes functional attributes or functional features
like functions as first class objects, etc. And it also, of course, it supports imperative
programming. And it has a wide variety of applications, you can do web development with
Python, you can do scripting, it's a lot use for system development for configuring machines
in general. And of course, you can also do data science, it has multiple applications
has a couple of interesting features like indentation, for defining blocks, etc, that
make it and very good language to get started with programming. So if Python is your first
language, you should be comfortable with it. It's a very good idea for me, it wasn't my
first language. And I hope it was, it wasn't. But I, I have taught people programming with
Python as their first language. Seriously, it's always been very good for them, because
Python doesn't have weird things like my have in JavaScript or Java. So it's a very concise
language and consistent language to be honest. So let's get started very quickly. First of
all, when you're going to install Python, your own computer or you can use notebooks
AI or Google call up. But if you're installing in your own computer, you might see that you
can install either Python two, or Python three, or actually, if you're reading tutorials online,
etc, you might see Python two and Python three, the reality is that Python two was deprecated
in 2020, so the you cannot you should not use it anymore. There are still ways to install
Python two, but it was deprecated. So you shouldn't use Python two, you should stick
with Python three, which is the evolution of the language. So ton of fixes from Python
to the bay where, where things happen in the language and used to confuse beginners. So
that's no longer a problem. Python three, again, is what you should use, you will read
in multiple tutorials, etc. What they are using Python two, you should try using Python
three, and sometimes the code will break, but the changes to fix it are not very hard.
So to get things started here, I will be drawing the problem of this and with regular syntaxes.
For example, this is the way you will define a function in for example, JavaScript. And
it's also very similar to something like C or Java based languages, the function keyword,
curly braces, etc. So I will be drawing a parlors and with these sort of languages.
So to get things out of the way to defined function in Python is in this way. And the
main characteristic of this language is that the way we're going to define blocks is by
Using different indentation levels. So this is a valid function in Python def is the key
where we use the name of the function the parameters it receives. And the way to define
that the body of the function is by just indenting. Everything one level to the right. Usually,
this is just for spaces. Another example is an if else statement. So
if this thing happens, do that if else do something else, right? This is JavaScript.
In Python, again, it's defined by indentation. If this thing happens, we indent one level
to the right, do this else do something else, if there was another if statement here, if
I don't know, language, ends with something like I don't know, three, then do something
else. Print pi, three, for example. So we're indenting everything to the right, every time
we start a new block, whenever the block finishes is just when you go back again, print this
as first block, right, that's the way it's going to work it by indenting. Our blocks,
this is very good, because first, we don't have debates of where we should place the
curly braces. And also, because it makes it a lot more readable, it's a lot easier to
read these code because there is obligated obligatory indentation to even make the code
work to. So you can see that's that's just how it works. How we're going to make comments
in Python, just by using the number pad symbol, there we go. And the way to define variables
is just by specifying the name. So it Python is a language that you don't need to declare
variables, you just declare and define everything and just one pass, you know, you find a variable,
as it goes. Python is dynamically dynamically typed. But it's also strongly typed. And these
might kind of cause confusions. But basically, you can assign variables to any value you
want. And you will see that collections etc, are heterogeneous in terms of types, etc.
It is a very dynamic language. Talking about types, I'm going to show you the most important
types that we have in Python, especially we have numbers, of course, integers, we don't
have so many like, like, you might find that other languages, like different precision
cetera, we have integers, there is also the concept of Long's that has changed with Python
two. To be honest, on Python three, to be honest, we use just integers, that's the way
we work. It's a, it's a smart enough type to save storage when needed. So that's, that's
good. And it will also have floats, right, which is the regular float type for floating
point arithmetic in other languages. And of course, it suffers if you want from strange
behavior from float floating point arithmetic, like in this case, you can prevent that by
using the decimal module, which, as you can see, doesn't suffer from from this issue.
So numbers, we have integers floats, and we also have decimals, strings are just a type
str, and they are defined literal, right, as in this in the st, you can see right here,
you can just type the string as it goes. There is a difference between there was a difference
already in Python two, between Unicode and strings, etc. In Python three, that has all
been fixed. So we Python three, this is all Unicode. And there is the concept or the difference
in terms of the concept of something being the type. The Unicode code points as it's
this string, and the underlying encoding will turn it into binary. So in Python three still
have we have a few ways to differentiate between whether it's a binary string or whether it's
a text based string. For you shouldn't worry about it, I just want you to know, if you're
writing a Python tutorial, for example, you might find a difference between Unicode strings
and regular strings, which is, is no longer something that we should be worrying about.
If you have a string that it's too long and it expands multiple lines, you can always
write it using three quotes can be double quotes or single or single quotes. So just to create multi line strings is extremely
simple. Boolean there are two Boolean type do Boolean objects are unique, right? It's
kind of a single tone which is the true or false objects. For example, They are of type
Bo. There is also the concept of No, in Python, which is none, we don't have no, we have none,
but it serves pretty much the same purpose. In Python, everything is an object. So even
this strange, strange objects, like none will have an associated class, if you want, everything
in Python is an object. So all these types of you have seen. So for example, we have
this string, which is H of a string. The type is str, you can use the int, str float bool
types, right, but it's the result of the type also as function. So in order to cast in this
case, a string into in order to cast a string into an integer, you will use it you will
do it using the end function, which is the same thing that you get with these, for example,
so this is the same as this, as you can see, what we have to show. So functions again,
death is the key word we use, we don't use function, we use death I, you can use define,
as a mnemonic, the name of the function parameters are optional, and finally have the return
keyword, you should always include a return you usually 99% of the time, the function
should return something. Because that's going to be the result assigned once we invoke the
function just this is pretty regular. If your function doesn't return anything explicitly,
if that means if you haven't written down a return statement anywhere in your function,
the function will still return something so that the fact that you haven't included a
return statement explicitly doesn't mean that the function is not returning anything implicitly,
actually, it is returning something, it's returning none, right. So by default, if you
don't include a return, Python will do this. Just for you to know a function always returns
something as specified parameters and passing parameters is pretty standard. Python has
some advanced features with parameters like for example, variable length arguments, we
can pass as many arguments, we want to make it very dynamic keyword arguments, named arguments,
etc. So all their ethic operators, you know, already, the shin modulus, in this case, were
doing a power its operation, all this is pretty standard. And the same thing happens with
all our Boolean operators greater than greater or equals then etc, there are type checking.
So this is when we have the strongly typed feature, even though Python is dynamically
typed. It is the types are enforced. In this case, you cannot compare a two with this doesn't
make any sense. And Python is going to complain about that. So this is an example of an error
in Python. The exception type error was raised on the same thing with bolens and not on or
operators. As we saw before control flow is defined by the indentation so every new block
is defined with an indentation level. Python includes if else and also l F, which is very
convenient. And this is an example If this happens, Elif, Elif, etc. Python does not
have a switch statement. For example, loops, how are you going to loop through something
in Python loops on lists, or collections in general, are very interconnected.
Because in reality, when you're looping the Python, you're not doing a regular in Python,
we don't have something like in, in Java, you're gonna have something like int i equals
zero. What else I it's been decades. And I this is I haven't
coding in Java. So I, I don't know, minus 10, less than 10 less than 10. And here we
do I put last There you go. So we don't, we don't have these in Python. We have a way
to mimic it. But we in Python, we always eat iterate over a collection. So what we're going
to do is we're going to create a range elements, and we're going to iterate over it. So the
way it works is very close to one other language is going to be a for each. Alright, so in
this case, we have all these elements and we're going to do for name in names, that's
it. And at any moment, the name is going to be associated with an element in the list.
while loops are part of the language, they are usually discouraged in favor of for loops.
If something can be coded with a for loop, it should be coded with a for loop and not
a while loop. Because as you might know, already, these my trigger or these might result in
an infinite loop if you're not checking the conditions correctly. So the collections we
have in Python, are the fundamental ones, the primitive ones, the most important ones
are first the list Python is we do a heavy usage of lists. And it's just a heterogeneous
data structure. So you can put anything in it. And actually, all these collections are
heterogeneous, you can mix volumes as you want. And in this case, we have three elements
that we have added one string, one integer, one string, and one Boolean. And let me say
something here. Even though pythons, Python supports mixed types in the collections, it
doesn't mean that you should do it. To be honest, we should, you should usually avoid
mixing types in collections, because that means we don't, we don't know what we're putting
in it, right. So it's, we should be consistent. So it's possible, revisit your code, if you
have too many different types in it. I'm checking the length length function accessing elements
is by by zero indexed, and we use square brackets. So in this case, give me the first element
given the second element. And also we can index starting from the from behind from the
end. So in this case, minus one, minus two, minus three. So in this case, minus one minus
two, again, give you different elements, you can check the operations associated with all
these elements. Very quickly, a list is L dot append, we're going to append the new
element. So the list now has that element at the end. And we can check if that element
is part of the list in this case is true in this case is false. topples are similar to
lists, they are also sequences, but the main difference is that they are immutable, there
is no way to add new elements to a tupple, or remove elements from a tupple once it has
been created. So in this case, we have created a list with three elements. Now tupple, sorry,
with three elements, we can access it, we can check if something is in it in the same
way that we did with a list. But in this case, with a tupple. Again, you cannot modify it
tupple never changes, you can't add elements to it. Another important data structure is
a dictionary. In Python, a dictionary is a key value, right and mapping, it's similar
to an object in JavaScript or hash table in in, in Java, it's a key value mapping type.
And in this case, we are going to associate values to names. So you can see this, the
way I like to explain it is if you create a topo list, right? So let's say we're going
to create a list, out of all these elements, give me one second, we're going to create
a list. There we go, we're gonna copy these elements. And we're gonna associate that to
our list. There you go. So these are a list, we could very well store the information about
our customers in a list, right? That works. I mean, I can get it done. The problem is
that whenever I need to access information about this list, we're going to say, for example,
I don't know I want to give me the email for this customer, I have to remember the position
that the email is located so in this case is going to be position number one, if these
information grows, instead of having 1234 values or four pieces of information for our
user, we have 100. Right, then it's gonna be very hard to access those individual volleys.
So that's why we create dictionaries, dictionaries are collections of values. The important part
is on the right, the important part is the value. But they are instead of just indexed
by the precision, we give them arbitrary names, we tell them very explicit names. This is
the name, this is the email. This is the age. And this is if they are subscribed or not.
So once we create these dictionary, we can access those values by the name, give me the
email of these user or is the age present of the user is the last name present of the
user in the user in the user dictionary. So again, it's a way to store information associating
later In order to make it simpler for us later, let me delete this. And I move four sets sets
are very common data structure, he is when you're learning about a collections and, and
and yeah, the instructions in general, it's not so common in too many languages. I mean,
it's not very popular in Python, we use it often because it has a very interesting feature,
first of all, and it's something that I forgot to tell you about dictionaries, both sets
and dictionaries, are what we call unordered. data structures, you never know, the order
of the elements. In Python, with recent versions, there have been changes, which make Python
dictionaries ordered. But for now, I'm going to say you shouldn't rely on it, you should
think your dictionaries as they are completely unordered data structures, and the same thing
for sets, sets are, it's a bag that contains elements, you know, it's a big bag, you keep
throwing elements inside of the set, there is no orphan in it. And what's gonna happen
with it, you're gonna odd elements, for example, to the set, or you're going to remove elements
to the set. And there is one important thing that makes this set so useful, and it's the
membership operation, I'm gonna write it down here, membership, ship operation,
there you go. So you can access these notebooks later. So in the membership operation, the the, the
process of checking if something now, nine in s, the process s of checking this is extremely
fast, it will be called oh one. And this is because as you might have seen here, when
I created this set, I included a couple of repeated elements, 333, write 11179, the resulting
set doesn't have those repeated elements, these are two features of the set, the set
will only contain unique values. And by the way, it's implemented behind the scenes will
make dot these unique values are extremely simple to check whenever you pass these membership
operation is extremely simple, or sorry, is extremely performant. It's very fast, different
from for example, a list. So keep it in mind sets are very, very useful when you're checking
for members. So again, as I told you before, we're going to iterate over collections with
the for loop. So in this case is if we have a list, it's going to be for element in list.
There you go. If you have a user dictionary, use a dictionary, sorry, in this case user,
we're going to the default iteration is by key, we're going to get for name email age
subscribed, and we have to extract the value out of the of the dictionary, we could also
do for value in user dot values. Oh, there you go. Or you can iterate over both key and
value with items. Key. And value. There you go. So each iteration in in in Python is very
readable to put it in a way. And again, remember, we're always using the for loop that assumes
that you're iterating over a collection, we don't have the for Ei equals zero equals zero,
I equals zero, i less than 10. i plus plus we don't have that right in Python, we can
simulate it with for i in range. Five, for example. Print. We've got simulated with the
range function, which generates pretty much those elements. Something that you might have
heard about Python is that it has a huge library of built in modules, right that you can just
import and just gonna work. There are so many things already coded in Python, that it makes
it very simple for you to create something on top. Do you want an a library for I don't
know security cryptography Math, numeric processing NumPy, right? machine learning web development,
creating games through is pi game, do you want to create a graphical user interface,
whatever you want to do, there is usually a library that has already been coded and
will make your job easier. On top of that, the bill team is down there library, right,
which is already included with Python, it's not third party. In this case, it's already
created by the Python core team. It's a huge library, so many modules. And the way it works
is by importing this module, so this is the way we work with packages and modules, there
are differences between modules and packages, third party ability, and this is a little
bit more advanced. But again, this gives that random number generator, it's already built
in. And you can check the docs right here. exceptions, whenever you do something that
doesn't work. So in this case, we say, if the age is greater than 21, but age is a string,
it's an it's not an integer, this is going to fail. We can catch exceptions before they
happen, that's going to be with a try and accept lock. Right. In that case, if this
fails, if anything here fails, these blocks going to be kicked in. And you can catch the
exception without the program fail failing. And you can be more explicit about the error
aspect. So again, this is just an introduction. It might be useful if you're coming from another
language, especially to keep this notebook as a reference. We're going to be using Python
a lot, of course, and it's a great language if you want to do scripting, work development,
of course processing with data, data analysis, etc, visualizations, machine learning, Python
is just great. So I hope this tiny tiny reuse lesson helps you port your knowledge from
other languages into Python. And that's it.
