Artificial Intelligence with Python | Artificial Intelligence Tutorial using Python | Edureka

[Music] undoubtedly artificial intelligence is the most trending technology at present tech giant such as Tesla Amazon and Netflix implement AI techniques to derive useful insights from data and grow their business hi everyone I'm Zuleika from at Eureka and today we'll be discussing all about artificial intelligence and how it can be implemented by using Python also before we get started make sure you subscribe to a Drake our YouTube channel in order to stay updated about the most trending technologies so let me begin by discussing today's agenda I'm going to begin the session by walking you through a couple of reasons as to why we choose fights in for artificial intelligence then we'll move on and look at the most important and the most popular Python packages which are specifically used for machine learning deep learning and artificial intelligence after that we finally introduced artificial intelligence where we will discuss - what led to the need of artificial intelligence we'll also discuss what exactly artificial intelligence is the different types of AI after that we'll discuss the difference between an artificial intelligence machine learning and deep learning then we'll move on to the machine learning module where I'll discuss with you what exactly machine learning is the different types of machine learning machine learning problems machine learning process machine learning algorithms and we'll end the machine learning module by implementing a small demo by using Python alright next we look at the limitations of machine learning and what led to the introduction of deep learning here we'll understand what exactly deep learning is how deep learning works what are neural networks perceptrons and multi-layer perceptrons will end the deep learning module by discussing a practical implementation with Python right after that we move on to our last module which is natural language processing here we'll discuss what exactly NLP is where it is used its applications and the various terminologies in natural language processing so guys there's a lot to cover in today's session let's move ahead and take a look at our first topic so why exactly are we using Python for artificial intelligence why aren't we using any other language right now the couple of reasons as to why Python is so popular when it comes to AI machine learning and deep learning the first reason is less coding is required now artificial intelligence has a lot of algorithms if you have to implement AI in any code or in any problem then there are going to be tons and tons of machine learning algorithms involved deep learning algorithms involved right now testing all of these can become a very tiresome task that's why Python usually comes in handy now the language has something known as check as you code methodology which eases the process of testing right you can check your program as you code it basically as you're typing each sentence your errors or your any sort of mistakes in your code will be given to you right so testing becomes much easier when it comes to Python the next important reason why were choosing Python is it has support for pre-built libraries that Python is very convenient for AI developers because all of the algorithms machine learning algorithms and deep learning algorithms are already predefined in libraries right so you don't have to actually sit down and code each and every algorithm that will take a lot of time and that's a very time-consuming task and thanks to Python you don't have to do that because they have libraries and packages that have all the algorithms built in them right so if you want to run any algorithm all you have to do is you have to call the function and load the library that's all it's as simple as that now the next reason is ease of learning so guys Python is actually the most simplest programming language right if you ask me I think it is it's the most easiest programming language it's very similar to English language right if you read a couple of lines in Python you'll understand what exactly the code is doing it has a very simple syntax and this simple syntax can be implemented to solve simple problems like addition of two strings and it can also be used to solve complex problems like building machine learning models and deep learning models so ease of learning is a major factor when it comes to why Python has chosen for artificial intelligence right now we have platform-independent so a good thing about python is that you can get your project running on different operating systems right and what happens when you transfer your code from one operating system to another operating system is we find a lot of dependency issues to solve that Python has a couple of packages such as that is a package known as pi installer right this PI installer will take care of all the dependency issues when you're transferring your code from one platform to the other platform so all of this support is provided by Python the last reason is massive community support this is a very important point because it is important that you have a large community that will help you out with any errors or with any sort of problems in your code right so python has several communities and several forums and groups on Facebook so if you have any doubts regarding any error you can just post those errors in these groups and you'll have like a bunch of people helping you out right so guys these are a couple of reasons as to why Python is chosen for artificial intelligence it's actually considered the most popular and the most used language for data science AI machine learning and deep learning to prove that to you here is a stat from Stack Overflow Stack Overflow recently stated that Python is the fastest growing programming language if you look at the graph you can see that it is taken over JavaScript and Java and C hash C++ and PHP and so python is actually growing at an exponential rate especially when it comes to data science and artificial intelligence a lot of developers are very comfortable with the Python language because you know it's a general-purpose language first of all so most of the developers are already aware of Python and then using the same language in order to solve complex problems like artificial intelligence machine learning and deep learning is something every developer wants right they want a simple language in order to code or the complex algorithms or the complex models and so that's why Python is the best choice for artificial intelligence for those of you who are not aware of Python programming and don't know much about Python I'm going to leave a couple of links in the description box right you can go through those links and sorry a little bit more but how Python works or how the coding part works all right I'm going to be focusing mainly on artificial intelligence and I'll be showing you a lot of demos so those of you are not aware of Python make sure you check the description box right next I'm going to discuss the different Python packages for artificial intelligence now these are the packages that are specifically for machine learning deep learning natural language processing and so on so let's take a look at all these packages so first we have tensorflow if you are currently working on a machine learning project in Python then you must have heard of this popular open source library known as tensorflow that this library was developed by Google in collaboration with brain team dense flow is used in almost every Google application for machine learning now let me just discuss a few features of tensorflow it has a responsive construct meaning that with tensorflow we can easily visualize each and every part of the graph which is not an option when you are using other packages such as numpy or cycle right another feature is that it's very flexible now one of the most important tensorflow features is that it is flexible in operability maying that it has modularity and the path of which you want to make a standalone it offers you that option right it's very flexible in that way it'll give you exactly what you want now good feature about Anza flow is that you can train it on both CPU and GPU and so for distributed computing you can have both these options also it supports parallel neural network training so tensorflow offers pipelining in the sense that you can train multiple neural networks and multiple GPUs which makes the models very efficient on any large scale system alright so parallel neural network training is supported by Delta flow right this is one of the most important features of tensorflow apart from this it has a very large community and needless to say if it has been developed by Google then there's already a large team of software engineers who work on stability improvements and all of that right the next library I am going to talk about is cyclone now cyclone is a Python library that is associated with numpy and side pipe right that's why it has a name cyclotron now this is considered to be one of the best libraries for working with complex data there are a lot of changes that are being made in this library and one modification is the cross validation feature which provides the ability to use more than one metric write cross validation is one of the most important and one of the most easiest methods for checking the accuracy of a model right so cross validation as being implemented in cyclone and apart from that again there are large spread of algorithms that you can implement by using cyclotron right these include unsupervised learning algorithms starting from clustering factor analysis principal component analysis - although unsupervised neural networks cyclone is also very essential for feature extracting in images and text so mainly cyclone is used for implementing all the standard machine learning and data mining tasks like reducing dimensionality classification regression clustering and model selection next up we have number now numpy is considered as one of the most popular machine learning libraries and python now let me tell you that tensorflow and other libraries they make use of numpy internally for performing multiple operations on tencel's the most important feature of numpy is the addi interface it supports multi-dimensional arrays right that's one of the most important features of numpy another feature is it makes complex mathematical implementations very simple right it's mainly known for computing mathematical data so numpy is a package that you should be using for any sort of statistical analysis or data analysis that involves a lot of math apart from that it makes coding very easy and grasping the concept is extremely easy been done by now numpy is mainly used for expressing images sound waves and other mathematical computations alright moving on to our next library we have Theano Tiano is a computational framework which is used for computing multi-dimensional attic right a no actually works very similar to tensorflow but the only drawback is that you can't fit piano into production environments but apart from that Tiano allows you to define optimize and evaluate mathematical expressions that involve multi-dimensional arrays and this is another library that lets you implement multi-dimensional arrays these are the Theano include tight integration with numpy an advantage of Tiano is that you can easily implement them by a DS in theano right that's why there's a connection between Theano and numpy because both of them effectively use multi-dimensional arrays transparent use of GPU now performing data intensive computations on much faster when it comes to Tianna because of its use of GPU right Theon o.o tool lets you detect and diagnose multiple types of errors and any sort of ambiguity in the model so casiano was actually designed to handle the types of computations are required for large neural network algorithms that it was mainly built for deep learning and neural networks it was one of the first libraries of its kind and it is considered as an industry standard for deep learning research and development siano is being used in multiple neural networks projects and the popularity of Tiano is only going to grow with time right a lot of people actually haven't heard of siano but let me tell you that this is one of the best ways to implement deep learning and neural network models moving on we have Kara's now Kara's is considered to be the most popular Python package it provides some of the best functionalities for compiling models processing your datasets and visualizing graphs it is also popular in the implementation of neural networks right it is consider to be the simplest package with which you can implement neural networks in fact in our today's demo for deep learning we'll be implementing Kira's in order to understand how neural networks work through the features of Kara's include that it runs very smoothly on both CPU and GPU it supports almost all the models of a neural network right from fully connected convolutional pooling recurrent embedding all of these models are supported by Kiera and not only that you can combine these models to build more complex models caris is completely button based which makes it very easy to debug and explore right since Python has a huge community of followers it's very simple in order to debug any sort of error that you find while implementing Kiera's so the libraries that I discussed so far would dedicated to machine learning and deep learning for natural language processing we have the most famous library known as the natural language toolkit which is an open-source Python library mainly used for natural language processing text analysis and text mining the main features include that it's study is in analyzes natural language text in order to draw useful information from all this natural language text it performs text analysis and sentimental analysis by performing tasks such as stemming lemma ties a Shinto organization and so on now don't worry if you don't know about any of those terms mean I'll be discussing all of those terms with you by the end of today's session so guys these were a couple of Python based libraries which are very essential for implementing machine learning and deep learning and artificial intelligence when you're using Python right these laboratories are perfect for implementing AI so guys if any of you have any doubts regarding the libraries or if you want to learn more about the libraries I will leave a couple of links in the description box you can go through those videos as well so now let's move on to the main topic of discussion which is the artificial intelligence now before we get started with the demand of artificial intelligence let me tell you that air was invented long ago AI goes back to the 19th century it was not something that was recently invented even though AI has recently gained a lot of popularity we can say that in the past decade AI has gained the maximum popularity but was actually invented in the 19th century now especially in the year 1950 there were some day known as Alan Turing I'm sure a lot of you have heard about the Turing test the Turing test basically you should determine whether or not a machine is artificially intelligent meaning that whether a machine can think Intel like a human being right this was the first proposition and this was one of the most important breakthroughs in artificial intelligence right somebody known as Alan Turing he published a landmark paper in which he speculated about the possibility of creating machines and that think right so the during test was the first serious proposal in the philosophy of artificial intelligence this was done in 1950 and after this we had errors of AI we had the game AI which was in 1951 now since the emergence of AI in 1950s we have seen an exponential growth in his potential right eeeyah covers domains like machine learning deep learning neural networks natural language processing knowledge bees and so on it's also made its way into computer vision and image processing but the question is if AI has been here for over half a century why has it suddenly gained so much importance right why are we talking about artificial intelligence now the main reasons for the vast popularity of AI are the following right the first reason is more computational power now AI requires a lot of computing power recently many advances have been made and complex deep learning models can be deployed and one of the greatest technology that made this possible are GPUs since the invention of GPUs we can compute much more with our computers initially we could barely process 1 GB of data right they only had hard disks to store additional memory and all of that now our computers can process tons and tons of data so now we have more computational power which is one of the main reasons behind why AI became so popular so by having more computational power it becomes much easier to implement artificial intelligence next reason is more data now big data is one of the most important reasons behind the development of artificial intelligence now AI and data science and machine learning deep learning all of these processes are here only because we have a lot of data at present and the main idea behind all these technologies is to draw useful insights from data now since we saw generating a lot of data we need to find third that can process this much data and draw useful insights from data such that it benefits an organization or it grows a business that's why artificial intelligence and machine learning comes into the picture right so more data led to the demand of artificial intelligence apart from this we also have better algorithms now right we have state-of-the-art algorithms most of them are based on the idea of neural networks and these are constantly getting better neural networks are actually one of the most significant discoveries in artificial intelligence because with neural networks you can take in thousand layers of input data right you can taken a lot of input data to perform computations so through neural networks were actually able to solve a lot of problems including healthcare problems fraud detection problems and so on another reason is broad investment so our universities and governments and startups and any tech giants like Google Amazon and Facebook they are all investing heavily in artificial intelligence which also led to the demand of a.m. so AI is rapidly growing both as a field of study and also as an economy right it's adding a lot to the economy and I think this is the perfect time for you to get into the field of artificial intelligence because right now AI is in a really high demand ai machine learning data science all of this are of really high demand at present and so this is the perfect time for you to get started with artificial intelligence now let me tell you that the term artificial intelligence was first coined in the year 1956 by a scientist known as John McCarthy now John McCarthy defined artificial intelligence as the science and engineering of making intelligent machines now let me give you a descriptive definition of artificial intelligence artificial intelligence is a theory and development of computer systems able to perform tasks normally requiring human intelligence such as visual perception speech recognition decision-making and translation between languages in a sense artificial intelligence is a technique of getting machines to work and behave like humans in the recent past AI has actually been able to accomplish by creating machines and robots that have been used in a wide range of fields including healthcare robotics marketing business analytics and many more right so AI is actually a very vast field it covers a lot of domains including machine learning natural language processing knowledge base deep learning computer vision and expert systems so these are a few domains that AI covers now let's move on and discuss the different types of artificial intelligence now guys AI is structured along three evolutionary stages you can say that AI is developed along three evolutionary stages we have something known as artificial narrow intelligence followed by artificial general intelligence and finally we have artificial super intelligence artificial narrow intelligence which is also known as a weak AI it involves applying AI or need to specific thoughts many of the currently existing systems that claim to use artificial intelligence are actually operating as weak AI which is focused on a narrowly defined specific problem now take Alexa Alexa is actually a good example of artificial narrow intelligence it operates within a limited predefined range of functions right there is no genuine intelligence or no self-awareness despite being a sophisticated example of vki Google search engine sofia and self-driving cars and even the famous alphago fall under the weak AI category then we have something known as artificial general intelligence this is also known as strong AI and it involves machines that possess the ability to perform any intellectual task that a human being can now guys machines don't possess human-like abilities we have a very strong processing unit that can perform high-level computations but they're not yet capable of thinking and reasoning like a human and there are also men you question whether it would be desirable for example Stephen Hawking's warned that strong AI would take off on its own and redesign itself at an ever-increasing rate humans who are limited by slow biological evolution couldn't compete and would be superseded right so we have a lot of tech and a lot of geniuses who are actually worried if strong AI is ever implemented it might take over the world right so as I may tell you that strong AI is something that has not been implemented yet we are only at the first stage of artificial intelligence which is artificial narrow intelligence also known as weak AI right we haven't yet reach strong AI or artificial super intelligence so artificial super intelligence is a term that refers to the time when the capabilities of a computer will surpass humans at ESI is currently seen as a hypothetical situation as depicted in movies and science-fiction books where machines have taken over the world alright so artificial super intelligence is something that is very far off but tech masterminds like Elon Musk believed that artificial super intelligence will take over the world by the year 2040 right a lot of people who are against the development of artificial general intelligence and artificial super intelligence a lot of us believe that we should stick to weak AI and not move any further and risk the existence of human civilization so guys let me know your thoughts about artificial intelligence in the comment section I'd love to know what you guys think about AI and whether you're believe yeah I will take over the world or not so now let's move on and talk about how artificial intelligence is different from machine learning and deep learning a lot of people are tend to assume that artificial intelligence machine learning and deep learning are the same because they have common applications right for example Siri is an application of AI machine learning and deep learning so how are these technologies related right or how are they different from each other now artificial intelligence is the science of getting machines to mimic the behavior of human beings machine learning is the subset of artificial intelligence that focuses on getting machines to make decisions by feeding them data deep learning on the other hand is a subset of machine learning that uses a concept of neural networks to solve complex problems so to sum it up to you artificial intelligence machine learning and deep learning heavily interconnected fields right machine learning and deep learning aids artificial intelligence by providing a set of algorithms and neural networks to solve data-driven problems however AI is not restricted to only machine learning and deep learning right it covers a vast domain of fields which include natural language processing object detection computer vision robotics expert systems and so on right so AI is a very vast field guys I hope I cleared the difference between AI machine learning and deep learning also a lot of you might be confused about data science data science is now an umbrella term right data science basically means to derive useful insights from data so data science actually uses AI machine learning and deep learning right so it implements all of these three technologies in order to derive useful insights from data right now let's move on to the most interesting topic in artificial intelligence which is machine learning now guys the term machine learning was first coined by a scientist known as Arthur Samuel in the year 1959 looking back that your was probably the most significant in terms of technological advancements in order to define machine learning if you browse the internet for what is machine learning you'll get at least 100 different definitions in simple terms machine learning is a subset of artificial intelligence which provides machines the ability to learn automatically and improve from experience without being explicitly programmed to do so in a sense it is the practice of getting machines to solve problems by gaining the ability to think now the question here is can a machine think or can a machine make decisions well if you fear a machine a good amount of data it will learn how to interpret process and analyzes data by using something known as machine learning algorithms to give you a basic idea of how the machine learning process works look at the figure on this slide a machine learning process always begins by feeding the Machine lots and lots of data now by using this data the machine is trained to detect any hidden insights and trends in the these insights are then used to build a machine learning model by using a machine learning algorithm in order to solve a problem the basic aim of machine learning is to solve a problem or find a solution by using data now moving ahead I'll be discussing the machine learning process in depth right so don't worry if you haven't got the exact idea of what machine learning is and the machine learning process involves building a predictive model that can be used to find a solution for a particular problem a well defying the machine learning process will have around seven steps it always begins with defining the objective followed by data gathering or data collection then we have something known as preparing data which is also called data pre-processing then we have data exploration or exploratory data analysis this is followed by building a machine learning model then we have model evaluation and finally predictions this is how the process of machine learning works to understand the machine learning process let's assume that you've been given a problem that needs to be solved by using machine learning let's say that the problem is to predict the occurrence of rain in your local area by using machine learning now the first step is to define the objective of the problem right as this step we must understand what exactly needs to be predicted in our case the objective is to predict the possibility of rain by studying the weather conditions but this stage it is essential to take mental notes on what kind of data can be used to solve this problem or the type of approach that you must follow to get to the solution the questions you should be asking yourself is what are we trying to predict right here we are trying to predict whether it will rain or not right you need to understand what are the target features target features are basically the variable that you need to predict here we need to predict a variable that will show us whether it's going to rain tomorrow or not then you must also understand what kind of data you will need to solve this problem apart from that you need to know what kind of problem you're facing is it a binary classification problem or is it a clustering problem now if you don't know what classification and string is don't worry I'll be talking about all these things in the upcoming slides so your first step is to define the objective of your problem you need to understand what exactly needs to be done here right how can you solve this problem moving on your next step is to gather the data that you need at this stage you must be asking questions such as what kind of data is needed to solve this problem is the data available to me and if it's not available how can I get the data right once you know the type of data that is required you must understand how you can derive this data data collection can be either done manually or it can be done by web scraping but don't worry if you're a beginner and you're just looking to learn machine learning you don't have to worry about getting the data there are thousands of data resources on the web you can just download the data set and you can get going coming back to the problem at hand the data needed for weather forecasting includes measures such as humidity level your temperature the pressure the locality whether or not you live in an institution and so on Sajida must be collected and it has to be stored for analysis this is where you collect all the data now moving on to step number three is data preparation the data that you collected is almost never in the right format all right even if you collect it from a internet resource if you download it from some website even then your data is not going to be clean right it's not going to be in the correct format there is always going to be some sort of inconsistencies in your data inconsistencies include any missing values or any redundant variables duplicate values all of these are inconsistencies removing all of this is very essential because they might lead to any wrongful computation therefore this stage you can scan the entire data set for any missing values and you have to fix them here itself now actually this is one of the most time-consuming steps in a machine learning process if you ask a data scientist which step he hates the most or which step is you know the most time-consuming they are probably going to tell you data processing and cleaning right it's one of the most tiresome tasks because you need to look at all the values that are there you need to find any missing values any data that is not relevant to you right all of this has to be removed so that you can analyze the data in a better way now step number four is exploratory data analysis so guys this thing is all about getting deep into your data and finding all the hidden data mysteries EDA or exploratory data analysis is like the brainstorming stage of machine learning data exploration involves understanding the patterns and the trends in your data so at this stage all the useful insights are drawn and any correlations between the variables are understood for example in the case of predicting rainfall we know that there is a strong possibility of region if the temperature has fallen low such correlations have to be understood and mapped at this stage area is actually the most important step in a machine learning process because here is where you understand your data you understand how your data is going to help you predict the outcome moving on to step number five we have building a machine learning model so all the insights and all the patterns that you got from your data exploration stage those insights are used to build the machine learning model so this stays always begins by splitting the data set into two parts that is training and testing data now remember that the training data will be used to build and analyze the model the model is basically the machine learning algorithm that predicts the output by using the data that you feed to it an example of machine learning algorithm is logistic regression and linear regression all of these are machine learning algorithms now don't worry about choosing the right algorithm right first we focus on what the machine learning processes but anyway choosing the right algorithm will depend on several factors right it depends on the type of problem you are trying to solve the data set and the level of complexity of the problem in the upcoming sections we'll discuss all the different types of problems that can be solved by using machine learning moving on to step number six we have model evaluate and optimization now after you build a model by using the training data set it is finally time to put the model to a test the testing data set is used to check the efficiency of the model and how accurately it can predict the outcome now once the accuracy is calculated and any further improvements in the model they have to be implemented at this stage methods like parameter tuning and cross-validation can be used to improve the performance of the model before I move any further I don't know if all of you know what training and testing data set means in machine learning the input data is always divided into two sets we have something known as the training data set and we have something known as the testing data set so in machine learning you always split the data into two parts right this process is known as data splicing now the training data set will be used to build the machine learning model and the testing data set will be used to test the efficiency of the model that you built this is what training and testing data set is they're not any different data that you derived they're the same as the input data set the only thing is you are splitting the data set so that you can train the model on one data and test the model on another data now remember that the training data set is always larger in size when compared to the testing data set because obviously you are training and building the model by using the training data set the testing data set is just for evaluating the performance of your model now let's move on and understand step number seven which is predictions and once the model is evaluated and you've improved the model it is finally used to make predictions the final output can be a categorical variable or it can be a continuous quantity and all of this depends on the type of problem you're trying to solve don't worry I'll be discussing the type of problems that can be solved using machine learning in the upcoming slides in our case for predicting the occurrence of rainfall and the output will be a categorical variable categorical variable is anything that has some categorical values for example gender is a categorical variable has either male/female or other it has a defined set of values that is a categorical variable so guys I was the entire machine learning process now as we continue with this tutorial in the upcoming sections I will be running a demo in Python in which we will be performing weather forecasting so make sure you remember all these steps that I spoke about because I'll be going through all these steps by using Python we'll be coding all of this that we just spoke about now the next topic we are going to discuss is the types of machine learning a machine can learn to solve a problem by following any one of the three approaches you can see that there are three ways in which a machine learns the three ways are supervised learning unsupervised learning and reinforcement learning these are the three methods in which you can train a machine to learn so first let's discuss supervised learning so what is supervised learning supervised learning is a technique in which we teach or train the Machine by using data which is labeled to understand this better let's consider an analogy as kids be all needed guidance to solve math problems at least I had a really tough time solving math problems yeah so are teachers always helped us understand what addition is and how it is done similarly you can think of supervised learning as a type of machine learning that involves a guide the label data set is the teacher that will train the machine to understand the patterns in the data the label data set is nothing but the training data set so to better understand this consider the figure right here we are fearing the machine images of Tom and Jerry and the goal is for the machine to identify and classify the images into two separate groups basically one group will contain Tom images and the other group will contain images of Jerry now pay attention to the training data set the training data set that is fed to a model is labeled as in my telling the Machine listen this is how Tom looks and this is how Jerry looks but basically labeling each data point that were feeding to the machine right if the image of Tom's we've labeled it as Tom and if the image is a Jerry image then we're going to label it as Jerry by doing this you're training the machine by using label data so to sum it up in supervised learning there is a well-defined training phase done with the help of label data like the rest of the process is the same after you feed the Machine label data you're going to perform data cleaning then exploratory data analysis followed by building the machine learning model and then model evaluation and finally your predictions also one more point to remember is that the output that you're going to get in a supervised learning algorithm is a labelled output this Jerry will be labeled as Jerry and the storm will be labeled as Tom basically you get a labeled output now let's understand what is unsupervised learning unsupervised learning involves training by using unlabeled data and allowing the model to act on that information without any guidance so think of unsupervised learning as a smart kid that learns without any guidance in this type of machine learning the model is not fed with any label data as in the model has no clue that this image is Tom and this image is Jerry it figures out patterns and the differences between Tom and Jerry on its own by taking in tons of data for example it identifies prominent features of Tom such as point to yours bigger in size and so on to understand that this image is of type 1 similarly it finds such features in Jerry and knows that this is another type of image may be typed too right therefore it classifies the images into two different clusters without knowing who is Tom and who is Jerry now the main idea behind the unsupervised learning is to understand the patterns in your data set and form clusters based on feature similarity basically it will feature similar images or similar data points into one cluster and it will form another cluster which is totally different from the first cluster so look at the output of where the unlabeled output is basically clusters or groups of two different data next we have something known as the reinforcement learning reinforcement learning is comparatively different right it's pretty different from supervised and unsupervised it is basically a power of machine learning where you put an agent in an environment and this agent learns to behave in the environment by performing certain actions and observing the rewards which it gets from these actions to understand reinforcement learning imagine that you were dropped off at an isolated island what would you do initially we'd all panic right but as time passes by you will learn how to live on the island you will explore the environment you will understand the climate conditions you will understand the type of food that grows there you'll know what is dangerous to you and what is not you will understand which food is good for you and which is not this is exactly how reinforcement learning works it involves an agent which is basically you stuck on the island that is put in an unknown environment which is the island where the agent must learn by observing and performing actions that result in rewards reinforcement learning is mainly used in advanced machine learning area such as self-driving cars alphago and so on so guys that sums up the types of machine learning before we go any further I'd like to discuss the difference between supervisor unsupervised and reinforcement learning now first of all we have the definition supervised learning is all about teaching a machine by using label data unsupervised learning like the name suggests there is no supervision over here the machine is trained on unlabeled data without any guidance reinforcement learning is totally different here you have an agent who interacts with the environment by producing actions and discovers some errors and rewards now the type of problem that is solve using supervised learning is regression and classification problems we'll discuss what regression classification and clustering is in the upcoming slide right so don't worry if you don't know what it is unsupervised learning is mainly to solve association and clustering problems reinforcement learning is for reward based problems now what is the type of data in supervised learning it is labeled data that the main difference between supervised than any other type of machine learning and supervised you have labeled data in unsupervised we have unlabeled data whereas in reinforcement learning we have no predefined data at all the machine has to perform everything from scratch it has to collect data analyze do everything on its own now the training in supervised learning is external supervision meaning that we have external supervision in the form of the labeled training data set in unsupervised there is obviously no super version there is an unlabeled data set therefore there is no supervision in reinforcement learning there is no supervision at all now the approach to solving supervised learning problem is basically you're going to map your labeled input to your known output and unsupervised learning the machine is going to understand the patterns and discover the output on its own reinforcement learning here the agent will follow something known as a trial and error method right it's totally based on the concept of trial and error popular algorithms under supervised learning are linear regression logistic regression support vector machines and so on under an supervised learning we have the famous key means clustering algorithm under reinforcement learning we have the cue learning algorithm which is one of the most important algorithms it is basically the logic behind the famous alphago game I'm sure all of you have heard of that so guys these were the differences between supervised unsupervised and reinforcement learning now let's move on and discuss the type of problems that you can solve by using machine learning now there are three types of problems in machine learning now any problem that needs to be solved in machine learning can fall into one of these three categories now what is regression in this type of problem the output is a continuous quantity for example if you want to predict the speed of a car given the distance that means it is a regression problem first of all what is a continuous quantity a continuous quantity is any variable that can hold a continuous value a continuous variable is any variable that can have infinite number of values for example the height of a person or the weight of a is a continuous quantity and I can have a weight of 50 point 1 kgs or 50 point 1 2 or 50 point 1 1 2 kgs this is a continuous quantity regression problems can be solved by using supervised learning algorithms another type of problem is the classification problem here the output is always a categorical value classifying emails into two classes for example classifying your email as spam and non-spam is a classification problem here again you'll be using supervised learning classification algorithms such as support vector machines naive bias logistic regression and so on then we have clustering problem and this type of problem involves assigning the input into two or more clusters based on feature similarity for example clustering the viewers into similar groups based on their interest or based on their age or geography can be done by using unsupervised learning algorithms like k-means clustering one thing you need to understand is under supervised learning you can solve regression and classification problems under unsupervised learning you can solve clustering problems reinforcement learning is something else altogether right he can solve reward based problems and more complex and deep problems so now let's move on and understand the different machine learning algorithms now I will not be going into depth for machine learning algorithms because there are lot of algorithms to cover but we have content around almost every machine learning algorithm out there so I'll be leaving a couple of links in the description box right you can check out all these links and understand how each of these machine learning algorithms work in depth so I'm just going to show you a hierarchical diagram of how the algorithms are structured so under machine learning we have three types of learning we have supervised unsupervised and reinforcement under supervised learning we have regression and classification problems and under unsupervised learning we have clustering problems reinforcement learning is completely different I'll be leaving a link in the description specifically for reinforcement learning you can check out the entire content of reinforcement learning there now regression problems can be solved by using linear regression algorithm such as linear regression decision trees and random forests can also be used in regression problems but usually decision trees and random forests all of these are used to sort classification problems famous classification algorithms include K nearest neighbor which is basically KN n decision trees and random forests logistic regression naive bias support vector machines all of these are classification algorithms coming to unsupervised learning we have clustering and Association analysis and clustering problems can be solved by using k-means and association analysis can be solved by using a priori algorithm a priori algorithm is mainly used in market basket analysis right for this algorithm as well I'll be leaving a link in the description we've performed a very excellent demo where enma showed how market basket analysis can be done by using a priori algorithm Markov model is also explained in one of the videos are believing that link in the description box now to sum up machine learning to you I'll be running a small demonstration in Python right like I promised earlier I will be using Python to understand the whole machine learning process alright so let's get started with that demo so guys for those of you who don't know Python I will leave a couple of links in the description box so that you understand Python but apart from that Python is pretty understandable if you just look at the code you'll know what exactly I'm talking about right so don't worry and also I'll be explaining everything in the cone so I'm using pycharm in order to run the demo right so guys like I said if you don't know Python and leave a couple of links in the description box you can go through those videos as well the main aim of our demo is to build a machine learning module that will predict whether or not it will rain tomorrow by studying the past data set now this data set contains around one hundred and forty five thousand observations on the daily by the conditions as observed in Australia right the data set has around 24 features and we will be using 23 features out of that to predict the target variable serene tomorrow so this data that are collected from CAG will write for those if you don't know Cargill is an online platform making fine hundreds of data sets and you know there are lot of competitions held by machine learning engineers and all of that it's an interesting website now the problem statement itself is to build a machine learning model that will predict whether or not it will train tomorrow this is clearly a classification problem the machine learning model has to classify the output into two classes that is either yes or no yes will stand for it will rain tomorrow and Noble basically don't know that it will not rain tomorrow right this is a classification problem so I hope the objective is clear right so we'll begin the demonstration by importing the required libraries so first of all for mathematical computations will be importing the numpy library will also be importing the panda's library for data processing next we will load the CSV file basically my data is stored in a CSV format in this file whether ostrich CSV is my data set so basically I've saved this file in this path right so that's what I'm doing here I'm loading my data set and I'm storing it in a variable known as DF next what we'll do is we'll see the size of our data frame let's print the size of the data frame we'll also display the first five observations in our data frame let's look at the output basically around one hundred and forty five thousand observations and twenty four features now twenty four features are basically the variables that are there in my data set you know for example date is available location is available minimum temperature till rain tomorrow all of these are variables so have around 24 features in my data set right now the variables that I have to predict is rain tomorrow okay if the value of rain tomorrow is no it denotes that it will not rain tomorrow but if the value is yes then it will denote that it will rain tomorrow so rain tomorrow is basically my target variable right I'll be finding out whether it's going to rain tomorrow or not so this is my target variable also known as your output variable input variables will be the other 23 variables date location minimum temperature rain today risk all of this will be my input variables now these variables are also known as predictor variables basically they are used to predict your outcome so these are also known as predictor variables now the next step is checking for null values this is basically data pre-processing that which is commented for you this is data preparation or data right so this stick is data pre-processing right here we start checking for any null values or any missing values this is exactly what I'm doing over here I am checking for any missing or null values in my data set if you notice the output it shows that the first four columns have more than 40% null values and so it's always best for us to remove features or such variables because they will not help us in our prediction now during data pre-processing it is always necessary to remove the variables that are not significant unnecessary data will just increase our computations that's why it's always best if we remove the unwanted or unnecessary variables now apart from removing these four variables will also remove the location variable and we will remove the date variable right I'll come to this variable in a minute we'll also be removing location and date variable because both of these variables are not needed in order to predict whether it'll rain tomorrow right we do not need to know the location and the date now we'll also be removing this risk mm variable risk mm variable basically tells us the amount of rain that might occur the next day right now this is a very informative variable and it might actually leak some information to a model by using this variable will easily be able to predict and there's no point of doing that this variable will give us too much information and so that's why we're going to remove this variable as well it'll leak a lot of information so after that if you print the shape of your data frame we have only 17 variables and so many observations now after this bilges look at any null values and we'll remove them this drop dot any function bulges to remove all the null values right then if you print the shape of your data frame we'll have around 112 thousand rows with 17 variables this is the shape of the data set after removing all the null values and all the redundant or unnecessary variables now it's time to remove the outliers in the data so after you remove any null values we should also check our data set for any outliers an outlier is a data point that is very different other observations outliers usually occur because of miscalculations while collecting the data these are some sort of errors in your data set so in this whole code snippet which is getting rid of outliers this is the output that we get all our outliers next what we'll be doing is we will be assigning zeros and ones in the place of yes and no the only thing is we're going to change the categorical variables from yes and no to zero and one right that's exactly what we're doing over here now if there are any unique values such as any character values which are not supposed to be there we'll be changing them into integer values that's all we're doing over here after this we'll be normalizing our data set this is a very important step because in order to avoid any biasness in your output you have to normalize your input variables I had to do this we can make use of the main max scalar function which Python provides in a package known as SQL on you can use that package in order to normalize your data set so after normalizing our data set this is what our data set looks like this is before normalization you can see that these are in two digits whereas these values are in single digits right this causes a lot of biasness but once we normalize the values we know that all of the values are in a similar range we have everything in decimals and so normalization is something that has to be performed because if you have a data set like this your output is not going to be correct and that's why we perform normalization so now that we are done with pre-processing what we're going to do is it's time for exploratory data analysis let me just comment it for you this is exploratory data analysis so basically or what we're going to do is we're going to analyze and identify the significant variables that will help us predict the outcome to do this we'll be using on the select key best function which is present in the SQL on library there is a predefined function in python called select key best which will basically select the most significant predictor variables in our data set when we run that line of code we get these three variables to be the most significant variables in our data set now the main aim of this demo is to make you understand how machine learning works that's why to simplify the computation will assign only one of these significant variables as the input instead of taking all three or variables as input will select one variable and we'll take that as the input and the output is the reigned model variable so basically we are creating a data frame of all the significant variables basically we're choosing this variable in order to predict our outcome obviously our outcome is rain tomorrow variable so our input is humidity level and our output is to detect whether it'll rain tomorrow the next step is data modelling all of you are aware of what data modelling is to solve this we'll be using classification algorithms over here we'll use logistic regression we will use random for this classifier which is another machine learning algorithm we'll also use the decision tree classifier and a support vector machine right we'll be using all of these algorithms in order to predict the outcome we will also check which algorithm gives us the best accuracy so guys we're just using multiple algorithms or multiple classification algorithms on the same data set we're not doing anything very complex over here so we start by importing all the necessary libraries for the logistic regression algorithm we're also going to import time because we'll be calculating the accuracy and the time taken by the algorithm to get the output so the first step is data splicing I've already mentioned data splicing is splitting your data set into your testing data set and into your training data set that's exactly what we're doing over here so 25% of your data is assigned for the testing and the remaining 75% is your training data here you're creating the instance of the logistic regression algorithm this is an instance that you created then you'll fit the model by using your training data set so basically to build your machine learning algorithm you'll be fitting your training data set so extreme and white trained variables have your training data set after that you will be evaluating the model by using your testing data set then you'll calculate the accuracy score right I'll also be printing the accuracy using logistic regression and the time taken using logistic regression let's look at the accuracy don't worry about these warnings they are not important so accuracy using logistic regression is around zero point eight three percent which is eighty-three percent accuracy approximately 84% and this is the time taken so the accuracy is actually pretty good right 84 percent is a good number then we have random forest classifier your again will import the libraries that are needed to run random forest classifier then we are again calculating the accuracy and the time taken by the classifier data splicing like I mentioned splitting the data into testing and training data set then you're just building the model by using the training data set after that you'll evaluate the model by using the testing data set and you'll finally calculate the accuracy the accuracy using random forests is again approximately 84% which is a really good number then we have decision tree classifier here again we'll be importing the libraries needed for this classifier we'll be calculating the accuracy and the time taken by this classifier data splicing followed by building the model by using the training data set evaluating the model by using the testing data set and finally calculating the accuracy and printing the accuracy so let's see the accuracy using decision tree classifier again we have an accuracy of around 83 to 84 percent this is a pretty good number and last we're going to do this by using another classification algorithm known as support vector machine here again we are importing the needed libraries then we are calculating the accuracy and the time performing data splicing then we're building the model by using the training data set testing the model using the testing data set and finally printing the accuracy so guys all the classification models gave us an accuracy score of approximately 84% to 83% so this is exactly how a machine learning process works right you begin by importing all your data then you perform data pre-processing or data cleaning after that you perform exploratory data analysis where you understand the important patterns or the important variables in your data set after that you could build a model then you will evaluate the model by using the testing data set and finally calculate the accuracy I showed you all the steps in the machine learning process by using a practical demonstration in Python so guys give yourself a pat on the back because we just understood the whole machine learning process with a small implementation in Python now let's move on to our next topic which is limitations of machine learning before we understand what deep learning is it's important to know the limitations of machine learning and why these limitations give rise to the concept of the plan one major problem in machine learning is machine learning algorithms and models are not capable of handling high dimensional data right we can take in data with 20 to 30 feature variables but when it comes to data sets which have thousands of variables machine learning does not work machine learning is not capable enough to process that much data so high dimensional data cannot be analyzed processed and modeled by using machine learning another limitation is that it cannot be used in image recognition and object detection because these applications require of the implementation of high dimensional data another major challenge in machine learning is to tell the Machine what are the important features it should look for in order to precisely predict the outcome so basically you're selecting the important features for the machine learning model and you're telling them like these are the important features and this is what you should use in order to build the model this process is known as feature extraction now in machine learning this is a manual process you are going to manually input as a programmer you're going to tell that these are the important predictor variables but what happens when your data set has hundreds of variables how are you going to sit and choose every variable and perform analysis on each variable to understand which is a really significant period that's going to become a very tedious task right it's not possible for you to manually sit down with 100 variables check the correlation with each variable and understand which variable is significant in predicting the output so performing feature extraction manually is very tedious and that is one of the major limitations of machine learning now deep learning comes to the rescue to all of these problems so let's understand what deep learning is and why we have deep learning in the first place so deep learning is actually one of the only methods by which we can overcome the challenge of feature extraction this is because deep learning models are capable of learning to focus on the right features by themselves requiring minimal human intervention meaning that feature extraction will be performed by the deep learning model itself you don't have to manually tell that this feature is important that features important choose this feature for predicting the output although this is not needed in deep learning the model itself will learn which features are most significant in predicting the output also deep learning is mainly used to deal with high dimensional data right it is based on the concept of neural networks and is often used in object detection and image processing this is exactly why we need deep learning it solves the problem of processing high dimensional data and manual feature extraction now how exactly does the planning work now deep learning mimics the basic component of the human brain called the brain cell the brain cell is also known as a neuron so inspired from a neuron an artificial neuron was developed the planning is based on the functionality of a biological neuron so let us understand how we mimic this functionality in an artificial neuron now guys an artificial neuron is also known as cept Ron let's understand what this biological neuron does and how deep learning is based on this concept in a bio djegal neuron you can see these dendrites right in this image you see something known as ten tribes these dendrites are used to receive any input these inputs are summed in the cell body and through the axon it is passed on to the next neuron so similar to the biological neuron a perceptron or artificial neuron receives multiple inputs applies various transformations and functions and provides an output right so that's how artificial neural networks or that's how deep learning books now as the human brain consists of multiple connected neurons called a neural network similarly by combining multiple perceptrons we've developed what is known as deep neural networks the main idea behind deep learning is neural networks and that's what we are going to learn about so now let's understand what exactly deep learning is deep learning is a collection of statistical machine learning techniques used to learn feature hierarchies based on the concept of artificial neural networks so the main idea behind deep learning is to use the concept of neural networks a deep neural network will have three layers ok there's something known as the input layer followed by the hidden layers and then we have the output layer the input layer is basically the first layer and it receives all the inputs so all the inputs are fed into this input layer the last layer is obviously the output layer this layer will provide your desired output now all the layers between the input and your output layer are known as the hidden layers now the number of hidden layers in a deep learning network will depend on the type of problem you're trying to solve and the data that you have we'll get into depth of what exactly are in layer does but for now this is how a neural network is structured in deep learning so guys our deep learning is used in highly computational use cases such as phase verification self-driving cars and so on and so let's understand the importance of deep learning by looking at a real-world use case so I'm sure all of you have heard of the company people now PayPal makes use of deep learning to identify any possible fraudulent activities so the company makes use of deep learning for fraud detection now paper recently processed over two hundred and thirty five billion dollars in payments from four billion transactions by its more than 170 million customers to basically process this much data by using the planning PayPal uses machine learning and deep learning algorithms to mine data from the customers purchasing history in addition to reviewing patterns of any sort of fraud stored in the database and it will do this to predict whether a particular transaction is fraudulent or not now the company has been relying on deep learning and machine learning technology for around ten years initially the fraud monitoring team used simple linear models right they used machine learning but over the years of companies switched to more advanced machine learning technology called deep learning the shows how deep learning is used in more advanced and more complicated use cases the fraud risk manager and the data scientist at people he quoted that what we enjoy from more modern advanced machine learning is its ability to consume a lot more data handle layers and layers of abstraction and be able to see things that a simpler technology would not be able to see even human beings might not able to see this is exactly what he quoted he said that a simple linear model is capable of consuming around twenty variables but with deep learning technology you can run thousands of data points he also quoted that there is a magnitude of difference you will be able to analyze a lot more information and identify patterns that are a lot more sophisticated so by implementing the planning technology PayPal can finally analyze millions of transactions to identify any fraudulent activity this is how people makes use of deep learning not only PayPal we also have Facebook right Facebook makes use of deep learning technology for face verification you've all seen the tagging feature at Facebook where we tagged our friends in photos all that is based on deep learning and machine learning so guys I was a real-world use case to make you understand how important deep learning is now let's move on and look at what exactly a perceptron is right we'll be going in depth about deep learning a perceptron is basically a single layer neural network that is used to classify linear data it is the most basic component of a neural network now a perceptron has four important components it has something known as inputs weights and bias summation functions activation and transformation functions these are four important parts of a perceptron now before I discuss this diagram with you let me tell you the basic logic behind a perceptron there is something known as inputs right the input X here you can see X 1 X 2 2 X n so let me explain the structure of a perceptron what you're going to do is you're going to input variables into the perceptron right this X 1 X 2 2 X n basically stands for input W 1 W 2 till W n stands for the weight assigned to each of these inputs right there is a specific weight that will be randomly initialized in the beginning for each of your input next you have something known as the summation element here what you do is you multiply the respective input with the respective weight and you add all these products right that is basically a summation function after this is what is your transfer function also known as activation function right the activation function will basically map your input to your desired output so your input will go through these processes it will go through summation and activation function in order to get to the output so guys remember that the neural networks work the same way as the perceptron so if you want to understand how deep neural networks work you need to understand what a perceptron does a deep neural networks is nothing but multiple perceptrons so let me tell you how the entire thing works once again so basically all your inputs are multiplied with their respective reads now you add all the multiplied values and you call them as a weighted sum you use the summation function to add all of this after that you apply the weighted sum to the correct activation function or the transfer function activation function is very similar to a function in our brain the neurons become active in our brain after a certain potential is reached that threshold is known as the activation potential so mathematically there are a few functions which represent the activation function basically the Signum the sigmoid the tan edge all of these are activation functions you can think of activation function as a function that map's the input to the respective output then I spoke about something known as beats and biases all right now you must be wondering why do we have to assign weights to each of our input weights basically show the strength of a particular input or how important a particular input is for predicting the output in simple words the weight age denotes the importance of an input bias is basically a value which allows you to shift the activation function curve in order to get a precise output right so that's exactly what weights are I hope all of you are clear with inputs weights summation and activation function also one important thing I forgot to mention in a perceptron is a single-layer perceptron will have no hidden layers right they'll only be an input layer and output layer and a couple of transformation functions in between that's all we'll be there in a perceptron now a perceptron like I mentioned is used to solve only linear problems if you look at this data distribution how do you think we can solve this this data is not linearly separable so you cannot use a single layer perceptron to separate this data and that's why we need something known as a multi-layer perceptron with back propagation I'll be explaining this in the next slide so complex problems that involve a lot of parameters and high dimensional data can be solved by using multiple layer perceptron multi-layer perceptron is the same as a single layer perceptron the only difference is that a multi-layer perceptron will have hidden layers so the number of hidden layers in model depends upon various factors I told you it depends on the complexity of the problem you are trying to solve it depends on the number of inputs in your data and so on so it works in the same way all your inputs are multiplied with your weights and then you do the summation and then there is a transformation function or activation function while designing a neural network in the beginning itself I told you be initialized weights with some random values we do not have some specific value for each weight page initially we have selected random values it is always important that whatever weight values we have selected will be correct now whatever weight values we have assigned to each input it denotes the importance of that input variable so we need to assign the weights in such a way or we need to update the weights in such a way that it denotes the significance of that particular input so initially we are selecting some random value for weight and let's say that we use this weight value to get our output now what happens is the output is actually very different or it is not precise when compared to our actual output so basically the error value is very huge so how will you reduce the error the main thing in our neural network is the weight reads that you give to a input variable right depending on the weights that you give to an input variable you are telling the neural network how important that variable is now what if you randomly give some weight age and your output is wrong the first thing that comes into your mind is that you need to change the weight because the weight signifies the importance of a variable so basically what we need to do is we need to somehow explain to the model to change the weight in such a way that the error becomes minimum let's put it in another way so basically we need to train our model one way to train our model is called as back propagation so on back propagation what happens is once you've initialized a weight to each of the input you calculate the out put night you get an output and let's say you have a very high error value and that output what you'll do is you'll back propagate as in you'll go back to the weight and you'll keep updating the weight in such a way that your error becomes minimum this is exactly what back propagation is you'll be going back to the first layer you'll be updating each of the widths in such a way that your output is more precise so guys basically the weight and the error in a neural network is highly related by updating the weight in a particular way your error will decrease so you need to figure out how you need to update the weight do you have to increase the weight or decrease the weight once you figure out whether you have to increase or decrease the weight you have to just follow that direction in such a way that your error is minimized that's exactly what back propagation is so the final output of back propagation is you're going to select the weight that minimizes the error function and then you're going to use that weight to solve the whole problem and this is what back propagation is about now in order to make you understand deep neural networks let's look at a practical implementation so again guys i'll be using python to run the demo if you don't have a good idea about python check the description i'll leave a couple of links about python programming now in this demo i'll be walking you through one of the most important applications of deep learning I will demonstrate how you can construct a high-performance model to detect credit-card fraud right we'll be using deep learning models to do this now before that let me just tell you something about our data set right the data set contains transactions made by credit cards in the year September 2013 by European card holders this data set presents transactions that occurred in two days where we have four hundred and ninety two frauds out of two hundred and eighty five thousand transactions approximately two hundred and eighty five thousand transactions are to these transactions for ninety two were frauds and the data set is quite unbalanced right the positive class accounts for 0.172 percent so the positive class basically the fraudulent class so again we are going to start by importing the required packages we are going to import Kyra's match plot library Seabourn library and Escalon for pre-processing right again min/max gala which is for normalization we are going to import our data set and store it in this variable right this is the path to my data set my data set is in the CSV format or also known as comma separated version now we are going to print out the first five rows of our data set right let's take a look at the output so here is the time of the transaction v1 v2 v3 etc these are all the features of our data set I am now going to go into depth of what these features stand for because this demo is all about understanding the planning now these b1 b2 b3 these are all predictor variables which will help us predict our class so guys don't worry about what these features are these features are just information and details about your transaction such as the amount you spend or the time of transaction and so on so here we have the amount variable which denotes the amount spent after that we have the class variable now this class variable is your output variable or your target variable so your class is basically your output variable value 0 denotes that there has been no fraudulent activity but if you get a class of 1 it means that this transaction is a fraudulent transaction for example this transaction is not fraudulent that's why we have a value of 0 over here alright so this is our data set next what we're doing is we're counting on the number of samples for each class right we have class 0 and class 1 we're in class 0 denotes the normal transaction which is non fraudulent transaction and class 1 will denote the fraudulent transactions right so we have around 492 fraudulent transactions and around 280 4315 non fraudulent transactions so when you see this you know that our data set is highly unbalanced highly unbalanced means that one class has a really small number when compared to the other class right there's no balance between the two classes so here what we're doing is we are sorting the data set by class for stratified sampling stratified sampling is a statistical technique for sampling your data set now this type of sampling is always good if you have an unbalanced data set next what we are going to do is we're going to perform data pre-processing data pre-processing in the planning mainly has a method known as dropout method next what we are going to do is we are going to drop out the entire time column we do not need the time of the transaction in order to understand if the transaction was fraudulent or not alright so that's why we're getting rid of unnecessary variables and so we're dropping out that variable so after dropping out the time variable we are going to assign the first 3,000 samples to a new data frame right this DF sample will have our first 3,000 samples and we are going to use those 3,000 samples so here which is counting the number of class for each of these samples after that which is counting the number of samples for each of the class that we're doing the same thing again and here we get class zero has two thousand five hundred and eight samples and class one has 492 samples now this makes the data set quite balanced right it's very balanced when compared to our old data set next we just randomly shuffle our data set right in order to remove any sort of biasness in the data after that we'll split our data set into two parts one is for training and your other data set is for testing right this is also known as data splicing then we'd be splitting each data frame into feature and label meaning that your input and your output you'll be doing this for your training data and for your testing data right all you're doing is you're separating your input from your output next by looking at our training data set right well printing the shape of a training data set the training data set has around 2,400 observations and 29 variables or 29 features similarly we'll be printing out the size of our test data frame right that's exactly what we're doing over here after that we'll perform normalization all right for this we'll be using the min max kilo so a normalization will basically be scaling all our Preta wearables around the same range so that there is no bias nests in our prediction after this we'll be plotting a function for each of the learning curves for your training phase and for your testing phase you'll be plotting a learning curve now I'll show you the output of this in a couple of minutes for now let's move on to the main part which is model creation right in this demo we'll use three fully connected layers will also use dropout technique now dropout is a type of regularization technique that is used to avoid any sort of overfitting in a neural network technique where you select neurons and you drop them during the training phase we'll be using the relu as the activation function which is a type of activation function just like sigmoid and panitch so the type of model that we'll be using is the sequential model right sequential is the easiest way to build a model in Kira's right by using the Kira's library over here if you remember I imported that in the beginning it allows you to build a model layer by layer so each layer has weights that correspond to the layer that follows it after this you'll use the add function to add the dense layers basically your hidden layers you're going to add over here so in our model we'll be adding two dense layers or hidden layers you can say so here what we're doing is we're adding the first dense layer now guys a dense layer is standard layer type that works for most cases right in a dense layer all the nodes in the previous layer connect to the nodes in the current layer so guys don't get too involved into what exactly is happening here all I'm doing is I'm creating a sequential model and what is happening is I'm just assigning the number of inputs for each of the dense layer or for each of the hidden layer I am also assigning dropout value dropout is basically to prevent overfitting oh fitting might occur when your model memorizes the training data set overfitting basically reduces the accuracy of a model that's why we're using the dropout method to prevent overfitting so in the first hidden layer we have around 200 units right we have the activation function R a loop then we're adding the second dense layer with again 200 neurons and the rail u activation function kernel initializer is uniform meaning that is just sequential and normal then we're again adding a dropout layer of 0.5 the dropout value of a network has to be chosen very wisely okay a value that is too low will result in a minimal effect and a value that is too high will result in under learning by the network so 0.5 is a standard dropout value this last layer is our output layer in the output layer will obviously have only one neuron we'll have one neuron that will show us the output class either 0 or 1 0 will show us non fraudulent transactions and 1 will denote fraudulent transaction right that's why we have only one neuron over here and the activation function here is sigmoid right since the number of neurons is only 1 after that we're printing the model somebody now I'll show you the summary and everything before that ledges understand what exactly optimization functions are we'll understand what this optimization function dance now an optimizer takes care of the necessary computations that are used to change the network's weights and bias so basically your optimizers will take care of all your computation such as changing the weight or updating the weight if you'll remember I spoke about back propagation right where you'll update the weight and all of that that is run by using optimizers here we selecting an optimized are known as the atom optimizer so atom optimizer is one of the current default optimizers MD planning that it stands for adaptive moment estimation we don't have to get into the depth of all of this at all of these are predefined optimizers in our Kiera's package itself after this we are going to fit our model by using the training features but also setting 200 a box and also there's something known as a pox and bat size right we're setting a pox as 200 and bad size as 500 I'll tell you what exactly this means now bad sizes are basically used so that we don't over fit our model right we are going to basically split our data set into 500 batches so our input will be going in the form of batches right and our batch size is 500 inputs per batch and we'll be going through 200 epochs meaning that our training will iterate 200 times this is basically the number of times our training our model all right that's what I poke and bat size is after that we're just showing our training history and we're just printing the accuracy curve for our training phase they're also going to print our loss curves for our training phase basically the error curves and then finally we have the evaluation here we'll be testing our model by using our testing data set then we finally printing the accuracy on our testing data set after that which is going to plot a heat map which I'll be showing you all let me just show you the output so guys in this entire line of code all we're doing is we're printing an accuracy plot basically by renting a heat map I'll show you what the heat map looks like suggest to check the accuracy by comparing all the correctly predicted values to are incorrectly predicted values so this is our training history here blue stands for our training phase and this is a validation or a prediction stage that was our training curve and this is our loss curve and when you compare it to the actual validation stage is quite similar right meaning that our model is doing pretty well so guys this is the heat map that I was talking about this is basically going to give us the class for each of our predictions and it basically Lots the classes that we correctly predicted right basically for each data point it's just going to tell us whether we predicted it correctly or not it's sort of a confusion matrix in the form of a heat map so guys these are all our epochs basically the two hundred iterations that we went through right this is the 50th iteration is showing us our loss is showing us our accuracy as well and here we have 88 percent 90 percent 92 percent now if you carefully look at the epoch accuracy values you see that as we train our model even more our accuracy keeps increasing initially our accuracy was around 83 right at epoch number 15 our accuracy was around 83% but as we kept training our model a little bit more our accuracy kept increasing we have 90 we have 91 94 95 96 and so on and so basically the more you train your model the better it's going to be so guys this was our entire demo now in the end I'm printing out the false positive rate and the false negative rate right all of this basically denotes harmony of the data points was I correctly able to predict as fraudulent and how many did I predict wrongly that's all the false negative and the false positive rate denotes so guys this was the entire demo on deep learning now if you have any doubts regarding the deep learning demo please mention them in the comment section and I will solve your queries right now let's look at our last topic for the day which is natural language processing now before we understand what is natural language processing let's understand the need for natural language processing and a process known as text mining text mining and natural language processing are heavily correlated right I'll talk about both of these in the upcoming slides for now let me tell you why we need natural language processing or text mining so there's the amount of data that we generate these days is unbelievable it is a known fact that we are creating 2.5 quintillion bytes of data every day and this number is only going to grow with the evolution of communication through social media we generate tons and tons of data right the numbers are on your screen so basically we post around 1.7 million pictures on Instagram per minute right I'm talking about posts per minute all of these numbers are permanent values these are the amount of tweets three hundred forty-seven thousand tweets per minute right this is a lot of data may generating data while we're watching youtube videos when we're sending emails when we are chatting and all of that right even though IOT devices at our house right we have Alexa all of this is generating a lot of data a single click on your phone is generating a lot of data now not only that out of all the data that we generate only twenty one percent of the data is structured and well formatted right the remaining of the data is unstructured and the major sources of unstructured data include text messages from whatsapp Facebook Likes comments on Instagram the bulk emails and all of this at all of this accounts for the unstructured data that we have today now the data we generate is used to grow a business so by analyzing and mining the data we can add more value to a business this is exactly what natural language processing and text mining is all about text mining and NLP is a subset of artificial intelligence wherein we try and understand the natural language texts that we get from text messages and so on in order to derive useful insights and grow businesses by using these insights so what exactly is text mining text mining is a process of deriving meaningful insights or information from natural language text so all the data that we generate through text messages emails and documents are written in natural language text right and we're going to use text mining and natural language processing to draw useful in size or patterns from such data in order to grow a business now let's understand where exactly do we make use of natural language processing and smiling now have you ever noticed that if you start typing a word on Google you immediately get suggestions right this feature is known as autocomplete it will basically suggest the rest of the word to you we also have something known as spam detection right here's an example of how Google recognizes this misspelling Netflix and shows results for the keyboard that matches your misspelling let me show you a couple of more examples we also have predictive typing and spell checkers and features like auto correct email classification so predictive typing and spell checkers all of these are applications of natural language processing all of this basically involves processing the natural language that we use and deriving some useful information from it right or running businesses from it Netflix uses natural language processing in a really good fashion doing right it basically studies the reviews that customer gives for a particular movie and it tries to figure out if that movie is good or bad depending on the review so Netflix actually uses NLP in a very interesting manner it tries to understand the type of movies that a person likes by the way a person has rated the movie or by the way the person has reviewed a movie so by understanding what type of review a person is giving to a movie Netflix will recommend more movies that you like that's how important NLP has become now let's look at what exactly NLP is NLP which also stands for natural language processing is a part of computer science and artificial intelligence which deals with human language that it's basically the process of processing natural language in order to derive some useful information from it for those of you have studied natural language processing or have heard of natural language processing there is a huge confusion between text mining and natural language processing so text mining is the process of deriving high-quality information from text but the overall goal is to turn the text into data for analysis by using natural language processing so basically text mining is implemented by using natural language processing techniques rather various techniques in natural language processing that can help us perform text mining that's how text mining and natural language processing are related natural language processing is the techniques that are used to solve the problem of text mining text analysis and all of that has to get a couple more applications sentimental analysis is one of the major applications of natural language processing you see Twitter perform sentimental analysis Facebook Google all of these perform sentimental analysis sentimental analysis mainly used to analyze social media content that can help us determine the public opinion on a certain topic then we have chat BOTS now chat BOTS use natural language processing to convert human language into desirable actions we also have machine translation NLP is used in machine translation by studying the morphological analysis of each word and translating it to another language advertisement matching is also done using NLP in order to recommend ads based on your history right these are few of the applications of NLP let me tell you the basic terminology is under natural language processing so tokenization is the most basic step in natural language processing the organization means breaking down the data into smaller chunks or tokens so that they can be easily analyzed so the first step is you'll break a complex sentence into words then you'll understand the importance of each of the word with respect to that sentence in order to produce a structural description on an input sentence so for example take the sentence how would I perform two organizations on the senton let's say that tokens are simple is the sentence and I want to perform tokenization on the sentence this is what I am going to do I'm going to split the sentence into different words I'm going to understand each word with respect to that sentence right this is done to simplify operations in natural language processing right is always simpler to analyze a single token instead of analyzing an entire sentence then we have something known as stemming now look at this example right here we have words such as detection detecting detected and detections we all know that the root word for all of these words is detect so stemming algorithm basically does that it works by cutting off the end or the beginning of the word and taking into account a list of common prefixes and suffixes that can be found in an inflected word stemming basically helps us in analyzing a lot of words we know that detection is detected and detection basically mean the same thing so all we're doing is we are going to ease our analysis by removing prefixes and suffixes which not make sense right we just need to understand the morphological analysis of the word right so that's why we are randomly cutting the prefixes and suffixes in such a way that we only get the important part of the word this is called stemming now this cutting of words can be successful in some occasions but not always that is why we say that stemming approach has a few limitations in order to get over these limitations we have a process known as Lammert ization read Lammert ization on the other hand takes into consideration the morphological analysis of the words it does not randomly cut the word in the beginning and the ending it understands what the word means and only then it cuts the word for example let's consider the word recap if we perform stemming on the word recap we'll get cap right and the output will be cap but cap and recap do not have the same meaning do they they have absolutely different meanings that's why stemming is sometimes not considered to be the right thing to do but when it comes to damnit ization it's going to understand the meaning of recap only then will it perform any sort of change in the word or it will cut down the word so basically it groups together different inflected forms of a word called lemma limit ization is similar to stemming because it Maps several words into one common root but the output of a limit ization process is always a proper word an example of limit ization is to map gone going and went in to go go on going went all of them mean go so basically by limit ization you can just output the words as go that is what limit ization is next we have something known as stop words right stop words are basically a set of commonly used words in any language right not just English any language the reason my stop words are critical to many applications is that if we remove the words that are very commonly used in a given language we can finally focus on the important words for example in the context of let's say you open up Google and you look for strawberry milkshake recipe instead of typing strawberry milkshake recipe let's say you type how to make strawberry milkshake now here what Google will do is it will find results for how to and make instead if you just type strawberry milkshake recipe you will get the most desired output that's why it's always considered a good practice in natural language processing to get rid of stop words right stop words will just increase your computation and it will just add additional work to us they are not very helpful when we are analyzing important documents that we need to focus on the important keywords in the documents instead of all of these commonly use words example of stop words include though how when why not yes no all of these are stop words right so in order to better analyze our data we need to get rid of stop words now the last terminology I'm going to discuss is document or matrix it is important to create something known as the document or matrix in natural language processing or DTM or a document or matrix is basically a matrix that shows the frequency of words in a particular document let's say that we are trying to understand if the sentence this is fun is available in one of my documents so if it is there in my document one I'm going to put a one corresponding to each of the words that is available in my document for example in document - I have this is but I do not have the word fun similarly in document four I have the word this but I do not have the word is and fun so basically your document or matrix is like the frequency matrix of a document so during text analysis you always begin by building or document or matrix right here you try to understand which words frequently occur and which words are important and not important in the dot so guys these were a couple of terminologies in natural language processing and with this we come to the end of today's session right we discuss a couple of topics here we discuss machine learning deep learning natural language processing we also saw the whales Python packages that are needed in order to understand artificial intelligence with Python so if any of you have doubts regarding today's session please let me know in the comment section apart from that I'll be leaving all the important links in the description box so guys that's all for today's session I hope all of you enjoyed it and make sure you subscribe to our channel to stay updated about recent technologies until next time happy learning I hope you have enjoyed listening to this video please be kind enough to like it and you can comment any of your doubts and queries and we will reply them at the earliest do look out for more videos in our playlist and subscribe to any rekha channel to learn more happy learning

Channel: edureka!

Views: 272,796

Keywords: yt:cc=on, artificial intelligence with python, artificial intelligence tutorial, AI Python, artificial intelligence, artificial intelligence python, artificial intelligence using python, Artificial Intelligence Tutorial using Python, applications of artificial intelligence, Introduction to Artificial Intelligence, What is Artificial Intelligence, Machine learning basics, edureka, python edureka, ai python tutorial, ai python examples, artificial intelligence edureka

Id: 7O60HOZRLng

Length: 102min 23sec (6143 seconds)

Published: Fri Jul 26 2019