🔥 Natural Language Processing In AI 2023 | Natural Language Processing Course 2023 | Simplilearn

hello everyone and welcome to this fantastic natural language processing full course by simply learn before we begin with the video if you enjoy watching these kind of videos and find them interesting then subscribe to our YouTube channel because we bring the best videos for you daily also hit the Bell icon to never miss any updates from Simply learn so without any further delay let's go through the agenda for this video we'll brief you with a detailed introduction to natural language processing after that we'll see the NLP engineer road map followed by the salary of an NLP engineer we'll walk you through some fantastic Concepts like AI versus machine learning versus deep learning and after that we'll see text mining in NLP following that we'll cover topics like what is generative Ai and large language models in artificial intelligence after covering all these topics we'll see algorithms and projects like sentiment analysis and email spam classifiers speaking of interviews we have covered you along with the most frequently Asked interview questions to help you crack the toughest interviews but before we move on to what NLP is if you are an aspiring Ai and machine learning engineer then there's no better time to train yourself in the exciting field of artificial intelligence if you're looking for a course that covers everything from the fundamentals to Advanced Techniques then accelerate your career in Ai and machine learning with a comprehensive post-graduate program in Ai and machine learning boost your career with this Ai and ml course delivered in collaboration with the puru university and IBM learn in demand skills such as machine learning deep learning NLP computer vision reinforcement learning generative AI prompt engineering chat GPT and many more you'll receive a prestigious certificate and ask me anything sessions by IBM with five capstones in different domains using real data sets you will gain practical experience master classes by PD faculty and IBM experts ensure topnotch education simply learns job assist help help you get noticed by Leading companies this program covers statistics python supervised and unsupervised learning NLP neural networks computer vision G scaras tensorflow and many more search skills admission to this postgraduate program in Ai and machine learning requires a bachelor's degree with an average of 50% or higher marks along with the basic understanding of programming Concepts and Mathematics and candidates with two plus years of work experience are preferred to enroll in this school so roll now and unlock exciting Ai and machine learning opportunities the link is mentioned in the description box below so without any further Ado let's send over this session to a training expert hey everyone in this video we'll be talking about NLP in the next 10 minutes so what can you expect from this video what's in it for you first we'll give you a basic understanding of NLP we'll tell you why we need NLP and we'll talk about what NLP exactly is next we'll look at NLP pipeline L NLP pipelines cover the basic processes involved in text passing in NLP which means converting text to a form suitable for machines to understand as humans would understand it and finally we'll talk about some real world applications of NLP which are all around us and a lot more common than you might have realized so let's get started so what exactly is NLP before we talk about NLP let's talk about how humans talk and interact with each other humans communicate with each other mostly via speech or text to convey your intent to someone you may directly talk to them or send them a message the human way of communicating is known as natural language around the globe many languages are spoken and messages are shared virtually every single day all these conversations feedbacks and messages are data in themselves this data is extremely valuable as it can give us customer information and insight into human sentiment however this data is not useful to computers as it is not in a form that can be understood by machines machines communicate using ones and zeros and not via words they cannot understand English French or Spanish only binary and this is where NLP comes into the picture NLP stands for natural language processing natural language processing is a branch of artificial intelligence that deals with the interactions between humans and computers using the natural language the ultimate objective of NLP is to read decipher understand and make sense of human languages in a manner that is valuable and to build systems that can make sense of text and perform tasks like translation grammar checking or topic classification NLP basically is the task of processing written forms of languages and making a computer understand them companies are increasingly using NLP equipped tools to gain insights from data and to automate routine tasks a sentiment analyzer for instance can help Brands detect emotions in text such as negative comments on social media NLP at the end of the day is nothing more than a cumulation of artificial intelligence computer science and the human language the next topic that we'll be looking at are NLP pipelines so what exactly are Pipelines a pipeline is a set of data processing Elements which are connected in series where the output of one element is the input of the next one it basically represents the various steps which have to be taken in our computation and the order in which they occur in this pipeline we are going to be parsing our data parsing means converting our data from one form to another in this case we are passing our input Text data to a file which can be understood by via computer the text is first broken into segments and the segments are then converted into even smaller tokens the word stems of all of these tokens are found followed by finding the root words of these stems in a process known as lemmatization we will then recognize which part of speech the word belongs to which means if the word is a verb a noun or a pronoun and finally we will separate the instances of everyday popular entities from our words to better understand what the sentence is trying to convey the final file that we'll get will be a file which will be understood by a computer segmentation the first process of a pipeline is segmentation now what exactly is segmentation sentence segmentation or text segmentation is basically dividing the given text into logically decipherable units of information we divide this sentence into its constituent Subs sentences usually along the punctuations like full stops or commas or along line breaks and Page components for HTML files dividing a document into its constituent sentences allows us to process it without losing its Essence and the necessary information that it contains in this case let's consider a sentence the lemonade quenched her thirst but not her longing after sentence segmentation we are going to divide this sentence into two the first sentence is going to be the lemonade quenched her thirst and the second one is going to be but not her longing the next step of our pipeline is called tokenization tokenization is nothing but the process of dividing a sentence into its constituent words the sentence that is given to us will be separated and all the words in the sentence will be stored separately this is done so we can understand the syntax IC and semantic information contained in each sentence thus we decipher the relevance of a sentence by analyzing it word by word thereby making sure that no loss of information occurs the computer does not understand punctuations and special characters hence we can remove any punctuations and special characters which may occur let's take a part of our previously segmented sentence over here let's consider the lemonade quenched her thirst after toen ization we are going to separate every single word in this sentence so after tokenization we are going to get five different tokens the lemonade quenched her and thirst all of these are going to be treated as separate entities after tokenization we perform stemming stemming is a process of obtaining the word stems of a word word stems are also known as the base form of a word and we can create new words by attaching a fixes to them in a process known as inflection stemming is a process of recognizing the word stems of individual words this is done by removing affixes such as in s ed etc for example consider our sentence jump jump is the word stem of various different words like jumping jumped and jumps if we remove all of these affixes we will get our basic word stem which is Jump this is basically what we want at the end of stemming the next process in our pipeline is called lemmatization lemmatization is the process of figuring out the root form or root word which is nothing but the most basic form also known as the Lemma of each word in the sentence lemmatization is very similar to stemming where we remove word affixes to get the base form of a word the difference is that the root word is always a word which is present in the dictionary but the root stem may not be so lemmatization uses a knowledge Base called word net let's consider three different words went going and gone at the end of the day all of these words have originated from a single word which is go in this case go is our Lemma all the other words which are derived from go can be traced back to it the next part of a pipeline is called part of speech tagging part of speech tagging is a process of converting a sentence to different forms it can be a list of words or a list of tuples the tag in case of is a part of speech tag and signifies whether the word is a noun adjective verb and so on we are basically splitting our verbs into the grammatical components to understand the meaning of any sentence or to extract relationships and to build a Knowledge Graph part of speech tagging is a very important step as the same word can have different part of speeches in different sent sentences for example let's consider the sentence give me your answer in this sentence answer is a noun but if we consider another sentence answer the question over here answer will be a verb using part of speech tagging we can take our different tokens and find the different part of speech that it belongs to in this case the is a determiner lemonade is a noun quenched is a verb her is a pronoun and thirst is a noun the final step in our NLP pipeline that we are looking at here is nothing but named entity recognition named entity recognition also known as named entity identification entity chunking and entity extraction is a subtask of information extraction that seeks to locate and classify named entities which are mentioned in unstructured text into predefined categories extracting the main entities in a text helps us sort unstructured data and detect important information which is crucial if you have to deal with large data sets the subcategories that we are considering are person whether the named entity is a person whether it's a quantity like kilogram a location an organization the name of a movie or whether it's a monetary value like dollars or Euros so far we've looked at what NLP is and how we can perform natural language processing but what are some applications of it and where is it used in the real world one of the applications of NLP is in chatbots chat Bots can help you solve issues while performing natural language generation in other words they can hold a conversation in plain English a chatbot is nothing but a software application which can be used to conduct an online chat conversation either through text or speech in place of providing direct contact with a life human agent you might have seen those talk to one of our agents section on websites those are usually chat Bots a lot of companies also use WhatsApp chat Bots to make the process seem less mechanical another application of NLP is speech recognition probably the most popular example of NLP in action are virtual assistants like Google assist Siri and Alexa natural language processing understands and translates the human language like Hey Siri where is the nearest gas station into numbers making it easy for machines to understand they recognize when you are talking converting speech to text and understand what you requested over the years virtual assistants have become streamlined enough to be able to emulate human speech patterns almost flawlessly another application of NLP is autoc correction autocorrection also known as text replacement replace as you type or simply autoc correct is an automatic data validation function commonly found in word processors and text editing interfaces for smartphones and tablet computers it acts as a spell check and corrects any spellings or grammar mistakes which may arise as you are typing some language check softwares like grammarly paper r reverso and others can even check how unique and engaging your articles are and all of this is done using NLP hello everyone welcome to this video on the natural language processing engineer NLP engineer road map by simply learn to stay updated with our content make sure to subscribe to our YouTube channel and hit the Bell icon so in this video we will Begin by examining the role of NLP engineer progressing further our Focus will shift toward the Essential Knowledge and Skills required to excel as an NLP engineer having comprehensively explored these subjects we will then proceed to explore the Ines and consing of a professional career road map tailored specifically for individual aspiring to become accomplished NLP Engineers on that note if you are an expiring AIML engineer then there is no better time to train yourself in the exciting field of machine learning if you are looking for course that covers everything from the fundamentals to Advanced Techniques like machine learning algorithm development and unsupervised learning look no further than our ctech program partnership with IBM this AIML boot camp in collaboration with keltech will help you advance your career as AIML specialist the AIML boot camp includes live classes delivered by industry expert handson L industry relevant projects and master classes by ctech Professor key features of this amazing machine learning course includes earn up to 22 cus from Kelch ctme online convocation by Kelch ctm program director simply career service helps you get noticed by top hiring companies on a boot camp certificate get from CeX ctm industry relevant project in three domains CeX ctm Circle membership 25 plus Hands-On project across industry verticals with integrated Labs so why wait join now seats are filling fast find the course link from the description box below so without any further to do let's get started so what is NLP engineer an assure language processing NLP engineer is a professional responsible for crafting and cening computer system that can process asss and produce humanik language NLP bres out into numerous captivating domains including semantic analysis sentiment analysis face recognition machine translation and speech recognition the reach of NLP extend across a diverse spectrum of application notably featuring virtual assistant chat boards sentiment analysis customer service interaction and language translation the task of NLP engineer involves ER acting these Innovative application and seemingly embedding them into the existing framework the the average salary for NLP engineer in India is 10 lakh per year and in the US average annual salary for NLP engineer is $120,000 so moving forward let's see skills required as an NLP engineer so becoming an NLP engineer entails acquisition of a diverse set of application and knowledge some of these encompasses first one is Proficiency in programming Mastery of at least one programming language such as python Java or C++ coupled with the solid graphs of data structure algor and principle of software engineering the second one is grasp of machine learning building a strong foundation in machine learning algorithm and compassing both supervised and unsupervised learning familiarity with Advanced Techniques like CNN and RNN is vital the third one is NLP techniques cultivating a deep understanding of NLP techniques which encompasses facets like text pre-processing centic and semantic analysis and the ingrate domain of sentiment analysis the fourth one is NLP libraries and Frameworks navigate easily through NLP libraries and Frameworks such as natural language toolkit nltk spy Stanford code NLP and genim harnessing their capabilities to streamline your NLP and davors the fifth one is data science acuma developing a sound gaps of data science principles and compassing facets such as data visualization statical analysis and the Art of data manipulation armed with these multiac skill set you will be primed to embark on a journey as an accomplished NLP engineer ready to navigate the complexity of natural languages processing and contribute to the exciting world of NLP Innovation moving forward let's see the professional NLP engineer road map here is a comprehensive professional trajectory for those aspiring to become NLP Engineers the first one is Educational Foundation embark on your journey by pursuing a degree in compitent fields such as computer science Linguistics or artificial intelligence many accomplish NLP Engineers hold Advanced degrees either a master or PhD in these domains the second one is internship or entry-level Venture considering immersing yourself in Practical experience through internship or entry-level job within the industry these opportunities not only equ with the Hands-On expertise but also Aid in constructing a robust portfolio the third one on the list is skills and knowledge acquisition develop a comprehensive grasp of essential proficiency request of NLP engineer this encompasses programming processes and machine learning and NLP techniques and data science the fourth one is portfolio development curate a diverse portfolio teaming with projects that illuminate your capabilities and expertise projects like suffocated chatboard a sentiment analysis tool or a language model can sustain your aptitude for solving real world NLP challenges and validate your skill set for potential employers the fifth one is networking endears engage in conference meetups and industries even to forge connection with fellow NLP professionals the sixth one is specialization consideration p under the prospect of specialization in specific NLP realm be it machine translation speech recognition or conversational agent this specialization not only differentiate you in the job market but also enables you to attain expertise within a particular need in summation the part toward becoming an NLP engineer entails a fusion of educational attainment hands-on experience and the Mastery of electric area of skills and knowledge domains hello everyone and welcome to this new video on NLP engineer salary with the world advancing towards a more informed and databased future learning skills that help us to be relevant in the job market is crucial one such skill in the field of data and AI is natural language processing let us start by understanding what NLP is natural language processing or NLP is a branch of artificial intelligence or AI that deals with understanding and manipulating human language it seeks to develop algorithms and systems that can understand and process both structured and unstructured language data to support decision making processes LLP is a field that has been gaining momentum in recent years thanks to advance in machine learning algorithms and the availability of large data sets such as social media conversations customer reviews and medical records NLP is used in various applications such as language translation sentiment analysis text summarization and text generation if you want to master NLP and become an expert NLP engineer then you should enroll an amazing Ai and machine learning program by CTIC ctme you will gain skills like machine learning deep learning reinforcement learning NLP generative AI prompt engineering chat GPT and more this Ai and machine learning boot camp includes live classes delivered by industry experts Hands-On Labs industry relevant projects and master classes by calch instructor s so hurry up what are you waiting for enroll the course now find the course Link in the description box so before we move ahead consider subscribing to Simply learn and hit the Bell icon to never miss any updates from us so let's move ahead by knowing what does NLP include at its score NLP involves a series of steps number one is pre-processing the first step in NLP is to prepare the language data for analysis this involves activities such as cleaning tokenizing ation and normalization of the text Data step two involves feature extraction the next step is to extract relevant features from the prepared data this involves identifying keywords stems and parts of speech tags that are important for the task at hand step three involves model training the extracted features are then used to train a machine learning model such as neural networks or a support Vector machine to predict the target variable and the last step is model evaluation once the model is trained it needs to be evaluated to ensure it performs well on new data now let's move ahead by knowing NLP job roles and responsibilities several job roles that involve NLP and each role is focused on different aspects of NLP here are some most common NLP job roles the number one is NLP engineer this role involves developing and implementing NLP Solutions including building machine learning models designing pipelines and developing apis NLP Engineers typically have a background in engineering or computer science and have knowledge of machine learning algorithms software development and data structures the next we have is text analytics specialist this role involves analyzing large amount of Text data to extract useful information text analytics specialist typically have a background in data analysis statistics and machine Lear learning and they use NLP tools and techniques to process and analyze text Data language model engineer this role involves developing and improving natural language processing models such as language models and machine translation models language model Engineers typically have a background in computer science machine learning and natural language processing on number four we have NLP researcher this role involves conducting Research into new NLP methods and techniques NLP researchers typically have a background in computer science machine learning and natural language processing and they publish their research in various academic journals and conferences and the most awaited let's start with salary of NLP Engineers let us now discuss the salary of NLP engineers in India and USA for the sake of better understanding we will split up the salary ranges based on the experience level of a professional so let's start with entry level NLP engineers in India the salary ranges from 2.5 million rupees to 5 million rupees however in us the salary ranges between $60,000 to $90,000 per year enter level NLP Engineers typically have background in computer science engineering or a related field and have some experience with NLP but may not have yet worked on large scale Enterprise projects moving towards midlevel NLP Engineers these Engineers earn 5 million to 8 million rupees per year however in United States they earn between $90,000 to $120,000 per year mid-level NLP Engineers have few years of experience working on NLP projects and are typically comfortable working on both Technical and business related aspects of NLP projects they may have some leadership responsibilities and are often responsible for ensuring that NLP projects are completed on time and to speec next we have a senior level NLP Engineers these Engineers earn up to 8 million rupees to 15 million rupees in India per year however in United States they earn between $120,000 to $165,000 per year senior level NLP Engineers have extensive experience working on Enterprise level NLP projects and are often considered authorities in NLP Technologies methods and tools they may serve as subject matter experts lead teams of engineers and other NLP professionals and be responsible for ensuring the long-term success of NLP projects human versus artificial intelligence humans are amazing let's just face it we're amazing creatures we're all over the planet we're exploring every nii and Nook we've gone to the Moon uh we've gone into outer space we're just amazing creatures we're able to use the available information to make decisions to communicate at with other people identify patterns and data remember what people have said adapt to new situations so let's take a look at this so so you can get a picture you're a human being so you know what it's like to be human let's take a look at artificial intelligence versus the human artificial intelligence develops computer systems that can accomplish taxs that require human intelligence so we're looking at this one of the things that computers can do is they can provide more accurate results this is very important recently did project on cancer where it's identifying markers and as a human being you look at that and you might be uh looking at all the different images and the data that comes off of them and say I like this person so I want to give them a very good um Outlook and the next person you might not like so you want to give him a bad Outlook well with artificial intelligence you're going to get a consistent prediction of what's going to come out interacts with humans using their natural language we've seen that as probably the big biggest development feature right now that's in the commercial Market that everybody gets to use as we saw with the example of Alexa they learn from their mistakes and adapt to new environments so we see this slowly coming in more and more and they learn from the data and automate repetitive learning repetitive learning has a lot to do with the neural networks you have to program thousands upon thousands of pictures in there and it's all automated so as today's computers evolved it's very quick and easy and affordable to do this what is machine learning and deep learning all about imagine this say you had some time to waste not that any of us really have a lot of time anymore to just waste in today's world and you're sitting by the road and you have a whole lot of and a whole lot of time passes by here's a few hours and suddenly you wonder how many cars buses trucks and so on passed by in the 6 hours now chances are you're not going to sit by the road for 6 hours and count buses cars and trucks unless you're working for the city and you're trying to do City Planning and you want to know hey do we need to add a new truck route maybe we need a Bicycle Link we have a lot of bicyclists here that kind of thing so maybe City Planning would be great for this machine learning well the way machine Learning Works is we have labeled data with features okay so you have a truck or a car a motorcycle a bus or a bicycle and each one of those are labeled it comes in and based on those labels and comparing those features it gives you an answer it's a bicycle it's a truck it's a motorcycle this look a little bit more in depth on this in the model here it actually the features we're looking at would be like the tires someone sits there and figures out what a tire looks like takes a lot of work if you try to try to figure the difference between a car tire a bicycle tire a motorcycle tire uh so in the machine learning field this could take a long time if you're going to do each individual aspect aspect of a car and try to get a result on there and that's what they did do that was a a very this is still used on smaller amounts of data where you figure out what those features are and then you label them deep learning so with deep learning one of our Solutions is to take a very large unlabeled data set and we put that into a training model using artificial neural networks and then that goes into the neural network itself when we create a neural network and you'll see um the arrows are actually kind of backward but uh which actually is a nice point because when we train the neural network we put the bicycle in and then it comes back and says if it said truck it comes back and says well you need to change that to bicycle and then it changes all those weights going backward they call it back propagation and let it know it's a bicycle and that's how it learns once you've trained the neural network you then put the new data in and they call this testing the model so you need to have some data you've kept off to the side where you know the answer to and you take that and you provide the required output and you say okay is this is this neural network working correctly did it identify a bike as a bike a truck as a truck a motorcycle as a motorcycle let's just take a little closer look at that determining what objects are present in the data so how does deep learning do this and here we have the image of the bike it's 28x 28 pixels that's a lot of information there um could you imagine trying to guess that this is a bicycle image by looking at each one of those pixels and trying to figure out what's around it uh and we actually do that as human beings it's pretty amazing we know what a bicycle is and even though it comes in has all this information and what this looks like is the image comes in it converts it into a bunch of different nodes in this case there's a lot more than what they show here and it goes through these different layers and out comes and says okay this is a bicycle a lot of times they call this the magic Black Box why because as we watch it go across here all these weights and all all the math behind this and it's not it's a little complicated on the math side you really don't need to know that when you're programming or doing working with the Deep learning but it's like magic you you don't know you really can't figure out what's going to come out by looking what's in each one of those dots and each one of those lines are firing and what's going in between them so we like to call it the magic box uh so that's where deep learning comes in and in the end it comes up and you have this whole neural notwork it comes up and it says okay we fire all these different pixels and we connects all these different dots and gives them different Waits and it says okay this is a bicycle and that's how we determine what the object is present in the data with deep learning machine learning we're going to take a step into machine learning here and you'll see how these fit together in a minute the system is able to make predictions or take decisions based on past data that's very important for machine learning is that we're looking at stuff and based on what's been there before we're creating a decision on there we're creating something out of there we're coloring a beach ball we're telling you what the weather is in Chicago what's nice about machine learning is a very powerful processing capability it's quick and accurate outcomes so you get results right away once you program the system the results are very fast and the decisions and predictions are better they're more accurate they're consistent you can analyze very large amounts of data some of these data things that they're analyzing now are pedabytes and terabytes of data it would take hundreds of people hundreds of years to go through some of this data and do the same thing that the machine learning can do in a very short period of time and it's inexpensive compared to hiring hundreds of people so it becomes a very affordable way to move into the future is to apply the machine learning to whatever businesses you're working on and deep Learning Systems think and learn like humans using artificial neural networks again it's like a magic box performance improves with more data so the the more data the Deep learning gets the more it gives you better results it's scalability so you can scale it up you can scale it down you can increase what you're looking at currently you know we're limited by the amount of computer processing power as to how big that can get but that envelope continually gets pushed every day on what it can do problem solved in an end to end method so instead of having to break it apart and you have the first piece coming in and you identify tires and the second piece is identifying uh labeling handlebars and then you bring that together that if it has handlebars and tires it's a bicycle and if it has something that looks like a large Square it's probably a truck the neural networks does this all in one network you don't really know what's going on in all those weights and all those little bubbles uh but it does it pretty much in one package that's why the neural network systems are so big nowadays and coming into their own best features are selected by the system and it this is important they kind of put it as it's on a bullet on the side here it's a subset of machine learning this is important when we talk about deep learning it is a form of machine learning there's lots of other forms of machine learning data analysis but this is the newest and biggest thing that they apply to a lot of different packages and they use all the other machine learning tools available to work with it and it's very fast to test um you put in your information you then have your group of uh test and then you held some aside you see how does it do it's very quick to test it and see what's going on with your deep learning and your neural network are they really all that different AI versus machine learning versus deep learning concepts of AI so we have a concepts of II you'll see natural language processing uh machine learning an approach to create artificial intelligence so it's one of the subsets of artificial intelligence knowledge representation automated reasoning computer vision robotics machine learning versus AI versus deep learning or Ai and machine learning and deep learning so when we look at this we have ai with machine learning and deep learning and so we're going to put them all together we find out that AI is a big picture we have a collection of books that goes through some deep learning the Digital Data is analyzed text mining comes through the particular book you're looking for maybe it's a genre books is identified and in this case uh we have a robot that goes and gives a book to the patron I have yet to be at a library that has a robot bring me a book but that will be cool when it happens uh so we look at some of the pieces here this information goes into uh as far as this example the translation of the handwritten printed data to digital form that's pretty hard to do that's pretty hard to go in there and translate hundreds and hundreds of books and understand what they're trying to say if you've never read them so in this case we use the Deep learning because you can already use examples where they've already classified a lot of books and then they can compare those texts and say oh okay this is a book on automotive repair this is a book on robotic building the Digital Data is in analyzed then we have more text mining using machine learning so maybe we'd use a different program to do a basic classify uh what you're looking for and say oh you're looking for auto repair and computers so you're looking for automated cars once it's identified then of course it brings you the book so here's a nice summation of what we were just talking about AI with machine learning and deep learning deep learning is a subset of machine learning which is a subset of artificial intelligence so you can look at artificial intelligence as a big picture how does this compare to The Human Experience in either uh doing the same thing as a human we do or it does it better than us and machine learning which has a lot of tools uh is something that learns from data past experiences it's programmed it's uh comes in there and it says hey we already had these five things happen the sixth one should be about the same and then uh then there's a lot of tools in machine learning but deep learning then is a very specific tool in machine learning it's the artificial neural network which handles large amounts of data and is able to take huge pools of experiences pictures and ideas and bring them together real life examples artificial intelligence news generation very common nowadays as it goes through there and finds the news articles or generates the news based upon the news feeds or the uh backend coming in and says okay let's give you the actual news based on this there's all the different things Amazon Echo they have a number of different Prime music on there of course there's also the Google command and there's also Cortana there's tons of smart home devices now where we can ask it to turn the TV on or play music for us that's all artificial intelligence from front to back you're having a human experience with these computers and these objects that are connected to the processing machine learning uh spam detection very common machine learning doesn't really have the human interaction part so this is the part where it goes and says okay that's a Spam that's not a Spam and it puts it in your spam folder search engine result refining uh another example of machine learning whereas it looks at your different results and it Go and it uh is able to categorize them as far as this had the most hits this is the least viewed this has five stars um you know however they want to wait it uh all exam good examples of machine learning and then the Deep learning uh deep learning another example is you know because you have like a exit sign in this case is translating it into French sorti I hope I said that right um neural network has been programmed with all these different words and images and so it's able to look at the exit in the middle and it goes okay we want to know what that is in French and it's able to push that out in French French and learn how to do that and then we have chatbots um I remember when Microsoft first had their little paperclip um boy that was like a long long time ago that came up and you would type in there and chat with it these are growing you know it's nice to just be able to ask a question and it comes up and gives you the answer and instead of it being were you're just doing a search on certain words it's now able to start linking those words together and form a sentence in that chat box types of AI and machine learning types of artificial intelligence this in the next few slides are really important so one of the types of artificial intelligence is is reactive machines systems that only react they don't form memories they don't have past experiences they have something that happens to them and they react to it my washing machine is one of those if I put a ton of clothes in it and they had all clumped on one side it automatically adds a weight to reitter it so that my washing machine is actually a reactive machine working with whatever the load is and keeps it nice and so when it spins it doesn't go thumping against the side limited memory another form of artificial intelligence systems look into the past information is added over a period of time and information is shortlived when we're talking about this and you look at like a neural network that's been programmed to identify cars it doesn't remember all those pictures it has no memory as far as the hundreds of pictures you process through it all it has is this is the pattern I use to identify cars as the final output for that neural network we looked at so when they talk about limited memory this is what they're talking about they're talking about I've created this based on all these things but I'm not going to remember any one specifically theory of Mind systems being able to understand human emotions and how they affect decision-making to adjust their behaviors according to their human understanding this is important because this is our page mark this is how we know whether it is an artificial intelligence or not is it interacting with humans in a way that we can understand uh without that interaction is just an object uh so we talk about theory of mind we really understand how it interfaces that whole if you're in web development user experience would be the term I would put in there so a theory of mind would be user experience how is the whole UI connected together and one of the final things is as we get into artificial intelligence is systems being aware of themselves understanding their internal States and predicting other people's feelings and act appropriately so as artificial intelligence continues to progress uh we see ones are trying to understand well what makes people happy how would they increase our happiness uh how would they keep themselves from breaking down if something's broken inside they have that self-awareness to be able to fix it and just based on all that information predicting which action would work the best what would help people uh if I know that you're having a cup of coffee first thing in the morning is what makes you happy as a robot I might make you a cup of coffee every morning at the same time uh to help your life and help you grow that'd be the self-awareness is being able to know all those different things types of machine learning and like I said on the last slide this is very important this is very important if you decide to go in and get certified in machine learning or know more about it these are the three primary types of machine learning the first one is supervised learning systems are able to predict future outcome based on past data requires both an input and an output to be given to the model for for it to be trained so in this case we're looking at anything where you have 100 images of a bicycle and those 100 images you know are bicycle so it's they're preset someone already looked at all 100 images and said these are pictures of bicycles and so the computer learns from those and then it's given another picture and maybe the next picture is a bicycle and it says oh that resembles all these other bicycles so it's a bicycle and the next one's a car and it says it's not a bicycle that would be super vised learning CU we had to train it we had to supervise it unsupervised learning systems are able to identify hidden patterns from the input data provided by making the data more readable and organized the patterns similarities or anomalies become more evident uh you'll heard the term cluster how do you cluster things together some of these things go together some of these don't this is unsupervised where can look at an image and start pulling the different pieces of the image out because they aren't the same the human all the parts of the human are not the same as a fuzzy tree behind them CU it's slightly out of focus which is not the same as the beach ball it's unsupervised because we never told it what a beach ball was we never told it what the human was and we never told it that those were trees all we told it was hey separate this picture by things that don't match and things that do match and come together and finally there's reinforcement learning systems are given no training it learns on the basis of the reward punishment it received for performing its Last Action it helps increase the efficiency of a tool function or a program reinforced learning or reinforcement learning is kind of you give it a yes or no yes you gave me the right response no you didn't and then it looks at that and says oh okay so based on this data coming in uh what I gave you was a wrong response so next time I'll give you a different one comparing machine learning and deep learning so remember that deep learning is a subcategory of machine learning so it's one of the many tools and so they we're grouping a ton of machine learning tools all together linear regression K means clustering there's all kinds of cool tools out there you can use in machine learning enables machines to take decisions to make decisions on their own based on past data enables machines to make decisions with the help of artificial neural networks so it's doing the same thing but we're using an artificial neural network as opposed to one of the more traditional machine learning tools needs only a small amount of training data this is very important when you're talking about machine learning they're usually not talking about huge amounts of data we're talking about maybe your spreadsheet from your business and your totals for the end of the year when you're talking about neural networks you usually need a large amount of data to train the data so there's a lot of training involved if you have under 500 points of data that's probably not going to go into machine learning or maybe you have like the case of one of the things 500 points of data and 30 different fields it starts getting really confusing there in artificial intelligence or machine learning and the Deep learning aspect really shines when you get to that larger data that's really complex works well on a low-end systems so a lot of the machine learning tools out there you can run on your laptop with no problem and do the calculations there where with the machine learning usually needs a higher-end system to work it takes a lot more processing power to build those neural networks and to train them it goes through a lot of data when we're talking about the general machine learning tools most features need to be identified in advanced and manually coded so there's a lot of human work on here the machine learns the features from the data it is provided so again it's like a magic box you don't have to know what a tire is it figures it out for you the problem is divided into parts and solved individually and then combined so machine learning you usually have all these different tools and use different tools for different parts and the problem is solved in an endtoend manner so you only have one neural network or two neural networks that is bringing the data in and putting it out it's not going through a lot of different processes to get there and remember you can put machine learning in deep learning together so you don't always have just the Deep learning solving the problem you might have solving one piece of the puzzle with regular machine learning and most machine learning tools out there they they take longer to test and understand how they work and with the Deep learning it's pretty quick once you build that neural network you test it and you know so we're dealing with very crisp rules Limited resources you have to really explain how the decision was made when you use most machine learning tools but when you use the Deep learning tool inside the machine learning tools the system takes care of it based on its own logic and reasoning and again it's like a magic Black Box you really don't know how it came up with the answer you just know it came up with the right answer a glimpse into the future so a quick glimpse into the future artificial intelligence be using it to detecting crimes before they happen humanoid AI helpers which we already have a lot of there'll be more and more maybe it'll actually be Androids that'd be cool to have an Android that comes and gets stuff out of my fridge for me machine learning increasing efficiency in health care that's really big in all the forms of machine learning better marketing techniques any of these things if we get into the Sciences it's just off the scale machine learning and artificial intelligence go everywhere and then the subcategory Deep learning increased personalization so what's really nice about the Deep learning is going to start now catering to you that'll be one of the things we see more and more of and we'll have more of a hyper intelligent personal assistant I'm excited about that in this lesson you are going to understand the concept of text mining by the end of this lesson you will be able to explain text mining execute text processing task so let's go ahead and understand text mining in detail let's first understand what what text mining is text mining is the technique of exploring large amounts of unstructured Text data and analyzing it in order to extract patterns from the text Data it is aided by software that can identify Concepts patterns topics keywords and other attributes in the data it utilizes computational techniques to extract and summarize the highquality information from unstructured textual resources let's understand the flow of text mind there are five techniques used in text mining system information extraction or text pre-processing this is used to examine the unstructured text by searching out the important words and finding the relationships between them categorization or text transformation attribute generation categorization technique labels the text document under one or more categories classification of Text data is done based on input put output examples with categorization clustering or attribute selection clustering method is used to group text documents that have similar content clusters are the partitions and each cluster will have a number of documents with similar content clustering makes sure that no document will be omitted from the search and it derives all the documents that have similar content visualization technique the process of finding relevant information is simplified by visualization technique this technique uses text Flags to represent a group of documents or a single document and compactness is indicated using colors visualization technique helps to display textual information in a more attractive way summarization or interpretation or evaluation summarization technique will help to reduce the length of the document and summarize the details of the documents it makes the document easy to read for users and understand the content at the moment let's understand the significance of text mining document clustering document clustering is an important part of text mining it has many applications in Knowledge Management and information retrieval clustering makes it easy to group similar documents into meaningful groups such as in newspapers where sections are often grouped as business Sports politics and so on pattern identification text mining is the process of automatically searching large amount of text for text patterns and recognition of features features such as telephone numbers and email addresses can be extracted using pattern matches product insights text mining helps to extract large amounts of text for example customer reviews about the products mining consumer reviews can reveal insights like most loved feature most hated feature improve improvements required and reviews of competitor products security monitoring text mining helps in monitoring and extracting information from news articles and reports for national security purposes text mining makes sure to use all of your available information it is a more effective and productive knowledge discovery that allows you to make better informed decisions automate information intensive processes gather business critical insights and mitigate operation risk let's look at the applications of text mining speech recognition speech recognition is the recognition and translation of spoken language into text and vice versa speech often provides valuable information about the topics subjects and concepts of multimedia content information extraction from speech is less complicated yet more accurate and precise than multimedia content this fact motivates content based speech analysis for multimedia Data Mining and retrieval where audio and speech processing is a key enabling technology spam filtering spam detection is an important method in which textual information contained in an email is extracted and used for discrimination text mining is useful in automatic detection of spam emails based on the filtering content using text mining an email service provider such as Gmail or Yahoo mail checks the content of an email and if some malicious text is found in the mail then that email is marked as spam and sent to the spam folder sentiment analysis it is done in order to determine if a given sentence expresses positive neutral or negative sentiments sentiment analysis is one of the most popular applications of text analytics the primary aspect of sentiment analysis includes data analysis of the body of the text for understanding the opinion expressed by it and other key factors comprising modality and mood usually the process of sentiment analysis works best on text that has a subjective context than on that with only in objective context e-commerce personalization text mining is used to suggest products that fit into a user's profile text mining is increasingly being used by e-commerce re retailers to learn more about the consumers as it is the process of analyzing textual information in order to identify patterns and gain insights e-commerce retailers can Target specific individuals or segments with personalized offers and a discounts to boost sales and increase Customer Loyalty by identifying customer purchase patterns and opinions on particular products let's look at natural language toolkit library in detail natural language toolkit is a set of open-source python models that are used to apply statistical natural language processing on human language data let's see how you can do environment setup of nltk go to Windows start and launch python interpreter from Anaconda prompt and enter the following commands enter command python to check the version of python installed on your system enter import nltk to link you to the nltk library available to download then enter nltk doownload function that will open the nltk download window check the download directory select all packages and click on download this will download nltk onto your python once you have downloaded the nltk you must check the working and functionality of it in order to test the setup enter the following command in Python idle from nltk Doc Corpus import Brown brown. word parenthesis parenthesis the brown is an nltk Corpus that shows the systematic difference between different genres available words function will give you the list available words in the genre the given output shows that we have successfully tested the nltk installed on python let's Now understand how you can read a specific module from nltk corpora if you want to import an entire module from nltk corpora use asterisk symbol with that module name import command enter the command from nltk do book import asterisk it will load all the items available in NLT K's book module now in order to explore Brown Corpus enter the command nltk do Corpus import Brown this will import Brown Corpus on the python enter brown. categories function to load the different genres available select a genre and assign that genre to a variable using the following syntax variable name is equal to brown. wordss categories is equal to genre name now in order to see the available words inside the selected genre just enter the defined variable name as a command let's understand text extraction and pre-processing in detail so let's first understand the concept of tokenization tokenization is the process of removing sensitive data and placing unique symbols of identification in that place in order to retain all the essential information concerned with the data by its security it is a process of breaking running streams of text into words and sentences it works by segregating words using punctuation and spaces text extraction and pre-processing engrams now let's look at what engram is and how it is helpful in text mining engram is the simplest model that assigns these probabilities to sequences of words or sentences engrams are combinations of adjacent words or letters of length and in the source text so engram is very helpful in text mining when it is required to extract patterns from the text as in the given example this is a sentence all of these words are considered individual words and thus represent unigrams a two gr or Byram is a two-word sequence of words like this is is a or a sentence and a thre or trigram is a three-word sequence of words like this is a or is a sentence let's Now understand what stop words are and how you can remove them stop words are natural language words that have negligable meanings such as a n and or the and other similar words these words also will take up space in the database or increase the processing time so it is better to remove such words by storing a list of stop words you can find the list of stop words in the nltk data directory that is stored in 16 different languages use the following command to list the stop words of English language defined in nltk Corpus importing nltk will import the nltk Corpus for that instance enter from nltk Doc Corpus import stopwords will import Stop wordss from nltk Corpus Now set the language as English so use set function as set under braces stop wordswords set genre as English stop words are filtered out before processing of natural language data as they don't reveal much information so as you can see in the given example before filtering the sentence the tokenization of stop word is processed in order to remove these stop words and the filtering is applied in order to filter the sentence based on some criteria text extraction and pre-processing stemming stemming is used to reduce a word to stem or base word by removing suffixes such as helps helping helped and helper to the root word help the stemming process or algorithm is generally called a stemmer there are various stemming algorithms such as Porter stemmer Lancaster stemmer snowball stemmer Etc use any of the stemmers defined under nltk stem Corpus in order to perform stemming as shown in the example here we have used port stemmer When You observe the output you will see that all of the words given have been reduced to their root word or stem text extraction and pre-processing lemmatization lemmatization is the method of grouping the various inflected types of a word in order that they can be analyzed as one item it uses vocabulary list or a morphological analysis to get the root word it uses word net database that has English words linked together by their semantic relationship as you can observe the given example the different words have been extracted to their relevant morphological word using lemmatization text extraction and preprocessing POS tagging let's now look at different part of speech tags available in the national language toolkit Library a POS tag is a special label assigned to each token or word in a Text corpus to indicate the part of speech and often also other grammatical categories such as tense number either plural or singular case Etc POS tags are used in text analysis tools and algorithms and also in Corpus searches so look at the given example here Alice wrote a program is the source text given the POS tags given our Alice is a noun wrote is a verb a is an article and program is an adjective look at the given example to understand how POS tags are defined so the given sentence or paragraph contains different words that represent different parts of speech we will first use tokenization and removal of stop words and then allocate the different POS tags these are shown with different words in the given sentence POS tags are useful for lemmatization in building named entity recognition and extracting relationships between words text extraction and pre-processing named entity recognition now let's understand what named entity recognition is all about Neer seeks to extract a real world entity from the text and sorts it into predefined categories such as names of people organizations locations Etc many real world questions can be answered with the help of name entity recognition where specified products mentioned in complaints or reviews does the Tweet contain the name of a person does the Tweet contain the person's address as you can see in the given example Google America Larry Page Etc are the names of a person place or an organization so these are considered named entities and have different tags such as person organization gpe or geopolitical entity Etc NLP process workflow now you have an understanding of all nltk tools so now let's understand the natural language processing workflow step one tokenization it splits text into pieces tokens or words and removes punctuation step two stop word removal it removes commonly used words such as the is R Etc which are not relevant to the analysis step three stemming and litiz it reduces words to base for in order to be analyzed as a single item step four POS tagging it tags words to be part of speech such as noun verb adjective Etc based on the definition and context step five information retrieval it extracts relevant information from the source1 brown Corpus problem statement the Brown University standard Corpus of present-day American English also known popularly as Brown Corpus was compiled in the 1960s as a general Corpus in the field of Corpus Linguistics it contains 500 samples of English language text totaling roughly 1 million words compiled from Works published in the United States in 1961 we will be working on one of the subset data set and perform text processing tasks let us import the nltk library and read the ca10 Corpus import nltk we will have to make sure that there are no slashes in between hence we will use the replace function within pandas for the same let's have a look at the data once tokenization after performing sentence tokenization on the data we obtain similarly after applying sentence tokenizer the resulting output shows all individual words tokens stop word removal let's import the stop word library from nltk Doc Corpus import stopwords we also need to ensure that the text is in the same case nltk has its own list of stop wordss we can check the list of stop wordss using stopwords do wordss and English inside the parenthesis map the lowercase string with our list of word tokens let's remove the stop words using the English stop wordss list in nltk we will be using set checking as it is faster in Python than a list by removing all stop wordss from the text we obtain often we want to remove the punctuations from the documents too since python comes with batteries included we have string. punctuation from string import punctuation combining a punctuation with the stop wordss from nltk removing stop words with punctuation steming and lemmatization we will be using stemming and lemmatization to reduce words to their root form for example walks walking walked will be reduced to their root word walk importing Porter stemmer as the stemming library from nltk do stem import Porter stemmer printing the stem words import the word net litier from nltk do stem printing the root words we also need to evaluate the POS tags for each token create a new word list and store the list of word tokens against each of the sentence tokens in data 2 for I and tokenized also we will check if there were any stop words in the recently created word list we will now tag the word tokens accordingly using the POS tags and and print the tagged output for our final text processing task we will be applying named entity recognition to classify named entities in text into predefined categories such as the names of persons organizations locations expressions of times quantities monetary values percentages Etc now press the tagged sentences under the chunk parser if we set the parameter binary equals true then named entities are just tagged as NE otherwise the classifier adds category labels such as person organization and gpe create a a function named as extract entity names along with an empty list named as entity names we will now extract named entities from a nltk chunked expression and store them in the empty created above again we will set the entity names list as an empty list and we'll extract The Entity names by iterating over each tree in chunked sentences great we have seen how to explore and examine the Corpus using text processing techniques let's quickly recap the steps we've covered so far One Import the nltk library two perform tokenization three perform stemming and lemmatization four remove stop words five perform named entity recognition structuring sentences syntax let's first understand what syntax is syntax is the grammatical structure of sentences in the given example this can be interpreted as syntax and it is similar to the ones you use while writing codes knowing a language includes the power to construct phrases and sentences out of morphemes and words the part of the grammar that represents a speaker's knowledge of these structures and their formation is called syntax phrase structure rules are rules that determine what goes into a phrase phase that is constituents of a phrase and how the constituents are ordered constituent is a word or group of words that operate as a unit and can be used to frame larger grammatical units the given diagram represents that a noun phrase is determined when a noun is combined with a determiner and the determiner can be optional a sentence is determined when a noun phrase is combined with a verb phrase a verb phrase is determined when a verb is combined optionally with the noun phrase phrase and prepositional phrase and a prepositional phrase is determined when a preposition is combined with a noun phrase a tree is a representation of syntactics structure of formulation of sentences or strings consider the given sentence the factory employs 12.8% of Bradford County what can be the Syntax for pairing the statement let's understand this a tree is produced that might help you understand that the subject of the sentence is the factory the predicate is employes and the target is 12.8% which in turn is modified by Bradford County syntax parses are often a first step toward deep information extraction or semantic understanding of text rendering syntax trees download the corresponding exe file to install the ghost script rendering engine based on your system configuration in order to render syntax trees in your notebook let's understand how you can set up the environment variable once you have downloaded and installed the file go to the folder where it is installed and copy the path of the file now go to system properties and under Advanced properties you will find the environment variable button click on that to open the popup box tab of the environment now open the bin folder and add the path to the bin folder in your environment variables now you will have to modify the path of the environment variable use the given code to test the working of syntax tree after the setup is successfully installed structuring sentences chunking and chunk parsing the process of extraction of phrases from unstructured text is called chunking instead of using just simple tokens which may not represent the actual meaning of the the text it is advisable to use phrases such as Indian team as a single word instead of Indian and team as separate words the chunking segmentation refers to identifying tokens and labeling refers to identifying the correct tag these chunks correspond to mixed patterns in some way to extract patterns from chunks we need chunk parsing the chunk parsing segment refers to identifying strings of tokens and labeling refers to identifying the correct chunk type let's look at the given example you can see here that yellow is an adjective dog is a noun and the' is the determiner which are chunked together into a noun phrase similarly chunk parsing is used to extract patterns and to process such patterns from multiple chunks while using different parsers let's take an example and try to understand how chunking is performed in Python let's consider the sentence the little mouse ate the fresh cheese assigned to a variable named scent using the word tokenize function under nltk corpora you can find out the different tags associated with the sentence provided so as you can see in the output different tags have been allocated against each of the words from the given sentence using chunking NP chunk and parser you will now create grammar from a noun phrase and will mention the tags you want in your chunk phrase within the function here you have created a regular expression matching the string the given regular expression indicates optional determiner followed by optional number of adjective followed by a noun you will now have to parse the chunk therefore you will create a chunk parser and pass your noun phrase string to it the parser is now ready you will use the parse parenthesis parenthesis within your chunk parser to parse your your sentence the sentence provided is the little mouse at the fresh cheese this sentence has been parsed and the tokens that match the regular expressions are chunked together into noun phrases NP create a verb phrase chunk using regular Expressions the regular expression has been defined as optional personal pronoun followed by zero or more verbs with any of its type followed by any type of adverb you'll now create another chunk parser and pass the verb phrase string to it create another sentence and tokenize it add POS tags to it so the new sentence is she is walking quickly to the mall and the POS tag has been allocated from nltk corpora now use the new verb phrase parser to parse the tokens and run the results you can look at the given tree diagram which shows a verb parser where a pronoun followed by two verbs and an adverb are chunk together into a verb parse structuring sentences chinking chinking is the process of removing a sequence of tokens from a chunk how does chunking work the whole chunk is removed when the sequence of tokens spans an entire chunk if the sequence is at the start or the end of the chunk the tokens are removed from the start and end and a smaller chunk is retained if the sequence of tokens appears in the middle of the chunk these tokens are removed leaving two chunks where there was only one before consider you create a chinking grammar string containing three things chunk name the regular expression sequence of a chunk the regular expression sequence of your here in the given code we have the chunk regular expression as optional personal pronoun followed by zero or more occurrences of any type of the verb type followed by zero or more occurrences es of any of the adverb types the regular expression says that it needs to check for the adverb in the extracted chunk and remove it from the chunk inside the chinking block with open curly braces and closing curly braces you have created one or more adverbs you will now create a parser from nltk do regexp parser and pass the grammar to it now use the new parser to parse the tokens sent three and run the results as you can see the parse tree is generated while comparing the syntax tree of the parser with that of the original chunk you can see that the token is quickly adverb chin out of the chunk let's understand how to use context free grammar a context free grammar is a four Tuple sum ntrs where sum is an alphabet and each character in sum is is called a terminal NT is a set and each element in NT is called a non-terminal r the set of rules is a subset of NT times the set of sum u n t s the start symbol is one of the symbols in NT a context free grammar generates a language L capturing constituency and ordering in CFG the start symbol is used to derive the string you can derive the string by repeatedly replacing a non-terminal on the right hand side of the production until all non-terminals have been replaced by terminal symbols let's understand the representation of context free grammar through an example in context free grammar a sentence can be represented as a noun phrase followed by a verb phrase noun phrase can be a determiner nominal a nominal can be a noun VP represents the verb phrase a can be called a determiner flight can be called a noun consider the string below where you have certain rules when you look at the given context free grammar a sentence should have a noun phrase followed by a verb phrase a verb phrase is a verb followed by a noun a verb can either be Saul or met noun phrases can either be John or Jim and a noun can either be a dog or a cat check the possible list of sentences that can be generated using the rules use the join function to create the Poss list of sentences you can check the different rules of grammar for sentence formation using the production function it will show you the different tags used and the defined context free grammar for the given sentence demo two structuring sentences problem statement a company wants to perform text analysis for one of its data sets you are provided with this data set named tweets. CSV which has tweets of six US Airlines along with their sentiments positive negative and neutral the tweets are present in the text column and sentiments in Airline sentiment column we will be retrieving all tags starting with at the rate in the data set and save the output in a file called references. txt let us first import the panda library and read the tweets data set extract the features text and Airline sentiment we will iterate through the data set using reg X find the relevant tweets now we will import the itertools module it returns efficient iterators the result is stored in a file named references. txt let's extract all noun phrases and save them in a file named noun phrases for left Carro Airline sentiment right Carro review. txt here left carrot Airline uncore sentiment right carrot has three different values positive negative and neutral so three files will be created now we will iterate all the leaf nodes and assign them to noun phrases variable this means that the functions in iter tools operate on iterators to produce more complex iterators using the map function we will get all the noun phrases from the text putting it into list creating a file name in the name of review. txt great we have now seen how to explore and examine the Corpus using text processing techniques if you're looking for a course that covers everything from the fundamentals to Advanced Techniques then accelerate your career in Ai and machine learning with a comprehensive post-graduate program in Ai and machine learning boost your career with this Ai and ml course delivered in collaboration with the perdu university and IBM learn in demand skills such as machine learning deep learning NLP computer vision reinforcement learning generative AI prompt engineering chat GPT and many more you'll receive a prestigious certificate and ask me anything sessions by IBM with five capstones in different domains using real data sets you will gain practical experience master classes by perd faculty and IBM experts ensure top-notch education simply learns job assist help you get noticed by Leading companies this program covers statistics python supervised and unsupervised learning NLP neural networks computer vision G scas tensorflow and many more suchar skills admission to this post-graduate program in Ai and machine learning requires a bachelor's degree with an average of 50% or higher marks along with the basic understanding of programming Concepts and meth mathematics and candidates with two plus years of work experience are preferred to enroll in this school so enroll now and unlock exciting Ai and machine learning opportunities the link is mentioned in the description box below the term generative AI has emersed seemingly out of nowhere in recent months with a notable search in interest according to Google Trends even within the past year the spike in curiosity can be attributed to the introduction of generative model such as d 2 B and chgb however what does generative AI entail as a part of our introductory series on generative AI this video will provide a comprehensive overview of a subject starting from the basics the explanation Will C to all levels of familiarity ensuring that viewers gain a better understanding of how this technology operates and its growing integration to our daily lives generative AI is after all a tool that is based on artificial intelligence a professional who elit seeds to switch careers with AI by learning from the experts then try giving a short simply L postgraduate program in Ai and machine learning from perd University in collaboration with IBM the link in the description box should navigate to the homepage where you can find a complete overview of the program being offered take action upskill and get ahead what is generative AI generative AI is a form of artificial intelligence possesses the capability of to generate a wide range of Conta including text visual audio and synthetic data the recent excitement surrounding generative AI stems from the userfriendly interfaces that allow us users to effortlessly create high quality text graphics and video within a seconds now moving forward let's see how does generative AI Works generative AI begin a prompt which can take form of text image video design audio musical notes or any input that AI system can process various AI algorithm that generate new content in response to the given prompt this content can range from essay and problem solution to realistic created using images or audio of a person in the early stages of generative AI utilizing the technology involved submitting data through an API or a complex process developers need to acquaints with a specialized tool and writing application using programming language like python some of the recent and fully operational generative AIS are Google Bart D open AI chgb Microsoft Bing and many more so now let's discuss chat GP Dal and B which are the most popular generative AI interfaces so first is DAL 2 which was developed using open as GPT implementation in 2021 exemplify a multimodel AI application it has been trained on a v data set of images and their corresponding textual description Dal is capable of establishing connection between various media forms such as Vision text audio it specifically links the meaning of words to visual elements open a introduced an enhanced version called d 2 in 2022 which empowers user to generate imagery in multiple Styles based on their prompts and the next one is chat gbt in November 2022 chat GPT and AI powered chatboard built on open AI GPD 3.5 implementation gained immense popularity worldwide open AI enable user to interact with and fine tune the chatbox text response through a chat interface with interactive feedback unlike earlier version of GPT that was solely accessible via an API charges GPT brought a more interactive experience on March 14 2023 open a released GPT 4 chat GPT integrated the conversational history with a user making a genuine dialogue Microsoft impressed by the success of new chgb interface announced a substantial investment in open Ai and integrated a version of GPT into its Bing search engine and the next one is Bard Google bard Google was also an earlier Fortuner in advancing Transformer AI technique for language processing protein analysis and other content types it made some of these model open source for researchers but were not made available through a public interface in response to Microsoft integration of GPT into beinging Google hardly launched a public facing chat B name Google b b debut was met by an error when the language model incorrectly claimed that the web telescope was the first to discover a planet in a foreign solar system as a consequences Google stock price suffered a significant decline meanwhile Microsoft implementation of chat GPT and GPT power system also face criticism for producing inaccurate result and displaying ER actic behavior in their early iritation so moving forward let's see what are the use cases of generative AI generative AI has broad applicability and can be employed across a wide range of use cases to generate diverse form of content recent advancement like GPT have made this technology more accessible and customizable for various applications some notable use cases for generative AI are as follows chatbot implementation generative AI can be utilized to develop chatbots for customer service and Technical Support enhancing interaction with users and providing efficient assistance the second one is language dubbing enhancement in the real in the realm of movies and educational accountant generative AI can contribute to improving dubbing in different languages ensuring accurate and high quality translation and the third one is content writing generative AI can assist in writing email responses dating profiles resumes and term papers offering valuable support and generating customized content tailor to specific requirement and the fourth one is Art generation leveraging generative AI artists can create photo realistic artwork in various Styles enabling the exploration of new artistic expression and enhancing creativity the fifth one is product demonstration videos generative AI can hun to enhance protect demonstration video making them more engaging visually appealing and effective in showcasing product features and benefits so generative AI versatility allow it to employ it in many other application making it a valuable tool for Content creation and enhancing user experience across diverse domains so after seeing use case of generative AI let's see what are the benefits of generative AI so generative AI offers extensive application across various business domains simplifying the interpretation and comprehension of existing content while also enabling the automated creation of a new content content developers are actively exploring ways to leverage generative AI in order to enhance the optimize existing workflows and even to reshave workflows entirely to harness the potential of Technology fully implementing generative AI can bring numerous benefits including automated content creation generative AI can automate the manual process of writing content saving time and effort by generating text or other form of content the next one is efficient email response responding to emails can be made more efficient with generative AI reducing the effort required and improving response time and the third one is enhanced technical support generative AI can improve responses to specific technical queries providing accurate and helpful information to users or customers and the fourth one is realistic person Generation by leveraging generative AI it becomes possible to create realistic representation of people enabling applications like virtual characters or Avatar and the fifth one is coherent information summarization generative AI can summarize complex information into a coherent narrative distilling key points and making it easier to understand and communicate complex concept the implementation of generative AI offers a range of potential benefits steamingly processed and enhancing content Creation in various areas of business operation so after seeing advantages of generative AI let's move forward and see what are the limitations of generative AI early implementation of generative AI serve as Vivid examples highlighting the numerous limitation associated with this technology several challenges arise from the specific approaches employed to implement various use gifts for instance while a summary of a complex topic May more reader friendly than explanation incorporating multiple supporting sources the ease of readability comes at the expense of transparent identifying the information sources so the first one is when implementing or utilizing a generative a application if ation it is important to consider the following limitation I repeat the first one is lack of source identification generative AI does not always provide clear identification of content Source making it difficult to trace and verify origin of the information the second one is assessment of bias assessing the bias of original sources used generative AI can be challenging as it may be difficult to determine the underlying perspective or agendas of the data utilized in the training process the third one is difficulty in identifying inaccurate information generative AI can generate realistic content making identifying inaccuracy or falsehoods within the generated output harder and the fourth one is adaptability to a new circumstances understanding how to fine-tune generative AI for a new circumstances or specific context can be complex requiring careful consideration and expertise to achieve desired result and the fifth one is glossing over bi Prejudice and hatred generative AI results May amplify or preate biases prejudices or hateful content present in the training data requiring Vigilant scrutiny to prevent such issues so awareness of these limitation is crucial when the implementing of utilizing generative AI as it helps users and developers critically evaluate and mitigate potential risk and challenges associated with the technology so future of generative AI furthermore advances in AI development platforms will contribute to the accelerated progress of research and development in the realm of generative AI the development will Encompass various domains such as text images videos 3D contact drugs Supply chains logistic and business processes while the current loan tools are impressive the true transformative impact generative AI will realize while these capabilities are seemingly integrated into the existing tools with regular use hello everyone and welcome to the video on large language models large language models are basically very very Advanced artificial intelligence systems that can process and generate massive amounts of text Data they are designed to learn from and understand natural human language and can be used to perform a wide range of language related tasks such as translation speech recognition and automatic summary generation one of the key advantages of large language models is their ability to learn from vast amounts of data which allows them to generate highly accurate and realistic responses to complex natural language prompts additionally they can be trained on multiple langu simultaneously which means they can be used to perform language translation between different pairs of languages making them an invaluable tool for businesses and organizations that work with people from diverse cultural backgrounds not just language related task but llms can also have the potential to revolutionize Fields such as research science and Healthcare by allowing researchers to quickly analyze and process vast amounts of complex text Data they can accelerate progress in areas such as drug Discovery Healthcare Diagnostics and AI development that being said large language models also present some challenges for instance they require a significant amount of comput sources which can be prohibitively expensive for some organizations additionally because they are trained on huge amounts of data they may inadvertently reinforce biased or discriminate KN patterns in language use which is something that researchers and Engineers are continuously working to address overall large language models are an exciting and rapidly developing field with tremendous potential to transform the way we live work and communicate so if you want to embark on a journey of AI and ml then try giving a sh to our postgraduate program in Ai and ml that is in partnership with IBM this artificial intelligence course covers the latest tools and Technologies from the AI ecosystem and features master classes by celtech faculty and IBM experts hecaton and ask me anything sessions this program showcases celtech ctm excellence and IBM's industry progress the artificial intelligence course covers key Concepts like stat ICS data science with python machine learning deep learning NLP and reinforcement learning through an Interactive Learning model with live sessions enroll now and UNL exciting AI anml opportunities the link is mentioned in the description box below and with that having said hey everyone welcome to Simply lears YouTube channel but before we dive into that don't forget to like subscribe and share in this video we will cover topics like what are large language models after that we'll look at what large language models used for four and after that we'll cover how are large language models trained after this we'll look at how do large language models work and at the last we'll see some applications of large language models so without any further Ado let's get started so we'll start with what are the large language models large language models such as gpt3 generative pre-trained Transformer 3 Advanced artificial intelligence systems designed to understand and generate humanik Tech text these models are built using deep learning techniques and have been trained on vast amounts of text Data from the internet these models use self attention mechanisms to analyze the relationships between words or tokens in a text enabling them to capture contextual information and generate Cent responses these models use self attention mechanisms to analyze the relationship between different words or tokens in a text enabling them to capture contextual information and generate coent responses these models have significant implications for a wide range of applications including virtual assistants chat boards content generation language translation and aiding in research and decision making processes their ability to generate cerent and contextually appropriate text has led to advancement in natural language understanding and human computer interaction now we'll see what are language models used for so large language models are utilized in scenarios where there is limited or no domain specific data available for training these scenarios include both few short and zero short learning approaches which rely on the model's strong inductive bias and its capability to derive meaningful representations from a small amount of data or even no data at all now we'll see how are large language models trained large language models typically undergo pre-training on a broad all and compassing data set that shares statistical similarities with the data set specific to the Target task the objective of pre-training is to enable the model to acquire high level features that can later be applied during the fine-tuning phase for specific task and the training process of a large language model involves several steps the first is text pre-processing the textual data is transformed into a numerical representation that can be effectively processed by the model this conversion may involve techniques like tokenization encoding and creating input sequences the next we have we have is random parameter initialization the model's parameters are initialized randomly before the training process begins the next is inputting numerical data the numerical representation of Text data is fed into the model for processing the models architecture typically based on Transformers allows it to capture the contextual relationships between the words and tokens in the text and the next is loss function calculation a loss function is employed to measure the discrepancy between the model's predictions and the actual next word or token in a sentence the model aims to minimize this loss during training and the next is parameter optimization the model's parameters are adjusted through optimization techniques such as gradient descent to reduce the loss this involves calculating gradients and updating the parameters accordingly gradually improving the model's performance and the next is iterative training the training process is repeated over multiple iterations or repo until the models of outputs achieve a satisfactory level of accuracy on the given task or data set by following this training process large language models learn to capture linguistic patterns understand context and generate coherent responses enabling them to excel at a wide range of language related task and now we'll see how do large language models work so large language models leverage deep neural networks to generate outputs based on patterns learned from the training data typically a large language model adopts a Transformer architecture which enables the model to identify relationships between words in a sentence irrespective of the position in the sequence in contrast to recent neural networks that is RNN that rely on recurence to capture token relationships Transformer neural networks employ self attention as their primary mechanism self attention calculates attention scores that determines the importance of each token with respect to other tokens in the text sequence facil itting the modeling of intricate relationships within the data now we'll see applications of large language models large language models have wide range of applications across various domains and here are some notable applications the first one is natural language processing large language models are used to improve natural language understanding task such as sentiment analysis named entity recognition text classification and language modeling the next is chat boards and virtual assistants large language models power conversational agents chat boards and virtual assistants providing more interactive and humanlike interactions with users and the next is machine translation large language models have been used for automatic language translation enabling the translation of text between different languages with improved accuracy and the next we have is sentiment analysis large language models can analyze and classify a sentiment or emotion expressed in a piece of text which is valuable for market research brand monitoring and social media analysis and the next we have is content recommendation these models can be employed to provide personalized content recommendations enhancing user experience and engagement on platforms such as news websites or streaming services these applications highlight the versatility and potential impact of large language models in various domains improving language understanding Automation and interaction between humans and computers everyone knows the algorithm is a step-by-step process to approach a particular problem there are numerous examples of algorithm from figuring out sets of number to finding Roots through maps to showing data on a screen let's understand this by using an example every algorithm is built on inputs and outputs Google search algorithm is no different the input is the search field and the output is the page of result that appears when you enter a particular phrase or keyword also known as sep or search engine result page Google has a search algorithm so it can sort results from various websit and provides the users the best result when you start a search you will see the search box you will attempt to guess what you are looking for in order to better understand what the user is looking for the algorithm is trying to gather as many as suggestions from them as possible the results from the search field that best matches the query will be ranked the choose F website will Rank and in what position using more than 200 ranking variables now let's take an example of coding program and see how the algorithm works here we will use a case of computer program wherein we want to print the multiplication table of any number let's take two the algorithm start here and then it assign a value to a variable the variable I is having an initial value of one the system will read the number the number in case is two now the system has a condition a condition can now either be true or false if the value of I reaches 11 then the loop will end otherwise value of I will multiply by the number the initial value of I is 1 so for the first time the system output will be two now value of I will be increased by one according to the loop condition the system will then move back and check for the condition again the new value of I is 2 which is still less than 11 the system will again print 2 into I which is 2 into 2 on the screen the new output result will be four the system will keep following the same procedure repeatedly until the value of I becomes 11 once the value of I becomes 11 then only the algorithm will terminate after discussing how an algorithm work let's move forward and see some popular machine learning algorithms some popular machine learning algorithms are first one is linear regression algorithm second one is logistic regression algorithm and the third one is decision tree and the fourth one is support Vector machine algorithm svm and the fifth one is KNN K nearest neighbor algorithm and the sixth one is K means clustering algorithms and the seventh one is random Forest algorithms and the last but not the least algorithm is aiy algorithms let's go through them in detail one by one linear regression is one of the most famous and straightforward machine learning algorithms utilized for predictive analysis linear regression show the linear connection between the dependent and the independent factors the equation of line is y = to mx+b here y stand for the response variable or a dependent variable whereas X is for the Predator variable or an independent variable it attempts best to fit line between the dependent and independent variables and this best fit line is known as line of regression or regression line let's take a real application example in predicting consumer Behavior businesses use linear regression to forecast things like how much a client is likely to spend things like targeted marketing and product development May benefit from this Walmart for instant use linear regression to forecast which good would be in high demand across the N moving forward let's see types of linear regression there are two types of linear regression algorithm the first one is simple linear regression and the second one is multiple linear regression in simple linear regression if an independent variable is utilized to forecast the worth of a mathematical dependent variable then at that point such a linear regression algorithm is called Simple linear regression the equation of line will be y = a0 + A1 X and the second one is multiple linear regression if the dependent variables declines on the Y and the independent variable on the X then such a relationship is known as negative linear relationship the line of equation will be minus of a 0 + A1 X moving forward let's see logistic linear regression logistic regression is the supervised machine learning algorithm utilized to anticipate all the categorical factors or discrete values it could be very well used for the grouping issues in machine learning and the result of the logistic regression can be either yes or no zero or one men or women and so on it gives the values which lies between zero and one for example a credit card business is interested in knowing whether the transaction amount and the credit scope have an impact on the probability that a particular transaction would be fraudulent the business can use logistic regression to determine how these two Predator values can relate the probability that a transaction is fraudin the response variable in the model has two possible outcomes first one is the transaction is fraud and the second one is the transaction is not fraudulent in logistic regression rather than fitting a regression line we fit an S form logistic capability which predicts two greatest value zero or one the logistic regression equation can be calculated from linear regression equation the steps to get logistic regression equations are the equation of a straight line can be return as y = b 0 + B1 X1 + b2x 2 till B and xn in logistic regression y can be between 0 and 1 only so for that let's divide the above equation by 1 - y then the equation will be y upon 1 - y that is 0 for y 0 and Infinity for y = 1 but range between minus infinity to plus infinity then we have to take the logarithm of equation and now it will become log of Y upon 1 - y = b 0 plus B1 X1 plus B2 X2 so on till B and xn let's move forward and see types of logistic regression there are three types of logistic regression that can be classified first one is binomial in binomial logistic regression there can be only two possible types of dependent variables like yes or no pass or fa men woman and many more and the second one is multinomial in multinomial logistic regression there can be three or more possible unordered ways of dependent variable such as horse cow and sheep and the last one is ordinal in ordinal logistic regression there can be three or more possible ordered ways of dependent variable such as small medium or large moving forward let's see decision trees in detail a decision tree is a tree structured classifier that could be used for classification and regression a decision tree is a tree in which each non-leaf node is assigned to an attribute additionally each are contain one of the available values for its parent node which is associated with each Leaf node that is the node from where the arc is directed let's see some decision tree terminology first one is root that contains the entire data set the next one is node attach for the data of a certain attribute and the third one is branch which connect the node to internal node or the internal node to Leaf node and the fourth one is leaf node the terminal node that predicts the outcome let's move forward and see decision Tre algorithms the first one is select the best attribute to use the current node in the tree the second one is for each possible values select the attributes the third one is partition the examples using the possible values of this attribute and assign these joint subset of the example to the appropriate child node recursively generate each child node until ially all examples for a node have the same label like class moving forward let's understand the Deion Tre for building a decision tree step one is Select an attribute then split the data into its children in a tree continue splitting with available attributes and keep splitting until Leaf node are pure like only one class remains a maximum depth is reached a performance metric is achieved let's move forward and see svm algorithm support Vector machine algorithms a support Vector machine is a well-known supervised machine learning model it is utilized for both information classification and regression it is regularly utilized for the grouping issues we can involve it in different life care system and we can involve it in typically happy or side look Arrangements we can involve it in filters if we make specific looks it would add the particular filter according to the expression the scope of articulation lies between happy and sad support Vector machine helps him to recognize handr return characters use yd like checks continue to be the significant part of the majority of non-cash transaction and are frequently return by the pupil the current check processing system in many developing Nations involves a bank employer to read and manually enter the information in a check while also verifying the data like signature and date a handwritten tax recognition system can reduce expenses and labor hours because a bank must handle several checks each day moving forward let's see the algorithm of svm the objective of support Vector machine is to make the best line or choice limit that can isolate n dimensional space into classes so we can undoubtly put the new data of interest in the right category later on this best decision boundary is known as a hyperplan let's move forward and see types of support Vector machine support Vector machine can be of two types first one is linear sbm second one is nonlinear sbm let's move forward and see linear SPM linear SPM is utilized for linearly detachable information which implies if a data set can be ordered into two classes by utilizing a straight line then such information a named linearly separable information and a classifier is utilized called linear SPM classifier moving forward let's see nonlinear sbm nonlinear sbm is utilized for non-directly isolated information and that implies in the event that a data set can't be categorized by utilizing a straight line such information is named non-directed information and the classifier utilized is called a nonlinear SPM classifier moving forward let's see KNN algorithm in detail KNN is a supervised learning technique Canon classifies new data into our targeted classes depending on the features of its neighboring points and also be used for the regression problems it is an instance-based learning algorithm and a bit lazy learning algorithm Canon calculation stores every one of its accessible information and orders another information Point based on the likeliness this means that when new data information appears it usually tends to be successfully categorized into a good suit classes using the Canon algorithm let's imagine we have an image of animal that resembles a cow or Ox however here we are not sure if it is a cow or Ops SN method is based on a likeus Matrix it will identify the properties of new data that are related to the image of cow or ox and based on those quality it will classify the data is belonging to either cow or o group moving forward let's see how does scn work the steps to implement KNN algorithms are step one decide on the neighbors SK numbers step two calculate the ukan distance between K Neighbors in step two third third one is based on the determined ukle and distance select the K closest neighbors step four is count the numbers of data points in each category between these SK neighbors step five assign the fresh data points in the category where the highest neighbors count and then KNN model is ready let's say we need to add a new data point to the vital category at first we will decide on the numbers of neighbors therefore we will pick K = to 5 then the ukan distance between the data points and then can be determined the distance between two points known as the uclean distance can be determined by under root of X2 - X1 sare + Y2 - y1 whole squ then we determine the closest neighbors by calculating the ukine distance there are three closest Neighbors in category a and two closest neighbor in category B this new data point Must Fall with with category a because as we can see its three closest neighbors are also from group a after understanding KL algorithm let's move forward and see K means algorithms in detail the K means is a cluster false under that is an unsupervised learning algorithm it is used to address machine learning clustering problems and utilized to tackle the grouping issues in machine learning it permits us to Bunch the information into various Gatherings it is a helpful method for finding the classification of groups in the unlabeled data set without the requirement of any training this K means algorithm groups the data into similar classes let's see some application of K means clustering let's see some applications of K means clustering diagnostic system the medical profession uses K means crusting in creating a smarter medical decision support system especially in the treatment of lever alignments the second one is search engines clustering forms a backbone of search engine when a search engine is performed the search result need to be grouped and the search engines very often use clustering to do this moving forward let's see how K means algorithm works the steps to implement K means algorithms are step one select the number K to set the number of clusters step two select a random K points or centroid step three assign each data point the closest centroid dat forms to predefined K cluster step four determine the variance and a set a new Gravity points for each cluster step five repeat the third step this means reallocating each data point to the new closest centroid cluster step six if a reassignment occurs go to step four otherwise go to exit step seven the model is ready to use so now we have an clear understanding of how K means algorithm work let's move forward to see the graphical represent presentation of K means algorithm consider that there are two variables M1 and M2 this is a scatter plot of these two variables along the X and Y AIS we should accept the number of K of bunches that is K = to 2 to recognize the data set and to place them into various groups it implies here we will attempt to Bunch these data set into two unique groups we will really want to pick an irregular keyo or centroid to frame this group these centroids can be either the focus of the data set or some of other points thus here we are choosing under two points as SK points which are not the piece of our data set we will assign every data of interest in the scatter plot to the nearest K point or centroid we will register it by applying some ma that we consider to find the distance between two points that is ukan distance thus we will draw a median between both the centroids from the graph the left half of the line are close points to the right and the K1 centroid green is near the orange centroid we should veriy them as green and orange for Clear representation as the need might arise to track down the nearest group we will repeat the cycle by picking another centroid to pick the new centroid we will figure out the center point of gravity of these centroids and we'll track down new centroids then we will reassign every piece of information to highlight new croid for this we will will repeat a similar course of tracking down a middle line the middle will be like as seen in the picture one orange point is on the left half of the line and two green points are on the right thus these three points will be appointed to the new centroids as reassignment has occurred we will again go to step four tracking down new centroids or k points we will repeat the cycle by tracking down the center point of gravity of centroids so the new centroids will be displayed as like this we now have new Android so once more Define the middle boundary and reassign the data of in by this graph there are no unique pieces of information on one of the other side of the line implying our model is shaped by the previous graph there are no unique pieces of information on one or the other line implying our model is shaped as our K means model is ready and the two last groups will be displayed as like these now we have an clear understanding of how K means clustering algorithm works now let's move forward to understand random Forest algorithm random Forest is an adaptable simple to utilize machine learning algorithm that produces even without the hyper boundary tuning an extraordinary outcome more often that not it is likewise quite possibly the most utilized algorithm because of its effort and variety like it tend to be the utilized for both grouping and classification task random Forest is a classifier that contains various Choice trees on different subset of the given data set and takes the normal to work on the present exactness of the data set instead of depending on the choice tree the random Forest takes the forecast from each tree and in light of the larger part of vs of expectation it predicts the final result now let's move forward and see how does random Forest work we should see the random forest in order since the arrangement is now and again thought to be the structured block of machine learning this is what a random Forest would look like with two trees the random Forest has a similar hypermeter to a decision tree or a badging classifier luckily there is a compelling reason need to consolidate a decision tree with a badging classifier since you can undoubtly utilize the classifier classes of random Forest with random Forest you can likewise manage task using the algorithm regression random Forest add extra AB bidness to the model while deing the trees rather than looking for the M element while parting a node it looks to the best component among an irregular subset of highlights these outcomes in a wide variety often result in a superior model subsequently in a random Forest just a random subset of the element is T about the algorithms you might make trees more random by involving random edges for each component instead of looking for the most ideal limits like a typical Choice tree does let's move forward and see some application of random Forest algorithms random Forest is involved at work by researchers in numerous Ventures including banking stock exchanging medication and many more it is utilized to predict things which assist these businesses with running productively like client activity patient history and safety in banking random Forest is used to identify clients who are more likely to pay back their debts on schedule additionally it is utilized to forecast who will make more frequent use of Bank Services even fraud detection uses it the Robin Hood of algorithms indeed random Forest is a tool used by stock traders to forecast future stock Behavior retail businesses utilize it to make product recommendation and forecast client satisfaction random Forest can be used in healthcare industry to examine a patient medical history and detect disorders random Forest is a tool used by pharmaceutical expert to determine the ideal mix of ingredients in treatment or to forecast drug sensitivity by seeing application of random Forest algorithm let's move forward and see some difference between decision trees and random Forest let's see the difference between random forest and decision tree the first one is while building a random Forest the number of rows is selected randomly in decision trees it builds several decision trees and find out the output the second one is it combines two or more decision trees together in decision trees whereas the decision is a collection of variables or data set or attributes the third one is it gives accurate results whereas it gives less accurate results the fourth one is by using random Forest it reduces the chance of overl whereas decision trees it has the possibility of over lifting the fifth one is random Forest is more complicated to interpreter whereas the decision tree is simple it is easy to read and understand after seeing what is zom Forest how it works let's move forward to see ariary algorithms in detail the AI algorithm utilize standard item sets to create affiliation rules and is intended to chip away the information basis containing exchanges with the assistance of these affiliation rules it decide how firmly or feebly two objects are associated this algorithm utiliz a breath for search and history to work out the item set Association effectively it is the iterative interaction for finding out the successive item set from the huge data ass said let's move forward and see steps for AI algorithms the steps for AI algorithms are Step One is establish minimal support and confidence for item set in the transactional database the second one is take all transaction supports with a greater support value the minimum or choosen support value in step two the third one is track down all the rules in these subsets with confidence value greater than the threshold value the fourth one is arrange the rules to lower the lift at last we will see some advantages and disadvantages of a PRI algorithm the advantages of AI algorithms are easy to understand an algorithm and the second one is the join and prune steps of the algorithms can be easily implemented on the large data set the disadvantages of AI algorithms are the AI algorithms work slowly as compared to the other algorithm and the second one is the a PRI algorithms times and space complex it are o of 2D which is very low compared to the other ones let's move forward and see some handson lab demo so we will see a hands on lab demo on linear regression as we know linear regression is a way to find or to predict the relationship between two variables generally we use X and Y so first we will open command prompt to write command to open jupyter notebook so we will here write Jupiter notebook and then press enter so this is the landing page of jupyter notebook and select open new python file so this is how jupyter notebook UI looks like so at first we will import some major libraries of python which will help us in mathematical functioning so the first one is numpy numpy is a python Library used for working with arrays it also has functions for working in domains like linear algebra and matrices it is an open source project and you can use it freely numi stand for numerical python so we will write like import numpy s NP here NP is used for denoting npire so we will import the next libraries pandas pandas is a software Library written for Python programming for like data manipulation and Analysis in particular it offers data structures and operation for manipulating numerical labels and time series so we will write import Hondas as PD here PD is used for denoting pandas so our next library is matplot lip mat plot lip is a graph plotting library in Python that serves as a visualization utility M BL is a open source so we can use it freely and it is most WR in Python a few segments in C and JavaScript for platform compatibility for importing M plot Li you have to write import Mt plot Li blot Li do pip plot s PLT so after importing the libraries we will move ahead and import data set so for importing data set we have to write data set equals to PD do read CSV and here we have to give the path of the file data set. CSV here PD is for Panda's labrary and read is used for reading the data set from the machine and CSV CSV is used for the type of file which you want to read so after reading let's see our data so we will write data set dot head and press enter here head is used for retrieving the first five lines from the data so our data is coming properly so moving ahead now let's first Define X and Y axis for X we have to write X = to data set do iock bracket colum comma colum again minus one dot values for x-axis and for y axis we have to write yals to data set do iog bracket column comma 1 dot values so if we will use data set. iog for minus one values for x-axis it will select till the second last column of the data frame instead of the last column and I know this is as the second last column value and the last column value for the row is different whereas if you use for the y- axis values WR a series Vector a vector does not have a column size so moving ahead let's see the value for x and y axis first we will see the value for x like just you have to write X and press enter so these are the arrays value for x and for y axis you have to type Y and then enter so these are arrays value for y axis so after this now let's split the data set into training and test testing separating data into training and testing set is an important part for evaluating data mining models typically when you separate a data set into a training set and a testing set most of the data set is used for training and a similar portion of data set is used for testing so we will split it into 70 and 30 ratio so for splitting we need to import some more libraries for this process so we will write from skar Dot model underscore selection selection import train underscore test underscore split psyched learn like SK learn is most useful and robust library for machine learning in Python it provides a selection of efficient tools for machine learning and statical modeling including classification regression and clustering so after importing let's write code for the splitting data so we will write here xcore train comma xcore test comma Yore train comma Yore test equals to P underscore test underscore split in Brackets we have to write X comma y comma test uncore size equals to 0.3 comma random _ State equals to 0 random with random State equals to 0 we get the same train and test sets across different execution so after this let's see the values together so we will write xcore train comma xcore test comma Yore train comma Yore test so these are the values of array X and array y to together moving ahead now let's work with regression first we need to import the library for regression so we will use from skon do linear underscore model model import linear regression we already discussed escalan is used for machine learning and linear regression is a major part of machine learning so after this let's make a function for regression as R for easy use so we will write R equals to linear regression and brackets first we try to train and then test and compare so we will write here r dot bit bracket xcore train comma Yore train now let's predict values but prediction values are always different so we will predict values for y first so we will write y underscore predict = to r dot predict sorry predict or xcore test so like Yore predict then enter this is when the predict functions come into the picture python predict function enables us to predict the labels of the data values on the basis of training model the predict function accept only a single argument which is xor test here usually the data to be tested so when you will check the value for y test you will see the different values like Yore test so you can see there are totally different values from x-axis now let's display on graph for training data set so we have to write PLT do scatter xcore train comma Yore train comma color equals to you can choose by yourself I will choose red and then PLT do plot xcore drain comma regression point for predicting values for xcore train comma color equals to Blue so this color is called the regression line and the PLT do title linear regression okay regression salary versus experience and I will prefer this size would be uh 180 so let's set the labels for x and y axis so X label year of employee of employee comma size will be I will prefer size be 50 then for y axis we will write here like salaries of emplo I will prefer this same size 15 let's show the plot by writing PLT do show then hopefully we are for everything is going to be fine perfect so you can see we have linear regression line fitting through our data set so this is how the linear regression work for training data set let's see how it will work for testing data set now let's predict for test data set you can copy for the same and like you can change it so we can copy here so we can paste here so we have to just change here train to test data set test here then again test here then again test here everything everything we will reduce some size to 16 let's see then it will 12 and 12 so perfect so you can see we have linear regression line fitting through our data set so this is how the linear regression works for testing data set if you're looking for a course that covers everything from the fundamentals to Advanced Techniques then accelerate your career in Ai and machine learning with a comprehensive post-graduate program in Ai and machine learning boost your career with this Ai and ml course delivered in collaboration with the puru university and IBM learn in demand skills such as machine learning deep learning NLP computer vision reinforcement learning generative AI prompt engineering chat GPT and many more you'll receive a prestigious certificate and ask me anything sessions by IBM with five capstones in different domains using real data sets you will gain practical experience master classes by PD faculty and IBM experts ensure top-notch education simply learns job assist help you get noticed by Leading companies this program covers statistics python supervised and unsupervised learning NLP neural networks computer vision G scas tensorflow and many more such skills admission to this post-graduate programm in Ai and machine learning requires a bachelor's degree with an average of 50% or higher marks along with the basic understanding of programming Concepts and Mathematics and candidates with two plus years of work experience are preferred to enroll in this school so enroll now and unlock exciting Ai and machine learning opportunities the link is mentioned in the description box below let's first understand what text mining is text mining is the technique of exploring large amounts of unstructured Text data and analyzing it in order to extract patterns from the text Data it is aided by software that can identify Concepts patterns topics keywords and other attributes in the data it utilizes computational techniques to extract and summarize the high quality information from unstructured textual resources let's understand the flow of text mining there are five techniques used in text mining system information extraction or text pre-processing this is used to examine the unstructured text by searching out the important words and finding the relationships between them categorization or text transformation attribute generation categorization technique labels the text document under one or more categories classification of Text data is done based on input output examples with categorization clustering or attribute selection clustering method is used to group text documents that have similar content clusters are the partitions and each cluster will have a number of documents with similar content clustering makes sure that no document will be omitted from the search and and it deres all the documents that have similar content visualization technique the process of finding relevant information is simplified by visualization technique this technique uses text Flags to represent a group of documents or a single document and compactness is indicated using colors visualization technique helps to display textual information in a more attractive way summarization or interpretation or evaluation summarization technique will help to reduce the length of the document and summarize the details of the documents it makes the document easy to read for users and understand the content at the moment let's understand the significance of text mining document clustering document clustering is an important part of text mining it has many applications in Knowledge Management and information retrieval clustering makes it easy to group simil ilar documents into meaningful groups such as in newspapers where sections are often grouped as business Sports politics and so on pattern identification text mining is the process of automatically searching large amount of text for text patterns and recognition of features features such as telephone numbers and email addresses can be extracted using pattern matches product insights text mining helps to extract large amounts of text for example customer review about the products mining consumer reviews can reveal insights like most loved feature most hated feature improvements required and reviews of competitor's products security monitoring text mining helps in monitoring and extracting information from news articles and reports for national security purposes text mining makes sure to use all of your available information it is a more effective and productive knowledge discovery that allows you to make better informed decisions automate information intensive processes gather business critical insights and mitigate operational risk let's look at the applications of text mining speech recognition speech recognition is the recognition and translation of spoken language into text and vice versa speech often provides valuable information about the topics subjects and concepts of multimedia content information extraction from speech is less complicated yet more accurate and precise than multimedia content this fact motivates content-based speech analysis for multimedia Data Mining and retrieval where audio and speech processing is a key enabling technology spam filtering spam detection is an important method in which textual information contained in an email is extracted and used for discrimination text mining is useful an automatic detection of spam emails based on the filtering content using text mining an email service provider such as Gmail or Yahoo mail checks the content of an email and if some malicious text is found in the mail then that email is marked as spam and sent to the spam folder sentiment analysis it is done in order to determine if a given sentence expresses positive neutral or negative sentiments sentiment analysis is one of the most popular applications of text analytics the primary aspect of sentiment analysis includes data analysis of the body of the text for understanding the opinion expressed by it and other key factors comprising modality and mood usually the process of sentiment analysis works best on text that has a subjective context than on that with only in objective context e-commerce personalization text mining is used to suggest products that fit into a user's profile text mining is increasingly being used by e-commerce retailers to learn more about the consumers as it is the process of analyzing textual information in order to identify patterns and gain insights e-commerce retailers can Target specific individuals or segments with personalized offers and discounts to boost sales and increase Customer Loyalty by identifying customer purchase patterns and opinions on particular products let's look at natural language toolkit library in detail natural language toolkit is a set of open-source python models that are used to apply statistical natural language processing on human language data let's see how you can do environment setup of nltk go to Windows start and launch python interpreter from Anaconda prompt and enter the following command commands enter command python to check the version of python installed on your system enter import nltk to link you to the nltk library available to download then enter nltk doownload function that will open the nltk download window check the download directory select all packages and click on download this will download nltk onto your python once you have downloaded the NL TK you must check the working and functionality of it in order to test the setup enter the following command in Python Idol from nltk Doc Corpus import Brown brown. word parenthesis parenthesis the brown is an nltk Corpus that shows the systematic difference between different genres available words function will give you the list available words in the genre the given output put shows that we have successfully tested the nltk installed on python let's Now understand how you can read a specific module from nltk corpora if you want to import an entire module from nltk corpora use asterisk symbol with that module name import command enter the command from nltk book import asterisk it will load all the items available in NLT K's book module now in order to explore Brown Corpus enter the command nltk do Corpus import Brown this will import Brown Corpus on the python enter brown. categories function to load the different genres available select a genre and assign that genre to a variable using the following syntax variable name is equal to brown. wordss categories is equal to genre name now in order to see the available words inside the selected genre just enter the defined variable name as a command let's understand text extraction and pre-processing in detail so let's first understand the concept of tokenization tokenization is the process of removing sensitive data and placing unique symbols of identification in that place in order to retain all the essential information concerned with the data by itself security it is a process of breaking running streams of text into words and sentences it works by segregating words using punctuation and spaces text extraction and pre-processing engrams now let's look at what engram is and how it is helpful in text mining engram is the simplest model that assigns these probabilities to sequences of words or sentences engrams are combination of adjacent words or letters of length n in the source text so engram is very helpful in text mining when it is required to extract patterns from the text as in the given example this is a sentence all of these words are considered individual words and thus represent unigrams a two gr or Byram is a two- word sequence of words like this is is a or a sentence and a three GR or trigram is a three-word sequence of words like this is a or is a sentence let's Now understand what stop words are and how you can remove them stop words are natural language words that have negligible meanings such as a n and or the and other similar words these words also will take up space in the database or increase the processing time so it is better to remove such words by storing a list of stop words you can find the list of stop words in the nltk data directory that is stored in 16 different languages use the following command to list the stop words of English language defined in nltk Corpus importing nltk will import the nltk Corpus for that instance enter from nltk Doc Corpus import Stop wordss will import Stop wordss from nltk Corpus Now set the language as English so use set function as set under braces stop wordswords set genre as English stop words are filtered out before processing of natural language data as they don't reveal much information so as you can see in the given example before filtering the sentence the tokenization of stop word is processed in order to remove these stop words and the filtering is applied in order to filter the sentence based on some criteria text extraction and pre-processing stemming stemming is used to reduce a word to stem or base word by removing suffixes such as helps helping helped and helper to the root word help the stemming process or algorithm is generally called a stemmer there are various stemming algorithms such as Porter stemmer Lancaster stemmer snowball stemmer Etc use any the stemmer defined under nltk stem Corpus in order to perform stemming as shown in the example here we have used Porter stemmer When You observe the output you will see that all of the words given have been reduced to their root word or stem text extraction and pre-processing lemmatization lemmatization is the method of grouping the various inflected types of a word in order that they can be analyzed as one item it uses vocabulary list or a morphological analysis to get the root word it uses word net database that has English words linked together by their semantic relationship as you can observe the given example the different words have been extracted to their relevant morphological word using lemmatization text extraction and preprocessing POS tagging let's now look at different part of speech tags of ailable in the national language toolkit Library a POS tag is a special label assigned to each token or word in a Text corpus to indicate the part of speech and often also other grammatical categories such as tense number either plural or singular case Etc POS tags are used in text analysis tools and algorithms and also in Corpus searches so look at the given example here Alice wrote a program is the source text given the POS tags given are Alice is a noun wrote is a verb a is an article and program is an adjective look at the given example to understand how posos tags are defined so the given sentence or paragraph contains different words that represent different parts of speech we will first use tokenization and removal of stop words and then allocate the different POS tags these are shown with different words in the given sentence p s tags are useful for lemmatization in building named entity recognition and extracting relationships between words text extraction and pre-processing named entity recognition now let's understand what named entity recognition is all about Neer seeks to extract a real world entity from the text and sorts it into predefined categories such as names of people organizations locations Etc many real world questions can be answered with the help of name entity recognition were specified products mentioned in complaints or reviews does the Tweet contain the name of a person does the Tweet contain the person's address as you can see in the given example Google America Larry Page Etc are the names of a person place or an organization so these are considered named entities and have different tags such as person organization GP e or geopolitical entity Etc NLP process workflow now you have an understanding of all nltk tools so now let's understand the natural language processing workflow step one tokenization it splits text into pieces tokens or words and removes punctuation step two stop word removal it removes commonly used words such as the is r Etc which are not relevant to the analysis step three stemming and litiz it reduces words to base form in order to be analyzed as a single item step four POS tagging it tags words to be part of speech such as noun verb adjective Etc based on the definition and context step five information retrieval it extracts relevant information from the source M1 Brown Corpus problem statement the Brown University standard Corpus of present-day American English also known popularly as brown Corpus was compiled in the 1960s as a general Corpus in the field of Corpus Linguistics it contains 500 samples of English language text totaling roughly 1 million words compiled from Works published in the United States in 1961 we will be working on one of the subset data set and performed text processing tasks let us import the nltk library and read the ca10 Corpus import nltk we will have to make sure that there are no slashes in between hence we will use the replace function within pandas for the same let's have a look at the data once tokenization after performing sentence tokenization on the data we obtain similarly after applying sentence tokenizer the resulting output shows all individual words tokens stop word removal let's import the stopword library from nltk Doc Corpus import stopwords we also need to ensure that the text is in the same case nltk has its own list of stop wordss we can check the list of stop wordss using stopwords do wordss and English inside the parenthesis map the lowercase string with our list of word tokens let's remove the stop wordss using the English stop wordss list in nltk we will be using set checking as it is faster in Python than a list by removing all stop wordss from the text we obtain often we want to remove the punctuations from the documents too since python comes with batteries included we have string. punctuation from string import punctuation combining the punctuation with the stop words from nltk removing stop wordss with punctuation stemming and lemmatization we will be using stemming and lemmatization to reduce words to their root form for example walks walking walked will be reduced to their root word walk importing Porter stemmer as the stemming library from nltk do stem import Porter stemmer printing the stem words import the wordnet litier from nltk do stem printing the root words we also need to evaluate the POS tags for each token create a new word list and store the list of word tokens against each of the sentence tokens in data 2 for I and tokenized also we will check if there were any stop words in the recently created word list we will now tag the word tokens accordingly using the POS tags and print the tagged output for our final text processing task we will be applying named entity recognition to classify named entities in text into predefined categories such as the names of persons organizations locations expressions of times quantities monetary values percentages Etc now press the tagged sentences under the chunk parser if we set the parameter binary equals true then named entities are just tagged as NE otherwise the classif fire adds category labels such as person organization and gpe create a function named as extract entity names along with an empty list named as entity names we will now extract named entities from a nltk chunked expression and store them in the empty created above again we will set the entity names list as an empty list and we'll extract The Entity names by iterating over each tree in chunked sentences great we have seen how to explore and examine the Corpus using text processing techniques let's quickly recap the steps we've covered so far One Import the nltk library two perform tokenization three perform stemming and lemmatization four remove stop words five perform named entity recognition structuring sentences syntax let's first understand what syntax is syntax is the grammatical structure of sentences in the given example this can be interpreted as syntax and it is similar to the ones you use while writing codes knowing a language includes the power to construct phrases and sentences out of morphemes and words the part of the grammar that represents a speaker's knowledge of these structures and their formation is called syntax phrase structure rules are rules that determine what goes into a phrase that is constituents of a phrase and how the constituents are ordered constituent is a word or group of words that operate as a unit and can be used to frame larger grammatical units the given diagram represents that a noun phrase is determined when a noun is combined with a determiner and the determiner can be optional a sentence is determined when a noun phrase is combined with a verb phrase a verb phrase is determined when a verb is combined optionally with the noun phrase and prepositional phrase and a prepositional phrase is determined when a preposition is combined with a noun phrase a tree is a representation of syntactics structure of formulation of sentences or strings consider the given sentence the factory employs 12.8% of Bradford County what can be the Syntax for pairing this statement let's understand this a tree is produced that might help you understand that the subject of the sentence is the factory the predicate is employ and the target is 12.8% which in turn is modified by Bradford County syntax parses are often a first step toward deep information extraction or semantic understanding of text rendering syntax tree download the corresponding exe file to install the ghost script rendering engine based on your system configuration in order to render syntax trees in your notebook let's understand how you can set up the environment variable once you have downloaded and installed the file go to the folder where it is installed and copy the path of the file now go to system properties and under Advanced properties you will find the environment variable button click on that to open the popup box tab of the environment now open the bin folder and add the path to the bin folder in your environment variables now you will have to modify the path of the environment variable use the given code to test the working of syntax tree after the setup is successfully installed structuring sentences chunking and chunk parsing the process of extraction of phrases from unstructured text is called chunking instead of using just simple tokens which may not represent the actual meaning of the text it is advisable to use phrases such as Indian team as a single word instead of Indian and team as separate words the chunking segmentation refers to identifying tokens and labeling refers to identifying the correct tag these chunks correspond to mixed patterns in some way to extract patterns from chunks we need chunk parsing the chunk parsing segment refers to identifying strings of tokens and labeling refers to identifying the correct chunk type let's look at the given example you can see here that yellow is an adjective dog is a noun and the' is the determiner which are chunked together into a noun phrase similarly chunk parsing is used to extract patterns and to process such patterns from multiple chunks while using different parsers let's take an example and try to understand how chunking is performed in Python let's consider the sentence the little mouse ate the fresh cheese assigned to a variable named scent using the word tokenize function under nltk corpora you can find out the different tags associated with the sentence provided so as you can see in the output different tags have been allocated against each of the words from the given sentence using chunk in NP chunk and parser you will now create grammar from a noun phrase and will mention the tags you want in your chunk phrase within the function here you have created a regular expression matching the string the given regular expression indicates optional determiner followed by optional number of adjective followed by a noun you will now have to parse the chunk therefore you will create a chunk parser and pass your noun phrase string to it the parser is now ready you will use the parse parenthesis parenthesis within your chunk parser to parse your sentence the sentence provided is the little mouse at the fresh cheese this sentence has been parsed and the tokens that match the regular expressions are chunked together into noun phrases NP create a verb phrase chunk using regular Expressions the regular expression has been defined as optional personal pronoun followed by by zero or more verbs with any of its type followed by any type of adverb you'll now create another chunk parser and pass the verb phrase string to it create another sentence and tokenize it add POS tags to it so the new sentence is she is walking quickly to the mall and the POS tag has been allocated from nltk corpor now use the new verb phrase parser to parse the tokens and run the result you can look at the given tree diagram which shows a verb parser where a pronoun followed by two verbs and an adverb are chunk together into a verb parse structuring sentences chinking chinking is the process of removing a sequence of tokens from a chunk how does chunking work the whole chunk is removed when the sequence of tokens spans an entire chunk if the sequence is at the start or the end of the chunk the tokens are removed moved from the start and end and a smaller chunk is retained if the sequence of tokens appears in the middle of the chunk these tokens are removed leaving two chunks where there was only one before consider you create a chinking grammar string containing three things chunk name the regular expression sequence of a chunk the regular expression sequence of your here in the given code we have the chunk regular expression as optional person pronoun followed by zero or more occurrences of any type of the verb type followed by zero or more occurrences of any of the adverb types the regular expression says that it needs to check for the adverb in the extracted chunk and remove it from the chunk inside the chinking block with open curly braces and closing curly braces you have created one or more adverbs you will now create a parser from NLT k. regex parser and pass the grammar to it now use the new parser to parse the tokens sent three and run the results as you can see the parse tree is generated while comparing the syntax tree of the parser with that of the original chunk you can see that the token is quickly adverb chin out of the chunk let's understand how to use context free grammar a context free grammar is a four Tuple sum ntrs where sum is an alphabet and each character in sum is called a terminal NT is a set and each element in NT is called a non-terminal r the set of rules is a subset of NT times the set of sum u n t s the start symbol is one of the symbols in NT a context free grammar generates a language L capturing con constituency and ordering in CFG the start symbol is used to derive the string you can derive the string by repeatedly replacing a non-terminal on the right hand side of the production until all non-terminals have been replaced by terminal symbols let's understand the representation of context free grammar through an example in context free grammar a sentence can be represented as a noun phrase followed by a verb phrase noun phrase can be a determiner nominal a nominal can be a noun VP represents the verb phrase a can be called a determiner flight can be called a noun consider the string below where you have certain rules when you look at the given context free grammar a sentence should have a noun phrase followed by a verb phrase a verb phrase is a verb followed by a noun a verb can either be Saul or met noun phrases can either be John or Jim and a noun can either be a dog or a cat check the possible list of sentences that can be generated using the rules use the join function to create the possible list of sentences you can check the different rules of grammar for sentence formation using the production function it will show you the different tags used and the defined context free grammar for the given sentence demo two structuring sentences problem statement a company wants to perform text analysis for one of its data sets you are provided with this data set named tweets. CSV which has tweets of six US Airlines along with their sentiments positive negative and neutral the tweets are present in the text column and sentiments in Airline unor sentiment column we will be retrieving all tags starting with at the rate in the data set and save the output in a file called references. txt let us first import the pandas library and read the twe twet data set extract the features text and Airline sentiment we will iterate through the data set using reg X find the relevant tweets now we will import the iter tools module it returns efficient iterators the result is stored in a file named references. txt let's extract all noun phrases and save them in a file named noun phrases for left karro Airline sentiment right Carro review. txt here left carrot Airline uncore sentiment right carrot has three different values positive negative and neutral so three files will be created now we will iterate all the leaf nodes and assign them to noun phrases variable this means that the functions in iter tools operate on iterators to produce more complex iterators using the map function we will get all the noun phrases from the text putting it into list creating a file name in the name of review. txt in a world where email is crucial spam messages plag our inboxes disrupting our digital lives but fear not a hero emerges the email spam classifier this intelligent system learns from past encounters uncovering the tricks employed by spammers deceitful subject lines promising riches unreasonable low prices it sports signs like excessive capital letters misspellings and suspicious URLs yet the battle doesn't end here spammers constantly evolve their tactics but the email spam classifier doesn't Flinch it adapts refining its knowledge and staying up to date with the latest tricks every new spam email it encounters it provides insights enhancing its accuracy and ensuring our protection thanks to Dedicated minds and Ai and ml power the fight against email spam continues the email spam classifier stands as a faithful Guardian a symbol of Technology's victory over digital adversaries it reclaims our inboxes and restores order to our virtual world according to recent studies Ai and machine learning related job postings have increased by staggering 344 in the past 5 years companies across the global actively seeking professionals who can harness the power of data and build intelligent systems the average salary is $150,000 in the US and 15 lakhs perom in India accelerate your career in Ai and machine learning with a comprehensive postgraduate program in Ai and machine learning gain expertise in machine learning deep learning NLP computer vision and reinforcement learning you will receive a prestigious certificate exclusive alumni membership and hackathon and ask me anything sessions by IBM with three capstones and 25 plus industry projects using real data sets from Twitter Uber and more you will gain practical experience master classes by Purdue faculty and IBM experts andure topnotch education simply Larn job assist help you notice by Leading companies live sessions on AI trends like chat jpt generative Ai and explainable AI this program C statistics python supervised and unsupervised learning NLP neural networks computer Vision G caras tens of flow and many more skills enroll now and unlock exciting Ai and ml opportunities the link is mentioned in the description box below let's take a minute to hear from our Learners who have experienced massive success in their careers you need to keep updating your skills on a regular basis no matter what level you at I recently completed the professional certificate program in Ai and machine learning from Simply learn in partnership with P University the course material was comprehensive and the faculty was extremely experienced uh The Faculty was able to adjust their teaching style in order to cater to the overall skill set of the class in the rapidly evolving world of technology it's important to keep up Skilling for every working professional stay relevant with that having said hey everyone welcome to Simply LS YouTube channel but before we dive into that don't forget to like subscribe and share and now over to our training expert so hello and welcome to this new video by simply learn email spam classifier okay so first we will change this name to email spam classifier okay let's rename it so first what we going to do so first what we going to do we will import some libraries then the main main libraries like n skill and logistic regression algorithm and then we will move on to this importing of data set of spam mail and ham mail ham is known as like which is not a Spam okay so we will do some categorization of smam email and hem email then we will split into the training test and the testing of the data set okay then we will transform the text data to Fe feature vectors that can be used as input to the logistic regression and then we will convert the Y train and the Y test values as integers and then moving forward we will some see the features the model extractions and then we will train the model using logistic regression and after that we will predict the accuracy how much the training data is giving how much the testing data is giving and at the end we will build a predictive system which will tell whether a particular email is Spam or not spam Okay cool so let's move on to the coding part then okay so first I will import some major libraries like import numai as NP okay then numpy so numpy is a powerful python library for numerical Computing so it stands for numerical Python and provide support for efficient array operations and mathematical function L algebra random number generation and more with numai you can work with multi-dimensional array known as ND arrays and these arrays are highly optimized for performance and allow you to perform mathematical and logical operations on large data set efficiently so numai provides a wide range of function and methods to manipulate and analyze these arrays and the second one is import pandas as PD okay this PD and NP is nothing we don't want to write again and again pandas and napai so what is pandas so pandas is a powerful python Library used for data manipulation and Analysis it provides easy to use data structure and data analysis tool making it essential tool for working with structure data so the third third one is psychic learn okay so I will write here from okay I will write here from skar do model selection Bo train test split okay so what is escalar model selection and the train test split so escalar or psychic learn is a popular python library for machine learning it provides a wide range of tools and algorithms for various machine learning tasks including classification regression and the clustering dimensionally reduction and the model evaluation okay and this train test split this one so the train test split function is a utility in psychic learn that helps in splitting the data set into training and the testing subset later we will split the data into training and the testing so this function is commonly used in machine learning task to assess the performance of a model on unseen data okay so move on to the next algorithm we will write from we will write here from Escalon do feature extraction dot text import tfid vectorizer okay so what is TF ID vectorizer so tfid vectorizer is nothing is a feature of extraction technique commonly used in natural language processing NLP and text mining task so it is a part of psychic learn library in Python so tfid vectorizer transform a collection of text documents into a numerical representation that machine learning algorithm can work with Okay so move on to we will import from Escalon dot linear model support logistic regression okay so what is logistic regression so logistic regression is a popular classification algorithm used in machine learning it is a part of the psychic learn library in Python logistic learn and the regression is suitable for binary classification problems where the target variable has the two classes okay and the last we will import accuracy score okay so here I will write from escalan do Matrix import accuracy score okay so let me tell you first what is accuracy score the accuracy score function is a performance matric provided by the psy learn library in Python it is commonly used to evaluate the performance of a classification model by measuring the accuracy of the predictions okay so let me run it okay okay there is some error maybe the spelling mistake okay what I will do I will write it again running it again yeah it's working so now what I will do I will import the data set so I will write here DF equals to data frame = pd. read uncore CSV mail _ data. CSV you can find this data Link in the description box below okay I'll run it yeah so let's see the data frame what's in it for us so as you can see here is category then message so these are the you know the males ham means not spam and spam is Spam okay those males which are not spam known as ham so there are two columns 557 to rows so moving forward what I will do I will convert this male data okay to Nal so I will write here data equals to Raw raw data equals to DF dot where PD do not null not null is DF okay then some space okay let me give you some spaces for the better visual yeah so now I will print data do head so what is the meaning of head here head means if you want to print the top five lines of the particular data set it is you can use the head and if you want to put the last five rows of the data set so you can use the tail instead if you will write here 10 instead of giving it blank then run it so it will give you the top 10 rows of the particular data set okay so these are the spam mail this ham mails so later on what we will do we will train this data set using this particular data set okay so let's move on to and see the info okay of the data set so data do info so info you can see there are columns this null null count this data type so there are two columns like category and the message and the particular rows are the 5572 on the Bo and the data type is object okay see the memory usage and you can see everything and there is one more you can see the DAT shape is 5572 comma 2 the these are the rows and these are the columns two columns and 5572 rows okay so now what I will do I will assign the category to spam M and ham mail okay so what I will do here I will write here data dot Lo then and data vers this category equals to equals to spam comma category okay comma equals to zero so if the zero will show it is known as the spam and if one will show it is known as as the Hamil okay I will write here data dot lo location and data then category equals to ham then category will be one okay then let me run it okay it's running smoothly so I will write here x equals to data then message okay and the Y value will Beals to main s not main data then category okay then let me run it so now what I will do first so now what I will do I will write here print X okay so these are only the messages and category print y so now as you can see 1 1 0 1 1 0 so here zero means what is the zero means zero means spam and the one means it's Hamil okay so moving forward what I will do I will split this data set into testing and the training okay so I will write here x train this is the most important part and the X test okay then y train then y test okay equals to train test split X comma y comma test size equals to 0.2 then random okay random State equals to three okay I will write it so this is X train for the training set and X test is the testing okay then the same y you know that what is the X and Y here and here we are splitting using train test split then XA y test size equal to 0.2 okay 0.2 means it will be 80 and 20% and 80% will be draining and the testing will be 20% okay we will train data 80% of data using this data set will be trained and the for the test the 20% will okay the random State equals to three and the random state is a hyperparameters used to control any such Randomness involved in machine learning model to get consistent result okay when splitting a data set splitting a node in a decision tree or you know logistic regression initializing centroids in clustering takes place so that is why we use random stet okay so let me run it it's working fine so now I will show print x. shape okay then print xcore train. ship I will show you about the I'm talking about the test size now print xcore test. ship now see carefully okay this x will be Capital so as you can see total rows are 5572 total row of 80% is 4457 or else you can calculate and the 20% is for the testing is the15 same goes for the Y so moving forward okay let me show you for the Y also so I will copy paste it here so let's y y y see same goes for the Y also okay so now what I will do here I will transform the text data to feature vectors that can be used as input to the logistic regression so feature extraction equals to T FID vectorizer where the minimum difference will be one comma stop words okay stop words equals to English and the lower case okay here I'm transforming the test data to the feature vector and okay so here x okay let me tell you so this is the feature extraction we are using TF Dev vectorizer okay we have import here as you can see for the feature extraction of the text and this is the minimum difference equal to one and then what is the stop word stop words are the English words which does not add much meaning to the sentence they can safely be ignored without sacrificing the meaning of the sentence okay like for example you can see the words like the he have such words already computerized and the capture this is a corpus name and the Corpus okay and this lower case equals to true true so here I will write X Trin features to Features feature extraction fit transform xcope G xcore test features feature extraction do transform to the X test okay yeah so moving forward I will convert y train and Y test values as an integers okay y train because they are 0 1 0 1 0 train equals to Y train. as style in okay then y testal to Y t. as type in so okay there is spelling mistake we'll copy from here paste it here okay I will run it okay why test it is yeah it's rning so now what I will do I will print the X train train as you can see the X train so these are the messages and if I will print exrain features we created here now see extra features okay then you can see so these are the features accuracy okay say 0 0 0 means spam 1 one means ham okay so moving forward what I will do I will now the Train the model OKAY logistic regession so I will write here model equals to okay logistic regression so now we will you know train the model so here I will write model equals to logistic regression okay so we don't want to write again and again again and again logistic regession so I'm here assigning model to it so let me run it so here I will train the logistic regression model with the training data so here I will write model. fit extr features comma white train okay here we are training the logistic regression model with the training data let me run it yeah fine it's running fine so moving forward we will uh do the evaluating of the training model train model and the prediction of the training data so for that I will write here prediction on training data equals to model dot predict extrem features okay then I will write here accuracy on training data equal to accuracy score y train comma prediction okay I will write from here PR the training data okay let me run it so now we will see the prediction so I will write here prediction or simply I will write print accuracy on training data comma okay I will copy it leave it accuracy on training data okay let me run this so as you can see accuracy on training data is 0 .96 means almost 97% which is not bad very very good okay we will see print okay now we have to do for the testing data this is for the training data and let's see the accuracy for the testing data so for that I will write same prediction on test data equals to model do predict X test features okay then accuracy on test data to accuracy score then y test comma prediction prediction on test data okay okay accuracy score so here I will write print the accuracy on test data okay comma then I will write accuracy on test data okay so as you can see the accuracy on test data is almost same as the training data okay almost 95% 97% sorry so now what we will do we will build a predictive system now okay so I will write here like input okay you have to input the mail here then okay we will input later and we will now write the line for the convert text to feature Vector so for that I will write here input data features to feature extraction do transform input okay okay this is a function so I will write here input inut your main me okay so here we will make the prediction okay I will write the prediction model do predict input data teachers okay so I have some spam mails with myself so I will try to copy them and let's see see okay I have some mail so I will copy it okay and I will paste it here okay this is the second time we have tried to contact you you have want this this this these type of Mals are spam actually like you also know so I will try to run it let me run it okay I forgot to write print print predictions okay so zero so here as you can see already zero means spam okay so this is a spam mail actually this is a spam so okay wait let me write here if prediction is showing zero equal to equals to true then print print it to H and as then print [Music] spam spam me okay the prediction will be zero it's not equals to one okay so it will show the ham mail or else the spam mail simple pretty simple see Zero spam okay I will try to put it put here what one Hamil okay okay let me copy it paste it here okay let me delete this quickly okay let me run this so it is showing ham mail means not spam so quickly let me run one more spam mail okay okay copy this and paste it here you can try with your own like own email with your got okay so let me run this so it is spam mail okay see you can see you have been selected to stay like marketing okay see in so these type of males are like you know deals and the vouchers this type of males are the known as the spam mail see holiday value this dial this and this okay so here we are we have done with our email spam classifier as you can see if you want this code so so you can comment down below if you're looking for a course that covers everything from the fundamentals to Advanced Techniques then accelerate your career in Ai and machine learning with a comprehensive postgraduate program in Ai and machine learning boost your career with this Ai and ml course delivered in collaboration with the Peru University and IBM learn in demand skills such as machine learning deep learning NLP computer vision reinforcement learning generative AI prompt engineering chat GPT and many more you'll receive a prestigious certificate and ask me anything sessions by IBM with five capstones in different domains using real data sets you will gain practical experience master classes by perdu faculty and IBM experts ensure topnotch education simply learns job assist help you get noticed by Leading companies this program covers statistics python supervised and unsupervised learning NLP neural networks computer vision G SC class tensorflow and many more suchar skills admission to this postgraduate program in Ai and machine learning requires a bachelor's degree with an average of 50% or higher marks along with the basic understanding of programming Concepts and Mathematics and candidates with two plus years of work experience are preferred to enroll in this school so enroll now and unlock exciting Ai and machine learning opportunities the link is mentioned in the description box below we will take you through the Hands-On lab demo of how you can do the sentimental analysis of Amazon customer reviews before we start I hope the screen is clearly visible and the audio is fine if yes please type in yes if there are any issues do let us know in the chat section so that we can resolve them let's wait for some more minutes to let other people join let's wait for some more minutes to let other people join let me tell you guys that we have regular updates on multiple Technologies if you are a tech gig on a continuous hun for the latest technological Trends then consider subscribing to our YouTube channel and press that Bell icon to never miss any update from Simply learn great I think we can get started so in today's session we will go through the what is sentimental analysis after that we will see types of sentimental analysis we will see some application of sentiment analysis and at the end we will see Hands-On lab demo like how to implement sentiment analysis using python in detail we already have Amazon Customer data set with us we will perform different function and analyze data for sentiment analysis using python you can download data set from the description box below so before we move on the programming part let's discuss what a sentiment analysis actually is and proceed further for the same sentiment analysis sometime known as opinion mining is a technique used in natural language processing NLP to determine the emotional undertone of a document this is a common method used by organization to identify and group ideas regarding a certain good service or concept text is mined for sentiment and subjective information using data mining machine learning and artificial intelligence tools for sentiment analysis assist businesses in extracting information from unstructured un organized language found in online sources like emails blog post support tickets web chats forums and comments algorithm use rule based automatic or hybrid technology to replace manual data processing while automatic system use machine learning to learn from their data rule-based system execute sentiment analysis using predetermined lexicon based rules combining the two method result in hybrid sentiment analysis sentiment analysis or opinion mining can extract the subject opinion holder and polarity or the degree of positivity and negativity from the text and identify sentiment additionally other Scopes including document paragraph sentence and Subs sentence level can be used for sentiment analysis after seeing what is sentiment analysis let's discuss some types of sentiment analysis so the first one is find grain s sentiment analysis by segmenting sentiment each phrase or clause in a sentence is broken down and examined in relationship to the others simply said you can tell who review a product and what topics a person specifically discuss in their feedback and the second one is emotion detection instead of identifying positivity and negativity emotion detection recognize particular emotions example could include shock rage grief frustration and happiness and the third one is intent based analysis in addition to identify opinions in a text the intent based analysis also identifies behaviors for instance a frustrated online command about changing a battery can motivate customer care to get in touch to address that particular problem and the last one is expect based analysis collects the precise component that is being mentioned either favorably or unfavorably for instance a client May write in a product review that the battery life is too short the system will then respond that the battery life is the main complaint and not the product as a whole so after seeing types of sentiment analysis let's see some application of sentiment analysis organization can employ sentiment analysis Technologies for a number of purpose such as determining brand popularity reputation and awareness at a particular period or overtime customer service requests into categories and the next one is monitoring consumer response to new improvements or products and the fourth one is identifying the demographics or Target market and the fifth one is determining the effectiveness of a marketing effort so here is one question for you guys I will give you one minute for this you can comment or you can give answers in chat section so I can see if the answers given by you are right or wrong so the question is is text analytics also refers to as text mining I'm repeating again is text analytics also refers to as text mining the options are true or false so let us know your answer in comment section below or in chat section so I'm starting timer of 1 minute just type your answer in comment section or in chat section do let me know your answers so I'm starting timer of 1 minute I kindly ask that everyone take part in this to make this live session exciting so the question is is txt analytics also refers to as text mining 35 seconds are left do let me know your answers I kindly ask that everyone take part in this to make the live session exciting two one 20 seconds left you can comment or you can give your answer in chat section so I can see if the answers given by you are right or wrong so time up guys so after the allotted time has passed Those Who provided the correct response will receive a response and those Who provided the incorrect response will receive one okay now let's move to our programming part to perform sentiment analysis using Amazon customer data set so first we will open command prom to write command to open Jupiter notebook so here we will write Jupiter notebook press enter okay it's Jupiter notebook so this is the landing page of jupyter notebook and here you can select new python file it's loading so this is how the jupyter UI looks like so at first we will import some bunch of major libraries of python which will help us in mathematical fun functioning so the first one is numpy so first I will write numpy import numpy s NP NP is a python Library used for working with arrays it also has functions for working in the domain of linear algebra and matrices it is an open- source project and you can use it freely numai stand for numerical python so the next one is pandas I will write here import pandas s PD so pandas is a software Library written for a Python programming language for data manipulation and Analysis in particular it offers data structures and operation for manipulating numerical tables and Time series after that we will import import nltk so the natural language toolkit nltk is a Python Programming environment for creating application for statical natural language processing NLP for parsing classification stemming tagging and semantic reasoning it includes text processing libraries so after that we will import from nltk dot sentiment do Vader import sentiment density analyzer okay there is no colum yeah so the Lexicon and Rule based sentiment analysis tool Vader verer means Valance aware dictionary and essential reasoning is a customized precisely to Sentiments expressed on social media Vader makes use of variety of a sentiment lexicon is a collection of Lex elements such as words that are often classified as either positive or negative depending on their santic orientation Vader not only informs us of the positivity and negativity scores but also of the sentimentally of each score so the next one is import re python re module fully supports regular expression similar to those for in Pearl if a regular expression compilation or use error occurs the re modules raises the expression re two key functions that are used to manage regular expression will be covered so re means regular Expressions so the next one is from X blob import blob okay so a python 2 or three package called text blob is used to process textual data it offers a straightforward API to explore typical natural language processing NLP tasks like part of speech tagging noun phrase extraction sentiment analysis classification translation and more so the next one is import words from word cloud a word cloud commonly refers to as a tag cloud is an image of words popular words and phrases are highlighted using Cloud creators depending on their frequency and importance they give you immediate straightforward visual insights that can Inspire more through analysis and the next one is import cbor as SNS yeah so like an open- Source python Library based on mat plot Li is called cbon it is utilized for data exploration and data visualization where data frames and the Panda's Library cbon function with ease so the next one is import matplot lip dot P plot as PLT okay yeah seems good for Python and its numerical expression numpy met plot lib is a crossplatform data visual Iz ation and graphical charting package as a result it presents a strong open- Source substitute for matlb the API for M plot lib allow programmers to incorporate graphs into GUI applications and the next one is import pu links cuff L KS as CF perfect another pyth module called cuff link links plotly with panda so that charts can be easily created on data frames in essence it function like a plug-in so next in line at blot lip at blot lip so you can enable inline plotting by using the magic function percentage M plot Li inline which causes the plots and graphs to appear below the cell where you plotting commands are entered similar to a jupyter notebook it offers backend interactivity in front ends the next we will import what you can like we can import okay from shortly do offline import init uncore notebook underscore mode comma I plot okay in it uncore notebook notebook mode get through yeah cf. goore offline so yeah in order to display the plot inside the notebook you need to initiate plotly notebook mode as follows like this keep rest of the script as it is like it and run the notebook cell by pressing you can shift plus enter graph will be displayed offline inside the notebook so basically it is used for the offline graphing you can see okay so next we will import from plotly do subplots subplots are important import make underscore subplot okay yeah the non-leaf nodes in this figure schema are represented by a hierarchy of python classes that was automatically constructed and is found in the plot do graph objects which is normally imported as go these class instance are refers to as graph object the plotly defined fundamental classes okay so here we will import some like uh warnings you can [Music] say like to get not error so we will import let me give a line space import warning warnings yeah warnings dot filter warning ignore warnings. one and this uh like will not show and let me add something PD do set option display do maxcore Lums okay import import words from wordcloud okay there's an error let me check what's the error okay import this wordcloud from wordcloud it's capital and W is also Capital here will be Capital let it put is small only from wordcloud okay my bad here from wordcloud from wordcloud import wordcloud okay yeah seems good so where were we yeah in warnings despite the notification the software continues to execute to display warning messages utilize the warn found in the warning module the python built-in class expection is a subass of the warning module which is used to display warning messages import warnings okay for seeing full data every single columns like okay there is error C Bor no module as C so s will be small got it okay line this only like something like that in line okay any errors no I guess no error okay yeah for seeing full data like every single Columns of PD so here we are using this pd. set off option so we can use for it so moving forward let's import data set so for importing data set we will write here uh like DF equals to PD do read CSV Okay add data set name yeah so so you can download data set from the description box below so no worries of for that let me run this okay so like here is PD is for Panda's Library read is used for reading the data set from the machine and CSV is used for the type of file which you want to read so let's see our data so here I will write DF dot yeah so if you want to like if you want to see top five rows of your data set you can use head and if you want to uh like see five rows of your data set you can use tail in instead of head this one okay so this is our data set with only five rows if you want to see full data set you can like from starting from zero and ending it 4914 rows okay 4915 rows and 12 columns right so let me give something like this for the better visual yeah okay so moving ah ahead let's sort this Wilson bound column to ascending order for better analysis so I'm talking about this Wilson lower bound column this one for it's not 0 0 0 like in all the rows so let me show you so here you have to write like DF equals to DF do sort values Wilson spelling should be correct lower pound I make it to ascending order ascending sorry descending order so BF do drop named okay un name colum zero okay in place in equals to True comma XIs = to 1 okay so let's see of data so here you can see the data is sorted in as in as ascending order okay so the provided row or column is eliminated via the drop function the drop method eliminates the specified column when the column AIS is specified the drop method eliminates the designated row when the row AIS is supplied okay so I hope you guys understand till here if if you have any queries or like any queries regarding any code or there is something question till here just put as in comments our team will shortly provide you the correct solution okay so moving forward let's make a function for missing values so okay so I will write here like DF missing values analysis analysis okay it's DF for function we use DF yeah EF columns to co for columns for or Co n DF dot columns if DF column is null sum great than zero underscore missals to DF n underscore columns score is null sum dot sort values sortore values like in ascending ascending ascending equals to true okay like some like this and this that's fine I guess yeah so here I will write ratio underscore equals to DF nacore columns underscore dot isnull same thing that sum divided by F do shape 100 sort values standing to True fine okay okay th is okay and the DF part that sort value is equals to sending to True fine so here we will write missing underscore DF = to PD Dot conat dot round ratio underscore comma two perfect their XIs will be one okay is equals to that will give like total where I will write missing values missing values okay comma ratio thing underscore DF equals to PD dot data frame SC DF okay so let me return DF let me create one more function for like check data frame so DF Che underscore data frame okay okay equals to 5 comma tail equals to 5 so here I will write print okay okay dot center2 comma again for row Sprint r craes dot format what format do shape Z so for columns print do shape PR and types like of types of data Dot Center same as like that [Music] 882a in DF do D types Center just printing this this like you will see don't worry just stay with me you will see the beautiful result okay print values print missing values uh analysises like DF okay then M again get it values like there many how many duplicated values are there so dot same for the center D again this one print have dot duplicated don't worry I will explain you like line by line just stay with me let me write the whole code first print tiles like I'm printing multiple of things so just be with me dot ENT 82 comma EF Dot onti 05 comma one okay check data frame hoping like there will be no error yeah so you can see now like multiple information regarding data set is printed so this shape is for like how many rows and columns present in the data set like in types you can say like every column type like object in 64 float 64 and many more are like present so like you can see review text is like object and dat difference is N64 and wisell lower bound is float 64 okay so total missing values so total missing values are like uh one from here and one from here like one in from reviewer name and other is from reviewer text and like there is no duplicated values in our data set and here you can see so like after that quantiles a quantile defines a particular part of the data set that is a quantile determines how many values in the distribution are above or below a certain limit special quantiles are the like quartile quarter and you can say quintile that the fifth one and the percentile which is from 100 okay so quantile is for like quarter or the you can say the fifth one so why I wrote this 83 82 and this for this purpose for making this something look cool that is why okay and yeah so moving forward let's see unique values and in each column okay so I will write Dev check CL dat frame unique FAL to PD dot data frame variable variable frame columns comma classes colon data frame I I will write like an unique unique U or I in data frame columns okay like n unique [Music] Q DF equals to sort values classes ascending equals to false okay n unique underscore dfal to n unique EF do reset index drop equals to true return and unique qu and unique go BF check underscore class which one DF so let me run this first yeah it's running you can see here okay there is an error classes okay Jack class DF is saying that this is okay C is capital c is small let me run now yeah so here you can see the every column having unique values are sorted in descending order like review text have like more unique values so it come first and then reviewer name reviewer time day difference Wilson lower bound score average then goes to overall okay moving forward let's do some categorical variable analysis for overall yes so this will be like a amazing something amazing is going to be happen so constants equals to so I'm giving here like some color values so B3 4 d 22 e b 00 C okay so you can change you can like write whatever you want color hash oh I didn't write hashtag here hashtag three come on 0 C 992 EB EB z c and then D you like five right okay so DF let's make function Okay small DF categorical variable B okay DF comma column okay categorical variable summary yeah perfect so let make some figures so make subplots dos rows equals to 1 comma calls = to 2 let me write from here like subplot metals to count plot like one for percentage percentage okay so specs space equals to okay like type colon colon It's why okay same type colon here domain figure. add and then [Music] Trace dot bar that one bar and one that P chart okay y = to DF columns name value values do to list forx you can L string yeah St Str I or I in column name do value counts index TT equals to DF colum name value counts and then bracket dot values dot to list okay go for text font the size should be 14 is enough I guess so column M go name position so I'm doing for like making chart so stay with me to see that amazing chart okay so show Legend color go to constants so these are the like colors so I'm giving here value like VAR variable name con strs line equals to it's hashtag DB e 6 EC just one okay small okay let me just like go over through everything is correct or not for the column name value count index yeah it's fine text equals to DF column name do value counts do values to to list okay text font equals to T size then okay for size is enough next name equals to column name and text position equals to Auto and show Legend false and Mark dict like color equals to constant okay F line equals to color and this yeah this okay cool row = to 1 comma column equals to one seems good for pie chart we will make figure do add so this is done for the like line chart not line chart that bar chart and this I'm doing for the py chart okay Trace go Dot by y columns column name value is use to DF again that column name value count counts and Dot values text font T size I will prefer 18 is enough exposition like Auto is fine so Legend to no need of Legend false m equals to column name column name let's do like d then colorss to constants one one column = to two like in second okay you will see and you will get to know don't worry so figure dot update layout equals to colum n five comma X should be on Center then y anchor be on top let equals to shortly okay I plot figure let me run this okay like call equals to two now let's see the chart of like column overall one so we will write here category okay variable variable summary and then data frame name DF comma overall okay let's run this okay it's loading here it's loading yeah you can see this is this is our bar plot and this is our pie chart so like this is overall uh from overall you can say that column so like percentage bar and that bar plot so this is how our like pie chart and bar chart looks like I hope you guys understand till here if you have any questions or any query regarding adding any code okay like any code any sort of code so just put as in comments our team will shortly provide you the correct solution so moving forward our goal is to rank the com comments by the sentiment analysis so we don't get hung up on the other details I have sorted this data set according to certain rules before so moving forward we want to clean our data so let's see the sample data for cleaning so DF dot review text okay dot head let me run this so so this is how sample data looks likees so let's see like data full data for this okay let me see for what this I guess okay or like any particular row you can say I can take okay so I will write review underscore example example equals to DF dot VI view text uh I will take a random 2031 my guys okay so review underscore example so this is like at 2031 there is one review text okay so in that the whole data is here like uh like so my lovely wife bought me a Samsung Galaxy Tab for father this this this this like to find the exact model number this is is a long you know comment I got the this this this it works like a charm okay so after seeing this data what we going to do is like we will clean it from punctuation and numbers using Rex Rex what regular expression okay so here I will write review underscore example example okay equals to R do sub and what I will do is okay I will explain don't worry small L to capital A A to Z okay comma then space comma review underscore example exam but add sorry yeah give you so here we used re for regular expression okay and so all the punctuation and numbers are removed from particular data so let me check first okay this A to Z then A to Z okay fine so like all the punctuation and numbers are removed from particular data I will now convert the text to all lower case our machine learning algorithm recognize words that start with the capital letters as a different word and we will convert them to lower case so thus our machine learning algorithm will not perceive words that start with a capital letter as a different word so what I will do is here I will write review sample sample equals to G View again example lower I will split it split okay example so here you can see we have converted all the words in the lowercase letter by using lower function and why they are coming in new lines because we are splitting them using split function okay if you see only did lower case for single row like we did for the like 20 I guess 2 3 1 okay yeah so let's make all row same in lower case so I will write here RT equals to Lambda X St Str X TF view text E capital text DF view text text okay equals to DF view yeah do lower we have do head see all the data is in like lower case so I printed just top 10 rows and in like in reviews text column all the are in the lower case now so I hope you guys understand till here if you have any question or any query regarding any code or question till here like any code just put as in comments our team will shortly provide you the correct solution and if you need this full code you can comment down below okay so now the time is come to do sentiment analysis because all the data is sorted so what will I write here let me do it little bit up yeah so I will write from yeah from Vader sentiment do Vader both sentiment intensity analyzer okay have polarity subjectivity subjectivity to DF view text apply PD do Series yeah here you WR text blob then again text Dot like sentiment let me run for Loop for index comma Row in DF G VI text but it items 4 sentiment density analyzer polarity scores row netive score or neutral I will write score if for positive if this is something crazy if negative is greater than positive then DF do location index comma sentiment equals to negative netive okay then El if positive is greater than negative then in this case DF DOT log index comma sentiment to positive B capital else EF dot LC X timent to neutral okay so like this is for sentimental analysis and text blob this thing like text blob this blob okay it's blob text blob will return polarity and subjectivity polarity indicates your mood that is whether it is positive or negative or neutral it returns a value between 0 and one the closer to one the more positive and the closer to zero is more negative okay let me run this it will take a little bit time because this sentiment okay it is getting error like module Panda has no attribute series okay which One texx X sentiment okay here what is the thing s is capital let me run now it is executing like fit for a while wait so it will take time to run because like because of this sentiment intensity analyzer it take times G sentiment intensity analyzer okay spelling mistake again why index index is not defined sorry again index it will run I guess because everything seems good negative positive and this else loading so I have to wait like for like 2 minutes more I guess still loading guys it take time sentiment intensity analyzer so let me tell you again sentiment analysis like this text blob text blob will return polarity and subjectivity so polarity indicates your mood that is whether it is positive it is negative or it is it will be neutral it returns a value between 0 and 1 the closer to one the more positive the closer to Z more negative okay I guess is still running so it will take time so let me write code for that uh 20 interpretation I repeat moving forward let's identify the 20 interpretation now we can include the positive negative and neutral status of the command okay then DF DF sentiment equals to positive thought lose lower sending equals to false okay dot head it's five if I will run it may give errors to me because it's still running still running here it's still loading so let let let me write code for that unbalanced data problem so like we will category that data into positive negative and neutral okay so categorical variable summary DF ENT yeah it came like this because of this wait I will tell you yeah so here you can see the pie chart distributed in negative positive and neutral emotions okay or you can say ratings and here you can see the bar chart also so and you can see the percentage distributed in each like in positive negative and neutral one so I hope you guys must understood the concept of sentiment analysis how we can implement it using python if you're looking for a course that covers everything from the fundamentals to Advanced Techniques then accelerate your career in Ai and machine learning with a comprehensive post-graduate programming Ai and machine learning boost your career with this Ai and ml course delivered in collaboration with the Peru University and IBM learn in demand skills such as machine learning deep learning NLP computer vision reinforcement learning generative AI prompt engineering chat GPT and many more you'll receive a prestigious certificate and ask me anything sessions by IBM with five cap stones in different domains using real data sets you will gain practical experience master classes by PD faculty and IBM experts ensure top-notch education simply learns job assist help you get noticed by Leading companies this program covers statistics python supervised and unsupervised learning NLP neural networks computer vision G scas tensor flow and many more such skills admission to this post-graduate program in Ai and machine learning requires a bachelor's degree with an average of 50% or higher marks along with the basic understanding of programming Concepts and Mathematics and candidates with two plus years of work experience are preferred to enroll in this scho so enroll now and unlock exciting Ai and machine learning opportunities the link is mentioned in the description box below Jupiter notebook and press enter this is the landing page of jupyter notebook and it select open new python file here we will go new python file this is how the python G look likees Jupiter python going to look like so here first we will change the name WhatsApp group okay yes here first we will import some major libraries first we will import import R regular expression and then import projects I will explain one by one so don't worry import Hondas report numai as NP and the fifth one is import Express as PX okay then from collection import counter okay and import M Li dot P plot as PLT okay this is for the graph and from OS import okay and the next one is from p import then this is the major One Import date time and this one is the important word cloud Port what cloud M top wordss upwards and then image coloror okay then person M plot [Music] Li then press enter let's see uh collection should be [Music] collections from wordcloud import wordcloud Stop words image color generator yeah so cannot import name but cloud okay sorry it should be small okay so rest is fine so numai is a python Library used for this one lpai is a python Library used for working with arrays it also has a functions for working in the domain of Lan algebra and matrices it is an open source project and you can use it freely numai stand for numerical Python and the panda panda is a software Library written for the Python programming language for data manipulation and Analysis in particular it offers data structure and operations for manipulating numerical tables and time series this m plot lip P plot so for Python and its numerical extension number by met plot lip is a crossplatform data visualization and graphical charting package as a result it represent a strong open source substitute for MPL the API application programming interface format probably allow programmers to incorporate graphs into giu application okay then this re regular expression so a regular expression specify a set of string that matches it the function in this module let you check if a particular string matches a given regular expression or not or you can say if if a given regular expression matches a particular string which come down to the same thing okay then rex rex or regular expression is a sequence of character that forms a search platform okay or you can say search pattern reg can be used to check if a string contains the specified search pattern or not and this PX okay plotly Express this plotly Express module uses imported as PX contains functions that can create entire figure at once okay and is refers to as a plotly Express or PX blly Express is a built-in part of the daily library and it is recommended starting point for creating the most common figures okay then there is one more this counter Library so the counter is a dict subass for counting hashable objects it is a collection where elements are stored as a directory keys and their counts are stored as a dictionary values okay counts are allowed to be any integer value including zero or negative counts okay then there is one more this word cloud okay this is the kind of major one to visualize so not major one but it looks good at the end you will see so word cloud is a data visualization technique used for representing Text data in the which size of each word IND at its frequency or importance significant tal data points can be highlighted using wordcloud wordclouds are widely used for analyzing data from social network websites okay then stop wordss a stop word is a commonly used word such as the uh n in that a search engine has been programmed to ignore both indexing entries for searching and when retrieving them as a result in a search quer Okay so so let's proceed further and create the various function for the author for the date and time for the get data time for the you know data point for the Android and for the iOS like things like that okay so here I will addite DF starts with it and time okay it starts with data and time okay I will write here pattern right pattern equal to give this I will write here 0 okay 0 to 9 plus right here again these are for the numbers okay plus M okay or M okay so result equals to R do match if result done or else okay let me give some space for the better visual yeah so now we will find the author okay we'll create function for the author def find auor okay then here I will write s equals to dot it then of s was equals 2 okay uh else return fals now what we we'll create a function to get data point for the Android line okay so here I will write def get data point right 9 and date time equals to split line and date comma time = to date time do split okay then message equals to dot join to like split line wa line col find auor which let message split go to split Mage question do join with message order and 8 comma comma so this is the uh for the getting data point for the Android okay line with line this for the uh I will like copy pasted and this is for the the this one iOS okay so in this we are doing the same like split line the same we did for the you know Android okay so moving forward we'll create a function for the split count okay so for that to WR text here that I can like emoji Lambda then y then again English do Keys okay X to r dot compile I can write join I Dot Escape em or em okay emoji xcore set dot find all okay Enter yeah so now we will import our data okay so we will make a list to keep a track of data so it can be used for the you know Panda's data frame and we we'll create a separate column for the date Time author and messages okay so the major thing is okay let's import data first then I will tell you okay so this is the list to keep a track of data so it can be used for the data frame Panda's data frame okay so I will WR Converse conversation path so my file name is WhatsApp chat is Captain get B and placed student 22 this is my file so how you can export your chat from your WhatsApp group okay so this is the data this is my personal data so I can't give you so how you can import your text your chat okay so open any group just click on the top right corner okay three dots then you can go into more and then you have can explore chat including media and excluding media your wish okay so you can download you can send to the mail you can upload in the drive then you can use in your machine learning project okay so I will write these all the steps in the description box below so don't worry moving forward let's do open conversation coding equs to UTF eight then device question and first to FP do readline print first if or else Dev to and P dot read line message date comma time comma author n message contain four things dat Time author message okay so for the message this is the message buffer and for the there the DAT time and author okay so while dot read line not line devices device equal to iOS strip it and time iOS F start with date time iOS line so here I will as data Dot okay so this is the long it here so so what we did uh we created if devic is IOS so the line history will come and the function starts with date time and iOS this one we created already okay it will capture the date Time author and message okay then at the end the message buffer will append okay for the both okay this past data is for the we created for the data frame Panda data frame okay list so let's move and see okay while TR okay start with pattern is not defined Ern pattern so okay data point Android line maybe the spelling mistake as you can see Point get data point Android a is Capal and this is what line get data point right me copy it from here copy and P okay then let me run it split line okay split line split line is not def okay let me get back to you so as you can see everything is sorted so here you can see this is the group name okay B and play 2022 and this is date and this is time and this is the guy who created the group okay okay so here I will write if device plus equal to [Music] Android and dfal to PD dot data frame as data columns equals to eight [Music] then DF dat I'm creating the columns PD do to dat time we have okay then URL are then https plus okay so again I have ADD DF equals to for the URL count pattern okay U should be Capital comma X erl string length so is find then else here we are like initializing the data frame P data frame so DF equals to PD dot data frame data [Music] comma columns date comma time my yeah so this is the Pand data frame we have analized that DF = to DF do drop menu DF date and apply it time comma format is year okay percentage year then okay and url url pattern equals to yes canot assign to l copy it and paste it running so now let's see the data set so DF do head 20 okay first 20 rows of the data frame as you can see see uh this guy doctor uh something read so uh he sent the message greetings of all the DAT from this this this and in this there is one URL okay and this is the date and this is time when he sent or when she sent Okay so this there are three links in this two links okay is this related to placement this guy send okay so these are the top 20 the number of links you can see in the particular message 2 three and in the 0 0 this is the media okay so I downloaded uh the chat without media okay you can chat can come using media yeah let's see the information data so this is you can see we have five column name date Time author message URL account okay soal count date time this is object and this is object and URL count as N64 okay let's see all the author names only author names so here I will write DF dot author dot unique my bad these are the authors all the authors number of members you can say in particular uh group okay the all are the unique authors so moving forward let's see the you know like let's drop all the null values so for that we have to write DF do equals to DF do drop n so top end is done then so these are the some basic you can say the function of machine learning and python okay and moving forward let's see the total messages total this to DF do shape to print 87 total messages okay then media messages yeah which yeah zero print media why this is zero I already told you like I export chat without media okay so you can export with media too okay so write here URL done okay so this is the you can say the reject pattern this one okay this is the reject pattern and with the start with the STD https and the second one this this URL count return URL which matches with pattern and counts it through the all the Lambda function okay now I will write one more thing to NP do sum yeah what uh this will do this is sum up all the links okay so print links as you can see there are 47 links in the group chat okay so moving forward let's uh let us separate and media messages and the text message okay so this is the is the code for that so link message go to URL account okay deleted messages and this message was deleted okay both the statement and this media omitted this image okay this video for the video and the stickers you send we send okay then the drop and deleted message and the link message okay so for that let's press enter and message d F dot okay so now you can see we deleted dropped all the you can say the deleted messages or the link messages okay as you can see 87 now 56 only okay so let's calculate words and letters okay for that I have WR messages _ DF then account that account equals to messages BF and message can okay I will write here messages DF then word count isore DF which Dot okay messages EF then message count one okay so here you can see I wrote here split so word count will be count as like the space you can see space between the words okay splitting sentence with the white space you can say okay let's see Sage DF dot P 10 press enter here I have to write messages yeah so now you can see this author send this letter count 25 and the word count is this and the message count is one only so this package test dot things and this is things this doctor Argent Hing work from home kind of things and the word letter count is 2011 then one count 31 okay and this is sometime hex of you things and okay so these are the words and letters we calculate okay now we will see the stats of like author wise okay for that I will write the code and we'll get back to you after that so this is the code for the you can say stats okay there's an error word count where where where where where what message what count okay so these are the all the stats per user per author you can say message sent and words per message and media media sent is zero because like I haven't you know included media in my chat so that's why okay word per message this and these all the stats you can see and let me explain you the code so this is for the like filtering you know filtering out the message for the particular user this message DF message DF other friends and for this like rdf will contain messages of only one particular user so we are printing that okay and this shape this one so uh shell will print the number of rows which IND thatly mean the number of messages okay so this word count this word count so word count contains all the total words in one messages sum of all the words divided by total message will yield words words per message okay so and this is for the media messages this is zero and this is for the link consist of the total links okay so this is all the data so now moving forward times of the day let's see the times of the day so in which time the most messages gone like this so messages for DF that is time dot value do paragraph Okay horizontal okay then PLT do text label and pl. yel 10 times of the day at which the most number of message sent okay this a top 10 uh uh okay so because of okay let me do like this now it's good time a number of messages okay this is it what I can do this number of messages and the top 10 times most number of message now the time is came to see the word cloud okay now we'll make the word cloud so for that I will let me give text join join you can say review print this much words all the okay dot format be length and then stop wordss one of my favorite set stop words then we'll generate the world Cloud this is to generate the World Cloud okay sorry word cloud cloud upwards backround white do so we'll display the generated image and the pl uh using the You Know M plotly way so pl. figure SI equals to set it bilinear by linear and then L do XIs lt. show stop words and rate can see there are 9 to 35 words in all the messages so these are the sums like Gmail password pressure and this ready test email and this is student and see something like this these are the world Cloud okay so what if I want to see the particular author name World cloud like how many messes he sent and all like this so this is the doctor okay I took from here see Dr Reda and this Shand do something and this you can take the number also no issues so let's see this is author Dr Rapa he sent hiring job gen fresher student and this is Anusha ready join and same the Chand ready join okay so this is how you can analyze and you can see data uh you can analyze your whatsappp data okay I will add all the you know steps how to you how you can explore the chat if you're looking for a course that covers everything from the fundamentals to Advanced Techniques then accelerate your career in Ai and machine learning with a comprehensive post-graduate program in Ai and machine learning boost your career with this Ai and ml course delivered in collaboration with the perdu university and IBM learn in demand skills such as machine learning deep learning NLP computer vision reinforcement learning generative AI prompt engineering chat GPD and many more you'll receive a prestigious certificate and ask me anything sessions by IBM with five capstones in different domains using real data sets you will gain practical experience master classes by P faculty and IBM experts ensure top-notch education simply learns job assist help you get noticed by Leading companies this program covers statistics python supervised and unsupervised learning NLP neural networks computer vision G scas tensorflow and many more suchar skills admission to this post-graduate program in Ai and machine learning requires a bachelor's degree with an average of 50% or higher marks along with the basic understanding of programming Concepts and Mathematics and candidates with two plus years of work experience are preferred to enroll in this school so enroll now and unlock exciting Ai and machine learning opportunities the link is mentioned in the description box below hello everyone welcome to the session I'm Mohan from Simply learn and today we'll talk about interview questions for machine learning now this video will probably help you when you're attending interviews for machine learning positions and the attempt here is to probably consolidate 30 most commonly asked uh questions and to help you in answering these questions we tried our best to give you the best possible answers but of course what is more important here is rather than the theoretical knowledge you need to kind of add to the answers or supplement your answers with your own experience so the responses that we put here are a bit more generic in nature so that if there are some Concepts that you are not clear this video will help you in kind of getting those Concepts cleared up as well but what is more important is that you need to supplement these responses resp es with your own practical experience okay so with that let's get started so one of the first questions that you may face is what are the different types of machine learning now what is the best way to respond to this there are three types of machine learning if you read any material you will always be told there are three types of machine learning but what is important is you would probably be better of emphasizing that there are actually two main types of learning which is supervised and unsupervised and then there is a third type which is reinforcement learning so supervised learning is where you have some historical data and then you feed that data to your model to learn now you need to be aware of a keyword that they will be looking for which is labeled data right so if you just say pass data or historical data the impact may not be so much you need to emphasize on labeled data so what is labeled data basically let's say if you're trying to do train your model for classification you need to be aware of for your existing data which class each of the observations belong to right so that is what is labeling so it is nothing but a fancy name you must be already aware but just make it a point to throw in that keyword labeled so that will have the right impact okay so that is what is supervised learning when you have existing labeled data which you then use to train your model that is known as supervised learning and unsupervised learning is when you don't have this labeled data so you have data it is not labeled so the system has to figure out a way to do some analysis on this okay so that is unsupervised learning and you can then add a few things like what what are the ways of Performing uh supervised learning and unsupervised learning or what are some of the techniques so supervised learning we we perform we do uh regression and classification and unsupervised learning uh we do clustering okay and clustering can be of different types similarly regression can be of different types but you don't have to probably elaborate so much if they are asking uh for uh just the different types you can just mention these and just at a very high level you can but if they want you to elaborate give examples then of course I think there is a different question for that we will see that later then the third so we have supervised and we have supervised and then reinforcement you need to provide a little bit of information around that as well because it is sometimes a little difficult to come up with a good definition for reinforcement learning so you may have to little bit elaborate on how reinforcement learning works right so reinforcement learning works in in such a way that it basically has two parts to it one is the agent and the environment and the agent basically is working inside of this environment and it is given a Target that it has to achieve and and uh every time it is moving in the direction of the target so the agent basically has to take some action and every time it takes an action which is moving uh the agent towards the Target right towards a goal a Target is nothing but a goal okay then it is rewarded and every time it is going in a direction where it is away from the goal then it is punished so that is the way you can a little bit explain and uh this is used Prim or very very impactful for teaching the system to learn games and so on examples of this are basically used in alphao you can throw that as an example where alphao used reinforcement learning to actually learn to play the game of Go and finally it defeated the co world champion all right this much of information that would be good enough okay then there could be a question on overfitting uh so the question could be what is overfitting and how can you avoid it so what is overfitting so let's first try to understand the concept because sometimes overfitting may be a little difficult to understand overfitting is a situation where the model has kind of memorized the data so this is an equivalent of memorizing the data so we can draw an analogy so that it becomes easy to explain this now let's say you're teaching a child about some recognizing some fruits or something like that okay and you're teaching this child about recognizing let's say three fruits apples oranges and pineapples okay so this is a a small child and for the first time you're teaching the child to recognize fruits then so what will happen so this is very much like that is your training data set so what you will do is you'll take a basket of fruits which consists of apples oranges and pineapples okay and you take this basket to this child and uh there may be let let's say hundreds of these fruits so you take this basket to this child and keep showing each of this fruit and then first time obviously the child will not know what it is so you show an apple and you say hey this is Apple then you show maybe an orange and say this is orange and so on and so and then again you keep repeating that right so till the basket is over this is basically how training work in machine learning also that's how training works so till the basket is completed maybe 100 fruits you keep showing this child and then the process has happened the child has pretty much memorized these so even before you finish that basket right by the time you are halfway through the child has learned about recognizing the Apple orange and pineapple now what will happen after halfway through initially you remember it made mistakes in recognizing but halfway through now it has learned so every time you show a fruit it will exactly 100% accurately it will identify it will say the child will say this is an apple this is an orange and if you show a pineapple it will say this is a pineapple right so that means it has kind of memorized this data now let's say you bring another basket of fruits and it will have a mix of maybe apples which were already there in the previous set but it will also have in addition to Apple it will probably have a banana or maybe another fruit like a jack fruit right so this is an equivalent of your test data set which the child has not seen before some parts of it it probably has seen like apples it has seen but this banana and jack fruit it has not seen so then what will happen in the first round which is an equivalent of your training data set towards the end it has 100% it was telling you what the fruits are right Apple was accurately recognized orange were was accurately recognized and pineapples were accurately recognized right so that is like a 100% accuracy but now when you get another a fresh set which were not a part of the original one what will happen all the apples Maybe it will be able to recognize correctly but all the others like the jack fruit or the banana will not be recognized by the child right so this is an analogy this is an equivalent of overfitting so what has happened during the training process it is able to recognize or reach 100% accuracy maybe very high accuracy okay and we call that as very low loss right so that is the technical term so the loss is pretty much zero and accuracy is pretty much 100% whereas when you use testing the will be a huge error which means the loss will be pretty high and therefore the accuracy will be also low okay this is known as overfitting this is basically a process where training is done training process is it goes very well almost reaching 100% accuracy but while testing it really drops down now how can you avoid it so that is a extension of this question there are multiple ways of avoiding overfitting there are techniques like what do you call regularization that is the most most common technique that is used uh for avoiding overfitting and within regularization there can be a few other subtypes like Dropout in case of neural networks and a few other examples but I think if you give example or if you give regularization as the technique probably that should be sufficient so so there will be some questions where the interviewer will try to test your fundamentals and your knowledge and depth of knowledge knowled and so on and so forth and then there will be some questions which are more like trick questions that will be more to stump you okay then the next question is around the methodology so when we are performing machine learning training we split the data into training and test right so this question is around that so the question is what is training set and test set in machine learning model and how is the split done so the question can be like that so in machine learning when we are trying to train the model so we have a three-step process we train the model and then we test the model and then once we are satisfied with the test only then we deploy the model so what happens in the train and test is that you remember the labeled data so let's say you have thousand records with labeling information now one way of doing it is you use all the Thousand records for training and then maybe right which means that you have exposed all this thousand records during the training process and then you take a small set of the same data and then you say okay I will test it with this okay and then you probably what will happen you may get some good results all right but there is a flaw there what is the flaw this is very similar to human beings it is like you are showing this model the entire data as a part of training okay so obviously it has become familiar with the entire data so when you're taking a part of that again and you're saying that I want to test it obviously you will get good results so that is not a very accurate way of testing so that is the reason what we do is we have the label data of this thousand records or whatever we set aside before starting the training process we set aside a portion of that data and we call that test set and the remaining we call as training set and we use only this for training our model now the training process remember is not just about passing one round of this data set so let's say now your training set has 800 records it is not just one time you pass this 800 records what you normally do is you actually as a part of the training you may pass this data through the model multiple times so this thousand records may go through the model maybe 10 15 20 times till the training is perfect till the accuracy is high till the errors are minimized okay now so which is fine which means that here that is what is is known as the model has seen your data and gets familiar with your data and now when you bring your test data what will happen is this is like some new data because that is where the real test is now you have trained the model and now you are testing the model with some data which is kind of new that is like a situation like like a realistic situation because when the model is deployed that is what will happen it will receive some new data not the data that it has already seen right so this is a realistic test so you put some new data so this dat which you have set aside is for the model it is new and if it is able to accurately predict the values that means your training has worked okay the model got drained properly but let's say while you're testing this with this test data you're getting a lot of errors that means you need to probably either change your model or retrain with more data and things like that now coming back to the question of how do you split this what should be the ratio there is no fixed uh number again this is like individual preference some people split it into 50/50 50% test and 50% training Some people prefer to have a larger amount for training and a smaller amount for test so they can go by either 6040 or 7030 or some people even go with some odd numbers like 6535 or uh 6333 and 33 which is like 1/3 and 2/3 so there is no fixed rule that it has to be something the ratio has to be this you can go by your individual preferences all right then you may have questions around uh data handling data manipulation or what do you call data management or Preparation so these are all some questions around that area there is again no one answer one single good answer to this it really varies from situation to situation and depending on what exactly is the problem what kind of data it is how critical it is what kind of data is missing and what is the type of corruption so there a whole lot of things this is a very generic question and therefore you need to be little careful about responding to this as well so probably have to illustrate this again if you have experience in doing this kind of work in handling data you can illustrate with example saying that I was on one project where I received this kind of data these were the columns where data was not filled or these were the this many rows where the data was missing that would be in fact a perfect way to respond to this question but if if you don't have that obviously you have to provide some good answer I think it really depends on what exactly the situation is and there are multiple ways of handling the missing data or corrupt data now let's take a few examples now let's say you have data where some values in some of the columns are missing and you have pretty much half of your data having these missing values in terms of number of rows okay that could be one situation another situation could be that you have records or data missing but uh when you do some initial calculation how many records are corrupt or how many rows or observations as we call it has this missing data let's assume it is very minimal like 10% okay now between these two cases how do you so let's assume that this is not a mission critical situation and in order to fix this 10% of the data the effort that is required is much higher and obviously effort means also time and money right so it is not so Mission critical and it is okay to let's say get rid of these records so obviously one of the easiest ways of handling the data part or missing data is remove those records or remove those observations from your analysis so that is the easiest way to do but then the downside is as I said in as in the first case if let's say 50% of your data is like that because some column of the other is missing so it is not like every in every place in every Row the same column is missing but you have in maybe 10% of the records column one is missing and another 10% column two is missing another 10% column 3 is missing and so on and so forth so it adds up to maybe half of your data set so you cannot completely remove half of your data set then the whole purpose is lost okay so then how do you handle then you need to come up with ways of filling up this data with some meaningful value right that is one way of handling so when we say meaningful value what is that meaningful value let's say for a particular column you might want to take a mean value for that column and fill wherever the data is missing fill up with that mean value so that when you're doing the calculations your analysis is not completely we off so you have values which are not missing first of all so your system will work number two these values are not so completely out of whack that your whole analysis goes for a toss right there may be situations where if the missing values instead of putting mean maybe a good idea to fill it up with the minimum value or with a zero so or with the maximum value again as I said there are so many possibilities so there is no like one correct answer for this you need to basically talk around this and illustrate with your experience as I said that would be the best otherwise this is how you need to handle this question okay so then the next question can be how can you choose a classifier based on a training set data size so again this is one of those questions uh where you probably do not have like one size fits all on first of all you may not let's say decide your classifier based on the training set size maybe not the best way to decide the type of the classifier and uh even if you have to there are probably some thumb rules which we can use but then again every time so in my opinion the best way to respond to this question is you need to try out few classifiers irrespective of the size of the data and you need need to then decide on your particular situation which of these classifiers are the right ones this is a very generic issue so you will never be able to just by if somebody defines a a problem to you and somebody even if if they show the data to you or tell you what is the data or even the size of the data I don't think there is a way to really say that yes this is the classifier that will work here no that's not the right way so you need to still uh you know test it out get the data try out a couple of classifiers and then only you will be in a position to decide which classifier to use you try out multiple classifiers see which one gives the best accuracy and only then you can decide then you can have a question around confusion Matrix so the question can be explain confusion Matrix right so confusion Matrix I think the best way to explain it is by taking an example and drawing like a small diagram otherwise it can really become tricky so my suggestion is to take a piece of pen and paper and uh explain it by drawing a small Matrix and confusion Matrix is about to find out this is used especially in classification uh learning process and when you get the results when the our model predicts the results you compare it with the actual value and try to find out what is the accuracy okay so in this case let's say this is an example of a confusion Matrix and uh it is a binary Matrix so you have the actual values which is the labeled data right and which is so you have how many yes and how many no so you have that information and you have the predicted values how many yes and how many no right so the total actual values the total yes is 12 + 1 13 and they are shown here and the actual value NOS are 9 + 3 12 okay so that is what this information here is so this is about the actual and this is is about the predicted similarly the predicted values there are yes are 12 + 3 15 yeses and no are 1 + 9 10 NOS okay so this is the way to look at this confusion Matrix okay and out of this what is the meaning conveyed so there are two or three things that needs to be explained outright the first thing is for a model to be accurate the values across the diagonal should be high like in this case right that is one number two the the total sum of these values is equal to the total observations in the test data set so in this case for example you have 12 + 3 15 + 10 25 so that means we have 25 observations in our test data set okay so these are the two things you need to First explain that the total sum in this Matrix the numbers is equal to the size of the test data set and the diagonal values indicate the accuracy so by just by looking at it you can probably have a idea about is this uh an accurate model is the model being accurate if they're all spread out equally in all these four boxes that means probably the accuracy is not very good okay now how do you calculate the accuracy itself right how do you calculate the accuracy itself so it is a very simple mathematical calculation you take some of the diagonals right so in this case it is 9 + 12 21 and divide it by the total so in this case what will it be let's me uh take a so your your diagonal values is equal to if I say d is equal to 12 + 9 so that is 21 right and the total data set is equal to right we just calculated it is 25 so what is your accuracy it is 21 by your accuracy is equal to 21 by 25 and this turns out to be about 85% right so this is 85% so that is our accuracy okay so this this is the way you need to explain draw a diagram Give an example and maybe it may be a good idea to be prepared with an example so that it becomes easy for you don't have to calculate those numbers on the fly right so couple of uh hints are that you take some numbers which are with which add up to 100 that is always a good idea so you don't have to really do this complex calculations so the total value will be 100 and then diagonal values you divide once you find the diagonal values that is equal to your percentage okay all right so the next question can be a related question about false positive and false negative so what is false positive and what is false negative now once again the best way to explain this is using a piece of paper and Pen otherwise it will be pretty difficult to to explain this so we use the same example of the confusion Matrix and uh we can explain that so A confusion Matrix looks somewhat like this and um when we just take yeah it look somewhat like this and we continue with the previous example where this is the actual value this is the predicted value and uh in the actual value we have 12 + 1 13 yeses and 3 + 9 12 Nos and the predicted values there are 12 + 3 15 yeses and uh 1 + 9 10 NOS okay now this particular case which is the false positive what is a false positive first of all the second word which is positive okay is referring to the predicted value so that means the system has predicted it as a positive but the real value so this is what the false comes from but the real value is not positive okay that is the way you should understand this term false positive or even false negative so false positive so positive is what your system has predicted so where is that system predicted this is the one positive is what yes so you basically consider this row okay now if you consider this row so this is this is all positive values this entire row is positive values okay now the false positive is the one which where the value actual value is negative predicted value is positive but the actual value is negative so this is a false positive right and here is a true positive so the predicted value is positive and the actual value is also positive okay I hope this is making sense now let's take a look at what is false negative false negative so negative is the second term that means that is the predicted value that we need to look for so which are the predicted negative values this row corresponds to predicted negative values all right so this row corresponds to predicted negative values and what they are asking for false so this is the row for predicted negative values and the actual value is this one right this is predicted negative and the actual value is also negative therefore this is a true negative so the false negative is this one predicted is negative but actual is positive right so this is the false negative so this is the way to explain and this is a way to look at false positive and false negative same way there can be true positive and true negative as well so again positive the second term you will need to use to identify the predicted row right so if we say true positive positive you need to take for the predicted part so predicted positive is here okay and then the first term is for the actual so true positive so true in case of actual is yes right so true positive is this one okay and then in case of actual the negative now we are talking about let's say true negative true negative negative is this one and the true comes from here so this is true negative right nine is true negative the actual value is also negative and the predicted value is also negative okay so that is the way you need to explain this the terms false positive false negative and true positive true negative then uh you might have a question like what are the steps involved in the machine learning process or what are the three steps in the process of developing uh machine learning model right so it is around the methodology that is applied so basically the way you can probably answer in your own words but the way the model development of the machine learning model happens is like this so first of all you try to understand the problem and try to figure out whether it is a classification problem or a regression problem based on that you select a few algorithms and then you start the process of training these models okay so you can either do that or you can after due diligence you can probably decide that there is one particular algorithm that which is most suitable usually it happens through trial and error process but at some point you will decide that okay this is the model we are going to use okay okay so in that case we have the model algorithm and the model decided and then you need to do the process of training the model and testing the model and this is where if it is supervised learning you split your data the label data into training data set and test data set and you use the training data set to train your model and then you use the test data set to check the accuracy whether it is working fine or not so you test the model before you actually put it into production right so once you test the model you're satisfied it's working fine then you go to the next level which is putting it for production and then in production obviously new data will come and uh the inference happens so the model is readily available and only thing that happens is new data comes and the model predicts the values whether it is regression or classification now so this can be an iterative process so it is not a straightforward process where you do the training do the testing and then you move it to production now so during the training and test process process there may be a situation where because of either overfitting or or things like that the test doesn't go through which means that you need to put that back into the training process so that can be an iterative process not only that even if the training and test goes through properly and you deploy the model in production there can be a situation that the data that actually comes the real data that comes with that this model is failing so in which case you may have to once again go back to the drawing board or initially it will be working fine but over a period of time maybe due to the change in the nature of the data once again the accuracy will deteriorate so that is again a recursive process so once in a while you need to keep checking whether the model is working fine or not and if required you need to tweak it and modify it and so on and so forth so net net this is a continuous process of um tweaking the model and testing it and making sure it is up to dat then you might have question around deep learning so because deep learning is now asso assciated with AI artificial intelligence and so on so can be as simple as what is deep learning so I think the best way to respond to this could be deep learning is a part of machine learning and then then obviously the the question would be then what is the difference right so deep learning you need to mention there are two key parts that interviewer will be looking for when you're defining deep learning so first is of course deep learning is a subset of machine learning so machine learning is still the bigger let's say say uh scope and deep learning is one one part of it so then what exactly is the difference deep learning is primarily when we are implementing these our algorithms or when we are using neural networks for doing our training and classification and regression and all that right so when we use neural network then it is considered as deep learning and the term deep comes from the fact that you can have several layers of neural networks and these are called deep neural networks and therefore the term deep you know deep learning uh the other difference between machine learning and deep learning which the interviewer may be wanting to hear is that in case of machine learning the feature engineering is done manually what do we mean by feature engineering basically when we are trying to train our model we have our training data right so we have our training label data and uh this data has several let's say if it is a regular table it has several columns now each of these columns actually has information about a feature right so if we are trying to predict the height weight and so on and so forth so these are all features of human beings let's say we have sensus data and we have all these so those are the features now there may be probably 50 or 100 in some cases there may be 100 such features now all of them do not contribute to our model right so we as a data scientist we have to decide whether we should take all of them all the features or we should throw away some of them because again if we take all of them number one of course your accuracy will probably get affected but also there is a computational part so if you have so many features and then you have so much data it becomes very tricky so in case of machine learning we manually take care of identifying the features that do not contribute to the learning process and thereby we eliminate those features and so on right so this is known as feature engineering and in machine learning we do that manually whereas in deep learning where we use neural networks the model will automatically determine which features to use and which to not use and therefore feature engineering is also done automatically so this is a explanation these are two key things probably will add value to your response all right so the next question is what is the difference between or what are the differences between machine learning and deep learning so here this is a a quick comparison table between machine learning and deep learning and in machine learning learning enables machines to take decisions on their own based on past data so here we are talking primity of supervised learning and um it needs only a small amount of data for training and then works well on lowend system so you don't need large machines and most features need to be identified in advance and manually coded so basically the feature engineering part is done manually and uh the problem is divided into parts and solved individually and then combined so that is about the machine learning part in deep learning deep learning basically enables machines to take decisions with the help of artificial neural network so here in deep learning we use neural so that is the key differentiator between machine learning and deep learning and usually deep learning involves a large amount of data and therefore the training also requires usually the training process requires high-end machines uh because it needs a lot of computing power and and the Machine learning features are or the feature engineering is done automatically so the neural networks takes care of doing the feature engineering as well and in case of deep learning therefore it is said that the problem is handled end to end so this is a quick comparison between machine learning and deep learning in case you have that kind of a question then you might get a question around the uses of machine learning or some real life applications of machine learning in modern business the question May be worded in different ways but the the meaning is how exactly is machine learning used or actually supervised machine learning it could be a very specific question around supervised machine learning so this is like give examples of supervised machine learning use of supervised machine learning in modern business so that could be the next question so there are quite a few examples or quite a few use cases if you will for supervised machine learning the very common one is email spam detection so you want to train your application or your system to detect between spam and non-spam so this is a very common business application of supervised machine learning so how does this work the way it works is that you obviously have historical data of your emails and they are categorized as spam and not spam so that is what is the labeled information and then you feed this information or the all the these emails as an input to your model right and the model will then get trained to detect which of the emails are to detect which is Spam and which is not spam so that is the training process and this is supervised machine learning because you have label data you already have emails which are tagged as spam or not spam and then you use that to train your model right so this is one example now there are a few industry specific applications for supervised machine learning one of the very common ones is in healthcare Diagnostics in healthcare Diagnostics you have these images and you want to train models to detect whether from a particular image whether it can find out if the person is sick or not whether a person has cancer or not right so this is a very good example of supervised machine learning here the way it works is that existing images it could be x-ray images it be MRI or any of these images are available and they are tacked saying that okay this x-ray image is defective of the person has an illness or it could be cancer whichever illness right so it is tacked as defective or clear or good image and defective something like that so we come up with the binary or it could be multiclass as well saying that this is defective to 10% this is 25% and so on but let's keep it simple you can give an example of just a binary classification that would be good enough so you can say that in healthcare Diagnostics using image we need to detect whether a person is ill or whether a person is having cancer or not so here the way it works is you feed labeled images and you allow the model to learn from that so that when New Image is fed it will be able to predict whether this person is having that illness or not having cancer or not right so I think this would be a very good example for supervised machine learning in modern business all right then we can have a question like so we've been talking about supervised and um unsupervised then so there can be a question around semi-supervised machine learning so what is semi-supervised machine learning now semi-supervised learning as the name suggests it falls between supervised learning and unsupervised learning but for all practical purposes it is considered as a part of supervised learning and the reason this has come into existence is that in supervised learning you need labeled data so all your data for training your model has to be labeled now this is a big problem in many Industries or in many under many situations getting the label data is not that easy because there's a lot of effort in labeling this data let's take an example of the diagnostic images we can just let's say take X-ray images now there are actually millions of x-ray images available all over the world but the problem is they are not labeled so the images are there but whether it is effective or whether it is good that information is not available along with it right in a form that it can be used by a machine which means that somebody has to take a look at these images and usually it should be like a doctor and uh then say that okay yes this image is clean and this image is cancerous and so on and so forth now that is a huge effort by itself so this is where semi-supervised learning comes into play so what happens is there is a large amount of data maybe a part of it is labeled then we try some techniques to label the remaining part of the data so that we get completely labeled data and then we train our model so I know this a little long winding explanation but unfortunately there is no a quick and easy definition for semi-supervised machine learning this is the only way probably to explain this concept we may have another question as um what are unsupervised machine learning techniques or what are some of the techniques used for performing unsupervised machine learning so it can be worded in different ways so how do we answer this question so unsupervised learning you can say that there are two types clustering and Association and clustering is a technique where similar objects are put together and there are different ways of finding similar objects so their characteristics can be measured and if they have in most of the characteristics if they are similar then they can be put together this is clustering then Association you can I think the best way to explain Association is with an example in case of Association you try to find out how the items are linked to each other so for example if somebody bought a maybe a laptop the person has also purchased a mouse so this is more in an e-commerce scenario for example so you can give this as an example so people who are buying laptops are also buying the mouse so that means there is an association between laptops and mouse or maybe people who are buying bread are also buying butter so that is a Association that can be created so this is unsupervised learning one of the techniques okay all right then we have very fundamental question question what is the difference between supervised and unsupervised machine learning so machine learning these are the two main types of machine learning supervised and unised and in case of supervised and again here probably the keyword that the person may be wanting to hear is labeled data now very often people say yeah we have historical data and if we run it it is supervised and if we don't have historical data yes but you may have historical data but if it is not labeled then you cannot use it for supervised Lear so it is it's very key to understand that we put in that keyword labeled okay so when we have labeled data for training our model then we can use supervised learning and if we do not have labeled data then we use unsupervised learning and there are different algorithms available to perform both of these types of uh trainings so there can be another question a little bit more theoretical and conceptual in nature this is this is about inductive machine learning and deductive machine learning so the question can be what is the difference between inductive machine learning and deductive machine learning or somewhat in that manner so that the exact phrase or exact question can vary they can ask for examples and things like that but that could be the question so let's first understand what is inductive and deductive training inductive training is induced by somebody and you can illustrate that with a small example I think that always helps so whenever you're doing some explanation try as much as possible as I said to give examples from your work experience or give some analogies and that will also help a lot in explaining as well and for the interviewer also to understand so here we'll take an example or rather we will use an analogy so inductive training is when we induce some knowledge or the learning process into a person without the person actually experiencing it okay what can be an example so we can probably probably tell the person or show a person a video that fire can burn the fing burn his finger or fire can cause damage so what is happening here this person has never probably seen a fire or never seen anything getting damaged by fire but just because he has seen this video he knows that okay fire is dangerous and if fire can cause damage right so this is inductive learning compared to that what is deductive itive learning so here you draw conclusion or the person draws conclusion out of experience so we will stick to the analogy so compared to the showing a video Let's assume a person is allowed to play with fire right and then he figures out that if he puts his finger it's burning or if throws something into the fire it burns so he is learning through experience so this is known as deductive learning okay so you can have applications or models that can be trained using inductive learning or deductive learning all right I think uh probably that explanation will be sufficient the next question is are KNN and K means clustering similar to one another or are they same right because that the letter K is kind of common between them okay so let us take a little while to understand what these two are one is KNN and another is kin K stands for K nearest neighbors and K means of course is the cluster mechanism now these two are completely different except for the letter K being common between them K andn is completely different K means clustering is completely different K&N is a classification process and therefore it comes under supervised learning whereas K means clustering is actually a unsupervised okay when you have K andn when you want to implement KN andn which is basically K nearest neighbors the value of K is a number so so you can say k is equal to 3 you want to implement KN andn with K is equal to 3 so which means that it performs the classification in such a way that how does it perform the classification so it will take three nearest objects and that's why it's called nearest neighbor so basically based on the distance it will try to find out its nearest objects that are let's say three of the nearest objects and then it will check whether the class they belong to which class right so if all three belong to one particular class obviously this new object is also classified as that particular class but it is possible that they may be from two or three different classes okay so let's say they are from two classes and then if they are from two classes now usually you take a odd number you assign odd number to so if there are three of them and two of them belong to one class and then one belongs to another class so this new object is assigned to the class to which the two of them belong now the value of K is sometimes tricky whether should you use three should you use five should you use seven that can be tricky because the ultimate classification can also vary so it's possible that if you're taking K as three the object is probably in one particular class but if you take K is equal to five maybe the object will belong to a different class because when you're taking three of them probably two of them belong to class one and one belong to class two whereas when you take five of them it is possible that only two of them belong to class one and three of them belong to class two so which means that this object will belong to class two right so you see that so is the class allocation can vary depending on the value of K now K means on the other hand is a clustering process and it is unsupervised where what it does is the system will basically identify how the objects are how close the objects are with respect to some of their features okay and but the similarity of course is the the letter K and in case of K means also we specify its value and it could be three or five or seven there is no technical limit as such but it can be any number of clusters that uh you can create okay so based on the value that you provide the system will create that many clusters of similar objects so there is a similarity to that extent that K is a number in both the cases but actually these two are completely different processes we have what is known as KN based classifier and and people often get confused thinking that knive base is the name of the person who found this uh classifier or who developed this classifier which is not 100% true base is the name of the person b y s is the name of the person but naive is not the name of the person right so naive is basically an English word and that has been added here because of the nature of this particular classifier na based classifier is a probability based classifier and uh it it makes some assumptions that presence of one feature of a class is not related to the presence of any other feature of maybe other classes right so which is not a very strong or not a very what do you say accurate assumption because these features can be related and so on but even if you go with this assumption this whole algorithm works very well even with this assumption and uh that is the good side of it but the term comes from there so that is the explanation that you can give then there can be question around reinforcement learning it can be paraphrased in multiple ways one could be can you explain how a system can play a game of chess using reinforcement learning or it can be any game so the best way to explain this is again to talk a little bit about what reinforcement learning is about and then elaborate on that to explain the process so first of all reinforcement learning as an environment and an agent and the agent is basically performing some action in order to achieve a certain goal and these goals can be anything either if it is related to game then the goal could be that you have to score very high score a high value High number or it could be that your uh number of lives should be as high as possible don't lose lives so these could be some of them more advanced examples could be for driving in the automotive industry self-driving cars they actually also make use of reinforcement learning to teach teach the car how to navigate through the roads and so on and so forth that is also another example now how does it work so if the system is basically there is an agent and environment and every time the agent takes a step or performs a task which is taking it towards the goal the final goal let's say to maximize the score or to minimize the number of lives and so on or minimize the deaths for example it is rewarded and every time it takes a step which goes against that goal right contrary or in the reverse Direction it is penalized okay so it is like a carrot and stick system now how do you use this to create a game of chess or to create a system to play a game of chess now the way this works is and this could probably go back to this alphago example where alphago defeated a human Champion so the way it works is in reinforcement learning the system is allowed for example if in this case we're talking about Chess so we allow the system to first of all watch playing a game of chess so it could be with a human being or it could be the system itself there are computer games of Chess right so either this new learning system has to watch that game or watch uh human being play the game because this is reinforcement uh learning is pretty much all visual so when you're teaching the system to play a game the system will not actually go behind the scenes to understand the logic of your software of this game or anything like that it is just visually watching the screen and then it learns okay so reinforcement learning to a large extent it works on that so you need to create a mechanism whereby your model will be able to watch somebody playing the game and then you allow the system also to start playing the game so it pretty much starts from scratch okay and as it moves forward it it it's at right at the beginning the system really knows nothing about the game of chess okay so initially it is a clean slate it just starts by observing how you're playing so it will make some random moves and keep losing badly but then what happens is over a period of time so you need to now allow the system or you need to play with the system not just 1 2 3 4 or five times but hundreds of times thousands of times maybe even hundreds of thousands of times and that's exactly how alpha go has done it played millions of games between itself and the system right so for the game of chess also you need to do something like that you need to allow the system to play chess and then learn on its own over a period of reputation so I think you can probably explain it to this much to this extent and it should be uh sufficient now this is another question which is again somewhat similar but here the size is not coming into picture so the question is how will you know which machine learning algorithm to choose for your classification problem now this is not only classification problem it could be a regression problem I would like to generalize this question so if somebody asks you how will you choose how will you know which algorithm to use the simple answer is there is no way you can decide exactly saying that this is the algorithm I'm going to use in a variety of situations there are some guidelines like for example you will obviously depending on the problem you can say whether it is a classification problem or a regression problem and then in that sense you are kind of restricting yourself to if it is a classification problem there are you can only apply a classification algorithm right to that extent you can probably let's say limit the number of algorithms but now within the classification algorithms you have decision trees you have svm you have logistic regression is it possible to outright say yes so for this particular problem since you have EXP this now this is the exact algorithm that you can use that is not possible okay so we have to try out a bunch of algorithms see which one gives us the best performance and best accuracy and then decide to go with that particular algorithm so in machine learning a lot of it happens through trial and error there is uh no real possibility that anybody can just by looking at the problem or understanding the problem tell you that okay in this particular situation this is exactly the algorithm that you you should use then the questions may be around application of machine learning and this question is specifically around how Amazon is able to recommend other things to buy so this is around recommendation engine how does it work how does the recommendation engine work so this is basically the question is all about so the recommendation engine again Works based on various inputs that are provided obviously something like uh you know Amazon website or e-commerce site like Amazon collects a lot of data around the customer Behavior who is purchasing what and if somebody is buying a particular thing they're also buying something else so this kind of Association right so this is the unsupervised learning we talked about they use this to associate and Link or relate items and that is one part of it so they kind of build association between items saying that somebody buying this is also buying this that is one part of it then they also profile the users right based on their age their gender their geographic location they will do some profiling and then when somebody is logging in and when somebody is shopping kind of the mapping of these two things are done they try to identify obviously if you have logged in then they know who you are and your information is available like for example your age maybe your gender and where you're located what you purchased earlier right so all this is taken and the recommendation engine basic basically uses all this information and comes up with recommendations for a particular user so that is how the recommendation engine work all right then the question can be uh something very basic like when will you go for classification versus regression right when do you do classification instead of regression or when do you use classification instead of regression now yes so so this is basically going back to the understanding of the basics of classification and regression so classification is used used when you have to identify or categorize things into discrete classes so the best way to respond to this question is to take up some examples and use it otherwise it can become a little tricky the question may sound very simple but explaining it can sometimes be very tricky in case of regression we use of course there will be some keywords that they will be looking for so just you need to make sure you use those keywords one is the discrete values and other is the continuous values so for regression if you are trying to find some continuous values you use regression whereas if you're trying to find some discrete values you use classification and then you need to illustrate what are some of the examples so classification is like let's say there are images and you need to put them into classes like cat dog elephant tiger something like that so that is a classification problem or it can be that is a multiclass classification problem it could be binary classification problem like for example example whether a customer will buy or he will not buy that is a classification binary classification it can be in the weather forecast area now weather forecast is again combination of regression and classification because on the one hand you want to predict whether it's going to rain or not that's a classification problem that's a binary classification right whether it's going to rain or not rain however you also have to predict what is going to be the temperature tomorrow right now temperature is a continuous value you can't answer the temperature in a yes or no kind of a response right so what will be the temperature tomorrow so you need to give a number which can be like 20° 30° or whatever right so that is where you use regression one more example is stock price prediction so that is where again you will use regression so these are the various examples so you need to illustrate with examples and make sure you include those keywords like discrete and continuous so the next question is more about a little bit of a design related question to understand your Concepts and things like that so it is how will you design a spam filter so how do you basically design or develop a spam filter so I think the main thing here is he is looking at probably understanding your Concepts in terms of uh what is the algorithm you will use or what is your understanding about difference between classification and regression uh and things like that right and the process of course the methodology and the process so the best way to go about responding to this is we say that okay this is a classification problem because we want to find out whether an email is a spam or not spam so that we can apply the filter accordingly so first thing is to identify what type of a problem it is so we have identified that it is a classification then the second step may be to find out what kind of algorithm to use now since this is a binary classification problem logistic regression is a very common very common algorithm but however right as I said earlier earlier also we can never say that okay for this particular problem this is exactly the algorithm that we can use so we can also probably try decision trees or even support Vector meins for example svm so we will kind of list down a few of these algorithms and we will say okay we want to we would like to try out these algorithms and then we go about taking your historical data which is the labeled data which are marked so you will have a bunch of emails and uh then you split that into training and test data sets you use your training data set to train your model that or your algorithm that you have used rather the model actually so and you actually will have three models let's say you are trying to test out three algorithms so you will obviously have three models so you need to try all three models and test them out as well see which one gives the best accuracy and then you decide that you will go with that model okay so training and test will be done and then you zero in on one particular model and then you say okay this is the model will we use we will use and then go ahead and Implement that or put that in production so that is the way you design a Spam fi the next question is about random Forest so what is random Forest this is a very straightforward question however the response you need to be again a little careful while we all know what is random Forest explaining this can sometimes be tricky so one thing is random Forest is kind of in one way it is an extension of decision trees because it is basically nothing but you have multiple decision trees and uh trees will basically you will use for doing if it is classification mostly it is classification you will use the the trees for classification and then you use voting for finding the the final class so that is the underlyings but how will you explain this how will you respond to this so first thing obviously we will say that random Forest is one of the algorithms and the more important thing that you need to probably the interviewer is is waiting to hear is Ensemble learner right so this is one type of Ensemble learner what is Ensemble learner Ensemble learner is like a combination of algorithms so it is a learner which consists of more than one algorithm or more than one maybe models okay so in case of random Forest the algorithm is the same but instead of using one instance of it we use multiple instances of it and we use so in a way that is a a random Forest is an ensemble learner there are other types of Ensemble Learners where we have like we use different algorithms itself so you have one maybe logistic regression and a decision tree combined together and so on and so forth or there are other ways like for example splitting the data in a certain way and so on so that's all about Ensemble we will not go into that but random Forest itself I think the interviewer will be happy to hear this word Ensemble Learners and so then you go and explain how the random Forest works so if the random Forest for is used for classification then we use what is known as a voting mechanism so basically how does it work let's say your random Forest consists of 100 trees okay and each observation you pass through this forest and each observation let's say it is a classification problem binary classification zero or one and you have 100 trees now if 90 trees say that it is a zero and 10 of the trees say it is a one you take the majority you may take a vote and since 90 of them are saying zero you classify this as zero then you take the next observation and so on so that is the way random Forest works for classification if it is a regression problem it's somewhat similar but only thing is instead of vot what we will do is so in regression remember what happens you actually calculate a value right so for example you're using regression to predict the temperature and you have 100 trees and each tree obviously will probably predict a different value of the temperature they may be close to each other but they may not be exactly the same value so these 100 trees so how do you now find the actual value the output for the entire Forest right so you have outputs of individual trees which are a part of this Forest but then you need to find the final output of the forest itself so how do you do that so in case of regression you take like an average or the mean of all the 100 trees right so this is also a way of reducing the error so maybe if you have only one tree and if that one tree makes a error it is basically 100% wrong or 100% right right but if you have on the other hand if you have a bunch of trees you are basically mitigating that error or reducing that error okay so that is the way random Forest works so the next question is considering the long list of machine learning algorithms how will you decide on which one to use so once again here there is no way to outright say that this is the algorithm that we will use for a given data set this is a very good question but then the rest response has to be like again there will not be a one size fits all so we need to first of all you can probably shorten the list in terms of by saying okay whether it is a classification problem or it is a regression problem to that extent you can probably uh shorten the list because you don't have to use all of them if it is a classification problem you only can pick from the classification algorithms right so for example if it's a classification you cannot use linear regression algorithm there if it is a regression problem you cannot use svm or maybe no you can use svm but maybe a logistic regression right so to that extent you can probably shorten the list but still you will not be able to 100% decide on saying that this is the exact algorithm that I'm going to use so the way to go about is you choose a few algorithms based on what the problem is you try out your data you train some models of these algorithms check which one gives you the lowest error or the highest accurac and based on that you choose that particular algorithm okay all right then there can be questions around bias and variance so the question can be what is bias and variance in machine learning uh so you just need to give out a definition for each of these for example bias in machine learning it occurs when the predicted values are far away from the actual value so that is a bias okay and whereas they are all all the values are probably they are far off but they are very near to each other though the predicted values are close to each other right while they are far off from the actual value but they are close to each other you see the difference so that is bias and then the other part is your variance now variance is when the predicted values are all over the place right so the variance is high that means it may be close to the Target but it is kind of very scattered so the point the predicted values are not close to each other right in case of bias the predicted values are close to each other but they are not close to the Target but here they may be close to the Target but they may not be close to each other so they are a little bit more scattered so that is what in case of a variance okay then the next question is about again related to bias and variance what is the tradeoff between bias and variance yes I think this is a interesting question because these two are heading in different directions so for example if you try to minimize the bias variance will keep going high and if you try to minimize the variance buys will keep going high and there is no way you can minimize both of them so you need to have a tradeoff saying that okay this is the level at which I will have my bias and this is the level at which I will have variance so the trade-off is that pretty much uh that you you decide what is the level you will tolerate for your buyers and what is the level you will tolerate for variance and a combination of these two in such a way that your final results are not we off and having a tradeoff will ensure that the results are consistent right so that is basically the output is consistent and which means that they are close to each other and they are also accurate that means they are as close to the Target as possible right so if either of these is high then one of them will go off the track define precision and Recall now again here I think uh it would be best to draw a diagram and take a the conf ution Matrix and it is very simple the definition is like a formula your Precision is true positive by true positive plus false positive and your recall is true positive by true positive plus false negative okay so that's you can just show it in a mathematical way that's pretty much uh you know that can be shown that's the easiest way to define so the next question can be about decision tree what is decision tree proof Runing and why is it so basically decision trees are really simple to implement and understand but one of the drawbacks of decision trees is that it can become highly complicated as it grows right and the rules and the conditions can become very complicated and this can also lead to overfitting which is basically that during training you will get 100% accuracy but when you're doing testing you'll get a lot of Errors so that is the reason pruning needs to be done so the purpose or the reason for doing decision tree pruning is to reduce overfitting or to cut down on overfitting and what is decision tree pruning it is basically that you reduce the number of branches because as you may be aware a tree consists of the root node and then there are several internal nodes and then you have the leaf nodes now if there are too many of these internal nodes that is when you face the problem of overfitting and pruning is the process of reducing those internal nodes all right so the next question can be what is logistic regression uh so basically logistic regression is um one of the techniques used for performing classification especially binary classification now there is something special about logistic regression and there are a couple of things you need to be careful about first of all the name is a little confusing it is called logistic regression but it is used for classification so this can be sometimes confusing so you need to probably clarify that to the interviewer if if it's really you know if it is required and they can also ask this like a trick question right so that is one part second thing is the term logistic has nothing to do with the usual Logistics that we talk about but it is derived from log so that the mathematical derivation involves log and therefore the name logistic regression so what is logistic regression and how is it used so logistic regression is used for binary classification and the output of a logistic regression is either a zero or a one and it varies so it's basically it calculates a probability between zero and one and we can set a threshold that can vary typically it is 0.5 so any value above 0.5 is considered as one and if the probability is below 0.5 it is considered as zero so that is the way we calculate the probability or the system calculates the probability and based on the threshold it sets a value of zero or one which is like a binary classification Z or one okay then we have a question around K nearest neighbor algorithm so explain K nearest neighbor algorithm so first of all what is a k nearest neighbor algorithm this is a classification algorithm so that is the first thing we need to mention and we also need to mention mention that the K is a number it is an integer and this is variable and we can Define what the value of K should be it can be 2 3 5 7 and usually it is an odd number so that is something we need to mention technically it can be even number also but then typically it would be odd number and we will see why that is okay so based on that we need to classify objects okay we need to classify objects so again it will be very helpful to draw a diagram you know if you're explaining I think that will be the best way so draw some diagram like this and let's say we have three clusters or three classes existing and now you want to find for a new item that has come you want to find out which class this belongs to right so you go about as the name suggests you go about finding the nearest neighbors right the points which are closest to this and how many of them you will find that is what is defined by K now let's say our initial value of K K was five okay so you will find the K the five nearest data points so in this case as it is Illustrated these are the five nearest data points but then all five do not belong to the same class or cluster so there are one belonging to this cluster one the second one belonging to this cluster two three of them belonging to this third cluster okay so how do you decide that's exactly the reason we should as much as possible try to assign a odd number so that it becomes easier to assign this so in this case you see that the majority actually if there are multiple classes then you go with the majority so since three of these items belong to this class we assign which is basically the in in this case the green or the tennis or the third cluster as I was talking about right so we assign it to this third class so in this case it is uh that's how it is decided okay so K nearest neighbor so first thing is to identify the number of of neighbors that are mentioned as K so in this case it is K is equal to five so we find the five nearest points and then find out out of these five which class has the maximum number in that okay and and then the uh new data point is assigned to that class okay and with that we have come to the end of this NLP full course I hope you found it valuable and entertaining please ask any questions about the topics covered in the video in the comment box below and our team of experts will assist you in addressing your problems as as soon as possible so thank you so much for being here today we'll see you next time until then keep learning and stay tuned to Simply learn for more amazing updates staying ahead in your career requires continuous learning and upskilling whether you're a student aiming to learn today's top skills or a working professional looking to advance your career we've got you covered explore our impressive catalog of certification programs in cuttingedge domains including data science cloud computing cyber security AI machine learning or digital marketing designed in collaboration with leading universities and top corporations and delivered by industry experts choose any of our programs and set yourself on the path to Career Success click the link in the description to know more hi there if you like this video subscribe subcribe to the simply learn YouTube channel and click here to watch similar videos to ner up and get certified click here

Channel: Simplilearn

Views: 5,589

Keywords: natural language processing in ai, natural language processing course, natural language processing course 2023, nlp course, natural language processing, ml, nlp course for beginners, nlp for beginners, natural language processing tutorial, natural language processing in machine learning, nlp full course, nlp in artificial intelligence, nlp, natural language processing projects, nlp training for beginners, nlp tutorial, ai, simplilearn

Id: kqg-c3ACxgk

Length: 438min 0sec (26280 seconds)

Published: Wed Oct 18 2023