Intelligent AI Chatbot in Python

[Music] what is going on guys welcome to this ai tutorial in today's video we're going to build a chatbot using natural language processing and neural networks in python so let us get right into it now before we get into the actual python code we need to set up a so-called intense.json file this intense.json file contains certain categories of things that can happen for example the user can greet the chat bot the user can ask a question about the prices the user can ask a question about the name of the chatbot about the creator of the chat bot the user can say goodbye and those are all different categories provided with text for example we can have one tag called greeting and for each tag we also specify patterns like hi hello good day what's up hey greetings uh stuff like that so essentially we provide a couple of examples how a user could greet the chatbot and then the chatbot is going to take that as training data and see okay this is what uh greeting me looks like and i'm going to adapt it it's not only going to adapt to the exact things that you put into the chat bot so it's not going to uh if you type hello it's not only going to recognize hello but also stuff like uh hello how are you or similar things so so it's going to adapt also to a little bit more different data it's not super intelligent it's not going to to always recognize what you're trying to say but if you provide enough examples like hi hello hey it's going to generally recognize when the chat bot is being greeted and then we also have responses so a collection of static responses those are not going to be generated we're not going to have a text generating uh chatbot here this is going to be statically typed into the json file it's going to be what it's going to be the exact messages that the chatbot returns so the patterns are going to be more fluid more flexible but it's always going to return the exact responses so the idea is that we train the chatbot to recognize what the user is trying to get from it and then the chatbot is going to have some static responses to the questions or to the requests of the user so for this we're going to create a new file called intense.json intense.json and we're going to start with curly brackets so we're going to have a ordinary json file we're going to call it intense the whole thing is going to be called intense and then we have a list of dictionaries so we start with square brackets and inside of those square brackets we have individual categories and these categories have three pieces of information um sorry three pieces of information the first one is the tag the tag of this particular category let's say for example the first one is greetings and um then we have the patterns and the patterns are the examples of what a greeting would look like stuff like hey hi hello good day and so on so we specify a list of strings that give the that will later on give the neural network examples of what it looks like to be greeted by the user so we start with hello hey hi good day by the way you don't have to care too much about case about grammar because we're going to reduce all those words to the stems anyway so we're going to tokenize them later on uh so it doesn't matter if you say greetings uh with lowercase g or uppercase g doesn't really matter uh greetings what's up um did i forget something hello hi hey good day what's up maybe something like how is it going stuff like that and then we also have the responses and as i already said the responses are going to be hard coded so uh those here those those patterns are going to serve as training data so these they are not going to have to occur in exactly that uh constellation here but the responses are going to be the exact responses so here we need to take care about grammar uh take care of grammar take care of uppercase lowercase issues and so on so we're going to say hello or hey or what can i do for you and the idea is that once the neural network recognizes that the tag it's working with right now so that the input was in fact greetings it's just going to give a random response from that list so um it's going to be totally random you can sometimes it's going to say hello if the user says greetings it's going to say hello sometimes the user is going to say what's up and it's going to say hey sometimes the user is going to say hi and it's going to say what can i do for you it's totally random but the response is going to be exactly that the user input does not have to be exactly what's up it can also be what's up without that so like that um or it can be hi and hey or something like that and it's also going to be recognized so this is more flexible the responses are hard coded so now you can go ahead be creative and copy what we have just written and change it a bunch of times and introduce your own categories here in this case i have one for greetings i have one for goodbye age the pattern is always the same or actually not the pattern in in terms of patterns but the pattern of defining categories is the same you just copy it specify a name for the category then you specify some user input patterns and then some static responses and then you add a comma in the end start a new one and you can be creative and use your own categories i have one for greetings i have one for goodbye for asking for the age for asking for the name in this case the chatbot is called neural then if someone asks for products and i want to buy something what do you recommend what are you selling it goes to shop and gives a generic response here um then when it asks when are you guys open what are your working hours hours of operation 24 7. and then you can also add something like stocks what stocks do i own which stocks do i own and so on and the good thing is you cannot only do this with static text responses you can also later on in the neural network in the script uh you can you can check if the category that was recognized as stocks and in this case if you don't want to return just a basic generic message you can also execute a certain piece of code so you can say okay if the tag i recognized is stocks then go ahead download the stock data and show it to the user and you can also do it with something else so for example you can you can say tag shutdown if you recognize that the user wants to shut down the computer go ahead and shut down the computer for example so let's get started with the coding first of all we're going to create a training.py file because we're going to have one file for training the chatbot and a second file for using the chat bot and since those scripts are going to be quite large i'm going to look a couple of times onto my second monitor where i have the code already written so that i don't make any mistakes and uh yeah don't be confused by that we're going to start with some import statements here we're going to import random first for choosing the random response in the end we're going to import json for obvious reasons we're going to import pickle for serialization and we're going to import numpy as np if you don't have numpy installed use pip to install it just open up command line pip install numpy like that um then we use nltk as well again same thing if you don't have nltk installed the natural language toolkit you want to install it using pip and then we're also going to say from nltk dot stem import word net lemmatizer now this lemmatizer what it's going to do is going to reduce the word to its stem so that we don't lose any performance because it's looking for the exact word let me give you an example if we have the word work or working or worked or works and so on this is all the same words so we're not going to look for all these separate words we're going to reduce them to the stem and they're going to be considered the same word essentially this is what the lemmatizer does in a nutshell uh and then we're also going to import from tensorflow.keras.models we're going to import the simple sequential model then from tensorflow.keras.layers we're going to import a dense layer an activation layer and a drop out there and then we're going to import from tensorflow.keras.optimizers we're going to use sgd which is the stochastic gradient descent if i'm not mistaken yeah those are the libraries that we're going to need for this script so we're now going to go ahead and lemmatize the individual words for this we're going to first create a lemmatizer here lemmetizer equals wordnet lemmatizer and then we're going to load the json file intense equals json.loads open intense dot json dot read so we're reading the content of the json file as text we're passing that text to the loads function and as a result we get the json object uh which is a dictionary essentially in python um after that what we do is we create three empty lists first of all the words that we're going to have then the classes that we're going to have and then the documents which are going to be the combinations the belongings so to say documents and after that we're also going to specify the ignore letters ignore letters those are going to be the letters that we're not going to take care of or not going to take into account let's put it that way uh question mark exclamation mark period and comma there you go and now we have to iterate over the intent so we're going to say for intent in intense intense because now you have to imagine that this thing here is a dictionary in python so now we have the intense which is the object and then we need to access the key intense and there we have uh for for each intent we have certain uh sub sub keys and sub values so sub dictionary so the same and because of that we say four pattern in intent patterns so for each of those here for each of those patterns here those are the patterns in the patterns list for each of those we're now going to say word list equals nltk.tokenize and we're going to tokenize the pattern tokenize means that you're getting a text like hey i am john and what tokenize does is it splits it up into the individual words hey i am and john so this is what tokenize essentially does we're getting a list of words a collection of words and we're going to add that collection of words to the words list so to this list that we defined up here and after that we're going to also append it to the documents we're going to say documents append and we're going to append a tuple first of all of the word list and then also of the classes of the class of this particular uh intent so intent tag this is so that we know that this word list we just appended towards belongs to this uh category to this class to this tag and after that we're going to check if this class is already in the classes list so we're going to say if intent tag is not in classes we're going to append it to the classes like that and we can now go ahead and see what we got as a result here print documents in order to see if it works let's go ahead and we got an error here what is it line 24 word list equals nltk oh sorry word tokenize is the function not tokenized word tokenize but now it should work another error append takes exactly one argument to given i actually passed oh i didn't close the tuple all right now we should be done let's see if it works and we got another problem here oh no i didn't fix that problem okay come on it's getting ridiculous right now we say append word list like that i need to pass the tuple here now it should work and as you can see we got a list of hello this is a greeting hey this is also greeting hi this is a greeting greetings is a greeting what's up is the greeting and so on and we get all the individual things in the documents list related to each other now let's go ahead and do the actual lemmatizing here but before we do that we need to fix two mistakes and the first one is here we need to say words.extend instead of pen because extending means taking the content and appending it to the list and appending means taking the list and appending it to the list and this is not what we want to do and also we need to call the constructor here not just say word lemmatizer we need to call the constructor of the word lemmetizer um after we've done that we can say words equals list comprehension and then we say lemmatizer.lemmatize the word for every word in words if this word is not part if this word is not in the ignore letters list this would want to do and then in order to eliminate the duplicates what we're going to do is we're going to say words equals sorted set off words set eliminates the duplicates sorted turns it back into a list and sorts it and now we should be able to print this and see what happens we should get a list of lemmatize words and we do get one as you can see ken good goodbye greetings have i leaving see what as you can see all the individual words are now lemmatized which is pretty good so we can go ahead and also do the same thing for the classes i mean not not limitizing them but we can also say i'm not even sure if we would need that but i did it in the code so classes equal sorted set classes but we shouldn't have any duplicates anyway and the last thing we're going to do here is we're going to save them into files so we're going to say pickle dot dump words open into words dot pickle file as writing binaries so we're going to save them for later on and we're also going to do the same thing for the classes so we're now getting into the machine learning part and for this we need to understand why we're actually doing what we're doing in a second here and the problem is right now that we have a lot of characters and words and and classes and all that but those are not numerical uh values we do not have anything we can feed into a neural network a neural network needs numerical values we cannot just feed words into it and expect it to uh to to work on its own we need to represent these words as uh numerical values and in order to do that we're going to um use something called bag of words we're going to set um the individual words indices uh or the individual word values to either zero or one depending on if it's occurring in that particular uh pattern and then we're going to also do the same thing for the classes now before i confuse it too much let's get right into it we're going to say training equals empty list and we're going to create an empty output or output empty which is a template of zeros and we need as many zeros as there are classes so length classes and what we're going to do then is we're going to say for document in documents we're going to create an empty bag here so for each of those combinations we're going to create an empty bag of words and we're going to say word patterns are what we find in the document at index 0. and then we're going to say word patterns equals list comprehension lemmatizer.lematizeword.lower for word in word patterns and that's actually it we don't need to care about ignore letters and then we're going to say for word in words want to check if that word uh so so this year this words list is the one that we filled up here with all the words uh that occur in tense for each word we want to know if it occurs in the pattern so we're going to say if word um if words in word patterns then what we're going to do and we're going to write it in a different way here we're going to say back append 1 if this word occurs in word patterns and otherwise we're going to say back append 0 like that uh and now we're going to do the same thing for the output so we're going to say output row equals list output empty this is what we do in order to copy the list so we're not really type casting it we're copying it um and then we say output row equals or actually output row classes dot index dot one so the document one sorry wanna know the class which is at index one wanna know the index and then we're going to add uh to set this index in the output row to one and then we're going to append the whole thing here to the training list that we created so we're going to say back and output row we're going to append it to the training list that we have here so when we run that loop uh all the document data is going to be in the training list and we can we can work with this training list in order to train the neural network now we're going to do one final step of pre-processing before we get into building the neural network and this is first of all we're going to shuffle the data so random.shuffle the training data and then we're going to also turn it into a numpy array by saying training equals np array of training itself and then we're going to split it into x and y values train acts being training or actually list of training everything and first dimension oh zeroth dimension actually and the same thing for one and for the y values and those are the x and y values the features and the labels that we're going to use to train our neural network so we can now start to build the neural network model by starting with a model equals sequential so we're going to create a simple sequential model here and then we're going to add a couple of layers and the first layer is going to be the input layer it's going to be a dense layer with 128 neurons and an input shape that is dependent on the size of the training data for x so we're going to say input shape equals length of the training data for x index zero oh come on like that and then we also specify the activation function to be a rectified linear unit the relu so to say then we say model dot add drop out in order to prevent overfitting by the way i'm not going to explain all the individual layers and functions and optimizers here because i don't think that this is the video where i should explain neural network theory to you this is the application so if you're interested in neural network theory let me know down in the comments below uh so i can maybe make serious about that but now i'm not going to explain all the individual parameters and individual functions and so on after that we again add another dense layer with 64 neurons and also the activation function of relu rectified linear unit then another dropout layer drop out 0.5 and in the end we also want to have another dense layer but this time it's going to be a little bit different first of all we want to have as many neurons as there are classes so in a sense just as there are training data inputs training data elements so to say for the y uh for for the for the labels and then we also want to specify that the activation function that is used here is the uh softmax function because that's the function that is going to allow us to add up the results by the way i think i think i'm writing outside of the bounce here um there you go yes uh so now we say activation function equals soft max because that's the function that sums up or scales the results in the uh in the output layer so that they all add up to one so that we have sort of the percentages of how likely it is to have that output or that that result so to say that label then we're going to quickly define a stochastic gradient descent optimizer by saying sgd and then i'm going to enter a couple of parameters here that to be honest i don't understand all of them some of them i do understand but but as i already said i'm not going to go into the theory here so 0.01 is the learning rate then we have a decay of 1 e minus 6 is just another way of writing decimal places then we specify a momentum of 0.9 and uh we're going to say nesteroff i don't know what that is to be honest equals true and now we go ahead and compile the model by saying model compile the loss function that we're going to choose is called categorical cross entropy and then we say optimizer equals the sgd optimizer that we just defined and the metrics that we're interested in are accuracy or it's only accuracy then we go ahead and say model dot fit and we're going to use the training data so np dot array of the train x data here uh actually without specifying an index then of course we don't want to have as an output uh the y data so train y and what we need to specify then are the epochs when i have 200 epochs so we're going to feed the same data 200 times into the neural network in a batch size of five last but not least we're going to specify where bose equals one so that we get a medium amount of information and in the end we're going to save them all by saying model dot safe and we're going to say uh chat bot a model dot model or something like that and in the end we're going to print done so now we're going to run this script here and see if it works if everything went fine we should be able to see the neural network being trained in a second and for now it looks good yes you can see it starts training 200 epochs and we are done as you can see neural network is now trained and the next thing that we need to do is we need to create a chatbot application itself that uses the trained model so we start working in a new file and here we also need to import the same libraries so import random i mean we don't need to import the tensorflow libraries but other than that we need to import the same libraries import random import json import pickle import numpy snp import nltk from not frozen set from nltk.stem import wordnet lemmetizer and then we're also going to say import pandas as pd import or actually no we don't need that this is uh this is for another project sorry we don't need to import pandas um we only need to import tensorflow from tensorflow.keras.models we need to import the method or the function load model in order to load the model that we have um that we have created with the training script here so what we do then is we again create a lemmatizer we say lemmatizer equals worknet lemmatizer and then we do the same thing we say intense equals json.loads open actually we can copy that from the training from the training thing here and then we also need to say words classes and model we need to learn uh we need to load them and we've all saved we've saved all of them so we have words we have classes and we also have the model so we say words equals pickle dot load and we open up the file words.pickle in reading binary mode and we do the same thing with the classes so we say classes equals peak pickle load and then classes dot pickle and the model is just load model chatbot model dot model there you go and now what we need to do is we need to set up a bunch of different functions because the model is trained we can already use it but we need to use it the right way because what we get is again numerical data but we want to end up with words we want to end up with what do i actually need to do and for that we're going to define four different functions we're going to have a function for cleaning up the sentences a function for getting the bag of words a function for predicting the class based on the back based on the sentence essentially and then a function for getting a response so we're going to start with the cleaning up of the sentence first we're going to say def clean up sentence i'm going to have a sentence here and we're going to say sentence words equals nltk dot word tokenize so essentially again we're tokenizing the sentence um and once we have tokenized the sentence what we're going to do is we're going to say we're going to lemmetize the word so we're going to say sentence words equals lemmatizer lemmetize word forward in sentence words and this is it and then we return those sentence words and this is how we clean up a sentence so the next function we're going to write is called bag of words and it's going to convert a sentence into a bag of words so into a into a list full of zeros and ones that in uh that indicate if the word is there or not so using flax essentially in the same way that we did it in training script already so we're going to say def back off words sentence and then we're going to say sentence words sentence words i'm going to use a function that we defined above we're going to say clean up sentence sentence and then we're going to create an initial back full of zeros how many zeros as many as there are individual words that can be because we need to set for each word we need to set either zero one and then we're going to just say four word in for w in sentence words for i word in uh enumerate words we're going to say back or actually if if this word here is the same as w then we're going to say back i equals one otherwise it's automatically zero of course return and p dot array i'm going to return a numpy array off the back so this is how we create a bag of words so next we're going to define the predict function we're going to make use of those two functions here for that we're going to say def predict class based on the sentence and we're going to say give me a bag of words let's say back off words is going to be um bag of words of the sentence because that's what we need to feed into the neural network in order to get a prediction we're going to say result equals model dot predict and we're going to pass numpy array of the bag of words um do we need to actually i need to pass it as a list like that and then get the zeroth index to match the format so then we get a result and then we can specify an error threshold that allows uh for a certain uncertainty but if that uncertainty is too high we're not going to take it into a result so threshold and we can say this is 25 for now so remember we have the softmax function which means that we have uh each each position in the output is going to be a likelihood of that class being the result and if this is below 25 we don't want to take this in in order to realize that we're going to say result equals an index and the result itself for i are in enumerate the results but only if r is larger than the threshold so if the result is larger than threshold and of course we save the index so that we know which class it is and then we're going to sort the results we're going to say result dot or actually results dot sort the key for sorting is going to be an anonymous function a lambda expression if you don't know what it is check out my python tips and tricks tutorial series uh and we're just going to take the first index every time and reverse equals true so we're going to uh to sort it in reverse order so in descending order essentially and then we're going to have a return list it's going to be empty but then we're going to save for each uh for r in results we're going to say result list of return lists sorry return list dot append and here we're going to say intent classes are zero and then we're going to have a probability probability probability this probability is going to be string of r one of the index one of the result because that is the probability that is the actual result and the first one is the index um and then what we get is uh then what we do is we return the return list so in a nutshell what we do is we create a bag of words we predict the result based on those bag of words then we have a certain threshold that we don't wanna we don't wanna have too much uncertainty what we do is we enumerate all the results so that we get the index so the class uh and also the uh probability then we wanna sort by probability in reverse order so we wanna have the highest probability first and then we want to have a return list full of intents and probabilities so full of classes and probabilities now actually we need to fix two little mistakes here first of all when we train them all down here we need to save it first into a variable and then save it into an h5 file and pass the return value of the fit function to save it into a single file into an h5 file otherwise doesn't work and second one is a little bit uh more stupid we did pickle dump and before that it was uh saying words i forgot to change it to classes and because of that it didn't work and once we have fixed those two mistakes we can run the chat bot and we should be able to chat with this chat bot here so uh yeah i added a print up here for troubleshooting we can remove that as well uh and then you know it says go bot is running okay let's say uh hello let's see what happens it says hey uh who are you i'm the rule what is your name my friend i'm neural the assistant of florian how are you okay probably this was associated with how old are you and that therefore it tells uh me how old the owner is i could also say something like what are my stocks doing you own the following shares um then maybe we had buy gotta go okay i gotta go it's not recognized uh hi is recognized i guess hey what is up bro and as you can see it uh has decent responses i would guess uh or i would say of course you can improve it by adding more training data more and more training data more patterns more responses uh maybe even more text more intense but as you can see we have built a simple uh not so simple to be honest but a decent chat bot in python so that's it for this video if you enjoyed hope you learned something if so let me know by hitting the like button and leaving a comment in the comment section down below of course make sure you subscribe to this channel and you hit that notification bell to not miss a single future video for free other than that thank you very much for watching soon next video and bye [Music] you

Channel: NeuralNine

Views: 238,850

Rating: 4.9447412 out of 5

Keywords: python, coding, programming, ai, artificial intelligence, chatbot, intelligent, neural network, neural networks, tensorflow, nlp, natural language processing, iot, python tutorial, python ai, python chatbot

Id: 1lwddP0KUEg

Length: 35min 41sec (2141 seconds)

Published: Fri Dec 04 2020