Emotion Detection using Convolutional Neural Networks and OpenCV | Keras | Realtime

[Music] hey there this is welcome back to a new video and in this video we are going to make a brand new project that is about emotion detection model using keras and convolutional neural networks and we're also going to use opencv to test our model so first of all i'm going to show you what are we going to end with so let me run my code so this is the file which is using the cnn model which i have built using the kegel data set and we're going to build the data set on kegel kernel only so that you don't need to download the whole data set because it's of huge size right so we are going to build the model on kegel kernel and we are going to download that model and then we are going to write this whole code which is the opencv code and it is going to use your build model and it is going to show you the results so basically i'm focusing on these seven emotions that is angry disgusting fear happy neutral sad and surprised the accuracy of my model is around 72 percent that's a okay can you can say that's a good accuracy because you are having so much images and you are having seven classes right so that you can that that you can say that it's a good accuracy right so let's run the code and let me show you the output of our project so hey there everybody so first of all um i've downloaded some sample images on my phone so that i don't need to make emotions each and every emotions right so first of all i'm going to show you this face of a girl that is detecting it to be happy and you can see only that it's a happy face let me use the next image and you can see that so angry face and it is detecting very perfectly that its angry face and it is showing my face as neutral only because i am making a neutral face i am very bad at showing the fake emotions so let me show you the next image that's a kind of surprise or you can say fear but it is telling it that it's a surprised image okay that's not a problem and next image i have again it is telling this image is happy but i think it's a surprise so as you can see it is getting shifted towards surprise and happy i think it's a bit confused whether it is happy or it is surprise right because it is continuously changing its answer surprise or happy maybe next image is again a happy image you can see the next image is a kind of anger angle yeah it's angry mage shuffle lady next image is kind of sad it is detecting but i think it's a kind of confused image but it is telling that it's a sad image okay no problem and now i'm going to test with multiple images right multiple images in one photo so each and every phase is getting detected using phase cascade classifier and it is telling that they are having these these emotions right amazing now coming to the next image is again many faces and it is telling me that there are these are the respective output emotions right so this is the output that we are going to build uh in this project and without wasting more time i think we should first go to kegel and let's peep into the data set and then we are first of all going to make the model that is cnn model and we are going to download it and we are going to use it for our opencv right so lets start coding now as you can see i am on my kegel.com and i am on i am on the data set that i'm going to use for this video or this project and this is by jonathan o'hacks so a lot of thanks to the author for making such a beautiful data set and it is containing two directories i can see that training part and the validation directory and they are again containing seven seven directories which are for seven different emotions so if i go into the training part and i can see that there are seven different kinds of directories which are containing files in thousands right so this angry folder is containing the files of the uh they are containing the images of the angry which are which basically depict the anger anger face so that it can be fed to the model and it gets trained so like this you can just go through each and every folder and you can see so as you already know for each and every model we need a training part and we need some validation data so here also we are having one training directory and the other one is the validation directory which is again containing these seven different directories right so as you know that it is a detailed explanation video and each and i'm going to explain as much as i can uh do and if some things get missed you can tell me in the comment section i'm going to make a separate video for that particular topic right so this is the training part and this is the validation part so okay now let's go to kegel kernel i'm going to use kegel kernel so that i don't need to download this data set that is of 121 mb right so i'm going to use the kegel kernel and i'm going to download the model that we are going to build right let's first go to kegel kernel and this is the data set that i'm using that i have already shown you now first of all i'm importing the important libraries so first of all i'm importing the matplotlib and cbon these libraries are basically for data analysis and data visualization right i hope it is clear because i have covered it in many of my google uh videos and other videos also then i am importing os so that i can import or i can call the files the image files that are for training and validation so that i can show you which are my files using matplotlib i'm going to show you in the for the part don't worry then i'm importing the deep learning libraries that are going to be used for building the model the cnn model so for that i'm going to import the load image so to load the images and image to error so that i can convert my images into array so that i can fade into my model very easily because model will not take the image right it will take the array form of the image right then i'm going to import the image data generator and i'm going to explain what is the use of image data generator and when it is going to come into our process then i'm importing the important layers which are which are required for making the cnn model right so as you know that every artificial neural network is a collection of some layers the hidden layers or the input layer or the output layers like this we are going to import these layers which are used to build the model right so first of all it is the dense layer then input layer dropout layer global average pooling 2d then flatten then convolutional 2d batch normalization activation and max pooling tuning then we are going to import sequential model from keras.models then we are going to import some of the optimizers and lets see which one we are going to use i hope uh it is clear up till uh until now right now let's some let me run this thing and we are imported we have imported all the important libraries now we are going to display some of the images so that you get an idea uh what are the images we are going to use for building the model so i'm standardizing the size of each and every picture because it is not necessary that each picture is of one standard size so i have to standardize the size of each and every picture which i am choosing it to be 48 into 48 right so first of all i am defining the folder path uh so i am defining this path so as you can see which is containing my images that is training and validation so i am copying the path from here and i am pasting over it over here right then after this i am going to plot some of the images from my disgusting folder right which is containing the images of disgusting face right so let's uh do that so for that i am defining the expression it to be disgusting and if i change it uh to be anger or sad and happy you will see those images respectively right then i'm defining the figure size of my matplotlib graph that is 12 by 12 and for i in range 1 comma 10 to 1 so as you can see that if i go into my training part and i'm going to disgusting folder i can see that all these images are numbered uh from um from one number two different number which are having the same interval right so if i'm finding 3132 then i will will find 33 maybe not in series but there are but they are there right so all the images are in sequence right so i'm defining the images from 1 to 10 because i'm i want to plot the images from index 1 to index 9 that's why 1 to 10 and with the interval of 1 then i'm going to plot the subplot using matplotlib if you don't know you can follow my matplotlib video and i'm defining the subplot size to be three by three right so three by three so as you can see in my output three by three uh matrix is form subplot is formed using matplotlib and i'm defining the position so in subplot you need to define the position of that particular graph or of that particular image if i am more precisely over here right so it is i right then i am going to load the image so i am defining this load image function and i am defining the folder path plus strain slash expression so as you can see i have to go inside these images which have defined the path over here then i have to go inside the training part then i have to go inside the expression part which i've defined over here right so let me increase the form so that it is more clear to you right so i'm going inside the folder path then i'm going to set the training path then i'm going to set the expression that is disgusting and after this i'm going to show you show the images which are present over here with these indexes right so os dot list directory then folder path plus training an expression at i so all the images at index one then index two then index three then s4 till index nine then i'm going to those things and those folders those uh images i'm going to show you right target size is picture size comma picture size because i want to show i want to standardize the size with to 48 by 48 so i've defined the picture size which i've defined over here and it is defined over here that is 48 right so 48 by 48 is the standardized size which i have chosen right then i'm going to show the image so using plt i am show image this is the function to show the image using matplotlib right and after this i am defining plt dot show if you will run this you can see that these are the images so i want that it should be in dark theme so let me set these plt plt dot style plt dot style dot use dark background let me run it again and you can see that i can see the numbers also now because i'm a dark theme lover that's why i'm doing it right so nine by nine images i can see so first nine images it is showing me right so if i change the directory to angry then it will show me the first nine indexes uh images of angry folder let me run it so these are the angular images like this i can change it to happy and you can check each and every folder of you of your choice right so you can see that all these images are happy so these are the training images right make the training and the validation data it's like a trained test fit but as you can see that the data is already splitted into training and the validation right so here we have to do something different that what i'm doing first of all you need to define the bad side now what is the bad size so as you know that this video is completely detailed video but i think i may miss some of the points which you can tell me in the comment section i will make a separate video for that right so the hash size is basically a size or it's a number uh it depicts that how much training example how many training example your model should take in one iteration right so that says i am defining it to be 128 so it means that it should uh in one in one iteration it should take 128 training examples 128 training files you can say most more precisely right so then you i'm defining two variables that are datagen train and it has an uh validation right and i'm defining it to be as well image data generator right image data generator and image data generator right then after this i'm defining two variables that are train set and the test set or you can see the validation set it's totally up to you then these variables are going to contain the data which is coming from the directory so how we can define it so data is in train that is this variable which is the image data generator uh you can say object of image data generator dot flow from directory that means the data which is present in this directory that is folder path plus strain the folder path is or defined over here that is still images and after this i am adding the train string that is it should go inside the training folder and it should take all the images from it and it should give it to this state agent train and after giving it to the image data generator it should be stored in the training set here i am defining the target size of the image so as i have already told you earlier that we have to standardize the size of each and every image file so for that i'm defining the target size of the image each and every file that it is going to get iterated uh picture size comma picture set that is 48 by 48 color mode is gray scale because i want to read the images in grayscale batch size is bad size then i'm defining the class mode as categorical because there are seven difference there are seven different categories of outputs angry disgusting sad neutral like this so this class mode will be category then shuffle is true to shuffle the image it uh the name only tells it i'm going i'm going to do the same thing for my validation data right so training set trend training set is ready a validation set is ready or you can see the test set is ready right then it is telling me that it has found 28 800 and uh 21 images in my uh belonging to seven classes that is for training then 76 images in my validation folder seven classes right i hope it is clear to you now we are moving to the most important part that is defining the model so here we are going to build the artificial neural network that is going to detect different different emotions cool right so for that i am defining the model and i am going to import the optimizes which i've already imported so i think we don't need to write this statement and i think i'm not using it anywhere in the process right so i'm defining the number of classes as seven because there are seven different possible outcomes in model as sequential we are going to use the sequential model then this is the first convolutional neural network layer so you know so you know that in artificial neural network we have different different layers input layer is there dropout layer there flattened layer activation layer is there then hidden layers are there different different right so we are going to define these layers right so these all layers combined will make your artificial neural network or you can say the deep uh deep neural network right so first of all i am defining the convolutional neural network and 64 filters i am going to define which are of three by three three by three is a kernel size so if you don't know about this kernel size and what are the number of filters uh just tell me in the comment section i'm going to make a separate video for all this because i can't cover each and everything in this right so i'm defining and i'm trying to explain each and everything which can be possible so i'm defining 64 filters and three by three size so three by three size is the kernel size padding is uh padding and keeping it to be same from every side of your convolutional neural network layer so you don't know that i think you know that in convolutional neural network we are going to place the kernel on each and every part and we are going to shift shift the kernel at every iteration with your uh defined stride right so input shape is 48.48 comma one right so one because because our image is now grayscale image and 48 by 40 is your uh shape of your image right then i'm defining the patch normalization earlier after this we have the activation layer and activation activation layer is present at the end of the neuron and it basically decides which what data should be fired to the next uh layer right and we're defining the rectified linear because it does not start all the neurons it does not triggers all the neurons at the same time right so it is considered value is considered the best activation function for these kind of data sets right then we have the max pooling 2d and what does this max pulling today does first of all you need to define the pool size which you are defining two by two and it basically extracts the important information from that area where we are putting that pool uh which whose size is two by two right so it extracts the important uh feature from that particular area of that image right so for that we need to define max pooling 2d and we are defining dropout to to prevent our model to get overfitted right so these are the important uh just key points i have told you for each and every layer then after this again we are going to define the second cnn layer and we are defining convertery uh then then uh important step that is you you need to define the input shape only in the first area right you don't need to define the input shape in every every in each and every layer so here we are not defining the input shape then rest of the things are same and the kernel the number of filters we are defining 128 right then after this we are defining the number of filters at 512 then again i have defined this four cnn layer then this flattened layer the function of this flattened layer is to collapse the input size or you can say the input to one dimensional array which can be fed into a model very easily so this is the function of the flatten right then we are defining the fully connected first layer that is we are defining we are connecting all these layers using the dense right so we are connecting all these layers okay then after this what i am doing i am defining the atom optimizer so i'm defining this adam optimizer with my learning rate is 0.0001 and i'm defining this adam optimizer with into my variable that is opt then i'm going to define my model.compile and i'm defining the optimizer into it then loss which is categorical close entropy because there are different categories so we are going to use categorical cross entropy loss then the metrics i have kept it we kept it to be accuracy then i'm printing the model dot summary so this is the summary of my model and it defines all the important layers which i have used and their sizes like that data is there so now let's fit the model with my training and the validation data right so for that i'm importing these three important functions from my keras.callbacks class right so first of all it is your model checkpoint then early stopping and reduce lr on reduce learning rate on pattern uh you can say right so checkpoint consider consider it as a police checkpost right so police checkpoint is just checks uh you off uh when there is a some criminal cases going on like it's there are some certain checkpoints of queries right so the checkpoint is the same thing so it is going to check your model at each and every point and it is going to save your model in this path so what i have done i am saving my model in my this output uh kegel so this is the default output folder which you get from kegel so when you will copy this path and you are going to get this two thing that is dot then slash and this is the name of your model which you want to save right so i'm defining it to be model.h5 file then monitor is i'm capturing it to be validation accuracy so it is going to monitor my validation accuracy then verbose is one then save best model is true then mode is max and if you don't know about any of this factor you can tell me in the comment section and you can just check it on the net also you will just type verbose meaning of verbose in machine learning and you are going to get the answer right so certain things i am just leaving for you so that you should also do some research so these small small parameters you can just search on the internet right then i'm also defining the early stopping so if i'm getting the best model and the validation uh if and my accuracy is not increasing then it is going to early stop so that it uh it should not over fit my model right if my if my model accuracy is not increasing then just continuing with the number of epochs is just a time wasting right that's why we need to define the early stopping so and i'm keeping the monitor at validation accuracy then these are again certain uh fun parameters so restore the best weights that means it is going to restore the best weights which i'm getting the best model right then i have to define the reduced learning rate so if my model is uh not able to cop off with the particular learning rate which i have defined earlier then it is going to reduce the learning rate factor is 0.2 patience is 3 verbose is 1 minimum delta is 0.0001 right then i am defining callbacks list which is going to contain it is basically a list which is going to contain all my three parameters that is early stopping reduce learning rate and my uh what is checkpoint right right okay then after this i'm defining the epochs as 48 then i'm going to fit my model with my training set and my test set which i have which i'm ready uh yeah this training set and the test set so which i have uh taken from my folder the data set folder right i have defined the class mode i've defined the batch size i have defined the color mode right so i'm reading with i'm ready with the training set and my validation set so i'm going to define the generator as train set steps per epoch is the so you you already know the formula of steps per ipo which is the length of your training set that is train set dot n that is for training set length divided by the batch size so training set dot patch size then you need to define the epochs which is 48 then you need to define the validation data which is test set then validation steps formula you also know that is the length of test set divided by the batch size of the test then you need to define the callbacks which are these three things early stopping checkpoint and reduced learning rate then when i did all this stuff i saw that my accuracy was 67 percent and my validation accuracy is around 53 and it has almo almost changed because when i first did it uh when i was writing the statement uh when i was writing this kernel then i was getting the accuracy was around 70 and my validation accuracy was around 69 so it has decreased so it can change right so when you will do it it may come above also and it may come below also so you should try it and you can change all these factors and when you get the best accuracy that's good right so the model which i'm using for that opencv and which i have provided on the github uh link is given in the description that is having validation accuracy is around 69 and my accuracy of this model is around 72 or 71 it was right okay that's not a big deal then i'm just plotting the accuracy and the losses so when you will plot all this stuff that i'm using just a simple matlab lip thing and when you will plot this accuracy and your loss then you will see that this is these are my laws which are getting decreased from time to time and these are my accuracies which are getting increasing which are in which is increasing from uh time right okay that's a good thing and when i will uh i can let me see if i am getting the model over here i think i just refreshed that kegel kernel that's why my model.h5 file is gone so i'm not going to train my model again because it is going to take a lot of time but yeah when you will run it then you will get a file over here that is model.h5 which we have built just download it you will get a menu and just click on that and you will get the download button so when you will download it you will get your uh model.h5 file like this so model.h5 file i've just downloaded it so that's why i'm getting my model or x5 file then just copy it and you can paste it in your folder that is your project folder so in my python project folder i have defined this emotion detection cnn folder and i've defined this main dot py that is for opencv model which i'm going to just make in front of you then i'm defining then i'm pasting that model.h55 which is going to be useful when i'm making that opencv thing right then this hard gasket frontal phase default so i'm requiring this hard casket for detecting my faces right so for this you need to get this frontal case and this is the kl kernel file uh which i've just like this this file i'm talking about which i've just done right so i'm giving all these four things that is pre-built model and then this main pi our cascade and your uh ipython notebook these four things i'm giving in my github repository you can check that out right so let's go to main dot py and now let's start coding the uh opencv right opencl model so for opencv i'm going to import this uh load model because i need to load my model that model.h55 i need to import it right so for that i'm going to uh i'm importing this load model uh function from my keras.models class then from preprocessing i'm importing uh image to array so that i can then after this i'm importing image from my keras pre-processing then i'm importing cb2 and numpy so first of all i'm uh defining my cascade classifier that is phase cascade classifier and this is the location so you don't forget to write the raw path that's why i'm defining this r and this path the whole path is generally denoting this whole path right so this part so i'm defining the hard casket frontal face so after this just write it over here just copy paste all this all that path and just write one more backslash and write that casket frontal face default.xml right then after this i am defining my model.h5 so using load model i am going to define my model that is model right 5 the whole location is pasted over here then the emotion labels so i am defining a list which is containing all my possible emotions output possible outcomes that is angry disgusting fear and so on then i am defining the cap that is for video capture and defining zero for my default web camera of my laptop then using the while loop so i've already told you how you can just play with your webcam right in my opencv tutorial playlist you can check check that out and using the while loop you can just define two parameters that is the boolean that is whether my cap is getting recorded or not then this frame this frame is useful because this is the image read by your opencv so then i'm defining the labels then i'm defining i'm converting my frame into grayscale right so this spaces thing is going to contain all the faces present in my gray image right now this face is using the for loop it is going to return four parameters that is x and y then w and h for drawing the rectangle so for each and every phases so as you have seen in the starting part that it was able to detect multiple faces also right so for that we need to use the form and for each face it is going to give me four parameters that is x y w and h and it is going to draw the rectangle around that face right so that yellow rectangle was because of this so you do a rectangle then frame on which we are going to uh draw the rectangle then the x comma y then x plus w y plus h the color of the rectangle then the width of that line rectangular line right then region of interest i am interested only this face right i'm not interested in any other thing so that's why i'm defining the region of interest and uh gray scale only so for that gray i'm defining it to be y to y plus h x to x plus w because my face is present in these constraints only y y plus h x x plus one right then i'm dif i'm just resizing my image to 40 by 48 by because why i'm doing it because before i speak just comment down why i'm doing 48 by 48 because my model needs a 48 by 48 image because my model is trained on 48 by 48 images that's why i'm converting this region of interest into 48 by 40. now to standardize my region of interest i am dividing it by dividing it by 255 then i am converting this region of interest into gray that is sorry not gray that array so my model is trained on getting the arrays so i am converting my image into array so image to array region of interest then using the classifier that is my model i am just predicting the outcome right so it is going to predict the particular emotion uh in my region of interest right then i am using this prediction.rt which is telling so when it is going to predict the emotions it is going to give me certain numerical values for each and every emotion right let's say if the image was of angry then the argument that is the number given by the classifier will be maximum for my angry part right so i am going to find which which prediction is having the highest numerical value so i'm going to take that and that index that index is going to uh the emotion presented that index that is this array i have defined this array now that index will be my output my answer right so i'm doing that thing only so emotion level present at this index will be my label so label position will be x comma y minus 10 why i'm doing y minus 10 because when i was not when i'm not doing this y minus 10 then that output answer was just getting sticked uh to that rectangle that's why i'm just shifting it a little above so for that i need to do this minus 10 right so i'm just printing that text that is my label or using the cv2 output text using the using this font using this label position and using this label the color and the width and your text size right then if my model is not able to find any faces then it is going to tell that it is not containing any faces or no faces then this is the location where i want to put the text this is the font this is the text side this is the color this is the thickness right then i'm going to show my frame then if i click queue from my keyboard it is going to destroy everything let's run it and let's finish this tutorial right i'm running my code okay i forgot that this is not python it's running and i'm getting with i'm getting an error okay okay okay actually my webcam is already on because i'm using the obs studio i'm just going to switch off my uh this thing so that this camera should go for my opencv not for obs studio right running it is running yes so as you can see this is my face and it is telling that a neutral face is there so again if you want to check the output i'm just [Music] i can just show you okay first of all i need to tell you that i was telling about this area right so as you can see that there is a gap between this neutral and this uh yellow rectangle right so if i just remove this uh y minus 10 that i was telling that i've kept y minus 10 yeah if i remove this 10 thing then it is going to get stick to my yellow rectangle that i that i was just willing to avoid where is it where is my file okay i think it is okay yeah it has not updated i want to just quit it and then again run it again then my yes this one so as you can see the neutral is getting touched with this yellow rectangle that i don't want right so i think this much is enough for this video and like this you can detect any kind of emotions and not just emotions right i have told you how you can build a model which is categorical in nature using that data set and you can just detect anything now using the categorical values of your data set right so i so before i end this session i want to tell that there is a telegram channel and you can just join in the link is given in the description so if you have any doubts regarding any video and you want to give me some suggestions on which topic i should make different videos you can just tell me in the telegram channel right so yes till the next content keep coding keep innovating and thanks a lot

Channel: Akshit Madan

Views: 115,311

Keywords:

Id: Bb4Wvl57LIk

Length: 31min 50sec (1910 seconds)

Published: Thu Jan 21 2021