Real Time Face Mask Detection with Tensorflow and Python | Custom Object Detection w/ MobileNet SSD

hey man great to see you wearing a mask oh thanks so much man uh never mind [Music] what's happening guys my name is nicholas and in this video we're going to be building our very own real-time face mask detector now what's really cool about this video is that these same techniques can be applied to a whole range of use cases so if for example you wanted to train a custom object detector for your specific use case you can use these exact same techniques do that let's take a deeper look at what we're going to be going through so in this video we're going to be covering everything from start to finish in order to build a real-time custom object detector now we're obviously going to be doing it on face mask but you could apply this to a whole range of other use cases so if you wanted to build a custom object detector for your use case you could use these exact same techniques to do that so we're going to start out by labeling our images using the python label image package and then we're going to leverage techniques like transfer learning and the tensorflow object detection api to be able to train our model a whole heap faster and then last but not least we're going to be leveraging opencv to actually make real-time detections let's take a deeper look as to how this is all going to work so first up we're going to be preparing our images using the label image package so this is going to allow us to label different face masks on our face then what we're going to do is we're going to train our deep learning model using the tensorflow object detection api and specifically we're going to download a pre-trained model and then apply transfer learning to train this a whole heap faster and then last but not least we're going to make those real-time detections using opencv and python from within our jupyter notebook ready to do it let's get to it alrighty so there's quite a fair bit to get through in order to build our real-time object detector now broadly what we need to do is first up label our images then set up our training repository train and then make those detections now in order to simplify that process a little bit what i've gone and done is i've created a template project inside of github that you can download and start from now in this case what we're going to do is we're going to clone that repository and kick off from there now in order to get to this repository all you need to do is go to github.com forward slash knick knock knack forward slash real time object detection and you'll be able to clone that so let's first start out by cloning this repository so we can just copy this particular link then go to our alternate desktop and then what we're going to type in is git clone and then the name of our repository and you can see it's cloning and we've already got our file there so this might take a little bit of time to run so let's just let that run and we'll be right back okay so that's our baseline repository clone now if we step into it so in this case you can see it's called real-time object detection we've actually got a pre-written jupyter notebook which i've gone and pre-written which allows you to go through this entire tutorial without writing much code at all so if you wanted to skip through this and not write the code you can just go and use that if we step into this folder called tensorflow this is where the majority of the work is going to happen now inside of here we've got two folders so scripts and workspace and eventually what we're going to do is we're going to clone the tensorflow object detection repository into here as well inside of scripts we've got one file called generate tf record and this is from the official tensorflow object detection tutorial so that basically helps us build a tf record but we'll talk about that a little bit more later then if we open up workspace we've got a couple of folders here so we've got annotations so this is where our label map is going to go and all the stuff that we need to actually train our model we've also got an images folder so when we start our labeling we're going to throw our images in there we've also got a model folder and this is where the majority of our model stuff is going to go and we've also got a pre-trained models folder which already has a pre-trained model so in this case we've got a single shot detector and we're using mobile net and the 320 by pixel version but i'll show you where to download different models and how to use those as well once we get through that part alrighty what we're now going to do is we're going to download label image so label image is an awesome python package that helps you label your images for object detection so what we're going to do is we're going to go and grab that repository so similar to what we did for this particular one we're then going to go to the label image repository now in order to get this you just need to go to this github link but again all of these links are going to be available in the description below so you can grab them and just run with them so we're going to grab this one and we're going to clone that as well and again we're going to type in git git clone and then paste that link and this should appear within our same folder so you can see that we've got label image that's cloning now so let's let that clone down and then we can kick things off okay so that's label image now cloned now what we're going to do is we're going to grab our label image repository so in this case you can see it here we're going to cut that and we're going to paste it inside of our tensorflow folder so in this case we've now got scripts workspace label image and then eventually we're going to have models now what we need to do in order to set up label image is run a couple of commands so let's cd into that folder and then before actually going and using label image we need to also install a couple of dependencies now there's two key dependencies that we need so we need pi qt equals five and we need lxml so i've already got those pre-installed but let's take a look at those commands so in order to install pi qt we just type in conda install pi q t equals five and then we need to install lx mls so let's install that so to do that we just type in conda install dash c anaconda lxml let's make this a little bit bigger so we can see it so you can see we've just read in conda install dash c anaconda lxml so let's run that all right so now that that's done we just need to go and start setting up our binaries so inside of the label image folder so in this case we're still inside of real-time object detection tensorflow label image we just need to run this command okay so what we've written is p y r c c five so this is using our pi q t file and then we're passing through dash o resources dot pi dot resources.qrc this is in the official label image documentation so we're just going through the normal setup step so let's run this and then what we need to do is open up the label image folder and we need to grab this resources python source file and this resources.qrc file and paste it into the libs folder now if you're working on a windows machine then there's one last change that you need to do that i encountered when are setting this up and this is going to just make label image work when we're working with directories so what we need to do is we need to open up our label image folder and then what we need to do is scroll down to line 1114 which is down here somewhere and rather than having a forward slash what we need to do is just change this to os dot path dot sep and then hit save so that's just going to make sure that our label image library works well with the windows file directory all right so now that that's done we can go back and then we can start using label image now in order to use label image we're going to need some images so in this case i've already got some images here so inside about all images folder you can see some of them i have don't have a mask and then some of them i do have a mask now if you want to learn how to capture these images using python as well i'll include a link somewhere above to the video where we actually did this so for the last face mask detection tutorial which wasn't real time we actually went and captured these images ourselves using python and opencv in this case we're just going to reuse these images there's also a great kaggle library that has a great data set on face mask detection as well i'll include a link to that in the description below in this case we're all good so we're just going to grab this folder so all images and we're just going to cut it and then inside of our tensorflow folder inside of our workspace and inside of our images we're just going to paste that there so basically inside of our workspace inside of images we should now have three folders so all images test and train there's nothing in test and there's nothing in train we're going to set that up in a sec so now what we want to do is we actually want to start label image so to start label image what we need to do is we actually just need to run the label image python file so to do that inside of the label image directory we just run python label image dot pi and when we do that we're going to open up label image so what we can do from here is we can actually start to label all of our images now in this case the format that we're going to be outputting to is pascal voc so this is a common format that we use for object detection now in order to make our lives a little bit easier just make sure when you open up label image that you go to view and hit auto save mode so this is just going to make sure that all of your labels are saved now what we're going to do is first set up our save directory so this is where our annotations or our labels are going to go and then what we're going to do is we're going to open up our folder that has our images in it so first up let's change our save directory so to do that we just need to hit change save dear here and what we're going to do is inside of our real-time object detection folder what we're going to do is we're going to save it inside of our images folder or all images folder so basically the file path should be real time object detection which is our github repo tensorflow workspace images all images so we're going to save it in the same folder that our images are in and then to do that we just hit select folder and then what we're going to do is we're going to open up the images that we want to label so to do that we just hit open dear and in this case it's going to be the same folder that our images are in so we can go back select real-time object detection tensorflow workspace images and all images and all of our images are in here so we can hit select folder and this will open up all of our images ready for labeling now a few quick tips when you're working with label image the shortcut w is going to allow you to draw a label and a and d are going to allow you to skip between the next image and the previous image so if we hit w then we're going to be able to access our labeling tool and in this case we want to label from the nose down so this is going to give us the best results on a small data set so in order to label all we need to do is go say around about there and then what we're going to do is select that and then type in the name of our label so as soon as you select your square or draw your square you're going to have the option to type in your label name in this case we're just going to label it no mask and then what we can do to go to the next one is just select d so and then we can hit w again likewise and then no mask and then just keep going through so let's go ahead and power through the labeling of the no mask images and then we'll take a break and then we'll take a look at how to label images with the mask label so in this case let's hit ok and keep going okay so that's all of our no mask images done now what we want to do is a label our mask-based images so again similar process we're just going to hit w and then this time we're going to label our mask so rather than having a no mask label this time we're just going to type in mask and so that gives us our two different labels so the ones without a mask had a label of no mask and the ones with a label have a label of ma or the ones with a mask have a label of mask all right then keep going forward and that's about it so it looks like we've labeled all of our images so if we hit d we're not going next anymore now what we can do is actually take a look at what these annotations look like so if we open up our directory now you can see that we don't just have images we've got all of these xml files as well and ideally we should have double the number of files that we started out with because now we don't have just have the image we've also got our annotation so if we actually open up one of these you can see that we've got a fair bit of information in here so we've got the folder that it's in the file name so in this case whether or not it's jpg or png we've also got the file path uh the source the size and depth so in this case it's an rgb image so you're going to have the width the height and then the number of channels if you've got png you might have a depth of four because there's a transparency or opacity layer as well and then this is the bit that we're most interested in so this object so first up we can see the label so in this case this image is no mask so if we actually go and take a look at the image that's attached you can see that it's no mask and we can also see the bounding box so these two components are most important so this label and then this bounding box so here we've got the different coordinates for the box that actually represents our object so that's about it now you can see that we've got a whole heap but what we actually want to do is we actually want to split this up into training and testing now we're just going to grab the x or top x number to put into training and then grab the rest and put it into testing but really you could use a much better sampling method for this if you want it to be a little bit more accurate or precise in this case in the interest of time we're just going to grab a couple and throw them into train and then grab the rest and put them into tests so let's go ahead and do that so we're going to start up here and then let's make sure we've got some different styles of images down here looks like these are all masked we have some with no mask what we'll do is we'll grab these we'll put these inside of our training folder and then let's take a look make sure we've got some no mask ones and then we'll grab all of these and make sure you select both the image and the label file as well so we're going to grab all of those and throw them into our training folder as well so ideally if we take a look at all of our images you can see that we've got a mix of no mask and we've also got a mix of masks so if we go back into our testing folder or sorry into our all images folder likewise if we take a look we've got some images without a mask and we've got some images with a mask now again we don't have a lot of images so later on when we take a look at performance what you could also do is add more images you could also grab a kaggle data set and pull that down and use that as well and in this case we want to do it pretty quickly so let's take a look at how so we'll evaluate what the performance looks like towards the end and then we're going to grab all of these images and put the rest in test now what we can start doing is actually writing our python code to actually go and build and train this model so what we're going to do is we're going to jump into a jupyter notebook so in this case i've got a jupiter notebook here already and we're going to step into our real-time object detection folder and again if you didn't want to write this from scratch you can open up the tutorial jupyter notebook and it's got all the code you need to get started if you wanted to train this on a custom model or the core thing that you need to do is update this label map but again we're going to be writing all of this from scratch and taking a look if you don't want to write all of this from scratch you can skip to later in the video and we're actually going to be testing it out but in this case we're going to walk through it and actually build it up ourselves now rather than stepping through this entire notebook what we're actually going to do is we're going to write this code ourselves just so we get a better feel for what's actually there so in this case what we've got is we've got a i've also set up a template so this isn't in the github repository but let me know if you want me to post this up it's basically this tutorial without the code so we're going to open up our template and in this case we've got about eight different things we need to do so first up we need to set up our paths then create a label map so label map is what the tensorflow object detection library uses to represent our labels we then need to create our tf record so tf records are the representation of our data that the object detection api uses we're then going to leverage some of our pre-trained models we've already got some but i'm going to show you where to get others then what we're going to do is we're going to set up our model config train our model and then we're going to go and perform that real-time object detection so let's first start out by setting up our paths so in this case we've got a number of paths within our directory so if we actually go and open that up so basically what we're going to do is we're going to set up paths for everything that's within here so it just makes it a little bit easier when we're referring to it in our python code so let's go and do that okay so those are all of our paths set up so there's quite a fair few there but let's quickly go through them so we've got our workspace path and this is our main workspace so if we go back that's basically this let's go back to here that's basically this workspace that we've got in here and basically all the others just refer to the other part so if we take a look at the rest so we've got a scripts path that's where our generate tf record script is we've got our api model path so what we're going to be doing is we're going to be downloading the tensorflow object detection model library from github so we're going to grab that we've also got our annotation path so this is where our label map and our tf records are going to go we've got our image path this is where we did our labeling and where our images currently are we've also got a model path so this is where our trained model is going to go so when we actually train this model what we're going to get is a number of checkpoints that we can then restore from in order to build our real-time detection model or in order to actually make that real-time detection we've got our pre-trained model path so what we're actually going to be doing is we're going to be leveraging pre-trained object detection models to transfer learn and basically build our model a whole heap faster we've then got our configuration path so when we're using the object detection api what we're actually doing is we're using a config file and this basically gives us all the details for the deep learning model that we're actually going to be using and then we've got our checkpoint path so within this folder all of our checkpoints are going to be there so our config path is actually pointing directly to our pipeline.config file all right so now that that's done the next thing that we need to do is actually create our label map now our label map is just a representation of all the different objects that you can expect to find within our trained annotation so those xml files now remember we had one that was named mask and one that was named nomas so what we basically need to do is just create a representation of that so let's go ahead and do it okay so that's our labels dictionary now what we need to do is actually output this into a label map so when working with the tensorflow object detection library this needs to be in the format of pb text so it's protobuff file and this is just what the object detection library likes to use so let's go in and output that okay so that's our label map done now if we go and open up our folder directory we can close label image now we don't need so if we go and open up our annotations file you can see that we've now got this label map file so if we actually open that up you can see that what we've actually got in here is two items so in this case item one or item two and this one represents our mask and then this one represents our no mask down here oh these should actually be numbers so let's just quickly update that code so we just need to remove the quotes here let's try that again and we need to remove this slash save that and run that perfect that's looking better so we've got our mask as a string and our id as a number so if we actually go back now you can see that basically the code that we've written is we're going into our annotation path and we're creating a new file called labelmap.pb text and we're setting that to write because we're going to be writing to it and then we're looping through each one of these labels so in this case we've got two labels so label one and label two and then in order to work with our pb text file we just need to write these out into a specific format so this first line is writing that item line the next line is writing the name line the line after that is writing our id and then the final line is basically closing it out so that's our last set of curly brackets that you can see here okay so that's our label map done now our next step is to actually go and create our tf records now in order to do that what we're going to be doing is we're going to be leveraging the script that we've got within our scripts folder so this particular file here so generate tf record is actually from the official object detection api tutorial now if you actually go into that tutorial there's a whole heap of information on how to actually do this i've just simplified it a little bit but that generate tf record file is so critical because it makes sure that you get your data in the right format now in order to run this tf record or generate tf record script we're just going to be executing a command inside of our notebook so let's go on ahead and write that command all righty so that's all looking good so basically what we've done is we've gone and used the python command and then we've gone into our scripts path and then forward slash generate tfrecord.pi so this is basically allowing us to pass variables from our notebook to our command line so our scripts path which is basically this here so tensorflow scripts slash generate tf record dot pi and then we're doing a similar thing for all of these other components so then we're passing through our image path so in this case we're first up doing our train path and then we're passing through our label map which we just set up here and then we're specifying what we want the name of our output to be so in this case we're specifying dash o passing through our annotation path and then specifying train dot record now this has done it for our training images we also need to do it for our testing images so we can copy this as well and then all we need to do in order to do it for our testing images is change this to test and change this to test okay so if all of that has gone well what you should effectively get out of this is two lines printed out so successfully created a tf record file and then it will specify where those have actually been created in this case they've gone and been created inside of our annotations folder so if we go to that folder so remember our annotations folder is available through real so our main folder tensorflow workspace annotations and in this case here you can see that we've got our label map that we created but we've also got our test tf record which is going to be huge and it's not going to be readable but that's fine and we've also got our training tf record file as well again this one's going to be pretty big because it's got more images all right we don't need to take a look at that and we can close those all right so back to our notebook alrighty so now that that's done the next thing that we need to do is go on ahead and download the tensorflow model library so this is basically everything that you need to actually go and use this object detection api now a key thing to point out is that in order to run this tutorial you need to have the object detection package installed already and i'll include steps to actually do that and let me know if you want a video on that i can definitely make one up and go through that but basically you just need to run through these steps and i'll include a link to this in the description below but what we're going to be doing now is we're actually going to be downloading the tensorflow models github repo so let's go ahead and do that alrighty so under step 3 what we're going to do is just write that command all righty so the commander we're going to be triggering is first up going to cd into our tensorflow directory so this is going to clone it into our tensorflow directory to keep everything nice and neat and then what we're going to be doing is cloning this repo here so if we actually open this up this is basically the tensorflow model garden and it's also got all of this good stuff so this is what we actually need to go and train our model now we can just go and trigger that command and this is going to go and clone that repo so this might take a little bit of time because there's actually quite a fair bit in there alrighty now that that's all done and cloned if we go and take a look at what we've just gone and cloned so if we go into our tensorflow folder you can see that we've now got that available there so this is going to be key to when we actually go and train our model alrighty back into our notebook so the next thing that we need to do is actually go and set up our model config now whenever we're training using the object detection api we basically build a model from a model config now i've actually included one of these already inside of that pre-built repo so if we actually go into the tensorflow folder go into your workspace go into pre-trained models you can see that we've already got this pre-trained model here so it's called ssd mobile net fp and light or v2 fpn light 320 by 320 coco 17 tpu 8 now this is from the tensorflow model zoo if you wanted to go and use a different model however you could definitely go and do that all you need to do is go on ahead to the tensorflow model zoo and there's a whole heap of them here so if you wanted to use a model which is more accurate then basically all you need to do is look at the mean average precision score and then pick your model based on that if you wanted a model that's faster then you can pick one based on speed so in this case what we've done is we've optimized for speed because we're going to be performing real-time object detection and what we've done is we've actually grabbed this model here all you need to do in order to grab these is just click on the link and it will actually go and download that file and then what you can go and do is go and untie that using the tar command now i've actually included that code inside of the baseline tutorial so if you wanted to do that all you need to do is pass through the link to the model that you want to use and then just paste it into there in this case we're just going to leverage our existing model but what we are going to do is we are going to grab that config from that baseline file and use that in our own model config so you can see in there that we've got this file called pipeline config and that basically is a representation of the model so you can open that up and you can see in here we've got the number of classes available we've also got what the image resizing is going to look like so in this case it's going to resize to 320 by 320 and then further down here this is the section that we actually need to configure so we need to configure the fine tune checkpoint so where we want our model to start training from and this is where the transfer learning component comes in because we can basically leverage a pre-trained model and then just fine-tune from there rather than training from scratch we're also going to change a bunch of these other parameters as well to make sure that we get a model that's suitable for us so let's go back into our notebook and what we're going to do is first up copy that model config into our training folder and then we're going to make our changes alrighty so that's our config transfer now if we go and take a look at what that's actually gone and done if we can close that pipeline now so if we go and take a look inside of our workspace if we step into our models folder what this has actually gone and done is created a new folder called my underscore ssd underscore mob net and this is basically taken our pre-trained configuration file from that pre-trained model zoom model that we downloaded from the tensorflow model zoo and pasted it into here now what we're going to do is actually configure this pipeline config to be specific to our model so we're going to point it to our tf records and we're also going to change a couple of other parameters depending on our hardware so let's go back to our notebook and start making those changes so first up what we need to do so if we actually take a look at the commands that we wrote we basically specified the custom model name so if you wanted to change the folder name you can change that and then what we've gone and done is we've used the make d command to go and create that directory first up and then we've used the cp bash command to copy that pipeline config and in this case see this is where all of our pre-trained paths come in handy because we don't need to write them every single time and then we've copied it into the model path which was specified up here over there so slash models and then we've pasted it into our custom model folder which is called my ssd underscore mob net cool so that's now done let's go on ahead and update our config now we need to import a couple of dependencies first so specifically we need to import tensorflow we need to import our object detection utility so including config util and pipeline pb2 so this is going to allow us to work with protobufs and then we need to import text format from google protobuf so let's go ahead and import those okay so now that that's done what we want to do is we actually want to open up our configuration file from our models folder or our custom models folder so first up we're going to set up a configuration path so we're actually going to set up a path to that file then we're going to open it up make those changes so let's first up set it up grab it okay so we've now gone and grabbed our config now i just realized we skipped over our import so let's quickly go through that so we've imported tensorflow as tf and then from our object detection utilities we've imported config util which we're about to use here and then what we've also done is we've imported pipeline pb2 from our object detection protos library and we've also imported text format from google protobuf but we'll talk a little bit more about those once we get to them then what we've done is we've gone and set up our configuration path and this is specifically where our custom model pipeline config resides so in this case it's within our model path within our custom model folder which is my ssd mob net and then we've just grabbed the actual file and then what we've done is we've used our config utilities to go and grab that so specifically we've used the get configs from pipeline file method to go and grab that config so if we actually take a look at this now you can see that that same config that we opened up in vs code or visual studio i can't remember which one we've now brought it into our notebook now by doing this what we can actually begin to do is make changes to that so right now that particular config is configured to go and predict 90 different classes and it's also configured on classification so if we scroll further down you can see that right now our checkpoint type is on classification but we want to change this to detection now rather than going and doing this manually each and every time by opening it up we can actually write python scripts to go and make those updates to our config files so let's go and set up our new config file and this is where our pipeline pb2 function is going to come in handy so we're going to be able to create a shell config and then we're going to write it to our directory and then use that so let's go and make that change okay so now what we've done is we've created a template pipeline config so that's this line here and then what we're going to do is now make those changes so there's a couple of changes that we need to make so i'm going to power through them and then we'll go back and take a look at what we've actually gone and changed hmm alrighty so that's our pipeline config updated so there's quite a fair bit happening there so let's actually take a step back and take a look at what we've written so first up what we've done is we've gone and changed the number of classes that we want to predict and this is the same number as the number of classes that we've actually labeled within our images in our case it's two classes so mask and no mask then what we've gone and set up is our batch size so pipeline config train config dot batch size and we've set that to four this is really dependent on how much hardware capacity you've got available so if you've got a gpu you might be able to bump that up higher if you don't you might want to drop that lower then we've also specified our fine tune checkpoint so remember we downloaded our pre-trained models or i had them inside of that github repo what we're basically doing is we're telling our config where we want to start training from and in this case the checkpoint that we want to start from is called checkpoint0 so this file is actually within our pre-trained models directory so if we go back to our workspace go into pre-trained models go into this folder here which is just from the model zoo you can see that within our checkpoint folder we've got our checkpoint so these actually represent where our model was last trained to so we can pick up from there and then just fine tune from there so now if we go back in then we've also specified that we want to train a detection model we've specified where our label map path is in this case it's our training label map path then we've specified where our training records are so remember we created those tf records just up here or there what we're now doing is within our configuration we're saying that our training records are within our annotation path and then they're actually in the train dot record file likewise we're doing a similar thing but these two lines are doing it for our testing components and if we actually take a look at our pipeline config you can see that this has actually gone and made those updates but we haven't actually gone and updated that file within our directory yet we've just got it inside of memory within our notebook so what we actually want to do is go and write out this config file and once that's done we can actually go and train our model so let's go in and write it out to our directory okay so now what we've gone and done is we've written out this pipeline config or our new pipeline config to our directory so in this case what we've done is we've used our text format utility so remember we brought that in from google.protobuff and what we've gone and done is we've converted our protobuf message to a string then what we've done is we've gone and written it out using these two lines here so if we actually go and take a look within our directory now inside of our custom directory you can see that our pipeline was just updated so now we've gone and written down our new pipeline we can even open that up and you can see that we've got our two classes and if we scroll all the way down you can see that it's been updated with all of our new fine tune checkpoints and all of our pv.txt files as well as where our training and testing records are so that's looking good now that that's done we can actually start training our model so all right now in order to train our model again we're going to be now using the tensorflow models directory so this is where we start to get to the good bit and actually go and see how our model is going to perform now what we're going to do is we're going to write out our command inside of our notebook but then we're going to paste it inside of a separate command line terminal just so we can see the status as it's updating because sometimes it's a little bit glitchy inside of a jupiter notebook so let's go on and print out that command then we can copy and paste it and start training alrighty so that's all looking good so what we've gone and printed out in this case we've gone and used some string formatting to pass through our different paths again this is where they come in handy and our baseline script to go and train this model is python and then we're going and using the model main underscore tf2.pi so we're using tensorflow 2 for this particular training model and then what we're doing is we're passing through where our model directory is so in this case it's our custom model directory so tensorflow workspace models my ssd underscore mobnet then we're passing through where our pipeline config is in this case it's again within our custom model directory and it's called pipeline.config and then we're specifying the number of training steps so how long we actually want to train our model for and in this case we've specified 5 000 training steps but again you could drop it down you could change it it really depends on how accurate you want your model to be in this case i've tended to found that 5000 tends to work pretty well you can drop it down and it's still going to perform reasonably well as well and in this case what we're going to do is just throw it all on one line so it looks like that and we're going to grab this line so we're going to copy that and then we're going to go into a folder that we've actually got a directory in and we're going to create a new command line then in order to run this what we're basically going to do is navigate into our real-time object detection folder so let's do that okay so we're in that folder now now all we need to do is paste in that command and this is going to kick off our training run so it'll take a little bit of time to start but once it eventually kicks off you'll eventually start to see the loss metrics updating in real time so let's give that a second until we see our first training steps and then we'll go from there and there you go so that's our first loss metric appearing here so if we open that up you can see that these two lines here identify the fact that our model is training okay so you can also see the time it's taken to train per step as well as our current loss now ideally what i've tended to see is that if you get this loss to below 0.15 the model tends to perform a lot better but again train it a little bit see how it performs train a little bit and then test it again in this case we're going to step away and let this train for the next 5 000 steps alrighty so that's our model finish training so you can see here that we've trained for 5 000 steps and right now our loss is at about 0.183 so you can see it sort of varied throughout time but overall it looks like it's it's gotten pretty low again we can train for longer if we want at the moment let's run with that and take a look at how it looks once we perform our real-time object detection now i should also point out that you could also add additional photos so i'll include a link to the kaggle face mask directory so there's a whole heap of additional face masks or labelled face mask images that you can also add to this to improve your accuracy in this case we've only trained on maybe like 40 images or fine tuned on 40 images so that's not too bad for now so let's go back into our notebook and there's a couple of things that we've got left to write so specifically we need to now load in our trained model so if we actually go and take a look at our checkpoints we started off with the one we're now up to checkpoint six so we want to load that checkpoint into our pipeline and then we actually want to go and make detections and specifically real time detections using a webcam so let's go ahead into our notebook and get started on that so in this case we're now up to step seven so we've done quite a fair bit so far so we've done all of this now we're just up to seven so load our model and detect in real time so we're almost there now the first thing that we need to do is first up we need to load some dependencies so in this case we're going to be importing our label map utility as well as our visualization utility from the object detection libraries and we're also going to need the model builder method so let's go ahead and import those dependencies and then we'll take a look at those okay so those are our dependencies imported so what we've done is we've first up imported os operating system then we've imported three things from the object detection library so first up we've imported label map util so this is going to allow us to work with our label map remember we set that up right up here then we've also imported some visualization utilities so this is going to allow us to transform our detection to an overlay on our image so you'll actually see the bounding boxes and then we're also importing our model builder so this allows us to build our model from our checkpoint and our config file so now let's go ahead and load it up and then write our detect function okay so that's our model reloaded so let's take a quick look at what we've actually written there so first up what we've gone and done is we've grabbed our pipeline and so keep in mind this is the pipeline that we went and modified then we've gone and built our model using our model config and we've specified that it's not training at the moment then from that what we've gone and done is we've gone and restored our checkpoint so in this particular case we've gone and passed through our detection model to tf.compat.v2 and we've specifically passed through this detection model here and then we've gone and restored it now in this particular case what we've gone and done is we've restored checkpoint six so this is the latest checkpoint that we actually went and trained so if we take a look back in our training folder checkpoint 6 is our latest one if you train for longer you might have additional checkpoint numbers use the latest one or use the latest one that doesn't have any issues and this ideally should give you the best results but again it's dependent on the loss that you've got and whether or not your your model performed well all right so let's go back into our notebook so that's loaded now what we want to do is write our detect function so specifically we want a function that we pass our image to and then we get our detections so let's go and write that up okay so that's our detect function done so basically what we specified is first up this is a tensorflow function then we've called it detect function so we've basically used our standard python formatting here so def detect function and then we're passing through our argument which in this case is going to be our image then what we're doing is we're first up pre-processing our image so we're going to resize it down to the 320 by 320 and then we're going to go and use our detection model which we've defined up here to go and make that prediction then once we're done with that we're then going to pre-process it again and finally return our detections now what we can go and do is actually go and detect in real time so this is the final bit of this tutorial so once we've finished this detection function we're actually going to be able to leverage everything that we've just gone and done to go and detect in real time so in this case we're going to be using a webcam to make real-time detection so you'll actually be able to see the face mask or whether or not i've got a face mask on my face so let's go ahead and write up this function to go and make our detections in real time and also just before i keep going ahead remember that all of this is available in the pre-written tutorial so if you didn't want to write all this code or if you wanted to sort of skip through it you can just use that notebook and it's going to give you the same results all right so first up what we've done is we've imported cv2 which is open cv and we've also imported numpy so as mp so this is just going to help us work with our images now what we need to do is just create a category index so this is again a representation of our label map so that again it's come in quite in handy so let's go and create a category index and then what we're going to do is set up our capture from our webcam all right so that's our category index done so in order to create that what we've basically done is we've written label map or we've used the label map utilities and we've used the function create category index from label map and to that we've passed through our label map path now what we can do is we can actually output this so if we take a look at it it's going to be really similar to what we wrote just before for our label map except this time what you've got is a key per label map so in this case 1 is going to be mars key 2 is going to be no mask now what we can go and do is actually set up our capture from our webcam but we won't run it yet we'll wait till we've got everything written and then we'll trigger that so let's set up our capture and let's set up our real-time object detection all right so that's our video capture done so this first line here is basically cv2.video capture and that allows us to access our webcam now you might need to play around with this number so in this case my webcam is device zero yours might be one or two or three just play around with that if it's not working or drop a comment below and i can help you out and then we've also captured our width and our height now the next thing that we need to do is actually loop through set up our webcam make those detections and render the results to the screen so this is quite a large bit of code so let's write it all out and then we're going to step through like we usually do hmm you so first up what we've done is we've basically set up a loop so while true it's going to keep through and running this first up what we're doing is we're getting our frame from our video capture that we set up up here and then we're converting that frame into a numpy array so when we pass through our image we want it to be a numpy array in order to work with tensorflow then what we're doing is we're converting it into a tensorflow tensor and to that we're passing np dot expand dim so this is just going to put it inside of another array because tensorflow or our model expects multiple images to come through in this case we're only passing through one and we're setting the d type to float 32 then we're actually going and making our detection so here we're using a detect function which we defined here and we're passing through our input tensor then what we're doing is we're grabbing our number of detections because we're going to do some preprocessing on them over here and then we're adding those back and then last but not least we're going and grabbing our detection classes then we're specifying a constant for our label id offset so when we get our detections through they're going to start at zero we want them to start at one because our category index starts at 1 and then we're using and then we're making a copy of our detection so in this case we're grabbing our image.np up here and we're calling it image mp with detections this is because when we pass it through to this function here so visualize boxes and labels on image array it's actually going to transform that array so ideally we want to have a backup just in case we want to make some changes to it so to that function so we're using vis.utils and then visualize boxes and labels on image array we're passing through that image array that we had up here so this is our raw image from our webcam and then to that function we're passing through our detection boxes our detection classes as well as our detection scores so ideally when we go and visualize you should see the detection on our face and you'll also see the score and the label in the top left hand corner then we're passing through our category index whether or not we want to use normalized coordinates or not in this case you do want that because it's going to make sure that it positions the box correctly on your face then we've also got a number of other parameters so the number of boxes we want to draw in this case we just want the one the minimum score threshold in this case we're going to set it to 0.3 again you can play around with this and whether or not we want agnostic mode set to false or true so if we set that to false it's going to see that there's a detection but it's not going to differentiate between mask or no mass then what we're doing is we're displaying it back to the screen so we're grabbing that transformed array which we had up here and we're going and showing it using cv2.imshow and then if we want to break out of it we've also set up our regular break key setup that we've got for cb2 so then what we're going to do is we're going to start up our video capture so this will start up our capture for our webcam and then what we can do is we can actually start making these detections so all things holding equal ideally what you should see is a pop-up on the screen inside or outside of your jupyter notebook that's actually showing you those detections so let's go and run that and see what our results look like might take a little bit of time to start up as well but it should start up oh and we've got a bit of an error let's take a look and this looks like it's an error within a detections function so let's scroll back up let's take a look at that up this should be post process let's do that again and let's try that again and this should be a comma all full of bugs today and there you go so let me drag it onto the screen so you can see what i'm seeing so this is our real-time object detection so you can see it's obviously detecting that i don't have a mask if i cover my her face looks like it's still performing all right now if we do put a mask on you can see that it's accurately predicting that we've got a mask now keep in mind that this has only been trained with a couple of images and it hasn't been trained all that long so you're going to have varying levels of accuracy so i mean it's 96.97 but say for example you took the green screen down then you might have a different level of accuracy so if we bring this down you can see oh it's still performing all right there you can see every now and then it might blow up now the way to improve these models is to train for longer to provide more data and you can also perform a number of image transformations so say you performed a a shift or a split or you applied some blur um so there's ways to improve these models so let me know in the comments below if you'd like to learn more about how to perform some of those transformations to improve the accuracy of these models but you can see then it's actually performing pretty well if we put a mask on looks like it's detecting our mask we try with a different colored mask so in that case that one's black you can see it's detecting that one pretty well and that's real-time object detection so there you go so we've gone through end to end so we've labeled our images we've set up all of our pipelines and we've even gone and trained our models and in the end you can see that we've gone and built our real-time object detector which is detect still detecting in real time and it's performing pretty well and that about wraps up this video thanks so much for tuning in guys hopefully you found this video useful if you did be sure to give it a thumbs up hit subscribe and tick that bell so you get notified of when i release future videos and let me know what kind of object detector you want to be building thanks again for tuning in peace

Channel: Nicholas Renotte

Views: 235,730

Keywords: object detection, tensorflow object detection api, face mask detection, custom model, deep learning tutorial

Id: IOI0o3Cxv9Q

Length: 58min 42sec (3522 seconds)

Published: Sun Nov 01 2020