Building a Real Time Sign Language Detection App with React.JS and Tensorflow.JS | Deep Learning

[Music] i think you guys want a custom object detection video with tensorflow.js let's do it what's happening guys my name is nicholas trunot and in today's video we're going to be building our very own real-time sign language detector using tensorflow.js and react let's take a deeper look as to what we're going to be going through now in this video we're going to be covering a bunch of stuff so we're going to go from end to end from exporting our pre-trained tensorflow model to integrating it into a full-blown react jsr so the first thing that we're going to do is convert our pre-trained tensorflow model into a tf js friendly format then what we're going to do is take that same model and host it on cloud object storage so this is going to allow us to work with it inside of our application we're then going to leverage a pre-ridden computer vision template which i've used in a previous video so this is going to make it like a whole heap easier when building it up we'll then use that app to make real-time detections using react.js and our webcam and then last but not least you're actually going to be able to visualize these real-time detections on the screen so you'll actually be able to see all the different sign language models that we've actually got pre-trained now the cool thing about this video is that this could be easily applied to a whole heap of other different use cases so say for example you wanted to build a stop sign detection algorithm or say for example you wanted to build something that's more along the lines of microscope detection you can use this exact same app and these same steps to convert your model and deploy them all the way through to a front-end app and off on that let's take a deeper look as to how this is all going to work together alrighty so there's six key steps that we need to go through in order to get this up and running so first up what we're going to do is take our pre-trained tensorflow model and convert it to a tfjs graph model so we'll be able to take the model that we pre-trained in our sign language detection video and use that and i'll also include the model details for that so if you want to just download that you don't have to go and walk through that entire video yourself we're then going to host that on the cloud object store so i'll show you how to set all of that up we need to make sure that cause is set so that we can access it through our app so this basically means that our app can go out to other web pages and pull down our model we're then going to build it up in react.js and build our standalone front end application step five we'll use our webcam to detect images and make object detections in real time and we're going to wrap it all up together inside of our react js app and again we're going to be using tensorflow.js ready to do it let's get to it alrighty so without further ado let's jump right into this so in terms of what we've actually got to get through there's eight key steps that we need to walk through and we're going to be tracking to this markdown document thanks for the subscriber that gave this suggestion so we're actually going to be going through each one of these steps and as we get them done we're marking them off so just quickly we first need to export and convert our pre-existing python model and i'm going to make this entire repository available through a github link just check the description below as well as the completed code will be available there so the core thing that we're going to be taking a look at are these pre-trained checkpoints so these pre-trained checkpoints effectively represent our pre-trained model and are going to allow us to perform our real-time sign language detection so once we've exported that and converted it to a tf js friendly format then it's pretty straightforward from there so we've just got to export this and host it on cloud object storage so this is going to give us a place to host our model then what we're going to do is we're going to clone down a computer vision template so this is going to allow us to get a kickstart in building our app we're going to install and update some dependencies load our graph model from our url so from this particular step here we'll then make some detection so we'll clean up our computer vision template code to work with our new model we'll then create a drawing function and a label map so we'll be able to actually map our detected functions to our pre-trained labels and then what we're going to do is we're going to draw it to the canvas so we're going to specifically use a function called requestanimationframe to give us a nice smooth transition for that all right so let's get started so the first thing that we need to do is export and convert our python model now within the repository or within the baseline repository that i'm going to link below you're going to have a bunch of stuff but specifically you're going to have all the pre-trained models that you need to kick this off so it doesn't need to be gpu intensive you're going to be able to leverage all the stuff that we've done before specifically what we're going to be taking a look at is everything under this models folder so if you take a look inside of the folder let's just open it up so specifically we're going to be working inside of this custom object detection folder and we can just make it a little bit bigger so if we look inside of there we've got a bunch of files so we've got label image models scripts and workspace again we'll make this bit bigger so if we click on workspace and models and my ssd mob net so there's a bunch of stuff in here but specifically you can see all these checkpoint files so these checkpoint files are the files that we're going to need in order to host our model but what we really need to do is convert this into a tensorflow.js friendly format so we're just going to create a folder and we're going to name it converted and inside of that folder we're going to be exporting to there so basically everything that we're building from there we're going to be exporting to that so there's a pre-existing export folder and two pre-existing converted folders so if you wanted to leverage the ones that we've done before you can if you want to do it from scratch and take a look and specifically if you're working with different models you'll do it from scratch but we're going to do it now anyway so you can see what it looks like all right so inside of this particular folder so my ssd mob net we're going to create two new folders one called converted and one called export so first up we're going to freeze our inference graph and export it to here and then what we're going to do is we're going to convert that to a tensorflow.js friendly format and we'll save it in our converted folder so we can close that now what we're going to do is jump into a jupyter notebook and inside the repository you're already going to have a notebook called tutorial so we're going to be working with that so let's jump into that now so again this is our custom object detection folder we're going to open up our tutorial and you can see there's all the code here that allows us to train our model from scratch so if you wanted to do this for a completely different use case not sign language detection you could definitely do that so say for example you wanted to detect different street signs or if you wanted to detect issues or like for example skin cancer lesions you could definitely use this code there to go and run through that in this case we've just got two last steps that we need to go through in order to get our model up and running so we need to save our graph and convert it so let's go on ahead and start doing that so it's a bit of a long command but we're going to take it step by step and walk through it now before we go on ahead and export our model we're just going to make sure that we run our path script or cell so this is going to contain all of our paths so we can easily pass it through to our final command and we also need to run this one here which says custom model name now what we can do is actually start writing our python script so let's do it and we'll just make this a bit bigger and let's start writing our export script alrighty so that is our python script to save our graph done so let's walk through what we've actually written there so we're actually going to take this command and run it from within our directory but before we get to that let's actually walk through it so all right so what we're going to be doing is first up we're going to be using our exporter main v2 script so this comes from the object detection api library so what we're going to be doing is first traversing to there so tensorflow models research object detection exporter main v2. and then we're passing through a bunch of options so we're specifying our input type in this case it's going to be an image tensor we're then specifying our pipeline config path so this is intense flow workspace models my ssd mobnet and pipeline.config i'll actually show you where these are hold up if we just drag this over here and make that look like that so if we go into custom object detection so what we're actually doing first up is we are going into tensorflow models and then we're going into research and then object detection and then from here you can see that we've got a bunch of export scripts the one that we're going to be using is exporter main b2 so this one down here so that's what this first line is doing then what we're doing is we're specifying image sensor that's fine all kosher then what we're doing is we're specifying where our pipeline config part is so if you watch the sign language detection video from start finish you'll see how we actually build a pipeline config i'll include a link somewhere above that you can take a look for that alrighty so pipeline config so if we take a look this is going to be stored in tensorflow workspace models mob net pipeline.config so if we go to tensorflow workspace models myssdmobnet you can see that we've in fact got a pipeline.config file there and then what we're also doing is specifying where our train checkpoint directory is so tensorflow workspace models my ssd mob net and these are all of our checkpoint files here again this is going to be available in the repo that you can download and work with then we're specifying our output directory so tensorflow workspace models ssd mob net forward slash export so it's going to export into this folder now what we need to do is run this so we're just going to put this all onto one line just so it's a little bit cleaner and then we can paste it into a command prompt or a terminal if you're on a mac and we should be good to go let's make this full screen again alrighty so we can copy this and then if we open up a command prompt i'm just going to navigate to our folder and we are in custom object detection and if we take a look you can see that we've got all of our folders in there so what we can do is now run this python script and we should have an exported model so let's go ahead and do it looks like we didn't include so we just need to make sure we've got a space in between each of these otherwise we're going to get some errors let's just check that again all right let's rerun that so that error is basically saying that we didn't pass through an output directory but we did we just didn't have a space so it didn't detect it so let's try that again all right that's looking successful now so you can see that it's running and it's now going to export so we'll be right back as soon as that's done perfect so that looks like it's done so let's go and check it out so if we go into our export folder you can see that we've exported our model successfully so you can see we've got our checkpoints here we've also got our saved model and we've got our new pipeline.config now the next thing that we need to do is convert this into a tensorflow.js model so the first thing that we need to do is actually install the tensorflow.js python libraries so this is a converter library or a library that's going to help us get our model from python script to javascript so let's go on ahead and install that so really all we need to do is run the pip install tensorflow.js command all right so that's really the command there so we just need to type in exclamation mark pip install tensorflow.js and run that perfect so i've already got it installed so it was pretty quick but if you're going from scratch might take a little bit longer but really once you've got that installed then you can begin writing the command to convert this to a tfjs model so let's start writing that command then we'll break it down alrighty so there's quite a fair bit that we've just written so let's take a step back and take a look at what we've written so the first command that we're going to be running is tensorflow.js converter so this comes pre-packaged with tensorflow.js and it's going to help us export our model then what we've got are a bunch of input flags so specifically the first one is the input format and in this case we're passing through tf saved model because when we exported it we actually created a saved model format and if you take a look inside of your models and export a folder you can see that we do indeed have our savesmodel.pb or protobuff file so this is going to allow us to convert our model then we're specifying a bunch of output node names now this may change or differ depending on the type of model that you've trained in this case these are the ones that we're going to be using so specifically we're going to be capturing our detection boxes detection classes detection features detection multi-class scores detection scores number of detections raw detection boxes and raw detection scores again this is all going to be available in the repo so if you didn't catch it by all means do check that out then we've specified our output format so in this case we're going to be passing through a tfjs graph model because when we go and use tensorflow.js later on we're going to be loading our graph model using the load graph model method then we're passing through our signature name so when we train our object detection models using the object detection api you're going to get a signature name and by default or more often than not it's going to be serving default then what we're doing is we're specifying two file paths so the first one is where our saved model actually is so in this case it's going to be taking a look at tensorflow workspace models my ssd mob net export and saved model that's exactly what we're looking at here and then we're specifying where we want it to go so again it's going to be tensorflow workspace models my ssd mob net converted so in this case that's going to be this folder here so that's that blank folder that we created right at the start again what we're going to do is put this all onto one line and then execute it so let's go ahead and do that and again remember you just need to make sure that you have a space between each one of these flags and then we can copy this and again we're just going to go back to our command prompt and then run this sweet so that's our model effectively exported so if we go into our export folder or our converter folder sorry you can see that we've now got our model successfully converted so we've got our model and then we've got a bunch of these shard bin files so our model this is going to be the most important thing so if we actually open this up you can see it's a bunch of json that represents our trained model now what we're going to do let's just quickly check our to-do list so we have now successfully completed this one so we can now mark that off as done alrighty now the next thing that we need to do is just go on ahead and host our model inside of cloud object storage so the reason that we do this is that so our app can globally access our model so let's go ahead and do this so in order to host our model what we're going to be doing is using cloud object storage inside of ibm cloud now if you don't have an instance created you just need to go to cloud.ibm.com forward slash object storage forward slash create so if you just select the light tier it's all free more than enough to actually get this started and then hit create so this will create a new cloud object storage instance then what we're going to do is create a bucket so let's go ahead and do that so i'm just going to jump into my existing cloud object storage instance and create a bucket to hold our assets so what we now need to do is just hit create bucket over here so if you take a look you can see it's just there so let's go ahead and create a bucket and what we're going to do is just hit quickly get started so this is going to allow us to quickly set up a bucket so if we go and select next and we're just going to name it tensorflow jsrealtime model so you can name it whatever you want just make sure you follow the naming rules and then hit next so once we've named our bucket we just need to upload our files so in this case we're going to drag all of our model files that we need so our bin files as well as our model json and we're just going to drag it into here and we're just going to let that upload so as soon as that's finished uploading we're going to set our bucket to public and then we're going to use the ibm cloud cli to make sure that we've got cause enabled so cause basically allows us to access our files from our application oh yeah so we've got upload success and our files have been successfully uploaded what we can then do is just hit next and we can see all of our bucket details we can then go into bucket configuration and from here if we select access policies what we want to do is make sure that we've got public access so this is going to allow us to work with our bucket from within our app so in terms of this all we need to do is just hit public access from up here so if we hit access policies public access and then hit create access policy and enable this is fine don't worry about the warning so in this case we're going to we actually need public access in order to work with it from our application so that's all good now as i said the next thing that we need to do is just make sure that we've got cause enabled for our bucket so cause stands for cross-origin resource i think sharing or scripting so this makes sure that our app can work with this particular bucket now in order to do that what we first need to do is make sure that we have the ibm cloud cli installed so if you're doing this for the first time just make sure you select the right installer so in this case if we're working on mac select the mac one windows one select the windows one windows 32 so on and so forth so if you just click that and run through the installation pretty straightforward then what you need to do is you need to install the cloud object storage plug-in so it's just this command here so let's go and take a look at how to do that so i'm going to log into ibm cloud using the cli first so to do that we can just type in ibm cloud login and this is just going to require your regular ibm cloud login details and then just select an account so i've just selected my account and then what we're going to do is install this plugin so i've already got it installed but if we need to install it again we're just going to copy this command and paste it into our command prompt and in this case it's asking us do we want to reinstall it or not i don't need to reinstall it let's just hit no so once that's done so once you've finished installing the cloud object storage cli or the plug-in you just need to make sure you target the right region now in order to determine the right region if you just go back to your buckets and in this case we're going to type in the name of our bucket and it's going to search for the one that we need and i believe it was this one that we're using here so tensorflow.js real-time model so you can see that the location is in aucid so we're going to target that region so let's go ahead and write that command cool so we've now targeted that region now what we need to do is just pass through some configuration in order to enable cause now in order to do that we just need to write a little bit of json so i've pre-written it so we can take a look so if you take a look at what we need to write it's basically one big json string and again i'll make this available in the github repo so you can grab this relatively simple so it's basically stating that we're passing through cause rules and we're saying that we're allowing all headers allowing all methods or in this case sorry we're allowing the get method because we're going to be getting our model and then we're allowing all origins so this just makes sure that our app can connect to that model that we've set up so we're going to paste this inside of our d drive or inside of our model folder so let's go and grab it so this is our cause config i'll make this available in the github repo as well and we're just going to paste it in here then what we're going to do is we're going to run a command to send this config to our bucket so let's go on ahead and write that command okay so before we run that let's just quickly take a look at what we've written so basically again we're using the ibm cloud cli we're accessing the cloud object storage service and we're running the command bucket cause put so this is going to pass through those course credentials that we've just set up or those cause rules then we're specifying the bucket that we want to send it to so in this case we're specifying tensorflow.js real-time model which is the name of our bucket remember from up here and then we're stating what type of configuration that we're sending through so dash dash cause configuration and we're passing a file and that file is called cause config.json which is this file here and keep in mind we're inside of the same folder that our file is in right now so if it wasn't in the same folder then we'd need to pass through the full path so let's go ahead and run that and hopefully we should have a cause set perfect so we've got a successful message so it says okay and we've successfully set cause configuration on this particular model all right so that's our hosting done so if we go back to our to-do list we've now done this now onto the good beat so we've taken our python model we've converted it we've exported it we've hosted it now we can start building our js model so the first thing that we're going to do is we're going to clone our real-time computer vision template from github and then we're going to install our dependency so let's go on ahead and start doing that so in order to clone we're going to stay inside of our custom object detection app folder and we're just going to grab the link that we need so in this case the link that we're going to need is just github.com forward slash knick knock knack forward slash react computer vision template so i set up this template following a couple of the computer vision videos that we did so basically this gives you a bit of a shell to be able to build some real-time computer vision apps with react so we're going to grab this and we're going to go back to our command line and just type in git clone and then our link and then if we hit enter this should create a new folder inside of our custom object detection app folder which includes our template so let's run that reasonably easy step then what we're going to do is just open it up inside a vs code so we'll actually get a cd into our template and then open it up perfect so now what we've got is all of our template codes if you actually take a look inside of the source folder there's a bunch of stuff already pre-written let's make that a little bit smaller so we've already got all a bunch of to-do's that we can actually go and step through we're actually going to walk through this and build it up if we go and check our to-do list again we've now done that and what we now need to do is install and update our dependencies so by default when we clone our template we're not going to have our modules or our libraries or our dependencies installed so we've got to do that ourselves so in order to do that we're just going to go into our package.json file and this is going to contain all of the dependencies that we need for our particular application now it does include some other legacy ones that we use for our real-time object detection tutorial so we're going to rip a few at so specifically we don't need that one anymore which is tensorflow dash models forward slash coco ssd so we can remove that and we're just going to make sure that we've got an upgraded version of tensorflow.js so at tensorflow forward slash tfjs we're just going to make sure that we have 2.7 and we can also remove finger pose we're not going to need that as well so you can remove that line and hit save then what we can do to install these is just create a new or open up a new terminal and then to install all of the packages that we've got in here we just need to type in npm install and this will take a couple of minutes but once you've got that installed you'll have all your dependencies ready and good to go alrighty so those are our dependencies now installed so you can see our node modules folder is now appearing there and we've got all the stuff that we need now installed now let's go on ahead and quickly update our to-do list so we've now gone and installed and updated our dependencies so we can mark that as done as well so let's jump back into our application and start making our updates so the first thing that we need to do is we can actually skip this we don't need to import a pre-built model because we're going to be loading from our graph model that we hosted so what we're going to do is skip to do as well so we'll skip to do number two so we're going to do that in a second so we'll actually come back and do our drawing at the end the first thing that we need to do is actually load our network now we're going to be loading this from our hosted model so what we need is we need our link to our hosted model first up so let's go to our bucket and step into that and the link that we need is actually the link to the model.json file so let's wait for that to refresh so the way to get the link is just we're going to select our model.json file so these three little dots over here hit object details and then we just need this object public url so if we copy that go back to our model we're going to paste that in here temporarily and actually write our code to actually load it up from there so let's go ahead and load it up first and then we'll take a look at our code okay so we've got our link and we're just going to paste it in here so what we've written is we've created a new variable called net so this represents our neural network and we're going to make it asynchronous so we're going to wait for it to load and then using the tensorflow.js library that we loaded up here so this was part of the pre-built template we're going to use the load graph model method to load our model from that link that we just created so remember this is from our bucket and this is where it's currently hosted perfect so that's now done so again if we take a look at our to-do list we've now loaded our graph model from our url now the next thing that we want to do is actually start making our detections and clean up the rest of our code so let's actually start writing our detection code what we're going to do is we're also going to change this number here so we're going to set that to 16.7 so this basically specifies how frequently we're going to make detections using our webcam so 16.7 seems to be the appropriate number depending on your monitor's refresh rate but it's going to give us a smooth detection alrighty so what we want to do now is scroll on down to here and start making our detection so we can delete this line and start writing our code to make detection so let's go ahead and do that okay so that's our first bit of code that we're going to write so the first thing that we're doing so there's a couple of steps that we're actually going through here so we've got these five new objects that we've just gone and written so the first one uses tf.browser.frompixels and to that we're passing through our video so our video is actually being drawn from our react webcam component so this is pre-configured but if you want to see how all of this has been set up the facial landmark detection video that we created a while back is a great example of that so this is basically going to get a video feed from our webcam then what we're going to do is convert that to a number of pixels using the tf.browser.frompixels method and we're going to store it in a variable called image the next thing that we're going to do is we're going to resize this image so we're going to use the tf dot image dot resize by linear method and then we're going to pass through our image and reshape it to 640 by 480. so this is the size of our webcam feed that we're getting and then what we're going to do is we're going to cast it into an integer 32 so this is just going to make our model perform a little bit better because that's what it's expecting then we're going to expand dim so this is just going to put it in another set of arrays this is how a model expects the feed to come through and we're going to store that result inside of a variable called expanded then this is where the magic happens so using our network that we're passing through to this function so remember we set up our network up here then we're calling this function that we're actually writing in so that function's called detect we're passing through our network so using our network we're going to use the method execute async and then we're going to pass through this pre-processed image ideally if all things holding equal we should get some detections inside of our object variable so this is going to allow us to have new detections from our model but before we actually run this what we actually want to do is make sure that we manage our memory so tensorflow.js can be a really memory hungry application so what we want to do is make sure that we delete and get rid of any existing tensors so this is going to make sure that we keep our memory as tight and as up to date as possible so to do that we can use the tensorflow.dispose method so what we really want to do is we want to dispose of image resize casted expanded and object in preparation for our next detection so we're going to do that just under our draw something method down here so we're eventually going to draw our results to the our screen here but we want to make sure that once we've drawn it we get rid of anything that's remaining so let's go ahead and dispose of those cool that's good so and then what we're also going to do is we're going to console log out at objects at the moment we're not actually going to see any results rendered to the screen but we ideally want to make sure that we're going down the right track so if we console log out this object we should be able to see all of the results from our detections at least so let's save that and start up our app so in order to start up our app what we're going to do is just run the npm start command so if we type in npm start this will start up our react app and ideally it should open up a new browser and if you go to localhost 3000 if it doesn't open automatically you should be able to access your app there so let's just wait for this to compile and then we'll be able to see some of our results perfect so you can see that our webcam is on our screen you can see the green screen behind me if we just go on an inspector we'll actually take this down once we're starting to draw properly if we go to our console ideally in a second what we should start seeing is some detections and there you go so we can now start seeing some of our detection so from here we've got a whole heap of tensors now each one of these represents a separate thing so if you actually open these up you're going to be able to see each of the different detections now the ones that we're most interested in are number one which represents our different boxes so this represents a set of coordinates so x y width and height we also want number two which represents our classes so this represents what's actually been detected so is it thank you is it yes is it no is it hello is it i love you so those are going to be the different sign language models that we've actually got available here and then the last one that we're going to want is scores so this represents how confident it is that what's being detected is actually accurate now right now again this isn't all that pretty to look at we've just got a bunch of tensors being consoled logged out what we actually want to see is we actually want to see some stuff drawn to the screen so let's go back update our app and then we'll take a look of what this looks like once we're drawing to the screen and while we're at it we can update our to-do list so in this case what we've done is we've now started making detections we just need to complete step seven and eight and then we're all done so let's jump into our app and then what we really need to do if we can stop our app right now so what we really need to do is finish up this drawing step here and actually bring in our drawing here now in order to write our drawing script we're going to create a new file and we're going to call it utilities.js and then the first thing that we need to do is we need to define our label map and then we're going to define a drawing function so our label map basically is a dictionary that represents all of the different sign language poses that we're likely to encounter from our model so the five different ones that we're going to have are hello i love you thank you yes and no so let's go on ahead and define our label map now as well as defining those labels we're also going to define different color properties so this is going to allow us to have some nicer formatting when we actually draw to the screen so let's define it and then we'll take a look awesome so what we've now gone and done is we've defined our label map so in this case whenever we get our detections what we're actually going to get from our text is just a number so we're going to get 1 2 3 4 or 5. now we don't actually want to just display the number to the screen we want to display the actual label so when we actually go and display the label we're going to convert our number to this particular name here so if it's one we're going to print out hello if it's two we're going to print thank you three i love you for yes five no and now we've also specified some colors so when we actually go and draw the screen we're going to be using these colors so let's go ahead and now define our drawing function so our drawing function is just going to be called draw rect and to that we're going to pass a number of parameters so specifically we're going to pass our boxes we're going to pass it our classes our scores the threshold so we're also going to specify thresholds after which we don't actually draw to the screen we're also going to pass through our image width image height and our canvas so let's start defining this okay so this is our initial drawing template now we haven't actually done any drawing here but i wanted to take a step back and explain what we're actually going to be doing so as i said we're going to be defining a function called drawrect so this is going to be our rectangle drawing function and to that we're going to be passing those three different arrays that i mentioned that we're going to need so our boxes our classes our scores our threshold our image width our image height and our canvas then what we're going to be doing is looping through each one of the detections inside of our boxes array so for that we're going to be using a for loop and we're going to loop through each one of those we've then got a condition which checks if we've got boxes if we've got classes and whether or not our score surpasses the threshold so this basically ensures that we've got valid detections now let's wrap this up and actually do our drawing mm-hmm alrighty so that's our drawing function done so we've gone and written eight additional lines of code so let's take a step back and work through these so the first four are setting styling so in this case we're setting our stroke style and this is going to be the color of our boxes so in this case we're going to be setting it to the color of the detected label so in this case whether or not it's one two three four five and we're going to be grabbing the color so this allows us to grab these colors here so red yellow lime blue or purple so say for example our object detection model comes back and is detected class two well then that class actually represents thank you so the box is going to be drawn as yellow then we're setting the line width so this is basically how thick the line around the square is in this case we're setting that to 10 we're setting the color of our text to white and we're also setting our font to 30 pixels in ariel all of this you can play around with if you have a preference for how you want it to look you can change it by all means go for it then what we're doing is we're actually drawing so we're going to begin our path and then the first thing that we're going to do is put down our text so here what we're doing is we're grabbing our detected number so that's going to be a number and we're passing that through to our label map and grabbing our name so say our model comes back with number two there well then what we're going to be doing is passing label map two and we're going to be able to grab this name which says thank you then what we're doing is we're appending to that our score so this basically allows us to see how confident we are that that's an appropriate detection we're then scaling it so we're basically multiplying it by 100 and dividing it by 100 and putting it inside of math.round function so this is basically going to round it down so rather than having a huge confidence number we're going to have it to two decimal places and then what we're doing is we're placing it at the x and y coordinate now because we're scaling our image we need to multiply it by the image width and the image height and then we're also bumping it up a little bit just to make sure it's above our box then we're drawing our rectangle using the rect method so up here i should mention that we use the fill text method to put down the text here we're using the rect method to draw a rectangle and then we're passing through our starting point so x and y and again we're scaling those and then we're passing through our width and height and again we're scaling those and we're also dividing it because remember our the model that we trained was based on a 320 by 320 mobile net but when we're actually getting our image we're actually passing through an image which is 640 by 480. so if you divide 640 by 320 that gives you 2 and if you divide 480 by 320 that gives you 1.5 these are our scaling factors here so you can see 2 and you can see 1.5 now what we need to do is actually bring in this drawing function into our main application so up until now we're not actually doing anything with it so we need to import draw rect so if we go back into our app.js file so up here if we just use our standard import functions let's bring it in all right so that's our function imported let's just quickly check our to-do list because i think we've just marked off another thing yep we have so we've created our drawing function and our label map so we can now mark that as done one more thing we're almost there so if we go back now all we need to do is go down to step five so it should be around line 5960 and then we're actually going to draw with it so let's go ahead and do that so we're going to wrap this entire function call inside of a request animation frame method so this is basically going to give us much smoother detections when it comes to actually displaying to the screen so let's write it up and that's actually our code done so now what we're going to do is actually test this out so i'm just going to open up my blind so we get a little bit more light in here and then we'll see it live and in action alrighty now what we need to do is just start up our app hopefully we don't have any errors and let's take a look at how we go so again we can start up our app using npm start oh actually let's take a quick look at what we actually wrote to run our function so again we use request animation frame and then to that we pass through an anonymous function and that function is actually running our drawrect method so our drawrect method here we're passing through our boxes and specifically we're grabbing through the first detection because remember we're passing through one frame at a time and then what we're doing is we're also passing through our classes our scores and this is our threshold so you can play around with this if you want a more or less accurate set of detections then we're also passing through our video with our video height and our canvas so let's go ahead and start it up and take a look up so it looks like we've got a cup oh we haven't actually defined boxes and classes and stuff in here so we just need to make sure that we actually define where our boxes classes and scores are coming from so let's quickly do that save that see it doesn't look like we've got any errors now let's open our app back up all right so we've got our app running and it looks like we're successfully drawing stuff to the screen so you can see even on the green screen we're performing pretty well looks like our line width is a little bit small so we can play around with that but you can see that we're successfully drawing on the screen so if we take a look so we've got hello i love you thank you yes and no and there you go so we've now successfully gone and built our real-time object detection app so this is quite huge right so we've taken a python model we've converted it we've hosted it and we've now built a real-time app with it so if we take down the green screen and test it out a little bit more so we've got hello i love you thank you yes and no now it doesn't look like our line width is taken so let's quickly just check that that's because this line should be capital save it again jump back to our app that's looking way better now there you go so that's our real-time app so again we're detecting hello successfully i love you thank you yes and no and we can also do dual detection as well so say for example we had two hellos or two i love yous i love you is probably a little bit better let's get in front of them like or camera so there you go so we've got two i love yous two hellos two yeses two no's pretty cool right so there you go so we've gone and built our entire object detection model that allows us to make real-time detections now this can be applied to a whole bunch of different use cases so if you wanted to do street sign detection if you wanted to do something completely different you could use this exact same process to build your real-time detection app and you can see it's still pretty quick and the advantage of using that request animation frame is that you can see that it follows a hand and different objects pretty nicely so it's got a much nicer animation than sometimes when you get jittery apps alrighty so let's quickly check our to-do list i think we've actually finished it up so if we take a look we can now well we're now drawing to the canvas using request animation frame and we can now hit save so again if we take a look let's just make sure we've successfully done that so again we're getting hello i love you thank you yes and no so we've now wrapped it all up so we've gone through those eight steps so we exported and converted our python model and again this is all going to be in this side of the github reaper that you can pick up we've hosted our model inside of cloud object store we've cloned our template we've installed our dependencies and we've loaded our graph model from our url remember we grabbed it from here then what we did is we made our detections and cleaned up our code we wrote our drawing function and then last but not least we drew to the canvas and that about wraps it up thanks so much for tuning in guys hopefully you found this video useful if you did be sure to give it a thumbs up hit subscribe and tick that bell so you get notified when i release new videos and let me know how you went about building your app and if you're building anything custom thanks again for tuning in peace

Channel: Nicholas Renotte

Views: 78,050

Keywords: computer vision, deep learning, tensorflow, tensorflow js, tensorflow conversion, custom object detection

Id: ZTSRZt04JkY

Length: 46min 54sec (2814 seconds)

Published: Sun Nov 22 2020