Building an Object Detection App with Tensorflow.JS and React.JS in 15 Minutes | COCO SSD

i spy something beginning with um pete [Music] what's happening guys in today's video we're going to be building our very own real-time object detection app we're going to be using refjs and tensorflow.js to do this and in order to speed us along the way we're going to be taking a look at the real-time object detection template that i've set up that you're going to be able to access let's take a deep look as to what we're going to be going through so in today's video we're going to be covering three key things so first up we're going to be accessing our computer vision template so this is a template that i've set up for you on github that's going to allow you to kickstart your journey in terms of building your real-time object detection app we're also going to be setting up and coding with coco ssd so for this we're going to leverage the pre-built tensorflow js model so this is going to allow us to make detections and then last but not least we're going to make real-time detections using our app and our webcam so we'll actually be able to detect different objects within our frame let's take a look as to how this is all going to work so first up we're going to set up our react js app so this is all included inside of the template so it's pretty easy to set up then what we're going to do is capture images from our webcams our webcam is going to stream whatever's in that particular frame to our tensorflow.js model and then we're going to render those detections to the screen so you'll be able to see all the different objects that have been captured ready to do it let's get to it alrighty so in order to build our real-time object detection app using react and tensorflow.js we're going to be leveraging a couple of things that are going to help us along the way so first and foremost i've gone and set up this react computer vision template that's going to allow us to get up to speed a whole heap faster so we're going to be able to clone this down and build our app from there we've also got another reaper which is if you don't want to actually go on ahead and code and you just want to download the code and run it from the get go you can actually download this link so this is called the custom object detection react js tensorflow repo bit of a mouthful i know but everything that you need to actually run this without writing a single line of code is there in terms of how we've actually gone and built it we're using react and specifically we've gone and used the create react app library to go and do that and then we're also going to be leveraging tensorflow.js and if we go and take a look at the tfjs models we're specifically going to be using the object detection model so this model actually uses coco ssd to allow us to go and perform real-time object detection enough webbing let's actually get started so what we're actually going to do is we're going to clone this repo so the react computer vision template so we can copy that and we're going to open up a new command prompt and in order to clone this repo we're just going to go into our d drive or a drive that you want to clone this into and we're just going to type in git clone and then the name of our repository so if we just minimize that for a second so basically what we've written is git clone and then the link to our react computer vision template now all the links that i just mentioned i'm going to make available in the description below so if you want to pick those up by all means just grab them you'll be able to get started super quick so let's go on ahead and clone this awesome so that's now cloned now if we open up our d drive you should have a cloned repository so let's go ahead and open that up so you can see in fact we've now got a folder called react computer vision template and this contains all the code we need to get started now in order to go and build up from this we're going to open it up inside of an integrated development environment or a coding environment so in this case we're going to be using vs code so what we'll do is we'll first up go into that directory and then we can open it up using the code dot command this is only if you're using vs code so let's cd into it and then we can type in code and then dot and this will open it up inside of vs code and i'll just bring it onto the right screen and there you go so inside of this folder we've got a bunch of stuff so namely we've got a css file so app.css we've got an app.js file and this is where we're going to be doing the majority of our work we've also got an index.s or css file and an index.js file now as i mentioned the majority of our work's going to be inside of our app.js file but before we actually go on and start making some updates to our code what we're going to do is just make sure we install our dependencies so if you select package.json let's make this a little bit bigger you're going to be able to see all the dependencies that we've got within our application now in this case the library that is most important is or the package that is most important is the one named tensorflow dash models forward slash coco dash ssd so this is the pre-trained coco ssd model that we're going to be able to leverage from tensorflow.js now i've also got tensorflow.js we've also got react we can delete this fingerpose one out because that's from a previous code set and in order to go and install all of these all we need to do is open up a new terminal so i can just open one up by hitting control and the squiggly bracket i never know what it's called on a mac it's going to be commands wiggly bracket and then to go and install this stuff we're just going to type in npm install and let that run so this should take a couple of minutes to run but once it's done you're going to see a node underscore modules folder pop up so we'll be right back in a second alrighty so you can see that all of our packages have installed and we're back at our regular command line now what we're going to go and do is start making our updates to our app.js file so if you actually take a look at this so let's just make this a little bit smaller you can see that we've got a bunch of to do's now i've specifically called this out because it's basically going to walk you through the steps that you need to update in order to use this but likewise if you wanted to use this for other use cases you could do that as well here what we're going to do is we're going to go through steps one two three four and five and then by the time we've got through each one of those steps we should effectively have a real-time object detection reactor so the first thing that we need to do is import the required model here now the model that we're actually going to need is actually coming out of our coco ssd package so let's go on ahead and import that model first up alrighty so that's our first dependency imported so the line that we've just written there is import star as coco ssd from at tensorflow dash models forward slash coco ssd so this basically is allowing us to download our pre-trained tensorflow.js model the next thing that we need to do is actually import our drawing utility but we're going to hold off on that until the end because the last thing that we want to do is draw now the next thing that we would need to do is actually go on ahead and load our network into our model so let's go ahead and do that okay so that's our model loaded so in order to do that we've created a new variable called net and then we've made this because our function is asynchronous we're just waiting for that to load so here we're actually loading our coco ssd model and this is what we imported right up here and then we're using the load method to actually go on ahead and bring it in now if we go to step 4 the next thing that we want to do is actually start making some detection so really quickly we're already up to making some detections so let's go ahead and start doing that so once we get these detections we're going to be able to start our application and actually log these out so we'll actually be able to see how model is actually performing alrighty so that's the line of code that we need to actually make our detections so basically what we're doing here is we're creating a new variable called obj or object and we're using our network that we defined up here so we're actually passing it through to this detect function so we're using that network and we're passing through our video so this is the video from our webcam and we're passing that to our detect function so ideally we should be able to detect a bunch of objects then what we're doing is we're console logging that out so we should be able to see the output of each one of these detections in our console now what we need to do is actually start up our app so this is going to allow us to see our objects and whether or not our model is actually performing well so to start our app we just need to type in npm start and this is going to start our react app and open up a new browser so you can see it's opened up a new browser and it's gone directly to localhost 3000 so by default this is where our react app is going to start so let's give that a couple of seconds while it compiles and then ideally we should be able to see a couple of detections in our console all right so you can see that we've got our camera showing up on the screen now if we go and inspect our console and if we open that up you can see we're getting a bunch of detections here and if we open this up you can see that in fact we're making detection so we've got a couple of things showing here so we've got an array and inside of our array we've got a b box this represents our bounding box and each one of these represents a specific thing so this is our x coordinate this is our y coordinate and this is our box width and our box height we can also see our different classes so in this case you can see the class as person because it's detecting me and we can also see our score but at the moment we're not actually drawing anything to the screen so let's stop our app and actually finish our drawing function so we'll actually be able to see our results okay so the last two things that we needed to do were update our drawing utility and bring it in right up here so let's go on ahead and do that now to start building our drawing utility we're just going to right click on source create a new file and call that utilities dot js and in here we're going to define a function that's going to allow us to draw to the screen so this function is going to be called draw rect short for draw rectangle and it's going to allow us to pass our predictions to this function and draw them to our actual webcam screen or to our webcam canvas so let's go ahead and start doing that okay so we're going to start setting up our function and remember our function is going to be called draw rect alrighty so to our draw rest function we're going to be passing out detections and if you actually take a look this is actually going to be our object variable here and we're also going to pass through our canvas so this is going to be our canvas that we've already had predefined what we're then going to do is loop through each one of our detections using the 4h function so let's do that so the first thing that we're actually doing within our draw rect function is we're going and grabbing our x variable our y variable our width and our height so remember when we're console logging out our predictions we were able to see our class and our text these are exactly the same variables here we're just extracting them out now we're also extracting our class so in this case it was going to be person from our last prediction now the next thing that we want to do is set up some styling and actually go on ahead and draw our rectangles so let's power through that okay so we've finished up our code now what we've gone and done is we've first upset our styling so there we've created a new variable called color and this is going to hold the color of our box as well as the color of our text and we've also gone and set this should actually be stroke style so let's go and change that and we've set that to color we've also set our fill style to color and we've also set our font so in this case it's going to be 18 pixels and aerial then what we've gone and done is we've gone and drawn our rectangles and text so we've used our canvas and we've commenced our path we've then used the fill text method and to that we've passed the text that we've extracted from our prediction as well as the x and y coordinates in terms of their placement and then we're drawing our rectangle as well so to that we're passing our x variable our y variable our width and our height so this should ideally draw a rectangle around each one of our predictions and then last but not least we're drawing our stroke so this is going to actually apply it to the screen so we can save this now and then all we need to do is bring this into our app.js file and run it for each prediction so let's go ahead and do that so right up here on step two we can bring in our drawrect function so that's our drawrect function imported now what we need to do is just run it down here so in step five we just need to use our drawrect function and pass through our object so these were all about individual predictions and pass through our canvas so if we save that now ideally what should happen is if we go back to our browser we should be able to see our detection so you can see it's right up in the top screen but you can see that we're actually drawing our box now if we adjust our camera a little bit and take this down you can see that we're making a whole bunch of detection so we're detecting myself as a person the couch we're also detecting the chair pretty cool right now if we got some other stuff i don't actually have my phone on me but maybe if we tried out our bottle it's saying cup bottle there you go so you can see that we're doing we're creating real time detections using our webcam and using our pre-built tensorflow.js model now what we could do as well if we wanted to make this a little bit fancier or go and make detections in party mode we could do that as well so let's go ahead and do that so now what we need to do is we basically just need to update our drawing function so what we can do here is just go back into utilities.js and then rather than just automatically setting our color to green we're going to do a little bit of magic to make this a little bit more party mode so let's go ahead and define this so rather than our color just being green we're going to change it to pound sign and then plus and if we save it now and go back to our app you can see that our detections are now appearing in party modes our person is flashing and if we go and use our bottle you can see we're getting detections in party mode and that about wraps it up so we've gone through quite a fair bit now if we go back and take a look at our app we went through our importing of our model we created our drawrect function and imported it there we loaded up our neural network then we made our detections and we also drew it to our canvas and that about wraps it up thanks so much for tuning in guys hopefully you found this video useful if you did be sure to give it a thumbs up hit subscribe and tick that bell so you get notified of when i'm releasing new videos let me know what you think of my mo and also let me know how you went about building your real-time object detection app thanks again for tuning in peace

Channel: Nicholas Renotte

Views: 110,628

Keywords: object detection, tensorflow js object detection, tensorflow js image recognition tutorial, tensorflow js object detection tutorial

Id: uTdUUpfA83s

Length: 15min 55sec (955 seconds)

Published: Thu Nov 19 2020