Simple ESP32-CAM Object Detection

today in the workshop we're working with object detection you'll see how we can use a free Edge impulse account in an esp32 cam board to build an object detection system I'll also show you two ways to train your model using a webcam or the esp32 cam itself that's the object of today's video so welcome to the workshop foreign [Music] well hello and welcome to the workshop and today we're going to be working with some machine learning specifically object detection now object detection is the process of taking a machine that has a camera on it and training it on one or more different objects afterwards that machine will be able to recognize the objects and report back the position of the object or objects within its field of view now the thing that amazes me the most about object detection is actually the fact that it was only a few years ago that you needed huge computers in order to perform this task and now we can actually do it on microcontrollers and this has a lot of really cool applications especially for what we call Edge Computing where we have sensors that are smart that can recognize things and report back more useful information to the main processor and object detection with microcontrollers is certainly a big part of that now we've worked with this technology before several years ago we worked with something called a pixie Cam and this was pretty well the first hobbyist level object detection device that you could use and it worked pretty well a little while back we worked with the DF robot husky lens and this is an amazing product that does Advanced object detection image recognition a number of other things it has a little screen on it and it is powered by the powerful kendrite k210 processor but there is even a cheaper and more simple way of doing object detection and we're going to do that today we're going to be using an esp32 cam now the esp32 is not as powerful as the Kendra k210 but nonetheless it is capable of simple object detection that's exactly what we're going to be doing today now we're going to be building a model and we're going to be training the model using something called Edge impulse Edge impulse is an online product you can use it for free and you can use it to train models which can then deploy to the esp32 so let's get started by learning a bit more about object detection object detection is actually a combination of two different Imaging techniques image classification and image localization we can use these four pictures to illustrate image classification now here you see images of four different animals but for an image classification system there are only two classes cat and cow an image classification system would make no distinction between the different cats image localization techniques are used to determine the position of the object of interest within our field of view in many localization systems this will be Illustrated using a bounding box which is a box that surrounds the image in many microcontroller based systems we simply use the X Y coordinates of the top left corner of the bounding box the first step in any object detection project is to gather data and this involves Gathering a number of images of the object to be detected this can be done using a camera or it can be done using an image bank and it is not uncommon to need a hundred or more images of an object to get an accurate fix on it after obtaining and labeling our images we'll use them to build a model which we can use to recognize the object in the future we'll be building our model using a neural network which is a network that consists of a number of nodes or neurons that have multiple outputs which are then weighed to determine a response a neural network functions in many Fashions like the human brain once our model is built it can be deployed into production and used to recognize the object we can continue to train the model in order to improve its accuracy object detection techniques can work quite well but they are very CPU intensive in many cases gpus are employed for this sort of work this can result in a very expensive piece of equipment that can also consume a lot of power now with all of those requirements how could we possibly do this with an esp32 well the answer is that we do our data Gathering and model building offline on an online service or a local machine and then we export the results to a tiny ml or machine learning code we can use that tiny ml code to build an esp32 library and some code that we can then run on our esp32 cam tinyml is also known as embedded ml with ML meaning machine learning tinyml allows you to run machine learning workloads on a microcontroller this technique is used with Edge Computing and iot devices there are many methods of creating tiny ml code and the method we're going to be using today is a product called Edge impulse Edge impulse will allow us to collect and label our images build a model using a neural network and deploy that model to an esp32 Edge impulse accepts multiple data sources such as video audio and analog data it has a data forwarder and SDK for Linux so can be used with a Raspberry Pi it also exports to an Arduino library with sample code and this is what we'll be using with the esp32 edge impulse supports several different development boards it'll also work with a computer or a phone to run the experiments we're doing today you'll require a free Edge impulse account so go and sign up for one and get some objects out that you want to detect with your esp32 now of course the first step in working with Edge impulse is to go to their site and create an account and that's what I'm going to do here right now I've gone to edgeimples.com where they show you a number of different features of the product and I'll just click the get started button and after clicking the get started button I open up another window where I have to give a bit of information what should we call you that's basically what is your first name you pick a username you'll need to put your email in there and a password accept the agreement and hit the sign up button you'll get a confirmation message in your email click on the link in that and you're all set to start using Edge impulse now when you log into your Edge impulse account for the first time you'll be presented with a pop-up window that offers to run a wizard in order for you to do some data acquisition I would suggest is closing that because we're going to do everything manually and when you close it you'll arrive at a screen very similar to this you are actually in your first project it creates your first project for you and it gives it a name in my case it gave a dronebot project one but I can click on that to edit it and I can give it a different name if I wish I'm going to just keep that for now now down to side over here you'll see a number of different menu items some of them won't make a lot of sense to you at the moment but that's okay we will be covering them data acquisition and impulse design are the two areas that we're going to spend the most time in and up here in the corner you can go and change some of the aspects of your account you can give yourself an avatar or change your account name etc etc now this may seem a bit overwhelming but it actually is pretty simple to use and I think in order to understand how to use it the best we really need to look at the workflow that we're going to be using with Edge impulse now our workflow is going to consist of five different steps the first step will be to collect the images of our object the Second Step will be to label those images our third step is to train a model using those images the fourth step is to export that model into code that we can use and of course the fifth step is to actually run the code on our esp32 note that we'll be using Edge impulse for steps 2 3 and 4 and we can even use it for step one as well in our first step we're going to collect images and there are many methods we could employ to do this we could of course use a camera or some existing images in an image Bank a great method of collecting images is to use a webcam a really easy method of doing it would be to use your phone you could also use the camera on a Raspberry Pi or a single board computer and of course you can also use the esp32 cam itself today I'll be showing you how to use both the esp32 cam and the webcam now the second step is to label the images and we'll be using Edge impulse to draw bounding boxes around the images and to give them a label now this admittedly is the most time consuming step but the more of this that you can do and the more accurate lead you can do it the better results you're going to achieve in step 3 we'll take the images that we collected and use them to train a model we can run tests on our model to determine how accurate it is once we're happy with our model we can export it and we have a number of options here we can export it to a code library and this includes an Arduino library that we could use with our esp32 cam we can also export the model as firmware for a number of popular microcontroller boards including the esp-i board it can also be exported as Linux firmware for a number of different single board computers we can also use Hardware to test and work with our model and in this case we don't need to export anything at all it can just be used directly on your phone or on your computer and finally we can export our model to our esp32 Cam and test it out so keep this workflow in mind while we begin by collecting some images of our object for our esp32 cam project now here's a quick look at the cameras that we could be using for our experiment today and of course the classic one is this the esp32 cam board this is the nine dollar camera board with the esp32 on it it also has a Micro SD slot and we've used these in a number of different projects now if you've used it before you will know that there is no USB connector on this and so you're going to need to have a method of programming this board and the most common method is something called an ftdi adapter I've got one over here and you can use this with the esp32 cam I'm not going to show you how to do that today I've already shown you that in the video and article I did about the esp32 cam so you can check that out if you need to know how to hook that up for programming purposes there's also a jumper you're going to have to use the gpio pin zero you're going to need to ground it during the programming process an easier way to work with it is to use one of these boards and this is just a board that except the esp32 cam board directly and provides a micro USB connector and a couple of switches as well that you can use for boot and IO and you can often get these boards with the esp32 cam board on Amazon or on eBay so I'd advise you'd check it out if you're going to go that route another route you can go is this board and this is a board by expressive themselves and this is a very nice board it's called the esp-i board and it's got the camera on it also has a mems microphone and remember Edge impulse can also work with audio data so in your future experiments you could also add voice recognition into the picture and it's got a built-in micro USB connector on it already and so it's a lot easier to work with than the esp32 camboard raw and so either this board and this adapter or the espi would be my recommendation for a camera to use for the experiments that we're going to be doing today okay now that we're all set up we can begin by capturing some images and as I showed you in the workflow outline there were many different methods we can use to take images of the object we want to detect the method I'm going to be using right now is to use a webcam so let me show you how I've got my webcam set up and we'll start taking some images of some objects for our model now here's the setup that I'm going to be using to take the photographs to train my models and these are the two objects I'm going to be photographing I've got a lantern battery over here and over here we've got Robbie the robot this is a Radio Shack piggy bank robot that I purchased way back in 1975 and he's going to be my second object I'm going to be taking the photographs using a webcam and this is the webcam that I got it's a relatively inexpensive webcam that I picked up on Amazon the reason I picked up this webcam is as you can see it's got a tripod on it it's got a quarter inch Mount so you can mount it on any photograph of tripod I thought that would be pretty handy but of course you can use any webcam that you happen to have you could also use the camera that's built into a notebook computer if you can come up with an arrangement to keep the lid steady so that it's always in the same position now I've also added a couple of video lights over here I've got a couple of small video lights and this is just to give consistent Lighting on my objects there's also another light that you can't see that's up on the ceiling of the workshop that I use for video photography and it's on as well and I've put a white background over here in a white base to eliminate any other distractions so that we can just train ourselves on the objects so now that you seed my setup let's go and start the work with it so now that we're set up with the webcam it's time to use it to collect some data and so I'm back in the project on edge impulse and there are a couple of ways I can do this there is a data acquisition item on the menu on the side but the easiest way is this to go and collect new data from this box over here and I'll click on that and it brings up a pop-up and it gives me three different methods I can use to collect the data I can use my phone and they have a QR code that you can scan you can connect the device directly to Edge impulse to your computer with Edge impulse and start using it or you can just connect directly to your computer and use its webcam and that's what I'm going to be doing now so I'm going to be connected to the computer here I want to collect images I need permission to use the webcam and I'll get it to allow that and let the webcam set itself up it's got some Auto settings that seem to be adjusting itself but right now you can see that I've got Robbie in my image and I'm ready to start doing my captures so now we're ready to capture some images of Robbie over here there's a couple of things at the top the label and the category and in category you have training data and testing data and it automatically will split your data 80 20 between the two of them and that's probably a good thing to leave it at for a label though we're not going to have to give it a label so let's give it a label right now it's unknown I'm going to call them robot and we'll capture a picture of it and we'll capture another picture of it and we'll move Robbie around a little bit so we can give them a couple of different angles capture another picture get a side view of them again I think you're getting the idea I want to take a number of images of Robbie and I want to take them using different angles and different shots I'm going to move Robbie back and forth a bit as well get them closer to the camera at times get them further I want it to get a lot of different images of Robbie that it can use for training now of course I'm not going to make you watch me collect all these images so I'm going to go and continue to collect some images and I'll meet you back here so now I've taken 75 images of Robbie you can see the countdown over here and the more images you can take the more accurately you can train your model 75 should be a good amount but I could have taken more but I want to move on to my second object which is the lantern battery so let's take Robbie off the stage and replace him with this guy now before I hit capture of course I've got to go up and do my labels I got to change my label so here I'm going to change the label and we'll set the label for that and now I can start capturing my battery and it is of course the same procedure that I used when I was capturing the pictures of Robbie I just move it around to different spots different angles and try to get a bunch of training data for it so again I'm not going to make you watch me do all of this I'm just going to do it and gather some data and I'll show you the next step after that okay now that you've captured a large bunch of images we can move on with our workflow and the next steps in the workflow were to label our images to train the model and then to deploy the model to a format that we could use with the esp32 and we can do all of that within Edge impulse so let's start off by labeling the images that we've captured so now that I've taken a number of images of both my objects I've moved back to the main screen I have a pop-up window here that asks me if I'm building an object detection project and it sees that because it's already detected some images and obviously I am so I'm going to say yes after I clear everything you can see the screen that I'm on right now with a list of all the different photos that I've got and I can look at them over here if I wish and as it says on this one I have 122 training ones and 28 test ones so now that I have 150 images I'm going to need to label them yes I'm going to need to go through 150 different images and label them and let me show you how you do that you go into the labeling queue and there are 150 objects queued in here right now and you can see I've got a number of lines over here when I'm moving my cursor what I want to do is I just want to draw a bounding box around Robbie so I'm going to go and press my mouse down and draw the little box and it asks for label I'm going to call him a robot and as you can see it says robot up in the top corner and I can save the label and it advances to the next item and you'll notice that it actually has already put a bounding box around that item and it's not too bad and as I go through you'll see a few cases where it isn't bound incorrectly and what you can do is you can grab the box and you can resize it to make sure that it knows that this indeed is Robbie and as we hit save labels it'll continue to advance and you can adjust that and of course I'm going to have to go through this 145 more times and I don't think you want to watch me do that so let me do that and I'll meet you back after this one thing I need to note though is that you will need to change the label when you come to the battery and relabel that one battery and you'll do that just by clicking into the label thing here delete label and you'll do another save label and it'll give you a box to put the name battery into it so I'll do all that and I'll see you after that now before we move on I think it's a good idea to take a look at the workflow diagram and see where we are in the process and as you can see we've done step number one which is collecting our images and done step number two as well which is labeling the images so now we're on the third step which is to train the model and where I've gone to my screen is under impulse design create impulse now I have to confess I've been messing with this a little bit so on your screen you probably won't see the image and object detection items below it just yet until we complete this step but it's actually the same thing so don't worry about it now the first thing you'll note here is my image data is set to 96 by 96 and while that might seem a little bit small you actually want to create a small an image as possible because that'll mean less processing time on your esp32 in fact if this doesn't work well for you you could go through it again and train it as 48 by 48. the resize mode will save this fit shortest axis and we'll add a processing block in the processing block we're going to add is the only one available to us the image processing block and we'll also add a learning block and the learning block will be the object detection the one on the top over here and then we'll do saved impulse and we've saved our impulse and so now what we're going to do is go down into the image tag over here and we're going to do one thing over here in the image tag we're going to set the color depth to grayscale again this will be a lot easier for our esp32 to process and so we'll set that to save parameters and then we're going to click on generate features and this may take a moment or two and now we've finished generating the features for the reference that did take about two minutes on my machine and we actually have a pretty good diagram over here because we've got all of the samples for the robot in one section and all the ones for the battery and another section that's really a good sign because if the two of these are mixed up on the feature Explorer you're bound to get a lot of errors but right now it looks a lot of pretty good one you'll also notice that now image on the side is a green light and we can go on to the next thing which is the actual object detection now an object detection there are a few things that you might want to change over here now you might want to experiment with these a number of training cycles and the learning rate and the higher these numbers are the larger it's going to take the longer excuse me it's going to take to do this uh what we actually want to do over here is I'm going to leave these as they are but we're going to go down and take a look this over here the model that we're using and we'll choose a different model and this is the one we're going to choose faster objects more objects mobile net version 2.01 and the reason we're going to choose this is It's the only one that'll actually work with the esp32 and now we can just hit the start training button and this is going to take a few moments as well okay the training is finished and we've got the results over here now I wanted to show you something in the top over here you may have noticed that this had previously said Raspberry Pi 4. you can change the target to anything you want and I set it to espi which is the only esp32 device they have here now what that does is this gives you the specifications for how this is going to work on the espi and it's good but it's not perfect it's not a hundred percent so this is what's called a confusion Matrix and it looks like I might have some confusion between my battery and my background over here and you can see that over here that the battery is not being detected a hundred percent of the time whereas the robot seems to be just fine so we can expect we might have a few flaky results with this and the way to improve this of course is to go back collect more images put the bounding boxes as close as you can and run this through again you can also experiment with changing these training cycles and learning rates if you want to to try try to get these numbers better but we're going to go with these ones anyway and see what the results are knowing fully well that we might have been able to do a better job now our next step is to export our model and that's actually a pretty simple step over here we're going to go down to the deployment tab over here and open that up and as you can see there's a number of places we can export the model to but we want to make an Arduino Library so we're going to click on this and we'll do a build and it is built in Arduino library and it is also downloaded it to my PC in the form of a zip file and we are going to be able to use that with the Arduino IDE okay so now we have a zip file which contains a library that has been created by Edge impulse for our esp32 and the next step of course is to load this into our Arduino IDE and to use one of the handy example files provided by the library in order to test everything out with the esp32 cam so let's go ahead and do that so as you recall Edge impulse produced a library in the form of a zip file and we need to install that in our Arduino IDE so just go up to sketch go to include library and add zip library then go to wherever your library was downloaded to and pick out the library you'll notice the name of the library has your username in it and so that's how you can identify this pick that out and the library has been installed now the next thing we can do is go and use one of the example sketches that came with the library so go down into examples and go scroll down examples from Custom library and look for your username because your library is going to be named after that and here it is dronebot Project interfacing and you'll notice there's a number of different choices here I'm going to go into the esp32 and there's a program called esp32 camera so we'll load that and what this sketch does is it uses the camera to identify the objects it's been trained to identify and it prints on the serial monitor the location of that object as it sees it in its field of view now I'm not going to go through the whole sketch there's quite a bit here but there is one line you need to pay attention to and it's down over here and this is where you define the type of camera that you're using now by default it's going to use the espi and I'll probably use the espi as well simply because it's a lot easier to use because if it's built-in USB port but if you want to use the regular esp32 cam then you need to remark a line 32 an unremark line 33 for the AI thinker and that'll let you use the esp32 cam so all I need to do right now since I'm using the espi is to load this up to my espi camera and we can check it out and so we're ready to test out our espi camera and see its object detection capabilities now I've got the camera mounted onto a piece of perf board over here and this piece of perf board you'll notice has a number of different standoffs on it they're spaced out for Raspberry Pi and arducam cameras I've used this perf board before it's attached to a GoPro adapter which in turn has a quarter inch thread and it allows me to put it onto any tripod I want and I put it onto the same tripod that our webcam was using I've got everything aimed at my test area and as you can see in my serial monitor it's saying there are no objects found and that makes perfect sense because there are no objects so let's give it an object and see if it can find it put Robbie there and it seems to have a robot and you'll notice a number right beside the word robot that's how confident it is that it's a robot and it's pretty confident it's a robot one is perfect confidence so 0.98 is very confident we've got an x and y coordinate and that's the top left corner of the bounding box as well as a height and a width and that would be the size of the bounded box so let's take Robbie out of there and put our other object into here and it's detected it being a battery and if you noticed when I first put it in the confidence wasn't as high but after a couple of iterations the confidence got much much higher now let's see what happens if we put Robbie and the battery in of course I can't see the cameras field of view but it seems to see a robot and a battery again with a fairly High degree of confidence so it seems to be working pretty well let's take Robbie out of here and put a second battery into here and it sees two batteries and it gives the coordinates of the two batteries so our object detection system with the esp32 seems to be working pretty well so I thought I'd repeat the experiment using an esp32 cam because I figure that's probably the module you're most likely to use now remember you need to make a small change from the code up here line 32 needs to be remarked out where we've defined the model of the camera as the esp-i and you need to unremark line 33 which defines it as AI thinker and that's the only change you need to use by the way I was able to use the same entry in the boards manager the AI thinker esp32 cam for both the esp-i and the esp32 cam module now I've got my esp32 cam module mounted onto one of those adapters and I strongly recommend using that it makes the camera a lot easier to use now it doesn't have a mounting hole so what I've done is I've taken one of those clips that you use for holding wire when you solder and I've just clipped that on to the micro USB connector so that's all that's holding it up it's not the best arrangement but it does seem to be working and if you look of course I see no objects found because I have no objects and I'll give it an object and we'll see if we can find it and it does indeed find a battery with a pretty high level of confidence and so we'll get our friend Robbie out here foreign a robot as well let's turn Robbie to his side and see if it can spot him and it still has a fairly High degree of confidence that we have a robot here but we'll turn him the other way and again it can detect the robot we'll turn them This Way backwards and the confidence is down quite a bit but it still does detect a robot now of course a lot of this just happens to be the number of images of Robbie I took and from what angles I took them at and if I had trained with more images and more angles it could have a higher confidence but it does seem to be able to detect the robot at just about any angle and it seems to work just as well as the esp-i and they both seem to work very well and since this is a nine dollar module it's a very inexpensive method of doing object detection so now you've seen the entire workflow that's required to create object detection with the esp32 and that workflow started off with capturing the images now when I Illustrated this to you I used a webcam to capture my images but as I mentioned earlier there are a number of different ways that you can capture images for use with Edge impulse and one of those methods is to use the esp32 cam itself in some ways this makes a lot more sense especially if you're using the camera in a fixed position because you're using the same camera to capture your images and train your model as you're going to be running the model on now you can connect the esp32 cam directly to Edge impulse in the way that we use the webcam however this requires using the esp32cli or command line interface and it's a bit complicated there's another way of doing this that I like better and that's just to use the esp32 cam to capture our images and then upload those image is two egg impulse and after that proceed with the same steps we did before so that's what we're going to do now I'm going to be making use of a library from eloquent Arduino an eloquent Arduino has a number of different libraries for use with the esp32 cam including many of them that work directly with Edge impulse and there's an example file we can use to capture all of our images so let's go and load that Library up and get some images and go through the entire process and then when we play back the images we'll also use some features of eloquent Arduino instead of the sample code that came with the gimpulse let's go ahead and do that now the library that we're going to need to install is called the eloquent esp32 library from eloquent Arduino so go into your library manager and start typing e l o q that should get you down to the number of eloquent arduinos and you'll notice a number of libraries it's the eloquent esp32 cam library that we're going to install so click on install for that and now the library is installed now that we have the library installed we're going to use one of the example of sketches included with the library in order to capture our images so we'll go into file and we'll go into examples and we will scroll down to examples from Custom libraries look for eloquent esp32 which we come to here and we see a lot of different examples so you can get an idea of all the different things that you can do with this library and we're going to go for number 26 here collect images so let's make that bigger so we can see it now there's a couple of things you're going to need to modify on this sketch in order to get it to work the first thing you're going to need to do is to put your camera model in over here so you have a number of different types of cameras that you can place in so we're going to probably just keep using the esp32 cam and that's the AI thinker so this is correct for us over here but if you're using for example the esp-i camera you would replace this with I over here etc etc now we also need to add the SSID and password for our local area network so that we can connect so I'm going to modify this sketch and we'll load it up to the esp32 and we'll check it out now after you load the sketch onto the esp32 if you look in the serial monitor you'll see an IP address after it connects to your local area network assuming you gave it the correct credentials of course you can use that IP address in a web browser to go to the screen that I'm on right now or you can just go to esp32cam.local and assuming that your router supports mdns that'll also bring you to the screen and it's easier than remembering an IP address either way we're looking at an image right now of what the esp32 camera sees and it's a square image it's been designed that way on purpose now we're going to do something a bit different when we do our collection of images today first we're going to take an image of nothing in other words an image of the background and we're going to use that during training to eliminate the background from the other images it's a Technique we didn't use last time but it can sometimes increase the accuracy of your object detection so so you'll see a little button here that says start collecting I'm going to click on that button and it's making a collection of images right now it's just continuing to collect images and of course this is an image list of my background I probably don't need too many images of the background because it's never going to change so I'm going to pause the collection and I get a couple of buttons here I can hit start collecting to continue collecting more images I can clear all my images or I can download my images which is what I'm going to do right now so let's just download the images we have to give them a name I'm just going to call them back it's my background and it downloaded the file called back.zip and that zip file will have all the images that I just collected so now I can hit clear and I can put an object on so we'll go for Robbie over here and we'll position him now you'll notice a little lag sometimes in the image that's normal don't worry about that and we can do start collecting so let's collect some images of Robbie and I'll pause my Robbie collection and I'll move Robbie a little bit and we'll start collecting again and once again I'll pause and I'll move and I'll discontinue to do that in order to collect images of Robbie and the collection is still going on it's just I have to scroll the screen down to see it you can pause that give it a more distant image of Robbie and you'll note that it's collecting black and white images of Robbie too that's intentional and I think that's probably enough images of Robbie so let's download them and we'll get robbie.zip and I'll clear everything to grabeaut put my battery in and repeat the process for that and so I won't make you watch me do the battery it's exactly the same thing but the just is that what we're going to do is collect images and they'll all end up in a zip file so I downloaded the three zip files that were created for me by the utility and I extracted the contents of them all into one directory now you could extract them in the separate folders if you wish but because I labeled all of my files differently I can tell which ones are the batteries which ones are the backgrounds and which ones are Robbie and so now I'm going to go into Edge impulse and an edge impulse we're going to create a new project so go over to your own profile logo and click on projects and you'll see the one project that you have right now we'll do create a new project and we'll have to give the project a name and that's good enough and it's set as developer which is the type of account we want to use and we'll create a new project and we're now in our new project and we need to add some data to the project and of course that data is going to be the pictures that we just took so let's do add existing data and we can do upload data we'll have to navigate with the browse to where that data is and I put mine in the pictures in my esp32 cam images directory I'm going to take everything in this directory and I'm going to open that and I've got 235 files selected I'm going to leave automatically split between training and testing you can set them as training or testing images otherwise and we're going to label them and we're going to infer the label from the file name so every file name started off with a distinct label so there should be three different labels we get we're going to upload the data now don't confuse that with the labeling process with the bounding box that comes next and it asks if I'm building an object detection project because it detects its images and I say yes I am building one and we're complete over here and so now we have a number of pictures and of course we've got to go into our labeling queue so we'll go into the labeling queue and we're on the first image now this is very important to remember I did a number of images of the background I don't want to assign any kind of a label to these so I'm going to do save labels without actually entering a label here and if you'll notice the number that I have left to do is incrementing so although they look the same because they're basically the same picture we are saving labels we just got to wait until we get to the first image I took that actually had either the robot or the battery I think alphabetically it'll be the battery here we go that's just a robot I was wrong and so now we have to draw a bounding box and of course it's a very small image but try to do the best you can and we'll draw a bounding box around Robbie I'm gonna have to enter a label for this now I'm going to label him robot and we set the label in the bounding boxes there we save that label and now we've gone down this is the next image you remember the way I took it we're going to get a sequence of images that are all the same and we just go and save the robot label for Robbie now of course I've got uh quite a few images of Robbie so I'm not going to make you watch me do this but essentially I'm going to go through this until I get the images of the battery that I'm going to go up over here and I'm going to change the label by clicking on that little garbage can over there and label those ones battery and continue to save the labels and so I'll finish doing that and I'll meet you back here all right I've gone through all of my images and as it says here I have 235 of them and it split them into a training and test ratio of 80 to 20. now I can go back and edit the images if I wanted to but I think they're pretty good remember I have a number of background images that have no label assigned to them so the next thing we need to do is design our impulse so on impulse design over here I'll go to create impulse and you remember this screen from before now the images are currently 96 by 96 but eloquent Arduino actually recommends it to make them even smaller 48 by 48 so I'm going to do that and we'll keep the resize mode it fit shortest we'll add the processing block and the only one it really has for us is an image processing block and we'll add a learning block and we'll add the one in the top object detection and the output has two features a battery and a robot so remember I had three sets of images but the background image is not considered the feature so now I'll go and I'll save the impulse and the impulse has been saved and you'll notice some other options have opened up over here I'll go over to image right now and I'll look at the raw features in the image and want to set the parameters to grayscale because these are obviously not RGB parameters and I'll save those parameters and now the next thing I want to do is generate features and this may take a few moments okay and I have my features generated and it shows me the feature Explorer for the battery and the robot now it's not quite as good as the one that I had with the webcam as you can see the robots kind of spread over to a few different places the battery is all pretty well together but even that is in two different groups and this just lets you know how unique it can find everything and if you wanted to you could create more images and continue to do what we've done so far in order to get that a bit better but this should still work so we're going to continue and Go to the object detection tag now again eloquent Arduino recommends you change a few of these settings they say that you should set this to 30 training Cycles and .005 is what they recommend for a learning rate and you can experiment with these different uh settings if you wish now you can choose you need to choose a different model over here because there's only one model that'll work with the esp32 as before and it's this one here the fomo one the faster objects more object mobile net v201 and so that's the one we want to use and we can start training that and here we are finally completed that took about three or four minutes and uh we've got a great score over here of a hundred percent a hundred percent on our background our battery and our robot so it seems to really have done a very good job and if you take a look at it uh it tells you Peak RAM usage of 70.9 K in Flash usage of 58.6 K so it's a very very tiny model and the next step of course is to deploy it now the final step that we need to take in Edge impulse is to deploy our model and of course we want to do that in an Arduino Library so we'll go to deployment and in deployment will search deployment options just click on that and the first one is Arduino Library silk will select that we'll leave everything else as it is and we'll click on build and the build is completed it has downloaded another zip file with the library edit and we're ready to use that in the Arduino IDE so I performed the same procedure with the new zip file that I downloaded from Edge impulse I imported it as a zip Library into my Arduino IDE and then I went into the esp32 camera example now since I'm using the esp32 cam model I had to change line 33 so I could Define the AI thinker and remarked outline 32 for the espi I compiled it and sent it up to the esp32 cam which I still have mounted the same way on the wire clip and it's saying there are no objects found because there are no objects to find and so we'll put Robbie in what I hope is the field of view and we see a robot with some reasonable confidence confidence is pretty high it's almost around 90 percent and we'll remove Robbie and put our back Tree in there and it's also pretty certain it has a battery and see if we can get both of them into the same picture and it sees a battery and a robot with reasonable confidence a little bit lower on the robot but pretty impressive when you consider the size of the images we used we ended up using 48 by 48 at the very end and we use the esp32 camera in this case for both the image training and for the final optic detection so it should work pretty well since the setup is so consistent so as you can see you can perform reasonable object detection with an esp32 camera avoiding the high cost of other object detection options now there's another way that we can view the results of the object that we built and it's a bit easier to code for it comes with the eloquent esp32 cam library and if you look in your example sketches you'll see one that says Edge impulse fomo fomo is faster objects more objects it's not a fear of missing out and if we take a look at this sketch this sketch does exactly the same thing that the sketch that was given to us in the library we got from Edge impulse except it's a lot easier to work with look if you take a look at this sketch you can see that when we go over to the serial print it's very very easy to get data out of this and you can see there's a lot less needed to set up your camera you literally load a few objects over here you specify what type of camera you have so this is the AI thinker over here you can also replace it with one of these other types of cameras you can set things like the quality Etc and it's very easy to use and that's one thing that this Library makes things that's easy to use now you do need to modify this of course in order to work with the data that you just built and what you need to do is to go into the sketch that you got from Edge impulse and I'll go into this one here the esp32 camera one and if you take a look at that first Library they're including this is the custom library that has the data that you created for the object you built and so the best bet is this to copy this entire line over here and then go back into the other sketch and replace the top line here with the line that you just copied and now it'll contain your data so you can compile that and load it up to your esp32 cam and give it a test and so I'm running the eloquent Arduino code on the esp32 cam and it's using the object model with the esp32 cam was used to train in the first place and so the serial monitor is telling me there are no objects detected which makes perfect sense of course as there are no objects let's put an object in its field of view and it sees a robot and it's telling you how quickly it did at 184 milliseconds pretty good gives you the location of the bounding box as well as the size of it I can take my robot out replace it with my other object and it sees the battery that seems to work pretty well we'll see if we can see both of our objects and it sees both the battery and the robot so this does seem to work pretty good and if you looked at the eloquent Arduino code it's a lot easier to use than the code sample you're provided with from Edge impulse but by using Edge impulse and perhaps the eloquent Arduino a library as well you can build some pretty impressive object detection projects with the esp32 okay that wraps up our look at object detection with the esp32 and I hope that it has inspired you to start using object detection in some of your own projects another thing I think it could inspire you to do is to learn a little bit more about using Edge impulse remember Edge impulse doesn't only work with video signals it can also use audio signals and even analog signals so there are a lot of interesting applications that you can use at dimples 4 and the free account that you have can accomplish quite a few things so it's a good product to check out now if you want some more information about doing the tech center using Edge impulse if you want to get the step-by-step process you can just check out the article that accompanies this video on the dronebotworkshop.com website and of course there's a link right below the video to the article when you're on the website if you haven't yet please sign up for my newsletter just my way of occasionally letting you know what goes on here in the workshop and all I need from you is your email address for that and if you want to discuss object detection or anything else to do with microcontrollers or Electronics head over to the dronebot workshop Forum where you'll find a bunch of people who will help you out with any problems you're having with your projects and would love to hear about the things that you're working on and finally if you haven't yet please subscribe to the YouTube channel I make videos about electronics and microcontrollers and if you enjoyed this video I'm sure you'll enjoy my other ones subscribing is really easy all you need to do is hit that red subscribe button and also click on the Bell notification and as long as you enabled notifications in your web browser you'll get notified every time that I make a new video so until we meet the next time please take care of yourself stay safe out there and I will see you soon here in the dronebot workshop goodbye for now foreign [Music] [Applause] [Music] thank you [Music] [Applause] thank you

Channel: DroneBot Workshop

Views: 18,901

Keywords: esp32-cam, esp32cam, object detection, edge impulse, esp32

Id: HDRvZ_BYd08

Length: 54min 56sec (3296 seconds)

Published: Sun Jun 25 2023