Raspberry Pi LESSON 63: Object Detection on Raspberry Pi Using Tensorflow Lite

hello guys this is Paul McCarter with toptechboy.com and we're here today with episode number 63. in our incredible new tutorial Series where you're teaching your Raspberry Pi who's boss what I will need you to do is pour yourself a nice tall glass of ice cold coffee that would be straight up black coffee poured over ice no sugar no sweeteners none needed and as you're pouring your coffee as always I want to give a shout out to our friends over at Sun founder Sun founder is actually sponsoring this most excellent series of video lessons and in this class we will be using the sun founder Raphael kit for Raspberry Pi now most of you guys probably already have your gear but if you don't look down in the description there is a link over to Amazon you can hop on over there and pick your kit up and believe me your life and my life are going to be a whole lot easier if we are working on identical Hardware but enough of this Shameless self-promotion let's jump in and talk about what I am going to teach you today and we have a really exciting lesson today and what I'm going to teach you is I'm going to teach you how to how to do object detection on the Raspberry Pi using tensor flow light okay so we're going to do object detection now this is a continuation of the theme that we've had for the last six or seven lessons where we've been looking into Artificial Intelligence on the Raspberry Pi now we started by learning the basics of opencv how to use the camera then we started learning how to track an object based on the object's color then we learned how to identify and track faces in your field of view coming from your camera and now we're going to take it to the next level where we're going to go in and be able to identify objects in the field of view of the camera what do we mean by that we mean that it's going to look at a frame it's going to find the objects that it recognizes it's going to draw a box around them and it's going to label them how cool is that I think it is just pretty neat and I hope you enjoy this lesson because I've had a lot of fun playing around with this the last few days and actually I've been pretty impressed with the performance I was able to get out of my Raspberry Pi 4. so yes I am using a Raspberry Pi 4 if you're using a Raspberry Pi 3 your frame rates are probably going to be a little bit less now very important for this lesson and I gave you this as a homework assignment in last week's lesson you need to start with a fresh SD card if you like your old SD card you can keep your old SD card but take it out and flash a fresh a fresh SD card because for this to work smoothly for you we need to be on identical versions of the software so what you need to do is you need to get the 64-bit version of Bullseye okay the 64-bit version of Bullseye on a fresh card why a fresh card if you already have 64-bit Bullseye I have no idea idea what kind of nonsense you put on that card and what versions of software you've already put on there so if you start with a fresh card and I start with a fresh card you will get the same results that I get if you start with a different card on a different version with different stuff on it you might not get exactly the same results that I get okay so if you want to get exactly what I get you need to do things exactly the way that I do them okay so enough of this talking what we're going to do is I've got a fresh I've got a fresh fresh Raspberry Pi card all ready to go here booted up and ready to get going now the first thing that we're going to need to do is we're going to need to go in and install tensorflow Lite okay and it's actually not that hard to do I will probably make you happy if I get out of your way so you can see things a little better and there's just a couple of things we need to do to get this going and within a few minutes we'll actually be up and coding okay but we need to start by opening a terminal let me just let me take a second here and see if I can get you a this is a fresh card and what you need is you need a larger font on this man I told you guys it was a fresh card so don't be surprised if some things aren't exactly the way we want them as we go through it okay there it is I think you will be able to watch me pretty easily now and so I'm going to start by saying pseudo apt app get sudo app get update okay this can go fast or slow depending on how long it's been since you have done an update this is a fresh card so there's not too much stuff to worry about here and now I'm going to do a pseudo apt get up great so we're going to upgrade all of those things and even though this is a fresh card sometimes it takes a little while this actually went pretty uh pretty fast and so now we have a fresh card and it is all updated and it is all upgraded and we are ready to roll here now I'm going to go ahead and make a directory mkdir and let's look at what's in this home directory right now so I'll just do an LS and you can see we've got this the standard set of starting folders so I'm going to go ahead and do a make directory of python and since they uppercase the first letter of their directories I'm going to upper case the first letter of my directory so I'm going to have python I'm going to go on in CD into python like that okay LS it's empty as you would expect now let's make a folder to do this tensor light work in so I'm going to do an mkdir and I'm just going to say t light like that for tensor light okay now let's uh change directory entity light okay and now what we are going to do is we're going to go out and we're going to download those tensor libraries and all that tensor software that we need and we will do that from GitHub and so let me make sure that you can see this line because this line of code is going to be a this this command is going to be a little bit longer and I want to make sure that you can't see it so let's get our window all up in in a clearly easy to see place okay now we're going to do a get clone an https the S is kind of important git clone https and then slash slash get hub.com and then we are going to tensorflow and we are going to flash as much as I didn't want my head to cover that my head is covering that okay now you can see it git clone https slash slash github.com tensorflow slash examples and then a space and then a dash dash and then d e p t h space one and that would probably be good if that was all on one line okay so it's example space dash dash depth one and let's hope this thing finds uh finds everything and so now if I come here I should just be able to press enter and hope we see it says cloning at least it didn't yell at me it looks like it's getting stuff this is kind of exciting I think this doesn't take very long and so don't get don't get too worked up about this I'm not going to pause the video or anything I'll just sit here and chat with you as this uh as this downloads this you're probably having to do this too [Music] it's not it's not going real fast maybe my internet connection is a little bit slow here this morning but uh we'll we'll just wait here and we'll let it download okay but I am really excited about this I've uh I've been playing around and tensorflow on the Raspberry Pi it's a pretty cool thing to do [Music] you're up to 76 percent and almost there almost there okay now okay boom we're all done now okay so now we need to kind of navigate to a specific place and rather than give you a CD something slash something slash something slash something slash something that's kind of hard because if you make a mistake in there then you got to go all the way back so we're going to do it one at a time let's LS and now we're going to change directory into examples like that LS and now we're going to change directory into light because we are using tensorflow Lite okay we're going to LS and now we're going to change directory into examples so this is like examples inside of white inside of examples and then we're going to LS again and now what we see is we want to go to object detection so I'm going to LS and then I'm just going to put OB and here's a pro tip if you click tab at this point it fills out the rest for you because that was the only thing that started with ob somebody told me that about six months ago I thought man if I would have known that 10 years ago the last 10 years of my life would have been a whole lot easier not having to type in these long things okay now we're going to change directory uh element I lsed into object detection where I should have uh let me see yeah I should have changed directory into object detection like that okay now I'm in object detection.ls now we want to go into Raspberry Pi so change directory rasp and now I click Tab and it takes me to Raspberry Pi now LS okay here we are all right here we are and here's where the action happens do you see this uh sh file setup dot sh we're going to run that so we're going to say sh and then we are going to say setup dot sh and man it's like no matter what I do this path gets longer and longer and longer and I hate having it go to a different line so sh space setup dot sh and we are going to run that thing and see what happens okay it is sitting there okay so looking so it's it's doing some installations and it says it already has Pips so it's not uh so it's uh well it does look like it's collecting Pips so it's gonna it's gonna install the different things we need to run tensor light and the nice thing about this is we are not uh we are not having to go in and type in 20 different commands this is just a script and it does everything that you need in order to get tensor light working okay I think now I'm gonna probably make this bigger and come down and it takes two or three minutes for this to for this to go but I think again we'll just sit and wait on the download together but uh like I say I was happy to learn that it's relatively easy to get tensor light installed and we're going to have our first program written today so sort of like by the time this lesson is over you will be doing object detection along with me [Music] it's going pretty uh pretty fast so yeah I really like using separate SD cards when you're really going into a specialized area like this similarly if we do media pop in the future I'm going to recommend that you have a media pop SD card you have an SD card for tensorflow okay this is moving right along okay [Music] it's giving me some warnings now when I did this earlier uh everything worked okay so I didn't really have to worry about those warnings but uh I'm trying to look at that okay it didn't those those warnings didn't cause me any problems okay so it looks like that is all good okay that is all good now what we need to do is we need to LS and we need to kind of see what we have here all right now this is the thing this is the thing we are going to need this model this is the tensorflow Lite model that understands how to look for objects we're going to need that and we're going to need this utils.pi okay and those two files have to be in the same folder that our main Python program is so we could either every time we played with this navigate down to this folder or we could take these two files and put them in our main python folder and that's what I'm going to do because if you come back in three months you're going to forget where all this stuff was and so what I'm going to do is I'm going to do a MV for move and then I'm going to move eff and then who's your friend Mr Tab and that goes to zero because you can see that there's two files that start like that so to tell it it's the second one I'm going to tell it a period there and then I'm going to tab again now I've got the same thing and where do I want to move that I want to move that to my home phone folder the squiggle and then slash python that folder that we just created and if you've done things exactly like me your folder should be there and then when you click this voila it worked now what we also want to move is we want to move the utils dot pi and we want to put that in uh squiggle that's my home python okay and now LS all right now change directory to home LS change directory to python okay LS and there I've got those I've got those things that I need I've got the uh utils.pi and I've got the model OKAY sometimes guys get mad at me for moving through my directories one at a time but I just find it easier to follow along when uh when it's done that way okay now what we're going to need to do is we're going to need to open up Bonnie okay we've got Bonnie opened up so I'm going to come and I am going to say new and then what I'm going to do is I'm going to go ahead and save this because we've got to make sure that we save it in the right spot and so we are going to go to this python folder that we created and then there is our utils.pi the other one doesn't show because it's uh not doesn't have the uh suffix dot pi and so that's okay so what we're going to do is we're going to call this detect dot Pi in the dot py is kind of important so we're going to do that okay now we're going to jump in and we are going to start coding here so I'm going to start by saying import CV2 because you can imagine we are going to need to use opencv we're going to import time because we're going to want to be doing the frames per second and then we're going to want to use the pi camera now when I write this program I'm going to set it up where you can use the pi camera or you can use a webcam so you can switch back and forth between your cameras so then we're going to say from PI camera two we're going to import Pi camera to and be careful of case the first Pi camera all lowercase the second one the p is uppercase now we need to get those tensor libraries that we just installed so we're going to say from t f light T of light underscore support dot task we are going to import the core Library okay the core library now these are very similar similar libraries here so I'm going to go ahead and copy this Ctrl C and then control V and then control V and so from tflight support desk we get core and then we need processor and then we are going to need Vision like that those are all good and then we need to import utils and that was that file that we moved into this folder let's just make sure we can import these things so I'm going to run this and just see if we have a problem okay oh TF Flight Support uh did I misspell some tasks task cut that is the danger of copying and pasting it was task okay so now let's run but also that's why I like to run it as we go okay so that all works you see down here all right let me make sure that you can see this good okay that looks a lot better and you can you can see the terminal window down here so you can see what's happening as we uh as we go along okay now this you gotta kind of tap in carefully so we got to tell it what model we're using and the model we're using is efficient d-e-t underscore light 0 light 0 dot t f light like that t f white so get this exactly right efficient DET underscore l-i-t-e-0 dot TF for tensorflow Lite that is the model in the string and enter okay now we've got to put in some parameters and one is the number of threads num threads and this is like how many threads is tensorflow going to run to try to find objects so it's kind of like doing threading down in those libraries and I'm just going to put four that seems to work pretty well for me you can kind of play around with it and see what you want to do now we got to go ahead and tell tell it what size we want to do this at so I'm going to say display width is 1280 pixels and display height is equal to 720. now this might be too big and if it runs really really slow we can make it smaller in order to get it to run faster if that makes sense okay so now what we're going to do is we're going to come in and set the camera up so I'm going to say Pi cam 2 is equal to Pi camera two okay this should make sense if you've taken my other lessons and then what I'm going to do is pi cam 2 dot preview [Music] and then preview Pi cam 2 preview underscore configuration dot main dot size and this is telling the camera that we want the size to be display W comma display height which we just set up okay so that's preview underscore configuration dot main dot size and display with display height now we're going to come in and say Pi cam 2 okay and then we are going to do dot preview again and then we're going to do underscore configuration and we are going to do main dot format and that is going to be equal to the string uppercase r g b 888 so 8-bit red green and blue is going to be the format of our image and now we're going to do PI cam 2. preview and underscore configuration dot align and this just says if we ask for a size that isn't a standard size adjust what we ask for to something that is a standard size and that way if we make a little mistake in putting a size in it will adapt and still run pretty pretty good okay now I'm going to do PI cam2 dot configure and how are we going to configure it with the preview that we just set up p r e v i e w so we just set all that up and now we're going to say to use that close your parentheses so we configure it to the preview now let's start the camera Pi cam 2 Dot start like that open close okay now the Raspberry Pi camera is set up and ready to use now let's set up the webcam so I'm going to say the web cam is equal to for me it is an open string slash gev slash vid slash video two okay now how would you figure out what yours is let's come here and what you would do is you would give me just a second here you would do the command V for V4 l 2 Dash CTL and then Dash Dash and list devices this will show you your camera devices if I typed it in right okay now here they are and it's kind of strange the way this does this on the updated and upgraded Bullseye it gives us like a whole bunch of different paths to the same video but uh these up here I'm not exactly sure these might be like the Raspberry Pi camera well this one certainly is the Raspberry Pi camera here on video Zero and video one and then here this one you can see the CD this the C270 HD webcam is the one that I have yours is going to be different but you want to look for the thing that says webcam and then you can see my webcam is at video two video three and media 5. now I haven't tried to see if I put in video three if it still works or not but it works for me for with the first one video too okay so that's pretty good so you can see how I got that so you can kind of figure out what webcam you are on and now we need to set up that webcam and the way we are going to set up that webcam is the way we're going to set up that webcam is we are going to do a uh we are going to do a cam is equal to CV2 dot video capture video capture from what the web cam that we just defined okay now we've got to set it up so I'm going to say cam dot set and then CV2 so you CV2 okay dot cap prop and then frame and then underscore width and then what do we want that to be well we want it to be display width that we Define above okay this next these next couple are almost the same so I'm going to copy that and paste and I hope I'm not pasting mistakes again but we're going to have uh cap prop frame height as well [Music] height as well like that and then that is going to be display type and we need to set the frames per second so cap prop in here you just take that and say FPS for frames per second and we're going to set that to 30 frames per second okay so now we have that camera set up we've got both cameras set up now and even though we haven't done anything I'm going to run it just to see if I've got any errors in there and those things that I did okay display height is not defined yeah okay what is that display height is defined ah I misspelled it down here in line 17. so display height like that now let's see if it runs okay you can see that we get some errors that the camera manager is doing this it's it's like a warning it says there's a mismatch between unicam and cam helper for and man I tried to get to the bottom of that but it doesn't seem to be that there's any problems but it just gives you that warning and so after playing around with a few hours I decided to ignore the warning okay so now we have the two cameras set up so now let's get some uh let's get some other things ready to go here and so what I am going to do is I am going to come down here and I got a set up we're going to calculate the frames per second and so I got to tell it when we display it where do I want to put it well I want to put it over by about 20 pixels and then I want to put it down about 60 pixels so that would be the upper left that's just going to be the position of the label and so I'm going to say position is equal to that Tuple okay now I need to set up a font so I'm going to say the font is equal to CV2 Dot wallet underscore Hershey underscore Simplex like that okay so when we get down to start doing some of our CV2 commands they become a lot easier because we've set these parameters up up here and so I really like to do that I'm going to say I want a height of the text of 1.5 okay I want a weight how heavy it is w-e-a-e-i-g-h-t I want the weight to be three okay and then I need to give it a color so I'm going to say my color is going to be equal to let's make it let's make it blue so I'm going to say 255 comma 0 comma zero like that for my color how does that look I think that's pretty good okay now also we're going to be calculating frames per second so we need to initialize frames per second to zero just so the first time through the loop it does not crash out on us okay now it's going to come three lines that are kind of tedious four lines that are kind of tedious but we're going to set up our object detector right now we're going to set up our object detection right now by creating a detector and so the first thing we need to do is create some base options and those are going to be that core right and that core we imported up here okay that core Library we need to pass it some values so core Dot base options make sure you get the case right and now we've got to give it the file name okay and what is the file name it is equal to model all right and we should have set model up up here I hope we did yeah we set model up here so it's going to know what model is already and then we need to uh [Music] say use underscore coral and the coral is that Google GPU accelerator for the Raspberry Pi but I'm sure most of us don't have that and I don't have it so since we don't have it we're not going to use it so we're going to set that to false and now how many threads num underscore threads is equal to num underscore threads okay and right we defined num threads up here I think yeah we Define num threads up here so it should know what that is so we got num threads and now what else do we need to do num thread so that we've done the file model use Corel false and numb threats and so that should be it for the base options now we got to do the D Tech shun [Music] options and those are going to be equal to core or I'm sorry this is going to be the processor and what is the processor we imported that up here that's the library that we imported there so now we're going to have to pass that some parameters so the processor okay it's going to be the processor Dot [Music] detection options okay detection options singular detection plural options and make sure that you get your case right okay now what are the detection options well Max results and this is how many objects you want to find well if you say a hundred you're probably going to be pretty slow and if you say one it's not going to be very interesting because it would only find one thing so I'm going to say 8 which is probably more than I should but I'm just going to say that and we'll see what happens if we're running really really slow we might need to change that okay man I hope I wasn't in your way on this uh on this line here let me make sure that you could see this one above okay so base options is core dot base options file underscore name is equal to model comma use underscore Corral is equal to false use Corel is equal to false and then num underscore threads num underscore threads [Music] is equal to num underscore threads and then we end it there okay so I'm sorry if my head was in the way of that one line of code but I think you were able to now see it and so Max results is equal to eight and then score underscore threshold is equal to 0.3 like that okay and what does that mean that means if you want like a score threshold of 0.9 it has to be absolutely positively sure that it's right before it shows you and the lower the threshold the more things you're going to find but the more likely you're going to misidentify something if that makes sense okay so now we did that now we're going to come up to options is equal to Vision dot uh uh object uh object detector options okay so the first one was processor dot detect options processor dot detect options this one is Vision dot object detection options okay do you see that and this one is going to be a [Music] okay this one is going to be we're going to pass it base options equal base options okay because what we already determined base options up here and this should be base options plural okay base options plural and then we pass the vision object detector options we paste it pass it the base options which we have already set up and then detect shun options okay detection options is equal to detection underscore options okay so let's let's kind of just review this in this line of code I set up my base options okay in this line of code I set up my detection options and then now I have my over all options is Vision dot object detector options and I pass it the base options that we just created and then I pass it the uh detection options all right and this is basically then one more thing that we have to do to actually create our detector based on all these options being set up and so our detect tour our detector itself is going to be Vision dot object detect for object detector create [Music] from options and then what options all right so this is this could be kind of confusing but you just got to get this in and then you see what does this allow you to do it allows you to select the model it allows you to use the coral if you have one okay it lets you set the number of threads the number of results that you want and then what threshold you want for detection and then it comes together you build the detector this way and then you can go in and you can change it as you need it but I know if you haven't seen this type of thing before it's a little confusing the first time but now you got your block of code and you can just use it let's try to run this there's no telling how many errors I made in there but let's see okay Max results it doesn't like that ah Max reels X results like that let's try to run it I really just like running my code as I go along because uh okay object detector options so where is that that is in Line 39 object detector options okay so uh let me look at this and see which one that is that is Line 39 and that is my options is equal to vision vision yes object ah detector detector options hopefully you caught that okay hopefully you caught that let's run it again but you see it's nice to kind of debug your code as you go uh object detector man I've got all kinds of Errors into your line 40 and so detector is Vision dot object D Tech tour dot create from options man that looks right okay it doesn't like object detector uh object I object detector I hope you guys did not follow along with those mistakes hopefully you got it in right but code like that is very tedious because it's a lot of parameters and a lot of different things okay and so it says that it created the Delegate for the CPU so that's good that means that we have this thing nominally up and going right now so I think that is pretty good so now we're ready to go in and actually do some object detection so we're going to say while true when is true true is always true and then what are we going to do we're going to read the image well if I want to read it from the webcam I would just say the return value which you ignore and then I am for image and that is equal to the cam that we set up read and so this would read from the uh this would read from the webcam or if we want to read from the Raspberry Pi cam we would do a i m is equal to Pi cam 2 which we set up dot capture dot array like that okay and then I'm going to give it an ability if we need to to flip it on one of these so i m is equal to CV2 dot flip in case one of these is upside down we would take the image and we would flip it minus one which will do the correct the correct 180 degrees rotation okay so now I should have an image and before I go in and do object detection let's just see if we can grab the image and show it and so now what I'm going to just do is I'm going to say CV2 dot I am show like that and then what am I going to show I'm going to show the camera and then I'm going to show the image like that okay now what I've got to do is if CV2 Dot dot weight key and we will wait we will wait for one millisecond for a key to be pressed and then if that is equal to the ORD value of Q okay so that means if I press the Q button what do I want to do if I press the Q button I want to break out of this okay and now what would I do when I break out I would want to I would want to most certainly see the two dot destroy destroy all when all windows like that open close okay so now let's just see now I've got to decide which camera I want to use let's use the pi camera and so I'm just going to comment this one out like this and I think the pi camera I mounted on that pan tilt bracket so I think I do need the flip so let's run this and see what happens okay doesn't have any object capture what happened there okay so image is equal to Pi cam 2 it should be underscore and not a DOT capture array like that let's try it again [Music] boom look at that okay so I've got that camera working that's good now let's make sure the other camera works and so what I'm going to do is I'm going to come up here and I'm going to uncomment out that and that'll turn on the webcam and now I'm not going to use the Raspberry Pi camera like that and then I don't need to flip for the webcam because I think it is up the right way and let's see what happens there okay boom you've got my desktop there okay let's uh yeah that was my desktop so that's all working now the next thing I want to do is we really want to pay attention to the frames per second and so let's make sure I put the frames per second equals zero up there that is very good okay now I need to go ahead and get my start time if I'm going to do frames per second I need to say t start is equal to time dot time and so that's just looking when am I going into the loop what is the clock time when I'm going into the loop and then down here at the bottom of the the bottom of the while loop I'm going to go ahead and say t end is equal to time dot time okay so between the start time and the end time is the loop time so I'm going to say Loop time is equal to T end minus t start like that so now I have the time through the loop and then now what would the frames per second be well the frames per second would be equal to 1 divided by the loop time like that okay but now I want to put a low pass filter on here and so you guys have seen me do this many times if you don't understand this go in and look at my other earlier videos but I'm going to take point I'm going to take 0.9 times the frames per second from the last time around and then I'm going to add 0.1 times the present one and that will give me a smooth version of the frames per second so it's not jumping all over the place and so I should have frames per second now let me just print frames per second just to see what happens make sure this works so let's see what happens [Music] did not like that ah T end is not defined I must have a a typo in there kill the program okay T uppercase E that should fix it so let's see what happens now okay so it's running and I'm getting not a good frames per second down there it's better than that so I must have an error in there somewhere so let's look okay so we are going to see what I made I said t end ah I see what I've got to do I've got to go ahead and snag the T start for the next time through the loop T start is equal to time dot time and so what I'm doing is I'm saying this is the start of the loop the step before we go up there so I think this should work this time thinking [Music] yeah so it's getting on up there for frames per second of about 12 and for the webcam and for an image this size that's about all that I'm going to be able to get probably that is probably about all that I am going to be able to get I got kind of a busy desk going on here don't I [Music] okay I shouldn't play around with things so much but just thinking about the next few steps that we're going to be doing okay so let me come in here and quit out of that okay so now I've got frames per second the printing actually is a slow step so when you print things that slows you up so I'm going to take that out and then what I'm going to do is I'm just going to come in and put it on the screen and so before I do the CV2 dot I am show here I'm going to put the frames per second on there by saying by saying uh [Music] CV2 dot put text where do I want to put it I want to put it on the image and then I want to put what the string value of the integer value of frames per second because we don't need a lot of decimal points there and then I'll close the int I will close the string and then I will say append to that concatenate to that the labels FPS for frames per second so that's what I want to put on my image where do I want to put it at the position which we defined above and now height which we defined above my color which we defined above and wait you see how easy these things get if you define the things above let's try it now let's try it now argument font face okay font face has to be a mod okay oh did I did I forget to put a position and I forgot font that okay now let's try it did you guys see that font [Music] okay boom 12 frames per second got my desktop okay that looks really good so you guys are probably getting tired of me writing this code like when do we get to the artificial intelligence part I'm getting tired of this well I wanted to take you through this in case you're kind of new I didn't want to lose you so I'm sort of showing you all the steps here and if that offends you I'm sorry if that offends you I'm sorry okay so now let's go in and let's actually do the artificial intelligence so at this point I have my image right at this point I have my image but what is the quirky thing about opencv it captures the images in blue green red where opencv captures the images in blue green red and tensorflow wants them in RGB so what I'm going to do is I'm going to say I I am image RGB is going to be equal to CV E2 dot CVT color and what do I want to convert I am the image and then how do I want to do it CV2 Dot color Okay CV2 dot color underscore b g r 2 r g b and so this will do the conversion where it will take IM it will make a to take you from BGR to RGB and then put it in the file imrgb okay now I have an RGB image now I have to create my tensor image so the IM tensor this is the image format that is wanted by tensorflow and how do I do that that is going to be equal to Vision the library that we imported dot tensor image Dot [Music] dot create underscore from underscore array and then what do we want to create it from well the i m RGB that we just created now I've got the tensor image that I can send to the tensor library to do the detection so now I'm going to get my detections and those are going to be equal to detector which we already set up dot detect and I want to detect on what I am tensor I hope that even though you don't perfectly understand every detail I hope it kind of makes sense so we created I am tensor from imrgb which we created from capturing from the camera and now I should have all the objects detected and they are kept where in that data structure detections now what I want to do is I want to create an image which is going to be equal to utils.vis visualize okay and I want to do that on what I want to do that on M our original image and then I want to decorate it with detections Okay so we've got the detections that we found and now we're going to pass it those detections and we are going to pass it the original image and then it is going to create it is going to create a decorated image if that makes sense it's going to decorate this IM if I am thinking about this right okay and so then now now i m is going to be decorated it is going to be decorated with these uh with these annotations and stuff okay so if I am thinking about this right I will need you to hold your breath Shazam okay wow look at that it is finding lots of stuff so let's see the stuff that it found it found a mouse it found a mouse it found a mouse it found a keyboard it found a another keyboard it found two keyboards three keyboards four keyboards it found the cup over here but now let's see it found me as a person and we're operating at six frames a second that is pretty amazing now I'm tired of all these mice because you see I hit my Max of eight things so I'm going to get those mouse out of the way and now what did it find now it found the scissors for a brief moment it found the scissors okay yeah it found the scissors and let's see if I can get one of these keyboards out of the way okay because it's still it's hitting that limit so now I've got keyboard and I've got let's see one two three four five six seven eight I think I'm still hitting my limit there so let's see I'll put this over here and again it's seeing too many different things let me take that other keyboard out and let's see if I can find some different stuff okay you see it's finding the cup now and let's see if it can find this bottle if I put it somewhere okay I think it's trying to find the bottle yeah okay do you see that it found the bottle and let me move this keyboard out of the way give me a little more room here so I've got bottle mouse keyboard keyboard person and then I'm gonna man I'm trying to figure out what to take out of the picture I got so much cool stuff okay so now you can see it has scissors banana you see that actually I had a real banana here I had a real banana here but as I was going through the lessons I was going through the lesson getting ready for it I kind of got hungry and I ate the banana but I'm surprised that it is finding the peel but you see still the problem is it is the problem is it's seeing too many things so it doesn't get around a banana okay now it's got bottled banana person okay and now let's see if it will figure out what this is cell phone Banana Cup scissors keyboard and person operating at five frames per second that is actually pretty darn amazing if you ask me to be able to get that type of performance out of a Raspberry Pi so let's see if we can make it a little bit simpler here so that was five frames per second which is pretty darn good let's come up here now and let's switch cameras so I'm going to turn on the Raspberry Pi camera I'm going to turn off the webcam and we're going to run it again and see what happens that was pretty amazing [Music] Okay so we've got a person and now we have a person where where is this I need to point this down a little bit okay we have a person man everything is backwards for me okay we have a person with a bottle we have a person with a bottle and let's see if we can get this banana person with a bottle and a banana okay let's see if we can come here man this uh this camera angle is really quite annoying because it's uh it's becoming okay a person with scissors and a banana what could possibly go wrong with a person with a scissor and banana that is pretty good okay what I'm seeing here is now with the Raspberry Pi camera we've dropped to about four frames per second so I'm not quite getting the speed that I was getting with the uh with the uh webcam let's see if we go to 640 by 360 like that [Music] okay and now it's still kind of slow there still kind of slow but we've got a person with a bottle and we didn't seem to save that much time by going to a smaller a smaller image but we got a person with a bottle and a banana peel that is just pretty cool you know I'm kind of curious uh I'm kind of curious why we're not getting better frames per second on that Raspberry Pi camera and somehow I have the feeling that somewhere along the way in one of these updates the pi camera isn't working as well but let's say if I turn off the detection stuff I just want to see what we're getting so I'm not going to do that I'm not going to do that I'm not going to do that and the other things are doing so I'm not doing the detection stuff now and so let's see what my frames per second are okay wow boom right there at up there at 30 frames per second let's go back to 1280 1280 by 720 and let's run that and see what kind of fps's we're getting here okay well so this is kind of an unexpected result for me I would think that I should be getting a better frames per second without doing anything off of that camera so we'll have to play around with that that just doesn't seem very good the other thing that we can do is we're going to come in here and instead of trying to detect eight results let's just try to detect three results and let's see if that runs any faster if we turn that down a little bit [Music] okay yeah seem to uh but I didn't sorry I didn't turn the other things on so we need to come back in here and we need to turn our detection stuff back on okay okay again seems pretty pretty slow here we're getting two or three or probably we'll get about four four frames per second it's thinking my microphone is a bottle that that would be a fail Okay so we've got a person with a bottle at four frames per second and we'll throw the banana in just for good measure and bottle gotta get it where you can see it banana bottle and person at four frames per second okay guys that is indeed pretty cool and I have to say that I was impressed with how accurate it was and I was also impressed that it could do it at the four frames per second because without a GPU that is really pretty good okay let's talk about the homework assignment for next week the homework assignment is just for you to show that you have this working and then show the different things begin to explore what things the tensorflow Lite model that we use this model up here what things can it detect you know you can try different household items different fruits and vegetables you know different things like cups and glasses and knobs and forks and scissors and just different objects and see this effect efficient detect light model what all it will detect okay so that is your homework assignment for next week but let me kind of show you where we're going here what do we like about this and kind of like this was a little bit of a long lesson but in one lesson we're to the point of doing object detection now but this is the kind of problem we let we let the library the utility Library do all the heavy lifting for us what if we wanted to find something and like track a banana right we've got the we've got the pan tilt camera there and with our pan tilt camera what if we wanted to do something and actually live motion track on the banana well we need to know not just get an image back that's already done the workforce but what we need to know is we need to know what is this data structure detections and how do we bust that apart and get the information we need in order to start doing cool and useful stuff so that's what we're going to do next week we're going to see if we can bust open this detections and start getting useful information out of it and then be able to do the text decorations or whatever like we would like to instead of just doing it like what they have done and I want to try this again I want to try one just one more time here with the with the stuff I I kind of like the results that we were getting from the webcam in this particular case a little better so let's try that again and I will come back over here all right here we are okay so five frames per second it's saying ah now it's just seeing four objects so it's not finding everything because it is just finding four objects but it is you can see jumping around and seeing the seeing the different things and so I think that is pretty neat but you see it gives red boxes and then it puts the text very small where you can't see it well what if I want green boxes and I only want to track the scissors okay we have to understand that detection data structure we have to understand that detection data structure in order to be able to do that okay guys I hope you're having as much fun taking this class as I am making it it's like been a lot of fun we've gone through a lot of work on opencv and now we're to the point of really doing some super cool stuff hopefully you enjoyed it if you enjoyed us give us a thumbs up also it helps us with the old YouTube juice if you'll leave a comment down below and then if you've not subscribed already make sure that you subscribe to the channel when you do make sure you ring the bell so that you'll get notifications When Future lessons drop and as always share this video with other people because the world needs more people doing coding and fewer people sitting around watching silly cat videos Paul McCarter with toptechboy.com I will talk to you guys later foreign

Channel: Paul McWhorter

Views: 22,357

Keywords:

Id: yE7Ve3U5Slw

Length: 61min 59sec (3719 seconds)

Published: Thu May 25 2023