AI on the Jetson Nano LESSON 11: Running the Raspberry Pi Camera in OpenCV

hello guys this is Palma quarter from top tech boy comm and we are here today with lesson number eleven in our incredible new tutorial series where you are gonna learn artificial intelligence on the Jets and nano I will need you to pour yourself a nice big mug of iced coffee and I will need you to get out your jets and nano gear what you don't have the gear yet look down in the description below and you'll see links to the hardware that I am running if you're gonna play along at home it does help if we are working on identical hardware also want to give a shout-out to you guys who are helping me out over on patreon it is an incredible support it's an incredible encouragement to me those of you who decide to become a part of this channel by helping me out over at patreon those of you who haven't yet think about going down in the description there's a link to my patreon account think about running over there and hooking a brother up but enough of this shameless self-promotion let's get into what we are going to learn today and I am Telling You I am so excited about today's lesson because the first 10 lessons have really been just kind of laying the groundwork laying the foundation for getting into the artificial intelligence and we're really going to do some exciting stuff today we are really gonna really get into it today and so what we're gonna do is I'm going to show you how to launch and work the Raspberry Pi camera from inside of Python using OpenCV and so I will probably need to get out of your way here okay I think that will work what I need you to do is go over and open up studio code Visual Studio code which we installed a few lessons ago alright and it should come up like this and if you look I am coming up in that demo program that we did last week um and close these other tabs okay let's see close that one okay so this was the demo program that we wrote last week just show that we had been able to install and open install an import open CV running Python 3.6 and so if I right mouse click and say run in Python file and what you can see is is that it prints out down here that we are on version 3.3 point 1 of open CV which is what we want okay now we're gonna go in and if you look you should have some setup like this where you've got the Raspberry Pi camera hooked up to the Jetson nano and remember we kind of did the physical build in lesson number one which was a particularly painful lesson because I think I had to take apart the acrylic case like four times and put it back together in order to get it put together right but we did persevere and we got the physical bill done and so you should have a setup that looks like this all right now what we really want to do now is instead of just having the little Raspberry Pi camera sitting up there looking cool we want to make it do something so we can just use this open CV one program and guys what we're going to do today is this is going to serve as the starting point for all of our future lessons because we're gonna set up a basic core program that will go out fire up the camera and read a frame and display a frame and so we'll have a kind of like a little thing that does the the core elements that we're gonna need for all the future lessons so you need to save a copy of this program and then always you know work from a copy of it don't modify this program after today because we want to kind of have this core starting point for all the future lessons alright let's jump in here and do something first thing you want to do is import OpenCV which we do with the import CV to command so that's already done now in order to fire the camera there's like a whole bunch of parameters that we're never going to change but there's kind of three parameters that we will change and so the first is we're going to call it display width and that's how wide the display is when it is shown on your screen and I'm going to put kind of a small one I'm going to put 320 and then I'm gonna put display height and I'm gonna say 240 now when we run the program we can come in and make this bigger or smaller but you want to kind of keep this aspect ratio okay you want to sort of keep this aspect ratio and that will serve you well now the other thing you want to do is you want to come in and say that flip equal to and that's one of the parameters on the Raspberry Pi camera that as we built this if you don't put a flip method in your pictures going to come up upside down and if you put flip equal to it'll come out looking like you want it to okay now this is the hard command so what I need you to do is I need you to type in this command alright and what this is is this is setting up a string okay and we have a variable are you getting all this are you getting it all this long command okay we have a variable called cam set okay what cam set is is it's a string variable and what I'm going to do even though you know under God that hates copying and pasting I'm going to put this command down in the description now in the description it might wrap around but what you've got to see is you've got to put it in Python you've got to put it as one long line like you see me here but because you're copying and pasting it I do want to kind of go in and tell you what this is and so this open parenthesis like an open quote this is starting the string and the first part of it is in Vargus camera SRC this this launches gstreamer okay it's a program called gstreamer to run the camera this is just the command that launches gstreamer now I could spend a year trying to teach you gstreamer but I'm not going to do that I'm just going to give you the parameters that will work that makes gstreamer work so then we have like a video format it's called video /x - raw memory : Envy mmm okay and then I actually don't want this so this will not be down below this was me playing around with parameter that we shouldn't play around with so I'm going to take that out and it won't be in the one that's pasted down below and now the width where it says width is 32 64 that's the native width of what's coming off the camera so we're reading the full resolution that's coming off the camera now how does that relate how does that relate to this w is 320 that's the display width and the display height is the 320 and 240 we're reading it in is full resolution and that way if we want to display it larger we've always got the native resolution that we want so we've got the native resolution of height equal 32 64 or width is 32 64 and height is 1848 then we want to format it as in v12 we want to have a frame rate of 28 / 1 so basically that's 28 frames per second the camera I think will run it like you can run it at 60 frames per second but you can't run it at 60 frames per second at full resolution and that's why I don't want you to really monkey with these values because these are the values that will work for the Jetson nano connected to the Raspberry Pi camera okay then we have this Nvidia conversion where we're doing a flip method now this is the one thing that you should sort of see you see how I ended the string here and then this plus is concatenating it with turning that flip variable which above was set to 2 we make a string out of that and we concatenate it so what that's doing is that's taking that flip variable above of 2 and it's just adding it into the string so we're going to have one string but this allows me to change those variables above and it build the string based on those I hope that makes sense to you but this is just inserting that to from above into the string and now we concatenate that with again opening the string and then we have this video X raw and then we have width and again this width is the width from above that we're going to display in this height is the height from above that we're going to display so you see I'm building this string as you're putting those numbers in above then we got to format it as BGR that's blue green red kind of confusing but OpenCV works with blue green red where normally we think of red green blue and then we've got to use the video convert method and we're doing video /x raw and format again is blue green red and then a PPS sync so you need to go down in the description you need to copy this whole thing and you've got to make sure that it pasted in as one line of code okay now we're going to hit enter and now we start coding again and we stop copying and pasting okay so now what we need to do is we need to create a camera object and that is and what we're gonna do is we're gonna launch our camera so what do we call our camera object let's call it C a.m. okay and what is C I am it's our library cv 2 right that's open CV we're giving like our first CV our first open CV command C V 2 dot and what we want to do is we want to do a video capture and notice that the the capitalization is important its capital V video capture capital C capital v capital C and then how do we launch it we just send that strain cam set right cam set from above we send that into the C v2 dot video capture function and now I have an object called cam now when I want to do something I just work with cam does that make sense cam is my camera now if I want to work with the camera I work with cam I hope that makes sense and do you see why I did it this way it's a very simple launch now all you got to do is just get that long string and then you got to send it to video capture now we have a camera ready to run now if you think of the camera the camera is a frame a frame a frame a frame or if we're going to want to read that we're going to want to read it inside of a while loop so what are we going to say we are going to say wow and how long do we want to loop forever so we say while true when it's true true true is also always true don't forget your colon at the end while true what do we want to do we want to read a frame when you read a frame it returns two values the first is kind of a zero or a one kind of indicating did you get a frame or not but with this setup we're really going to pretty much always have a frame so you just got to put that variable somewhere that first variable we just call it ret like what it returns that's the zero or one we won't ever use it but if you don't you know this function we're calling is going to throw two variables at us so we better have it set up to catch two variables the first is ret we will ignore that and then the second one is the image itself so we will put that in a variable that we call frame you could call it IMG you could call it my pic you could call it whatever but because it's a frame I'm gonna call it frame now what is that equal to well we got to go out to the camera what's our camera called C a.m. we want to read a frame how do we do that read open close that will go out and that will grant grab the latest frame the latest picture from the camera how easy is that okay now if we grab it what do we want to do we want to show it how do we do that again it's an open CV command it's - okay and then dot I am show so I'm gonna show and what am I gonna show well I got to give it a window to show in so we got to give it we got to give a name so let's call the window PI cam like that and now let's go out past the quote and now you have to say what are you going to put in this window called PI can I have a window it's called PI cam what am I gonna put in that window what would you guess you guessed it frame that frame that we just got so do you see what we're doing here we're grabbing a frame showing a frame grabbing a frame showing afraid I can't tell you how excited I am okay so we're gonna grab a frame and then show a frame now this is the thing and this is really important you know in our earlier programs when we wanted to leave Python we would just kind of crash it with a control D or something like that to just kind of crash out of the program from the command line on a program like this you have to exit gracefully because we've gone out and we have created a window and we've gone out and we've locked up the Raspberry Pi camera and if you just kill the program that's going to kind of have unfinished business out there that you've got to clean up so you've got to gracefully exit so what we are going to do is we're going to set up a key and if you press that key it will gracefully exit so how do we do that well we use a command if C v2 dot wait key and how long are we going to wait we're going to wait for like one millisecond all right if that is equal equal so our D I'll explain this in a minute and then open quote open single quote and then acute so what see v2 dot wait key is it just goes out for one millisecond to see is there a key being pressed and then if there is a key being pressed it looks and it sees is that key cue the ORD just kind of like reports the ordinance the order of the key but when you do this you don't have to worry about what the Orden umber is you just say if it's Q then you go into this if statement okay Q is for quit and so what do we do in this Q well what we are going to do is we are going to break out of our while true loop okay we are going to break out of our while true loop so it's going to loop and loop and loop and loop and if you press a key it'll read it if you press a key it'll read it if you press a key it'll read it but if you press Q it's gonna what it's gonna break out break out a foot break out of the while loop so it's gonna kill the program but as we kill the program we have got to clean things up what is one thing we want to do we want to let go of the cam so I'm gonna say cam release and if you don't do this and just kill the program and then rerun it it's like the cameras tied up and sometimes it doesn't air out sometimes it works but it starts acting real strange you really want to do this this is releasing the camera and then what you want to do is CV too and you want to destroy all windows and remember the capitalization matters the case the upper case lower case matters all right so look at that what I have is I have a little program are we gonna run this I think we should we do need to hold our breath on this when I've done a lot of coding I wonder how much must how many mistakes I mean this cam said I think should be right because I copied and pasted it and it'll be down in the description down below and see let's right mouse click run Python file in terminal hold your breath what happened it didn't like it what happened why did that not work why did that not work let's see if we got an error here somewhere it did like my CV to wait key what did I do Oh Oh remember how I was telling you about how capitalization mattered watch this the case should have been uppercase but watch this if I say wait look at that it was trying to tell me the right way to do it you see the wait key it was trying to tell me the right way to do it if I would have just selected that now let's write that one really scared me guys cuz I thought I'd done this right okay hold your breath run Python pilot our mold boom look down here look down here I have a window hi okay why am I so small why am I so small okay I'm small let's see let me turn it here where it will look more towards me more naturally towards me okay there I am okay and then I'm gonna adjust this and you know how I love to have all these things kind of looking good so here's a camera that I have looking down at the camera that is looking back at me but I need to get it to focus you know what guys I cannot stand having a camera out-of-focus so you're just gonna have to bear with me till we get a good focus okay now just stay there stay there don't get out of focus I'll call that good if you'll call it good I'll call it good okay so do you see the camera is looking back at me okay I am not exactly happy with that let me show you something down here if you guys will look down here it will tell you what the possible frames are that you can do and one of them is 3464 by 2464 so let's make sure that that's what we did here 32 64 by 1848 that does not look exactly right to me so it's either going to be 32 64 by 1848 or it's going to be 32 64 by 1848 do you see how the camera is telling you what settings it likes and so for my screen I think I was too much like this and so I'm going to for my screen go back to 24 64 like that okay now don't kill the program to kill the program you need to come and click on the window and type cue why because we did that wait key business ok 32 64 by 24 64 you guys look and see which one looks right you don't want to be scrunched this way or you don't want to be stretched out and it might depend on this screen but I'm gonna try to run this again and see what happens notice I am changing the string okay we've got to get that read value right so now I'm going to right mouse click run Python in terminal that looks more right to me that looks more right to me boom okay so like if you kind of compare me to me here that kind of looks right alright so what are you not so what how are we going to quit we're going to queue now how would we now don't mess with that string it's working don't mess with the string if you want to mess with something play with these you kind of would want the window bigger but try to keep this aspect ratio so I'm going to multiply by two and say 640 by 480 and now let's run that boom look at that look at that one two three four one two three four there's really like no latency in that and that really that is really looking good run in the pod c'mon the Jetson Nano yes okay did you do that with me you know they do that on football games I think we should do it in programming we should really do that more okay and let me quit out of that and let me just also just do kind of a quick check I'm just trying to kind of tweak this and make sure framerate it wants is 28 okay just to be good we use 32 by 64 and on that framerate it really seems to want a frame rate for that size of 21 and so I'm going to change this to 21 so if you use 32 64 by 24 64 you need to set this frame rate to 21 okay if you use 32 64 by 1848 if that looks more like the right aspect ratio on your system then you would set the frame rate to 28 you would put 28 here does that make sense okay this is really fine-tuning some people look at it and can't tell a difference but I just really don't like an aspect ratio to be wrong okay I'm going to make sure yeah that window is dad so I'm going to come over here and I'm gonna right mouse-click and I am going to run Python file in terminal okay yeah I think that I think that's better I think that's less less drop-frame one two three four one two three four I think that is good that looks really good okay I'm going to queue out of there and what if we did like the next one would be like 1280 by 4 this would be like 960 something like that let's see what would happen there 1280 by 960 run Python oh yeah okay we got a big one there alright oops okay that's right one two three four that really looks good right that looks good what you notice is it doesn't that the Raspberry Pi camera doesn't really have the white balance on toon that if you look at me here in the real shot that camera has auto it it's white balance incorrectly in the and the Raspberry Pi camera isn't quite balancing but we're not going to worry about that because this is right really it's like a high resolution there's no lag it is really looking good in my mind so I'm gonna queue out of there alright guys this is pretty exciting to me oh and just so that you know these are the different ways that you can configure the camera down here and really what the best thing is is to configure it in a high resolution mode and then just up here display it in a lower resolution so I would not mess with this string once you get it right and grab it it's high resolution and then display it at low resolution that is what I would suggest that you do okay and this shows you all the different ways you could you could set that up 32 64 by 24 64 you know you could come down to like 1280 by 720 and do a frame frame rate of 120 but don't mess with that stuff okay let's look down here and if you notice when we launch this thing we get this error message failed to load the camera module and that doesn't really hurt anything but I hate having an error message I can show you how to get rid of that let me make sure that I killed yeah the the camera is killed come over to a terminal and this is the command I need you to do I need you to do sudo apt-get install and then lib can Vera Li be ca in be e RR a - gtk - module so somehow on the nvidia somehow on the nvidia image that we got there's this little library you really need so let's go ahead and install it just to make sure that we don't get that error i don't think it hurts anything but i just don't like getting errors when i run programs okay that looks good now let's go back to our visual code studio and now let's run that again and let's see if we get the error this time run Python file in terminal boom there I am no error do you see over here no err okay let's quit out of this okay now most of you I think that have been playing along at home most of you got the Raspberry Pi camera but some of you are out there saying hey what happened to those of us who got the webcam those of us who got the logitech webcam because that's one of the that's one of the cameras that I'll link to you can either use the Raspberry Pi camera or you can use the logitech camera what do you do in that case well what I'm going to do is I'm going to show you how you launch that camera it's actually a lot easier so what I am going to do is I'm going to comment out this cam okay and then I'm going to come down here and I'm going to make a different cam and that is going to be cam equal C V 2 dot video [Music] vid video and there do you see how it has capture oh that was not good I'm trying to get kind of better ok video capture and then now instead of that long complicated string since it's a supported webcam all you got to do is either it's either going to be 0 or 1 well since I have the Raspberry Pi camera already on there that's probably zero for me but you can't launch the Raspberry Pi camera by just calling 0 you got to do the GStreamer but for a webcam all you got to tell it is a cam 0 or cam 1 if it's the only one you have it's likely cam 0 but if it's not cam 0 it is probably cam 1 alright and then everything else is the same I've just switched cameras and so let's see what happens hold your breath right mouse click run Python in terminal whoa look at that on the webcam on the webcam you see I'm putting my hand over here by the Raspberry Pi camera uh-huh it's not the Raspberry Pi camera it's the webcam ok but notice everything else is working the same and let's kind of check our latency here oh I like that 1 2 3 4 5 you see good moving not not getting all jittery oh I should put that where you could see it right ok 1 2 3 4 5 moving around not Jerry that is really neat so that's the webcam ok that's the webcam so you guys that are on the webcam it's a lot easier than those of you who are doing it but this is what I want I want this to be your course starting program and if you want to use the Raspberry Pi camera just comment out this and if you want to use the the yeah if you want to use the Raspberry Pi camera uncomment this and comment this if you want to use the webcam comment out this and come out uncomment out that so now always save this and just uncomment out based on the camera that you are used so now what I want to do is here in this open CV I want to create a new program so I'm gonna say new file and I'm just gonna call it PI scratch dot py and this is gonna be a program I play around with I'll just leave it there anytime we want to play around with something we can play around with it what I want to do is I want to come and get this program because right I don't want to mess with this one this is the one that will always be the starting point for what we're doing so I'm going to copy it and I'm gonna come over here and I'm gonna paste it okay like that now look at this I've got cam set to the Raspberry Pi or I have cam set to the webcam mm-hmm what should we maybe try to do do you have an idea if you're in the premiere tell me what you want to do just to try do you think we could run both cameras at once both cameras at once well you can imagine that I can't have the same object going to two different cameras okay so let's call this one let's call this one the PI cam and let's call this one the web cam all right now what do we need to do we need to come down here and we need to read from what PI cam and then we also need to what we need to ret and we need to read don't put it in frame because you can't put two pictures in the same variable we'll call it frame two and this will be from which camera web cam and that is going to be dot read okay now if I read twice I better what I better show twice so I'm gonna do C V 2 dot I am show now you can't be putting both images in the same window now you know once I say that it makes me want to do it what it's going to do is if I put it in PI cam and I put this frame - you know what this is going to do it's going to have one window but it's going to be going back and forth between the two cameras I think I shouldn't play around like this but I can't help it I just want to try that and see what happens now it seems to be it seems to just do the one so that just kind of didn't work at all it seems to what it seemed to do was just go to the webcam okay so what it ended up doing was just kind of the second one the webcam which was framed to it just kind of kept doing that one and so what we will do is we will make a second window in this window we are going to call it web cam you could call it whatever it's just something that sort of represents what it's soon so we should get how many windows this time to run Python file in terminal ah look at this boom boom okay why is pi cam so big because I'm PI cam we set the display within the display height right on the PI cam we set the display width in the display height when we launched it but when we did video capture when we did cv to video capture for the webcam that was just using the default values and this is kind of just the default values okay in fact let me kill this cue the program and then let's go to our programs up here and let's run cheese cheese is just a can program that runs the webcam and do you see it's like that same size that's the default size that's when we were doing it okay so cheese it's just a Linux program that runs the webcam and you see that is the same it's that same default size this if we did run Python file in terminal okay and then you see this looks like the same size as when we were running it in cheese okay let's see one two three four five alright so up there in the corner is the webcam okay the big one is the PI cam if we wanted to kind of make it about it seemed it got a little glitch there for a second okay if we want to make it the same size I'll quit out of there probably that was more like 640 by 480 okay right mouse click run file okay I think these are about the same size now boom look at that Wow okay man I'll tell you I am really excited about this the fact because darn it this is really cool stuff okay this is what I want you to see on this program let's go back to our first program where we were just running one we fire the camera here we fire the camera here and then we read a frame and then we show the frame okay you see this here between Reed frame and show frame that's where the magic happens because you think of a movie happening really quick at like the 28 frames a second or 21 frames a second or whatever you think of it happening really quickly we have all the time in the world to create mischief between reading the frame and showing the frame we can analyze the image we can add to the image we can take from the image we can change the image we can manipulate it analyze it do all types of different stuff because we sit in this spot between reading and displaying and that my friend is where the magic is going to happen and so as these new lessons go along what we are going to be doing is we are going to be doing some really really cool stuff between reading the frame and showing the frame okay I wonder like if I could even I always hate it because I always think that I'm gonna show you something and I really shouldn't I really shouldn't go in and start doing these things on the fly because they rarely work out let me see let me just see if I've got one here that will do something let's see I'm just gonna try to run some of the programs I've done before I really shouldn't do this you know that ah look at that this is the edge version of me what do you see you see edges okay well edges are actually really important they're really important things okay that was kind of cool that it actually worked okay and then let me see here let me see here it's always the challenge of me deciding to do things on the fly okay it's always the challenge of me trying to do things on the fly okay there's something not be interesting okay I got a sparkfun box I love sparkfun out order a lot of stuff from them let me see if I can come over here and run this let me just see if this or and some of these aren't really set up to run I've just been playing around so let's see if this one will run okay well it kind of messes it up because I'm wearing a red shirt I'm kind of wearing a shirt that's almost right but the purpose of this program is to find the color red and you see what I'm doing I'm tracking on red and what I guess I need to do is I need to get you see how I'm targeting it I'm finding it I'm outlining it and then I'm targeting targeting it you see that is really pretty cool now this is what you got to think about right because we have those GPIO pins I could be moving something based on those crosshairs right I could have servo setup and I could really be tracking this thing or aiming something at this so all of a sudden all types of crazy things start happening let's see what else I'm going to do here let's see I had some cool stuff here okay let me let me see if I can do this okay and guess these aren't just copy and paste things these are things you're gonna be doing from scratch and look at that what's it doing it's finding my face and it's finding my eyes I used to have a smile detector on it now I turn the smile detector off but it put a red box on a smile but I don't have that turned on right now boom do you see this and this stuff guys you're gonna be doing and you're not going to be doing it based on copying and pasting you're gonna be doing it based on sitting in that magic position between reading the frame and displaying the frame so we come in here and we recognize faces and then we put boxes around them or we could recognize specific people or we could recognize objects and then track them in so gasps that is where we are going and I am so darn excited about this I can't tell you how excited I am okay guys leave me a comment down below were you able to get these things working do you have your camera up and working are you as excited as I am about this all right guys we are really getting into it I know a lot of you have been impatient because of all the groundwork that I built you wanted to just jump in and do things but now you understand Python you understand matplotlib you have the right version of open CV on your on your system you have kind of done a review on Python man you've got this this groundwork down done now we are gonna do some really cool stuff so anyway appreciate you guys tuning in the the stuff is really going to get fun at this point those of you who have stuck with me this stuff is really going to get fun at this point would love to hear your comments down below think about giving us a thumbs up when you give me a thumbs up it actually helps me in the Google algorithm where Google will YouTube will start recommending this video series to people so you really do help me if you give me a thumbs up and again really appreciate you guys who have decided to become a part of this channel by helping me out on patreon really really appreciate it it's a huge encouragement to me okay this is polemic order from top tech boy com I will talk to you guys later

Channel: Paul McWhorter

Views: 30,957

Keywords: Jetson Nano, WEB Camera, Raspberry Pi Camera, AI, Artificial Intelligence

Id: AoyZHsXKw3Q

Length: 43min 21sec (2601 seconds)

Published: Sat Nov 09 2019