AI on the Jetson Nano LESSON 51: Improving NVIDIA Jetson Inference Library for RPi Camera

[Music] hello guys this is Paul McWhorter with top tech boy comm and we are here today with lesson number 51 in our incredible new tutorial series where you're learning artificial intelligence on the Jetson nano I will need you to pour yourself a nice big mug of iced coffee I will need you to get out your Jetson nano gear and I will need you to get ready to learn some cool new stuff what we're going to be doing today is we're going to be continuing in what we started last week and that is to really begin looking at these deep neural networks and deep learning on the Jetson Nano hey as always I want to give a shout out to you guys who are helping me out over at patreon your help and encouragement is what keeps this great material coming you guys that are not helping out yet look down in the description there is a link over to my patreon account think about hopping on over there and hooking a brother up but enough of this shameless self-promotion let's jump in and talk about what we are going to be doing last week we did our first program where we were really using deep neural networks or deep learning and the way we did that was we configured our system to use the Jets and the Nvidia jetson utilities and inference libraries to allow us to access some really sophisticated models for image recognition and with those models we could write simple programs that would allow us to use our webcam or a Raspberry Pi camera to begin to do some some really cool image recognition now if you remember though there was a little bit of a glitch things work great with the webcam but if we were using the Raspberry Pi camera depending on how we did it we kind of had a trade-off between if we went the opencv route where we read the frame using and CV then converted it to CUDA and then did the image recognition we got great frame rates but the problem is we had like a two second late latency and for an OCD guy like me latency is just something that I cannot tolerate I cannot live in a world where there is a two second latency we could do that or if we use the Jetson utilities and created the camera under the framework of the Jetson utilities it would load and it would do the image recognition and then do the display very quickly but the problem there is there's an error in the way that they set up that GStreamer command down in their library and if we set the frame size that we wanted it doesn't make the frame that big it takes the full frame from the camera and crops it and that throws everything off and then they don't really give us any handles to change it in our call to actually get the GStreamer to get the GStreamer string that we really want and so what we're gonna do today is we're gonna go in and fix that okay we're gonna fix that so you guys that are playing along at home with the webcam you can just watch and be entertained you guys that are playing along at home with the Raspberry Pi camera we're gonna fix it where you don't suffer moving forward in future lessons that your cameras gonna work right and work the way it was intended so we are going to come over here and I'm gonna get out of your way we're gonna open up visual studio we're gonna create a fresh new program I do believe I was working in the Nvidia folder inside a pipe ro I'm going to create a new Python file and I'm going to call it deep deep learning - 3 dot pi ok fresh new program hey I don't want to spend 30 minutes redeveloping the software that we developed last week so we're gonna run over to the most excellent WWE tech boy com and we are going to go to AI on the Jetson nano lesson 50 that was last week's lesson you can just search on that with this little green search icon search on that come to lesson number 50 and let's come down here and let's grab this code that we had ended up with last week so right mouse click copy come over here right mouse click paste and if we take a quick look at it just to remind ourselves we were loading up the jetson libraries and the opencv libraries we were actually creating the camera using the jetson utility GST camera and again we passed it width and height and then for the case of the raspberry pi camera you just pass it zero telling it you want camera 0 Canberra 0 is the first camera camera 1 is the second one if you have the 2 board if you have the 2 board model nothing else we can pass it but those two things so they've kind of tied our hands I wish they'd let us pass a few more parameters in this create camera utility but they didn't and then we pretty much use the Jetson utilities to load the frame and then to find the object in the frame or identified the object in the frame and then we convert that over to an open CV image and we do our display in open CV this is if I'm going too fast go back and watch last week's lesson all right so this is all looking good but then if you remember when we run this hopefully this won't take too long to run sometimes the first time you run them it takes a little while to get everything figured out and then subsequent times not so long but I'm hoping this thing will pop right up tensor arty is having to think about it a little while man have you what did you guys do did you guys go in and try different networks besides like here we're using Google net did you try those other ones like the Alex net and the Google net 12 and some of those other ones I had a lot of fun playing around with different libraries and some of them are better at finding some things than other ones and you can kind of play around with those in pick your favorite library boom there we are okay what do I like about this well I like that it didn't crash what do I also like I like that we are running at a whopping 20 frames per second I like that it seized my microphone and identifies my microphone and I like that there is no latency 1 2 3 4 5 there is no latency and we are finding items that's not lotion or what happened to pill bottle here's pretty it's not doing very well today okay you see there's too much confusion around it but you know we saw that it worked last week okay does get the microphone what is it that I don't like about this okay we set our image to 1280 by 720 but instead of taking that in scaling it to 1280 by 720 it took the huge 32 64 image and just cropped it so really this is kind of a low resolution image because it didn't resize it it cropped it okay and we don't like that at all because it doesn't give us an ability and plus the way this is done you really don't have any choices and the size that you would want to make it because if you just make up something 1280 by 720 is about the only one that you can put in there and anything else you put in there is going to cause it to crash okay because they didn't quite put the thought that they needed to when they did their gstreamer command so we are gonna live dangerously and we're going to edit their library and we're gonna fix it and we're probably gonna break it and we're probably gonna regret the day that we ever tried to do something so bold okay what I will need you to do is get out your G edit okay G edit so you come up here I probably went kind of faster you click on the upper-left icon which is kind of the start button you tap in G edit and you get this little text editor here you know I guess if I were braver I would open it in Visual Studio but I'm just gonna do it in G edit which is a text editor what you're gonna do is you're gonna say that you want to open and we're we're gonna do is we're gonna go to downloads okay we are gonna go to jets and inference that's where those libraries are we're gonna go to utilities and then we are gonna go to camera okay and then what we are going to do is we are going to edit GST dot CPP okay GST camera dot CPP so we're gonna open that all right now because we don't want to mess this up and then be in some sort of hopeless unbootable state we're gonna save a backup before we change anything so I'm going to save file and I am going to do save as and we'll save in the same place but under CPP I'm gonna put back up okay like that so I'm going to save it as backup so I have a fresh unedited one okay and then darn it that gave me the GST back up now is the working one so we'll come back over here open and then mine is right here I'm just going to go back to GST camera CPP okay so let's that was a lot of stuff really fast there so let's pause for a second let's pause for a second and let's make sure that you backed up GST Cameron CPP you backed it up as GST camera dot CPP back up and now you're back over here working on GST camera in fact I'm going to close this back up file so the only thing that I have open is GST camera dot CPP the problem lawn in this massive library library is about line 452 so come up here to search and then say go to line and go to line 452 I do believe let me make sure that that looks right ok I was a little bit off was that 428 was what it said 428 yeah that's probably 428 it's a little bit close a little bit closer so the the thing that we want is do you see here on about wine 428 and what this says is I wonder if I can make this easier for you to see pardon me while I see if I got a bigger font preferences ok editor font in colors let's make this about 18 okay that looks a lot better all right so if you look this is where we are in the library launching the GStreamer command and you can see that it's got several different cameras that it's looking at it says if the camera source is in Vargas that is not our camp oh yes it is okay this one where it says if it's GST source in V camera that is not our camera our camera is the GST source is in vargas and then you see that it's creating the gstreamer it is creating the G stream or string here okay it is creating the GStreamer string here and again that is about one 428 it's where you could also search on else--if and then you could search on this GST underscore source underscore n VAR g us okay then you begin to see here why I hate C I hate C with a passion because this is very very hard to see what they're doing here but what they are doing here is they are creating they are creating here the G stream or string okay this would be the string that let's see let me go to one of our earlier programs let me just go to like open CV one of our very first programs that we ever did and this was this cam set command so you see this in Vargas camera source ah this is not good I lost my spot so let me do a tools search go to line okay there it is alright so that's good so let me see if I can make these where you can see both of these things at the same time I think I can so what we are interested in is this is the string that launches G Street Dreamer so you can see it starts within Vargas camera source in Vargas camera source well we've just been not putting in the sensor ID but they left in here an ability to put in the sensor ID and so that's a string and then what they feed in is they feed in the variable M sense or CI okay and that part's fine all right but then what they do is they get this parameter video X raw memory in m NV mmm that's good right you see that matches what we're doing but then what they do is they set the camera width here at this first camera call but if you remember when we created this GStreamer string you don't set the size there you set that to full-size which is 32 64 by 24 64 and so this rather than putting in the variable this width needs to be 32 64 and this height needs to be 24 64 and then if you remember we make the size right way out here after this Nvidia convert we resize to the width and the height that we want so you launch the camera full strength and then you do the Nvidia convert to convert it down to a smaller size so I could put you through the torture of trying to edit this by hand I could do that to you because that's what I had to do but basically I am going to give you I'm going to give you a string and the string will be down in the description below and if it's not in the description then look in the first link because sometimes YouTube is strange and it won't let you put certain things in a description but you can't put it in a comment but what you will find is you will find these this all needs to be replaced it's everything from this SS through this SS all of this after this else if at about line 428 needs to be replaced and I'm going to replace it with this okay and you can get this in the description but I want you to see I want you to see that I have one SS I have a second SS and then I end with the : but you can see what I did was one thing I did was I didn't like my image that was coming off the Raspberry Pi camera because it was making me blue and washed out and so I put in this parameter of white balance mode equal three which warms the thing up a little bit and then we kept the sensor ID just like they did but what we did was you will see that rather than putting a variable and under width we putted it 32 64 at height we put it at 24 64 we had to change their frame rate from thirty to twenty one okay like if you look over here they had a frame rate of where is that silly frame rate they had a frame rate of 30 that has to be changed to 21 and then we do the flip method just like they do and then we have the video raw X coming after the Nvidia convert just like they did but then we inserted those variable width and height here which is a resize so you start with a full image and then you resize it down and then we basically have made their command here look like that nice gstreamer launch string that we have always used but do not be alarmed you're just going to get all of this all of this you are going to copy it you are going to come over here and you've got to make sure you want to replace that colon or that semicolon all the way to the SS don't select the curly bracket don't select the close parentheses right mouse click paste okay so now I have a new GStreamer launch command that will behave in the right way so we are going to save GST camera cpp we're going to save it we are going to close this okay and you're going to get that from the description at the bottom of the video okay and so now I will close this up now I will close this up now what we need to do is we need to do a couple of quick bookkeeping things so we've changed that file well we've got to recompile it alright so we're going to have to go to CD and we'll have to go to desk top right LS then we're going to CD where we are going to CD nope we want to go dot dot we're going to CD into downloads we are going to LS and now we are going to CD into jet Sun inference like that and then we're going to LS and then we are going to CD into build okay we are going to CD all the way in to build and then let me look here give me just a second here I got on the wrong page you're gonna have to give me a second because I want to make sure that I'm in the right spot here I want to make sure that you get exactly what you need okay I'm sorry for that little delay but we are in the right folder and in this folder we are gonna want to do a make okay so we've got to make this again because we change that 1c program so it's like we got to put the whole thing together again but this just takes just a second the make is going very quickly okay now we need to do a sudo make install like that all right now let's come back to our program and what I anticipate when we run at this time I'm not going to be filling the screen it's going to be showing the background okay it's going to be showing the background so let's right mouse click run Python file in terminal I feel pretty confident that this is gonna work it's moving ahead no air is so far you guys leave me a comment down below how many of you are working on the Raspberry Pi camera and how many of you are working on the webcam I kind of think there's pros and cons of both I like the Raspberry Pi camera better because it gives me more since you're launching from a g-string or command a GStreamer command there's more parameters that I can set using the Raspberry Pi camera okay boom look at that okay do you see how now you're seeing all the way from the wall way over here and it's recognizing microphone also hopefully you see there's a little bit more natural color balance there we are operating at 17 frames per second and watch this 1 2 3 4 5 no latency you see that okay no latency let's see if we can recognize something else I'd always get distracted cuz I always want to see it do something it thinks it's an oscilloscope remote control Cellular iPod okay that's pretty good that's pretty close iPod is about the closest it's kind of get let's give it something good to recognize you know what the problem is this is kind of like what I told you in recognition really kind of wants a clear view so there's too much in the screen for it to really for it to really do very good it's got the microphone okay that's good operating at 20 frames per second no no latency and we fix that problem okay also come in and look I'm going to quit out of this now you can actually resize this thing and it'll work okay so now let's come in and let's say that instead of 1280 let's say I went 640 by this would be what 360 okay I'm gonna resize it small so you see this is gonna fix it and with that other program or the library the way they wrote it you couldn't resize it because the you you had to set it to like one of three choices the camera gave you and they didn't build in the ability to resize so we're capturing the whole thing and then we're resizing it down okay boom there okay you see what I mean you see I made it smaller and we're still operating at 21 frames per second so now I can kind of make the window and the image any size I want okay guys I hope this hasn't been too tedious I guess what we should talk about is we should talk about what if you mess things up how do you get back to where you were if you mess things up if you mess things up what you would do is you would come back open up your filing cabinet you would come to desktop though I mean documents new not documents you would go to downloads you would go sorry I froze how's that okay from documents you would go to are from downloads you would go to Jetson inference from Jetson inference you would go to utilities and then from there you would go to camera and then remember how you made that backup okay you backed up your GST camera your GST camera dot cpp you created a backup well what you would do is you would kill this one that you just edited you would erase it and then you would rename this one the backup back to GST camera dot CPP so now you have your original one back by doing a rename and then you would come back and you would go to where we were you would go to CD downloads and then jet Sun inference and then build like that and then you would do the make and you would do the sudo make install again and then that should take you all back to where you started and then you could try again but if you copy and paste from the from the message below and you just make sure to put it at the right spot about line 428 it should work for you okay this has been a little bit of a tedious lesson but that thing bugged me so much that I didn't want to go through the rest of these lessons without being able to control the camera size or without being able to control the quality of the image and so unfortunately if I want to tweak the image I'm gonna have to go into that to that camera file and I'm gonna have to edit that G string or a streamer command and you see where it is and if you're a little bit more adept you can play with it yourself you know as far as tweaking the parameters to get the camera image that you want if you're not adept at it you can at least get something that works a lot better by copying the couple of lines of code that I put down below and that should help you okay guys I hope you enjoyed this lesson if you did give us a thumbs up think about subscribing to the channel and we will have an exciting new lesson for you next week palma quarter from top tech boy comm I will talk to you guys later

Channel: Paul McWhorter

Views: 5,062

Keywords:

Id: 60IaXdPn6RA

Length: 29min 18sec (1758 seconds)

Published: Sat Jul 18 2020