Jetson Nano B01 - Dual RPi Cameras + how to get faster frame rates

hello it's Jim from Jeson hacks calm on today's show we are looking at running dual CSI cameras on the Nvidia Justin nano developer kit version b01 recently Nvidia updated the Justin nano carrier board to version b01 this included support for to CSI cameras today I want to share with you how to access those cameras then we'll go over some optimization strategies for incorporating them into your OpenCV applications here are the two raspberry pi version 2.1 cameras that we are going to install notice the end of the ribbon cable connector one side has blue tape the other side has the contacts the contacts will go towards the Jetson module when we install it there are two camera connectors here for the CSI cameras gently pry up the plastic camera cable retaining clip as shown in the picture you can use your fingers or a small screwdriver just remember to be gentle next insert the camera cable ribbon make sure that the tape side is facing towards the outer edge of the board also make sure that the cable ribbon seats all the way into the connector press down on both sides of the camera cable retaining clip to capture the camera ribbon cable it will look something like this when you're finished repeat the process for the other camera and you're ready to go I'm using the helping hands from a solder station to help me with my demo on the jets attacks nano account on github there is a repository named CSI - camera let's clone that repository grab that repositories address then get clone now let's switch over to that repositories directory okay with that out of the way let's test our cameras here's a simple GStreamer command this sensor ID parameter tells us which camera to choose we have to so it can be 0 or 1 let's test out camera 0 tanks that's scary close that up no one wants to see that yeah let's test off the other camera let's try out old number 1 that's scary too ok if you're still here both cameras appear to work before we go any further notice that prints out the sensor modes let's put those sensor modes into a file just grab them here [Music] let's call it camera work text you can see that there are five different camera modes they range from 30 to 60 4 by 24 64 frame rate of 21 up to or down to depending on how you look at it 1280 by 720 120 frames per second so we'll keep that around that will be useful later we will be running OpenCV examples in Python 3 today in order to run OpenCV and python we will need to install numpy let's do that okay now we are ready to install numpy let's install sites on first numpy okay installation complete let's open up visual studio we are going to use that as our Python IDE let's open up our repository directory let's run simple camera PI oh so that appears to work it appears to be lagged also one of the things we're going to talk about in this video is how to kind of get your frame rates up I don't know which camera to look at again it's always a problem let's close this one up we also have dual camera so let's take a look at both cameras at once we'll run that example okay so here's the to CSI cameras one of the things I find interesting here is that even though the to CSI cameras are the same they're two separate cameras but they're two same make you get different color characteristics out of them you can see down here the webcam that's connected to the Nano is very saturated then the CSI cameras are less so but there's still quite a bit of color difference between the two CSI cameras so each camera has its kind of own little variation that you have to take into account now if you're just looking on how to access the cameras in your program all you have to know is the sensor underscore ID in Indy Argos in the GStreamer pipeline The Jetsons use GStreamer as their access to the cameras as mentioned before the sensor ID parameter - and the argos camera source is the key to this there are other parameters here you may find useful you can request a width of a frame a height of a frame you can also request a frame rate you can also request a sensor mode let's pull up our sensor modes again so here's our sensor modes this is kind of a shortcut here you can request any of these five the number even starts at zero so zero one two three four if you don't request a specific sensor mode it will try to configure it based on the parameters that you passed and then we construct the rest of the gstreamer pipeline then we end up here with app sync this tells gstreamer that an application program is accessing the gstreamer pipeline of course this is all set up for Python you'll have to translate it into whatever program language you're using here are the default sensor modes that we use in this particular case in this case we take it capture within a capture height which is requesting the frame size from the camera and then we convert that to this display width and display height by using a video convert in our case we're going to use sensor mode 3 1280 by 720 please return it in 1280 by 720 at 30 frames a second so let's go back over to simple camera pie there's an equivalent statement here if we look at the sensor modes the sensor mode 1280 by 720 can be provided in 60 frames per second here or 120 frames per second in our case we request 60 and the program itself is pretty simple basically in OpenCV we setup a video capture we set up a window we read a frame from the camera and then we show a frame from the camera and then in the third part here this is a little tricky the weight key is basically a processor yield it will weight up to 30 milliseconds and if it gets a key back it looks to see if it's equivalent to 27 which is the escape key this can act as a frame rate limiter in this type of loop so you'll always have a 30 millisecond weight in your loop it's important in your main input loops to have some type of processor yield it allows the operating system to multitask 30 is rather an arbitrary number typically the results of this are around 30 frames per second let's set it to 5 and let's run it make sure to save it so we see that it's lags a little bit here it doesn't really sink too much with my voice it appears to be about the same as the webcam a little bit laggy now we start guessing what that may be but because it's not instrumented we don't really know one of the reasons may be that we're requesting 60 frames a second from the camera and it can't display it that fast so let's change that [Music] we'll set the frame rate to 30 there's no underscore and now it's correct still it's a little hard to tell I don't know which camera are looking again so even though it's correct now we may want to actually instrument to this program tell us where the time is being spent when we are executing it it's not good form to have to try to guess through everything we'd also like to know how many frames are being displayed a second so the next part of this video will kind of go over some general strategies for doing that let's close this up we'll go over here to instrumented I've written a class to help us with this it's called timer uses some Python voodoo I just wanted you to know it's here rather than kind of go step by step through what it does basically what this class does is it provides a wrapper around the block of code that we are going to execute and runs a timer on it I won't go into the details here you can read through the code it's simple but it's kind of warped so let's go look at simple camera pie you know instrumented mode this is the same program as before except it has been instrumented so we take our time context class import timer then you can see that we instrumented this block of code here's how we might read this code we start a timer we will call it context underscore time we'll read a frame from the capture device and we will print out the time that has a lab since we started the timer next we show the frame that we captured on the window called CSI camera we print out the elapsed time from when the timer started and then we are going to wait in this case 20 milliseconds for a key will print out the elapsed time since the timer started again and then we'll print to this as a marker so let's run this see what happens camera is running you can see that all the instrumentation down here is coming out you can see that it's pretty messed up as far as the display let's kill this so we can see down here we enter the context this is all in seconds here three milliseconds six milliseconds and then it waits the 20 and that gives us a cumulative of almost twenty seven and a half so each frame takes about twenty eight milliseconds here you can see that that number actually varies across frames here's 23 milliseconds 28 milliseconds and so forth you can see the problem the camera is producing 60 frames a second you're only displaying 40 of them in most implementations the cameras are buffered so you're just falling behind as time goes by now of course we can cut down this wait time let's cut it down to 5 try that out you still have some type of lag here let's close that so this is pretty close you're getting down to looks like 15 milliseconds or so well you give these stutters occasionally if here's a 24 28 so you never quite keep up with it so let's change the frame rate to 30 let's run it now and now it works correctly so that runs a little counterintuitive you would think that if you crank up the frame rate on the camera that would actually display faster it doesn't it's limited by the amount of time that it takes to actually display it and read from the camera this is with one camera let's switch over to dual cameras this is kind of the naive implementation we take simple camera and basically just double the video capture pipeline we set it up as left cap and right cap we use our frame rate of 30 this one's instrumented also we set up our timer as before we captured the left image and then we capture the right image we combine the images into one a left image and a right image this is Python speak for combining the two images and then we show that then we are going to wait our 20 milliseconds as before and then we're going to break out of this if we hit the Escape key so let's run this one this has some interesting behavior the left and the right do not appear anywhere near synchronized let's close this up let's uncomment our elapsed times and run it again save and run let's take a look at it some of our numbers here we have 33 milliseconds 34 milliseconds 40 46 Oh 64 you can see that our numbers kind of vary quite a bit here in this particular case it doesn't take very long to read the cameras but to actually combine the images and to show them here's a 28 528 to 5 you see it's rather inconsistent 10 9 so it's a little choppy it can go anywhere from it looks like 25 milliseconds to 7 or so yet to subtract the time to actually read the frames I'm not really nodding back and forth I'm looking back and forth on the screen appears that I'm nodding no no no one thing that might be useful is to actually figure out how many frames per second are being displayed so one insight that I've shared with you already is to pick your frame size judiciously that seems kind of obvious the second one isn't quite as obvious we've been talking about these as sequential operations here's how we think about this program we read a frame we manipulate the frame we display the frame and then we wait a little bit this is all done sequentially as you know the Jetsons have more than one CPU on them one of the key ideas here is to take part of this loop and execute it on a different CPU for example we may want to take the frame reads and put them in a separate process that we can run on a different CPU now we can split this up to some degree we can read the frames on a separate process one of the caveats of all this is that you have to do GUI manipulation on the main thread the main thread is the thread that you call up when you start your program so in this case this is the main thread that we're looking what we are going to do next is set up our frame reads in a separate thread oh boy let's close this one up now let's open up dual camera underscore fps one of the features we'll add here is the number of frames per second that are being displayed let's look at setting up our cameras it's in a separate class it's here in the CSI underscore camera we will use the Python threading module we use a repeat timer that will help us with setting up our frames per second it will set up a class it's called CSI underscore camera when we instantiate a instance we call this all of these our instance variables typically we just initialize them to null and then we have some instance methods here the instance variable video capture holds the opencv video capture you can see that you pass it eg stream or pipeline as we've discussed before [Music] and it does the first read to start the capture now here's where it starts to differ a little bit from the previous versions now well we start the CSI camera we call this particular instance method and the key component here is that we create this new thread the target of the thread is update camera that's another instance method update cameras down here we'll go through that in a second and then we start the thread basically that starts up the thread on a different CPU if available it uses the underlying OS kernel P threads I believe next instance method to stop which will stop the thread and running in the thread is this instance method it does a couple of things it basically reads from the video capture device and then a tricky bit it's kind of beyond the scope of this video but you basically put a read lock on here so that you can synchronize between processes we saved the frame that we just read in the instance variable frame and we increment the number of frames read by one now when someone goes to read our frame we basically make a copy of the frame and hand it back to them that's what this does and then here's the kill method release we also keep track of the number of frames have been displayed that allows a another process to tell us that it's displayed something and then we have down here the familiar create GStreamer pipeline so we've encapsulated the camera as a class so let's go over to dual camera underscore fps and try this out so we basically read in our CSI camera class here's a function to draw a label on a CV image here's the function to read the camera we read our camera image into this variable and then on top of that we draw the number of frames have been displayed and the number of frames read here's some constants for what we want to play around with the size in this next function we create the cameras so left camera since right D is 0 sensor mode is 720 which is 1280 by 720 frame rate of 30 display height and display width that was 960 by 540 same thing with the right camera we also start the cameras here after opening them we create the CSI camera window a small sanity check to make sure that the cameras opened then here's our main block again we read the left image the right image we combine them into one image and then we show them and we tell the FPS counter the camera to increment by one after we do the display so let's try that out ooh that looks good you can see now that they are synced you can see that they're faster than the webcam is we are getting 35 frames per second displayed and we are reading 30 so let's crank that up and see what happens let's close this notice in our call to weight we specify 20 milliseconds so we would expect to see 50 frames per second in best case why aren't we getting somewhere near that sensor mode is 720 which is 1280 by 720 and then our display height and width is 960 by 540 there's a transformation that has to happen between the frame size and the display size we noticed that isn't an even divisor into 1280 by 720 let's comment this out and switch over to 640 by 360 basically quarter size of 720p save let's run it again you know we picked up a few frames for a second grew up into the 40s let's try cranking up the number of frames that we read let's go to 60 we'll do that on both cameras now we're reading 60 frames a second it looks pretty good on the screen but we are only showing 39 frames per second while we are here let's update our sleep time we'll make that 5 instead of 20 most we would expect is 200 frames a second at this point let's save that let's run it and see what happens okay so we can see that we're reading 60 frames a second and our frames displayed is 78 that's pretty good so again you can see by picking your frame rates and your display size you can adjust the frames per second being shown in general it's computationally thus expensive to use evenly divisible multiples of the frame sizes here for display okay that leads us back over to our face detect example let's close this up let's start clearing some stuff off let's open up our face detect example let's go through the code a little bit here here's the face detect method we are going to use frontal face our cascade for it and an I heart cascade opened up a window start counting our FPS --is we're going to use our timer so we can measure how long each one of these things takes and print it all out let's run it and see what happens so the camera shows up you can see it's pretty slow let's close that up so we just take example out of here doesn't take very long to actually read the camera takes a relatively long time to detect the face so you would expect four or five frames per second 17 15 16 maybe 6 frames a second let's look and see what camera mode we were using so 30 to 80 by 24 64 and her display width was 820 by 616 for any weight of 21 so we were going to full frame on that let's go over to the faster example we are going to use our CSI camera trick we will display the frames per second on the image 640 by 360 our display size we're using sensor mode 720 frame rate of 30 we crank down the sleep to 5 from 20 let's see what we get out of that baby so that about doubled or tripled our actual frame rates 9 10 somewhere in there I wonder what happens when you change the frame rates let's crank it to 60 as you would expect it's frame limited so it runs the exact same speed hopefully it should one of the interesting things here is that the model is trained for full-on frontal recognition of the face when looking directly at the camera you get pretty good recognition as soon as I go off access it doesn't recognize you of course this works on images also well it's pretty good the man here is a little bit off access oh it has difficulties with it you have to remember another thing about this is that it's very dependent on the lighting yet poor lighting it doesn't work quite as well just some examples let's close this up and then the third trick of vision processing is to use a small of image as you can to get higher frame rates let's cut this in half again now you can see that we're getting 30 frames a second and this is a major trick of computer vision and most machine learning methods you try to use a smaller frame as possible to reduce the computing time hey if you liked the video give it a thumbs up and if you have not already please subscribe obligatory sign-off stay safe I don't know what that means but everyone says it now thanks for watching [Music]

Channel: JetsonHacks

Views: 28,902

Keywords: JetsonHacks, NVIDIA Jetson Nano Developer Kit, Jetson Nano Developer Kit, Jetson Nano, Raspberry Pi Camera, RPi Camera, jetson nano tutorial, jetson nano dev kit, jetson nano camera, jetson nano opencv, facial detection, OpenCV facial detection, Python multiprocessing

Id: GQ3drRllX3I

Length: 33min 7sec (1987 seconds)

Published: Wed Apr 08 2020