Jetson AI Fundamentals - S3E1 - Hello AI World Setup

[Music] hey everyone dusty from nvidia here and welcome to the hello ai road portion of the jetson ai fundamentals course hello ai world is an open source tutorial for jetson that includes both inferencing and training of deep neural networks for image classification object detection and semantic segmentation let's start by taking a quick look at the project on github which you can find at the url shown below so hello ai world is both a tutorial and an inferencing library that uses nvidia's tensor rt underneath for real-time performance and that'll come in real handy when we go to run this on real-time camera streams and video feeds now the api is available for both c-plus plus and python so you can use that to make your own deep learning applications in either of those languages so the first thing we'll need to do to get started is to make sure that your jetson's been flashed with jetpack now i'm running this on the jetson nano 2 gigabyte developer kit so i would just flash the sd card image that you can download from the website linked to here if you're running on one of the other jetsons then you could use the sdk manager the jetson nano and the jetson xavier nx you could just simply flash the sd card image or jetson tx1 tx2 and jetson agx savior you would flash with this sdk manager tool so after your jetson's been all set up and you're at the ubuntu screen essentially what we're going to do is either build the hello ai world project from source or run the docker container so building it from source is not complicated it's a very simple set of instructions that you can find here that will build this natively on your jetsen and just install right into the root file system of your device on the other hand the docker container can just automatically be downloaded from docker hub and it includes everything pre-installed inside of there like pi torch and the deep neural network models and everything like that and since you can download the container in just a minute or two we're just going to use that option in these videos so in order to do that what we'll first do is actually clone the repo even though we're not going to build it from source there's a bunch of docker scripts in here that we use to run the container and mount all of our camera devices and directories and things like that so it's just easier to actually clone the whole project even though we're not going to actually build all the code there because it's already inside the container so we'll just get clone github dnv jetson inference okay this will take a minute the other benefit of cloning this project is the directory structures are already set up and everything for when we go to do training and we'll talk in a minute about how all these directories get mounted into the container so our data is automatically saved like custom data sets and models and things that we might train okay so after the repos clone cd into jetson inference and you'll see the project here inside of this docker directory are these scripts like the run script is the main one that we are going to be using so the first time you use the run script docker slash run.sh it's going to first download the container image from docker hub and then once that's downloaded it will prompt you to install some pre-trained models if you want to and those you can find here and the defaults include google net resnet those for image classification there's a bunch for object detection and semantic segmentation all told there's i think 40 models currently available that are available to download so we'll just choose the defaults here if you want to download models later additional pre-trained models that you didn't do before you can just easily go back and run that tool the model downloader tool again and that's linked to here on the github page and then you can download more models later so don't feel like you have to download them all right at the beginning if you don't want to okay so now it's downloaded all those pre-trained models we can see those on the host because they're actually downloaded to this data networks directory on the host which then gets mounted into the container which is nice because then you can download more models edit these models retrain them without having to rebuild the container itself and then after it does that it will actually launch the container and then give you the prompt to run commands inside of it and and when it does all this downloading and initial configuration that's only done the very first time you run the docker run script after that it already has the models and it'll just fire up instantly like if we quit the container here and then run it again now it's just going to start up straight away it only took a second or two there okay so now that we got our container running and if we see where we're at here it's got all the same project structure as it would if it's on the host and the directories that get mounted in are shown here under this mounted data volume section of the documentation notably it's this data directory which has the models and a bunch of images in it and then also these classification and detection training directories which we'll later in future videos we'll use for saving the data sets and the models too but just for sanity purposes let's real quick go into this data directory and we will make a file inside the container just a real simple abc123 there and if we go into the data directory on the host now we actually see that as shown up uh because that file is mounted into the container this whole data directory is and this will be very useful when we go to process these images and then save the images out because it's a lot easier to view the images from this file explorer than it is from inside the container technically okay so now that we've got the lay of the land here let's actually run some stuff we're going to make sure we got our camera set up correctly and run this video viewer test program here so the hello world project supports a variety of different streaming protocols and cameras and things like that the most popular are probably these v4l2 usb webcams like in this video series i'm using a logitech c920 the logitech c270 is also very popular mippy csi cameras those are like the raspberry pi camera module that's compliant with the jetson nano and the jetson xavier nx and other ones there's a whole list of cameras here on the nvidia developer page and also on the e-linux wiki that you can use but for these i'll just be using these v4 l2 cameras if you want to use a network stream it also supports rtp or rtsp and you can also just read in straight video files if you want like mp4 mkv avi and flv and those codecs that are supported are like h.264 h.265 vp8 vp9 and mpeg and mjpeg so that list is here and these use the jetsons hardware video encoder and decoder you can either read in files and write out files or both so it makes that very convenient and low overhead to use so you're not using the cpu when you're doing decoding okay so the video devices should automatically be mounted into the container i'm just going to make sure that i have mine here so mine is a video for linux 2 usb camera as i mentioned and that should show up as a dev video device because that's how v4 l2 devices are mounted into the system so there's mine dev video zero so if i just run this video viewer utility that's just a sample that comes with the hello ai world project and do dev video zero should see a camera feed pop up here there it goes so i've got a camera in front of my desk here say hello okay and it looks like everything is streaming correctly so this sample is basically serve as the skeleton for all of the other samples we'll later add in image recognition and object detection and segmentation on top of this video viewer utility which you can just find the source for in the repo but uh in later videos we'll actually go through coding our own applications uh but now that we know that the video streams working you know that's the first thing you want to test make sure you got your camera going before actually moving on to doing more complex deep learning stuff we can also reviews a bunch of video options that you can run and these command line arguments apply to the video viewer tool but they also apply to any of the samples that we're going to run for like image recognition or object detection and things like that so you can change the input resolution of the camera you can change the codec for example this c920 it's h.264 encoded because it's only a usb2 connection so that makes it much more efficient than just transmitting the raw data so via that i can get 1080p 30 because it's encoded and it uses the jetson's hardware decoder to decode and and display that so there's a bunch of other options in here if you want to do output streaming save video files or stream the video feed out to a remote host over rtp which will show that later if you're running headlessly without a display but the the point being all of these options are available for all the programs that we're going to use um here in this series and you can view these just by running uh the help so if we do video viewer dash dash help you know it prints out all the same stuff here so there's lots of options that you can toy around with depending on the video stream that you want and along those lines let's just do a real quick test of this detection sample here that will run some object detection on the live camera stream just to show that the command lines are indeed the same so before we did video viewer dev video zero now we're just going to do to technet dev video zero and we'll cover object detection in depth in a later video but just want to give you a feel for what it's like running the actual program here okay so we've got our camera feed up again and you can see now the only difference is now it's giving me these bounding boxes that come out of the object detection network and move around here say hello and we'll we'll cover later how you actually do this detection and the code for doing that but suffice it to say the architecture of all these programs is is very similar we're just going to swap out which neural network that we're using to process this video feed with and notably you can see this is running real time on the jetson nano 2 gigabyte at 23 frames per second even though this is a a complicated detection model called uh ssd okay so we close this down here so i briefly mentioned that we can actually do different types of network streaming as well so for those of you that don't have an hdmi display to connect your jetson to or you want to run headlessly then you can use this rtp streaming you can do both rtp input and output rtp is just a remote transport protocol commonly used in video streaming and essentially what we can do is instead of the jetson rendering to the display we will just have it output that video feed over rtp to like your pc or laptop or something where you can then view that feed without ever having a display attached to the jetson and the way we'll do that is just run this video viewer program again or detect net or imagenet program whatever deep learning example you're currently on you'll pass in the video camera this is under the transmitting rtp section here and then pass in the ip address or host name of your pc that you want to stream it to finally on your pc you'll want to open up a viewer of this stream you can either use g streamer or vlc player for that and so what i'm going to do is reset up my display here to connect to my pc and then we'll show you how that works as well so back in a minute okay we're back this time i've switched over to my pc it's actually a laptop but it's connected over wi-fi same wi-fi network that the jetson is on you can see here this machine is an x86 machine so this is not the jetson and what we'll do is ssh into the jetson using ssh my username is nvidia had nano nano is just the name the host name of my jetson on my network and nvidia is my username so here we are we are now logged into the jetson remotely over ssh if we run unnamed again now we see it's arc64 so uh now we're confirmed we are on the jetson all we're going to do is run this same video viewer but use an rtp output for it so to say the camera is still connected to the jetson the jetson's still running the inferencing and everything it's just going to be transmitting the video feed back to the pc for remote viewing and in order to do that we just pass in the first parameter like we were the video camera and then do this rtp protocol string for the output normally this was just implicitly the display but if you don't have a display attached you can output one that can be like a video file or an rtp stream so we'll just do a similar command to that here okay so let's test this video viewer dev video zero that's still the name of my camera it's on the jetson and then rtp sky x42 is the host name of my pc otherwise you would just put in like your ip address there of your pc and we can see this it started to capture on the jetson and we'll open up a new tab here and what we need to do now is actually open the viewer on the pc side so the recommended way i do that is just to run gstreamer because g streamer is actually running on the jetson side and it makes for very low latency experience we can just copy this g streamer pipeline here and this terminal is going to run commands from my pc so we see that this the display window already popped up and this video is being piped through the jetson from the camera it's being compressed with h.264 on the jetson side sent over to my pc over wi-fi decompressed and then displayed to the pc's display and in a similar vein let's actually run that detectnet program again so we'll just run the same command here but just change this to detect net as mentioned they all run the same input and output arguments for simplicity's sake so wait until this loads on the jets inside and we see it start doing detections okay so we see here it's starting to run the network and we'll just fire up the same pipeline again okay and here we go we're viewing the detections from the pc side the latency is pretty good actually it's not much of a noticeable difference at all especially if you're on like a five gigahertz network at home okay and again this was using g streamer you can install that on windows mac or linux it's most popular on linux to use but you can install on the other ones as well and i recommend that because the the latency is very low and it very it matches well with the side that's running on the jetson but if you want to you can also use vlc player which is like the most common media player on earth practically by this point and the way you would do that is just to create one of these sdp files here and copy in the contents of this so you can see mine i have mine right here on the desktop and the contents it just contains the same ones that are here on the github page it basically just tells you the ip address to look for the stream which is localhost because the way rtp works is it's broadcast from a device to a specific host and then if you using a different port than one two three four change that here and also there's like the codec setting and everything so if you change it to like h.265 codec or anything you'll need to make minor modifications to this sdp file and then the way you open it is just double-click the sdp file and it'll fire up in vlc player now you will notice the latency is a bit higher in vlc player than it is in through the g streamer which is why i recommend gstreamer but you can tweak vlc player a bit it's by default the buffering options are quite high on it it's not really meant for like real-time stream monitoring the way that g-streamer might be but if you go into the advanced preferences here and then scroll down to these uh caching ones the default of this is a thousand milliseconds so the default latency is even higher than this but you can try tweaking the network cache and the file caching milliseconds uh try to get those as low as possible but if you're still not happy with the latency after that then i recommend you try to install g streamer uh on your pc and go for that and all this is uh assuming that you don't wanna have a display attached to your jetson all the videos that i'm going to film from here on out will show the jets and attach to a physical display just for ease of use but by no means is that a requirement so that brings us to the end of this intro setup video in the next video we're actually going to be jumping straight into doing image recognition so we'll be running this step here and really getting started with deep learning and doing a bunch of classification things and real-time video streams and writing our own classification program in python so with that thanks for joining us in this intro video and we hope to see you next time i'm dusty from nvidia to learn more visit nvidia.com dli or email us at nvdli at nvidia.com

Channel: NVIDIA Developer

Views: 39,428

Keywords:

Id: QXIwdsyK7Rw

Length: 20min 32sec (1232 seconds)

Published: Mon Nov 02 2020