DIY Intelligence with NVIDIA Jetson Nano

[Music] hey everyone thanks for joining us today here at Nvidia were excited about the recently announced AI at the edge challenge with hackster and to dive in with you into using Jetson Nano to create your own AI power projects heading through the presentation we'll take your questions during QA with that let's get started awhile for those of you who aren't already familiar with Jetson it's an biddies family of embedded high-performance AI computers often used to deploy deep learning and computer vision for autonomous robotics and AI at the edge and which that's a nano we made it easier than ever to get started developing your own AI applications so we thought it would be a great opportunity to team up with hackster to put on the AI at the edge challenge there are three categories in the competition autonomous machines and robotics intelligent video Hana Linux and smart cities and artificial intelligence of things each category will receive three top awards including an all-expenses-paid trip to invidious GPU technology conference in Silicon Valley in the spring along with a chance to demo your project to world leading AI experts at the event and also a bunch of GPU hardware and cloud compute credits there's also an AI social impact award given to one of the projects among three categories that receives the top prizes and will be awarding participants along the way with free hardware and online courses Nvidia's deep learning institute project submissions are due by the end of january and we have a limited number of nanos to give away by November 15th so with that let's dive in awhile we launched Jetson Nano earlier this year and it's our smallest lowest power and least expensive computer that vidi has ever made this is the Nano Developer Kit and it's a complete embedded AI computer that has an NVIDIA GPU onboard capable of 472 gigaflops of compute with full support for cuda 10 a quad-core arm 64-bit CPU 4 gigabytes of RAM and it's only $99 it's fully compatible with clinics and the NVIDIA AI platform it's supported by invidious complete software stack and it has the performance and power efficiency to run modern AI workloads all in 5 to 10 watts of power and in a compact footprint the DEF gets built around the Jetson nano compute module which is a 45 by 70 millimeter embedded system on module that can be integrated into production products and devices the compute module contains all the processing memory and storage on board and it provides all of the i/o through it's 260 pins so dem style edge connector which a carrier board breaks out into the usable ports and connections the dev kit includes the preassembled nano module heatsink and an open source reference carrier board which embedded system designers can use to design their own customized carriers that are out the IO to the peripherals and connections that they desire here's a closer look at the specs and features of the dev kit it is 4 USB 3 ports HDMI and DisplayPort Gigabit Ethernet MDOT 2 with PCI Express nippy CSI camera connector 40 pen GPIO header including i-square sees spies and UART and a microSD card slot the camera flex cable connector is compatible with popular mid B CSI cameras like the I MX 219 based Raspberry Pi camera module v2 and others that are imaging partners I've created as well the 40 pen GPIO header on the dev kit is similar to the 40 pin header on the Raspberry Pi which means you can use a lot of the same devices and accessories with it there are three primaries ways to power the dev kit which we'll discuss in detail later I'll which use 5 volt DC you can use a micro USB charger the DC barrel jack adapter which is capable of delivering more power if you have a lot of peripherals plugged into the board or power over ethernet the Nano dev kit boots Linux from the removable microSD card slot the distribution of software for Jetson we call jet pack and the current distro is based on the boon to 18.04 the processor has an integrated 128 core Nvidia Maxwell GPU 64-bit quad-core arm cortex a 57 CPU 4 gigabytes of lpddr4 memory at 4k hardware video encoder and decoder capable of simultaneously decoding 8 HD video streams and coding for video streams at once Jetson Nano is made for processing lots of High Definition sensors and multimedia with AI and analytics you can get these developer kits from a variety of suppliers worldwide including aero amazon new a seed studio spark fund and silicon highway and through the challenge you can get a $5 rebate through seed studio as mentioned jetpack is our distribution of software that's available to run across the Jetsons and it's based on a boon to Linux along with the media's driver l40 BSP and tools like CUDA it's like a supercharged version of Ubuntu that comes preloaded with GPU accelerated deep learning and computer vision libraries along with multimedia tools for camera capture and the video codecs you can easily install machine learning frameworks that tend to flow PI torch and cafe on top of this and robotics frameworks like Ross and pretty much any of the programming languages that has a compiler or interpreter available for Linux there are tens of thousands of precompiled packages available from the arm 64 boomtube repositories to install with the click of a button or a few keystrokes altogether it forms a complete stack for creating and deploying a I powered applications in addition invidious release additional SDKs called deep stream and Isaac Isaac is our robotics SDK and simulator and deep stream is for multi stream video analytics there's also an interactive profiling and debugging tools called insight to help you tune the platform something wonderful about jetson is that you can easily install and run the full versions a popular machine learning and deep learning frameworks on it like tensorflow pi torch cafe cafe 2mx net Carris and others since jetpack provides full CUDA support including KU DNN these are all GPU accelerated and able to get great inferencing performance running onboard the device this really streamlines migrating from your training environment to the edge device for inference you can take your existing Python code using tensorflow or PI torch for example install those frameworks on any Jetson the full versions of them even and run that same code there these all have the ability to work with tensor RT for even better inferencing performance to deep learning frameworks aren't the only open source frameworks that you can run on Jetson you can also install Ross the robot operating system amazon Merlot maker and AWS Greengrass for IOT the l40 kernel supports docker so you can use docker containers another way people like to easily migrate their applications between the cloud and edge without extra maintenance and for folks running multiple jetsons and energy-efficient clusters you can use openmpi for multi processing just like on servers and this is just a small partial list check out the jetsons uu4 pre-built installers which there's a link when every Jetson desktop too so we mentioned NVIDIA tensor RT it's invidious DNN inferencing accelerator library optimized for GPU NVIDIA has Kuti and n which implements various Network layers in CUDA and tensor RT operates at the network level with knowledge of the network graph it's able to perform additional optimizations like kernel fusion which evaluates multiple layers at a time within the same CUDA kernel to reduce memory bandwidth usage and latency it'll tune itself for the platform and can fully utilize the FP 16 precision available on the nanos GPU typically you'll see multiple times speed up with tensor RT folks will often start running with the native frameworks on jetson still using GPU and then optimize using tensor RT to get the best real-time performance it can load models from lots of frameworks like tensor flow PI torch cafe and many others via the onyx open neural network exchange format next we're going to take a look at some inferencing benchmarks for our own jetson nano with tensor RT jetson nano can run a really wide variety of network models for doing everything from image recognition to object detection segmentation pose recognition and others here's a small slide show models trained using a bunch of different frameworks that are running on board the Nano using tensor RT Interactive real time frame weights are achieved even along the high-end networks up to HD resolutions in particular SSD mobile map v2 is a widely used object detector used in a lot of applications Yolo is another popular object detection model that we benchmark and that's trained with darknet units a semantic segmentation model that classifieds each pixel in the image semantic segmentation is like image recognition but instead of just classifying the overall image like ResNet inception and vgg do unit does per pixel classification which is really useful for perception and autonomous navigation another cool network is super ResNet which has high quality up sampling of images called super resolution that's useful for enhancing video of noisy low res sensors Oakland pose is what we ran for pose recognition which extracts the positions and orientations in 3d of human skeletal feature of people in a video stream like the locations of their arms hands fingers face feet legs and bodies with this info you can do behavior recognition and gesture recognition for example signaling with the robot or implementing collaborative behaviors in a factory or warehouse environment or your home ok next let's take a look at some areas in the competition where you could apply AI with the Nano Jetson powers deep learning and computer vision on many of today's most advanced autonomous robots like those for last mile delivery warehouse logistics and inventory management precision agriculture manufacturing and industrial inspection retail and home assistance robots on these platforms there are lots of high definition high frame rate sensors needing processed with low latency in order for them to function safely in their environment it takes a lot of compute horsepower and flexible software in a compact size and power efficient package to achieve autonomy on mobile platforms like these to help everyone get started we put together the jet bot an autonomous AI powered robotics kit with jets and nano it's fully open source and includes all the software and hardware Doc's for sourcing the components and putting it together the chassis is 3d printable and uses inexpensive parts so you can build a whole kit for under $250 including the price of banana jet pods programs through user friendly Jupiter ipython notebooks these contain the deep learning functions and logic to control the robot out of the box that you do obstacle detection object following and path following using pi torch and tensor RT from a low-cost monocular camera feed you can even collect new data from your environment right in the ipython notebooks and retrain the network's onboard the Nana so it's very easy to use and create new autonomous modes for we've also released Ross nodes supporting the jet bot Hardware along with the gazebo simulator model for it so if you're a Ross user you can bring Ross components to jet bot - let's watch a short video from the creator of the jet pod about what it can do so over here we have jet Hawk which is an open-source robot that we built using Jetson Nano and the idea behind Jeff OTT was to give people that wanted to start with AI or maybe AI with Jetson Nano a platform that they could just get up and running right away you buy the components of the robot we have everything on github so it's super easy to get started it's all done in Jupiter notebooks so we have like basic examples from just controlling from your web browser and Python it's super easy to do some more complicated examples like using AI for avoiding collisions and then we go all the way to road following to have like kind of an autonomous car navigating a Lego City so that's the basic idea is that you can start with this platform it's all open-source the hardware the software and then you can take what you learned you can take the components even and you could build something new and who knows what that would be it would be kind of up to you [Music] another open-source robotics kit that we put together his jet racer which takes many of the concepts from jet BOTS to high speed it's also built around the Nano and uses dnns trained onboard with PI torch to autonomously drive around courses and racetracks as quickly as possible there are versions that you can build with either one-tenth scale or 1/18 scale chassis z' for around $400 the technology has also been adopted into many other autonomous race car circuits out there like donkey car and DIY Robo cars musher racecar J the MIT race car project and the F 1/10 project from University of Pennsylvania to easily develop and deploy software for intelligent robots and autonomous machines the MIDI has created the Isaac SDK that we mentioned earlier it includes the Isaac runtime robotics engine with algorithms and DNS that we call gems for things like perception and autonomous navigation slam localization and mapping visual odometry path planning and stereo depth along with the high fidelity Isaac simulator and suite of simulation environments called Isaac Jim all fully optimized for GPU with CUDA and hardware graphics acceleration you can use it to train and deploy production autonomous machines and robots that require the best performance and you can download the Isaac SDK and simulator from our site and deploy it to your Jetson also included with Isaac are multiple reference robot designs like the carter delivery robot and kaya to get you started these reference designs include both the software component lists and plans for putting together the hardware here's an example pipeline of isaac gems used by the carter robot for navigation using a stereo depth camera and lidar for simultaneous localization and mapping or slam these are highly optimized with GPU acceleration to run a real time on Jetson and our camera ready so for the smart cities category of the challenge we have deep stream which invidious SDK a four edge to cloud intelligent video analytics here we're showing a video captured from nano running eight HD video stream simultaneously with object detection and time series and houses using deep stream deep stream takes advantage of the jetsons multi stream video encoder and decoder hardware along with the GPU and ten Sarty to get the best performance out of the system it's also optimized for zero copy so there aren't any extra memory copies of transferred being made between processing steps all of this is important to achieve the max throughput of the device which as it turns out is pretty impressive for a low-cost low-power board this example is processing over 500 mega pixels per second across eight 1080p streams each at 30 frames per second a resident base detector is used for accurate tracking on each stream running at the same time here we're detecting vehicles people and baggage and counting the number of occurrences of each type of object over time in each camera feed if you plan on entering the intelligent video analytics in smart cities category of the challenge you can build your application with deep stream and easily reconfigure the pipeline with different types of data streams inferencing an analytics processing that are tailored to your project okay now we're going to switch gears to walk through the resources at your disposal to help you get started with the Nano along with various peripherals and accessories available for it and tips and tricks to using the board there's already a lot of info out there on Judson Nano available for use like Barry's tutorials for deep learning that we have and open source projects from the community you can download the docs and software from our developer site and we publish guides to the e Linux wiki too there are lots of peripherals and accessories compatible with the Nano dev kit and people have started coming out with some of the first customized carriers and enclosures for it also and if you ever need support we have a very active developer forum where you can ask questions and get help hello ai world is an easy to follow open-source tutorial that we created to help you get started to point deep learning on jetson it covers inferencing with image recognition and object detection and comes with pre trained net works for you to use out-of-the-box and does live camera streaming you can follow it in around two hours or less on you Nano and during the tutorial you'll code your own image classification program in Python or C++ from scratch it all uses tensor RT underneath for great performance and the core code is built into a library that you can easily use to make your own deep learning projects let's quickly step through the different parts of the tutorial to follow the first thing to do is clone the repo from github and build it on board your jetson during the build process it will automatically download various Network models and test images that you can use go to the URL shown here to get started reading the documentation on each step as well once you've compiled the code we're going to classify some of the images that come with the repo where you can download your own images to play around with the repo includes built-in Google meant Alex net resonant and vgg classification models that were trained against the ILS VRC image net data set which are able to recognize a thousand different types of objects including many different species of animals fruits vegetables and different foods natural scenes along with manmade and household objects vehicles for own sides and lots of other things there's a link to the full list of objects that can classify on the tutorial so you can check that list against what it's able to do you can also retrain the networks on new data on board Nano using PI torch to be able to classify new objects how the classification works is there's one output neuron in the network for each of the 1000 classes and each output neuron corresponds to the confidence value of that class then you pick the arc Max or the neuron with the highest confidence and the index of that neuron corresponds to the object class that the network classified the image as okay next we step through the code to make your own image recognition program in Python underneath it's accelerated with tensor RT on the GPU for real-time performance first you import the jetson inference and judson util modules provided by the hello AI world vibe then parse a couple of command-line options that provide the image to load and the classification network to use then load the image and network and perform the classification which returns the class index that the image was recognized as along with the confidence value between zero and one one meaning one hundred percent confidence in the classification result you can map the class index to the text description or name of the class the built-in models that were trained on the ILS VRC imagenet data set of a thousand objects will return a class index between 0 and 999 which can then be looked up from a label file that stored on disk by the get class description function in this example we print out the classification results to the console however there are other examples included with the repo that overlay the results onto the input image and save it back out to disk like we saw in the previous slide there's also support for native C++ in hello ai world and an object API is provided for each of the vision primitives that the Python bindings use underneath for optimum performance the C++ code shown here shows the parts relevant to loading the network classifying the image with tensor RT and outputting the result this snippet is slightly abbreviated so from the slide without the part that loads the image from disk but even the full version is under 50 lines of C++ code and it's quite simple to follow every example from the tutorials provided in both C++ and Python so you can use either language throughout next we're going to run the inferencing in real time on the live camera stream you can connect a camera to your dev kit and pointed at different objects or in the house or little props to see how it classifies them it's pretty fun to play around with and like all the steps you can dig into the code from this program and see how to stream the camera as opposed to loading the images from disk the cameras it works with our MIT pcs I like the Raspberry Pi camera module v2 or USB webcams like Logitech c920 if you're using a USB camera see the directions in the repo for changing it over to use USB webcams instead of maybe CSI the program can run up to the full rate your camera supports in real time so for example if you have a thirty Hertz camera it will process 30 frames per second the second phase of hello world focuses on object detection which builds on the concepts of image recognition object detection gives you the bounding box coordinates and is able to classify multiple objects per frame as opposed to the image recognition networks used in the previous steps classified in the entire image object detection is really useful in robotics and analytics where you need to know precisely where the things you're looking for located in the frame like the previous examples we start with processing some test images from disk and detecting various objects in them you can look at the code for this program to see how the detections perform it's mostly the same as the recognition except that the detection network outputs a list of bounding boxes with a class and confidence value attached to each bounding box there are a handful of detection models included with the repo including SSD mobile net and Inception which were trained on the 90 class ms cocoa dataset and run in real time on the Nano like the steps for image recognition there's also a live camera app for object detection which overlays the bounding boxes of the detected objects over the camera Street real time along with their class and confidence values the SSD mobile net and inception models run on more than 20 frames per second on Nano in both Python and C++ you can see here a little demo video captured with SSD mobile mat v2 that shows it's detecting various objects around the room as mentioned the built in SSD models that were trained on the 90 class cocoa dataset so they can detect a bunch of different things out of the box like humans animals vehicles furniture and other various household objects that you may have lying around or on your desk there's a list of all the cocoa classes included with the tutorial that you can look over to since the network typically outputs high-quality results there's only a minimal amount of post-processing applied just some really basic clustering of the bounding boxes and is referred to as tracking by detection here's the Nano Det get that all of this is running on I have a mitt B si si R a spycam attached and also a Logitech USB webcam with a longer cord that I shot this video with the pre trained models can also detect common electronics like TVs monitors laptops keyboard and mice and cell phones okay the next vision capability covered in hello ai world this semantic segmentation which performs classification at the per pixel level using a fully convolutional network or f CN these models take a pre trained image recognition backbone like resin that 18 in this case and convolutional eyes them if a model capable of per pixel labeling segmentation networks are especially useful for environmental perception since you can discern seeing foregrounds and backgrounds and get a dense representation of the world it's commonly used in autonomous navigation for free space detection which lets the robot know where it can and can't go like for path following and obstacle avoidance there's a bunch of built-in segmentation models that come with the repo that run in real time on the Nano and we're pre trained on different data sets including cityscapes for urban environments deep scene for off-road trails multi human parsing proposed estimation Pascal vo C for common objects and animals and Sun RGB for indoor scenes most of these models support 21 segmentation classes and the performance on Nano is shown in the table here as with the classification and detection primitives there are static images and live camera demos provided for segmentation in both Python and C++ that you can run in real time on you Nana in the final steps of hello AI world we show how to retrain classification networks on custom datasets so that they can learn to recognize new objects that may not have been included in the pre trained models this process is called transfer learning when we can use PI torch running on board the Nano to do it in a few hours on some decently sized example datasets of 10,000 plus images like on different kinds of plants and cats and dogs there's also a camera capture tool provided for collecting and labeling your own data sets which typically can be trained much faster even on nano because those are probably going to be smaller data sets for large data sets and models we recommend training on a host PC or cloud instance with discrete GPU which are often used for computationally intensive training processes as the jetsons are out demise for low-power inference however you can still get started with training on your Nano particularly the PI torch framework seems to work well because it's pretty memory efficient ok that's it for now for hello AI world next let's cover some tips and tricks and walk through setting up a new nano dev kit out of the box the dev kit boots from a microSD card so all you need to do is download the SD card image from our website which already includes all the jetpack components pre-installed and flashed the image to your microSD card from any PC that has an SD card slot you can easily flash it with a common tool called echar that runs on Windows Mac or Linux and there are a number of other tools out there that you that can do it as well when you're done flashing you take the formatted SD card out of your PC and insert it into the slot on your dev kit the micro SD card slot is located on the underside of the nano module on the side opposite facing the connectors it's recommended that you use at least a 16 gigabyte uhs-1 micro SD card the faster the class of SD card the faster it'll boot up and load and you can use as large a micro SD card as you want there isn't a limit on the storage capacity and your user partition will automatically be expanded to fill the entire SD card you can also attach a solid-state drive with the USB 3 to SATA adapter for extra storage or faster read/write speeds as far as connections go you can use the dev kit either in a headless configuration and connect to it overseer port or Ethernet via SSH from a host PC or by plugging in an HDMI or DisplayPort monitor along with keyboard or mouse and use it like a normal desktop and develop right there on the Nano next let's talk about powering the dev kit for multiple ways that you can power it which makes it quite versatile to use in projects you can use a micro USB adapter capable of delivering two amps ideally 2.5 amps like the ones listed here and in the Jetson Nano supported components list if you have connected a bunch of peripherals to the board like USB devices or unpowered USB hubs and are running a demanding computational load then you may want to step up to using the DC barrel jack with a 5 volt 4 amp power supply if you want to use the DC barrel jack remember to fit a standard 2.5 4 millimeter jumper on the j48 power select header this switches the board between accepting power from the micro USB port to the barrel jack in this mode you can also connect a power source depends two and four on the J 41 expansion header 3 amps per pin or up to 6 amps total pins 2 and 4 should be 5 volt DC and pin 6 should be connected to ground you can also power the Nano for a battery pack like some of those portable USB battery packs with built in regulations seem to work pretty good now there's actually a fourth way to power the board and that's power over ethernet you can use a POV hat or use a voltage regulator to take it from the 48 volts standard that you see in our arethe net down to the 5 volts that the Nana uses also I called out an last slide that the board will automatically turn on and start booting up once power is applied you can change that behavior by fitting a jumper to pin 7 & 8 on the j40 button header which will disable auto power on and then you'll need a short pins 1 and 2 on j40 to power it up in addition to the power on signals j40 also has pins for resetting the board and entering it into recovery mode you can connect momentary push buttons to j40 which are more convenient to press or you can just use jumpers on them when you need to an extension of the discussion of power supplies is the power modes that the board is able to run nano has 5 and 10 watt power modes and by default it'll boot up in 10 watt mode which gives it the best performance NVP model is a software tool used to change power modes and tweak profiles depending on your use case you can run the env mild - queue command to query the current mode and NVP model - am to change modes when you change modes it'll persist after reboot and you can run the Jetson clock script to lock the class to their maximum for the current mode you can define your own custom power modes - they're located in the Etsy and VP model dot-com file and they look like the configuration shown here listing various maximum clock frequencies there's also a GUI widget located in the top right of the Ubuntu screen that will allow you to change the power modes from the UI now to keep an eye on system performance you would typically use the program like top or the abouttwo system monitor however there's a lot more information available on nano so we have a tool called Tegra stats which displays all the stats that you wouldn't get otherwise like the utilization and frequencies of the GPU and memory controller temperatures from the thermal sensors and power measurements from the on-board IMA sensors the IMA sensors measure the power consumption across different power rails including the usage from the CPU and GPU and the overall usage as well I recommend that you try running this in the background while your benchmarking or profiling your application so you can keep an eye on the core utilizations memory bandwidth and consumption and the dynamic clock frequencies the scale at run time depending on workload there's also some open source tools from the community like j top which visualized the output from Tegra stats the Nano dev kit has a 40 pen expansion header J 41 similar to what's found on the Raspberry Pi and is compatible with many of the same peripherals and hats it supported a native fruits blink a Python library and seed studios Grove library which all told support over 400 devices and sensors that you can use by default the nanos 40 pin header is configured to expose to spy ports to I squared C's you are audio GPIO S 3.3 volt and 5 volt power pens with the protocol interfaces like spy I squared C I 2's and you are are referred to as special function units or SF iOS using the pin muxing tool you can reconfigure the nanos device tree to change these SF IO pens over to GPIO or vice-versa to access the GPIO pins directly there are few api's you can use from user space for doing this jetpack includes the Jetson GPIO library from Python which provides the same interface as the our Pi GPIO library so assisting code and drivers using our PI GPO automatically can use the GPIO on nano tool for using GPIO from other programming languages and shell scripts from the command line the GPIO pins are exposed in sis FS under sis class GPIO directory which is the standard location for these in Linux you can read or write to these files using Kat and echo or the file IO API is from any programming language like F open and F read and C or C++ and actually the jetson GPIO Python library uses these sis FS files underneath to the special function units typically have their own software interfaces in the case of I squared C there's a cross language API called live i2c for C C++ and Python and spy has the standard spy dev user space module available as mentioned there are hundreds of compatible devices in hat that you can plug into the 40 pin expansion header however there are lots of other accessories that have already been created for the Nano including a number of 3d printable enclosures for the device aftermarket carrier boards with additional i/o and sensors like CSI imagers and stereo cameras you can find links to these on the e linux wiki and from our ecosystem partners and will anticipate a lot more becoming available this year the camera header on the Nano dev kit is compatible with the Raspberry Pi Canberra v2 and drivers for the IMX 219 sensor that it uses are included in jetpack so it works out of the box leopard imaging and ecomp systems also have a PC aside cameras available that work with the Nano 2 since jetson is frequently used to process live camera streams let's talk about how to capture video from software Nvidia has an optimized library called Argos which is part of the L 40 multimedia API which provides image acquisition for mipi CSI censors mipi cameras have the benefit of being chip-to-chip and being ingested directly into the Jetson hardware without even to go through the CPU or have extra memory copies arias is optimized for zero copy and low latency multi camera applications there are Argus api's for c++ and a wrapper library for python that you can find on our github site we also support GStreamer in jetpack we provide a number of hardware accelerated elements to use in gstreamer pipelines for accessing features like the camera interfaces and video codecs here's an example pipeline that you can launch from this command line that uses our env argus camera source GStreamer element which does camera capture with live Argus underneath the cheese streamer you can patch together a lot of different pipelines for compressing the video and recording it to disk or compressing it and stringing it out to a remote host over the network via RT p or RTSP lots of different things like receiving IP cameras over the network and decompressing those there's also v4 l2 or video for linux - which is a standard video api for linux v4 l two cameras get mapped to a device node under dev video AnnaLynne jetson mipi CSI cameras will show up there - v4 l - also supports USB webcams and various other camera protocols and you can use it by either accessing a dev video device file from any language or by using a wrapper library like those mentioned here for C C++ and Python there's also a gstreamer element for v4 l - camera capture called v4 l - source so it's easy to swap that out for USB webcams in your GStreamer pipelines another important part of jetsons multimedia capabilities are its multi-stream hardware video and decoder engines nano can decode up to 8 1080p 30 h.264 h.265 streams and encode up to 4 1080p 30 h.264 h.265 streams simultaneously it can also to code to 4k p 30 s and encode a 4k stream simultaneously and in addition to h264 h.265 compression it's a force of pp8 and JPEG profiles for encode and vp8 vp9 vc-1 mpeg-2 and JPEG profiles for decode at runtime you can mix and match the profiles number of streams resolutions and frame rates shown in the tables here up to the aggregate bandwidth of the encoder and decoder the encoder and decoder are independent Hardware units so they can each be utilized to their fullest at the same time without impact in the performance of each other choose these codecs from software there are a couple api's for them that nvidia provides through jetpack first there are accelerated gstreamer elements available for each of the profiles that do encoding and decoding for example omx h.264 ank and Omax h.265 eng here are some example GStreamer pipelines that include an h.265 video to disk in MKV format and then a second pipeline that decodes the MKV file and decompresses it from h.265 you can also easily connect the camera elements from the previous slide to the encoder and record it or stream out your camera video there are a lot more example gstreamer elements and pipelines included in our L 40 seller rated gstreamer user guide that you can download from our developer site and to see all the gstreamer elements available in your system you can list them with the GST inspect command there's also a hardware video image conversion and compositor engine available exposed in gstreamer through the env vid comp element for doing color space conversion cropping rescaling rotation and temporal noise reduction in hardware the second set of API is an Nvidia ships for the codecs our v4l 2 extensions from jet packs l40 multimedia api that support h.264 and h.265 compression and decompression these can be used directly in which case the file IO and D packaging to be from formed by the user or there are also exposed as gstreamer elements that can be integrated into a complete pipeline one final topic we're going to cover today is jetsons unified memory subsystem and 0 copy all the processor course and jetsons share the same common memory controller and physical DRAM including the cpu GPU camera ISPs codecs multimedia engines and high bandwidth IO controllers that means you don't need to have extra memory copies between the different cores they can simply reference the same memory used in previous processing step in the pipeline eliminating these extra memory transfers is called zero copy and it greatly reduces latency and the amount of memory bandwidth wasted freeing up the bandwidth for other purposes ultimately enables you to process more streams at the same time which coupled with the optimized infesting with tensor RT is one reason why Nano is able to handle 8 HD streams simultaneously when you're working on the GPU you may be familiar with the CUDA men copy function which is used to transfer data between the CPU and GPU on typical PC systems the GPUs connected over PCI Express and has its own discreet memory so it doesn't share memory with the CPU it needs to shuffle around data sometimes however on jetson you don't need to use kudamon copy because all the memory in the system is shared and you can just map the same pointer into the GPUs address space there are a couple api is included for accomplishing zero copy you can use mapped memory with cuda host Alec and CUDA host kit device pointer or you can use Cudas more recent unified memory feature with the CUDA malloc managed API which is also able to implicitly handle synchronization consult the cuda documentation to determine which approach is more beneficial for you and your application other engines can use zero copy in addition to the GPU also like the camera is P and codex with egl streams which are interoperable with OpenGL and CUDA 2 so you can do zero copy between more than just the CPU and GPU practically the whole system it's highly recommended to take advantage of these techniques with optimizing your application to achieve the peak performance and utilization from the device ok so that brings us to the end of today's presentation thanks again for joining us and we wish you the best of luck in the challenge and look forward to seeing what you create with AI later we'll post the PDF of these slides with clickable links so you can easily find all the resources we've covered here okay next we're going to get set up for QA so if you have any questions enter them into the Q&A window now we'll stick around for as long as we can but if we don't get to your question please feel free to post it at the forums and we'll get back to you soon so let's start so there was some general questions about you know I'm doing so-and-so project where does that project fit into the three categories of the challenge so I thought it would be good if we just did a quick overview maybe some examples of what type of projects would fit in each so the robotics category generally would require some type of mechanical actuation in it it doesn't need to be a mobile robot like the jet bot per se could be like a little robotic arm or some type of robotically triggered device that triggers a servo or an actuator but generally that would fall into the robotics category the intelligent video analytics category is mostly B for smart camera like applications that are using deep stream they don't need to be per se like traffic monitoring like the examples that we showed but if you're doing some type of analytics on video around your house or home you know there's like this smart doorbell application example that we have on our Jetson nano project site that could example fall under the IV a category and then finally there's this more general a IOT category the artificial intelligence of things and that can be you know very broad anything that is taking in some type of data and doing you know some type of machine learning or computer vision or some type of processing on it that you know you're either using locally or connected to the broader Internet it could be a local network doesn't necessarily have to be connected over a wide area network so hopefully that clears it up if you don't really know where it fits you know you are welcome to post to the forum and we'll try to help you and get back to you where we think it could best be submitted to okay so now for some more technical questions per say about the Nano there was a question does the Nano work like any other Nvidia video cards that you may be familiar with like a geforce card in your pc yes and no it the GPU on the Nano is very much just like a GPU in a PC it can run OpenGL graphics and any Linux stuff and CUDA and all of the typical graphics the difference with the Jetson is it has all of the other components in there like the CPU and the memory and is the complete SBC it's not just the graphics card there's a question about different ML frameworks and how do I install tensorflow or how can I install PI torch there's a couple different frameworks folks we're asking about there we we do make pre-built installers these pip wheels available for the popular frameworks like tensorflow and pi torch and you can find those on the jetsons ooh that's a page on the Linux wiki that's linked to from every Jetson desktop and basically you can just automatically download it so you can also build other ones from source which is the typical way if you don't already have a binary installer there were a bunch of questions about Wi-Fi cameras or basically IP cameras cameras that you stream over the network whether it's a wired connection or over Wi-Fi and you can use those on Jetson through gstreamer if they support RTP or RTSP which are common IP commerce streaming methods and if it's a Wi-Fi camera basically that just means you need to have a Wi-Fi adapter added to your Nano so that you can connect to the same network that's the camera I will say there are a lot of IP cameras out there that have vendor specific protocols or maybe some other protocol libraries that you need to install you in that case you would just need to install the libraries for those cameras onto your nano and use those interfaces but in general it's a good bet if they say they support RTP or RTSP cameras those can work pretty easily and RTP RTSP hammers are also supported natively in deep stream in addition to GStreamer another cool thing is you can setup your nano to act like an RTP or RTSP camera or you know a streaming source you can take in video from a camera locally and then stream that video back out over the network to another remote host using the same gstreamer elements there was a question about how do I if I get mine animals set up and install a bunch of stuff and may want to make a backup of the SD card image in case you know I just want the backup or I want to do some other modifications and maybe have that checkpoint or rollback on or when Nvidia does release new jetpack images and when I make it back up over that all you have to do is take the SD card image out of your nano and then plug it into your PC that you flash it from if it's Linux PC you can use like the DD command and it'll just backup the entire contents of that SD card image and then you can reflash it with the Etro tool or what have you and I think there are tools available for Windows and Mac that do a similar thing basically you're just cloning the SD card image and then you can reflash it at a later time but there's a question can I boot Nano from a USB Drive or like a solid-state drive yes you can do that you have to go into the ext linux.com file and change what device the kernel boots from and there's an article in Jetson hacks about doing that I if I mentioned Jetson hats but it's a very popular blogger that we like to work with who post a bunch of other videos and tutorials about using the Jetson so if you haven't already check out Jetson hack sees as a really good YouTube channel can I run the Nano headless or do I need to have a display attached to it you're welcome to do either or the Nano can be like a whole little PC and you know you just plug in HDMI and USB keyboard and mouse and it's just like working on a normal desktop or you can run it like a traditional embedded board where it's only connected over Wi-Fi or Ethernet and you basically just SSH into the board you can also set up the Nano completely headless if you don't even have an extra HDMI or DisplayPort monitor in that case you just connect it to your PC over the micro USB connection which emulates that UART terminal so you would just connect to it from putty for example and then that would allow you to set up your user configuration and then after you did that it would activate SSH and you could connect to it over the network there was also a bunch of questions about you know regarding networking can the Nano act as a Wi-Fi access point and the answer does depend on which Wi-Fi dongle or adapter you're using if that Wi-Fi device does support access point mode you can enable it just the same way that you would turn on access point mode on other systems generally that's using the host APD daemon that you can install through Linux so you can just install that and enable it in your Wi-Fi driver if necessary but generally you can just check if that particular Wi-Fi device that you want to use supports access point mode there was some questions about stereo cam and other depth sensors so I guess I'll just comfort those all at one time which stereo cameras can I use or what depth sensors either active or passive so the most common stereo camera that's used with the Jetson is called the Stereolab's II it's a USB 3 camera that does the stereo vision processing in CUDA so it's a really good match for the Jetson and there's the full-size Zed which like this big and then there's it mini there's also stereo cameras from econ systems and do a 3d that are supported on the Jetson and there's a bunch of like these time-of-flight RGB depth cameras as well like realsense d3 5i structure io sensors that i've seen used on the forums so and in general if you have a USB peripheral that you want to use as long as it has a driver that supports a recent Linux kernel version like Linux kernel 4.4 then it should be good to go you might need to install the driver if that driver is not natively built-in to to Linux or a lot of cameras just work because they're what's called UVC compliant which means it works with the video for Linux 2 interface others like the realsense for example you would need to install like the Lib realsense or like the Microsoft Kinect there's this lib free nekked - I think there's a current version that you can install to get those working along those lines there were some question about using Ross the robot OS and how do you install that and what versions are supported so the way you should install it it's just the normal way from the Ross website as they do maintain arm 64 a arch 64 builds now and the version that is supported is Ross Milotic because that's the version 4 if we went to 18 out of 4 which is also the version that the Nano currently runs and we do have Ross packages for deep learning available too and we also as mentioned have the Ross packages for get but question just came in about regarding to chat bond and jet racer is do they use different images and yes the jet racer does use a slightly different image than the jet pod just because the ipython notebooks that it uses and also the underlying hardware drivers are slightly different but it is mostly the same and the instructions for building your own jet bot or jet racer image are covered in those wiki's - so the pre-built images as with the main Nano image are really provided for convenience if you do wanna you know tweak and tailor your own nano image you can pretty easily do that just by following those instructions or if you want to create your own master nano image you can use what's called the NVIDIA SDK manager which would allow you to pick and choose which components you want to flash to the Nano as opposed to you know just pre installing everything in general you know just the main nano image is much easier to use okay so let me look for some other questions here what can I use to train my own neural network let's say you know we do provide a bunch of pre-built networks that are trained on common objects but it's part of you know project it could be very conceivable you want to train it on some specific source data in that case you know you can train smaller data sets on the Nano depending on how long you want to wait you know the examples from the helloworld tutorial typically run overnight for multi gigabyte data sets that train for you know 30 training epochs or you can either use smaller data sets or train it for less amount of time but for really large data sets you know 20-plus gigabytes or so you would probably want to look into using either a cloud instance with GPU like an AWS or Microsoft Azure instance that has a larger GPU because that's really what most you know data scientists are using for training neural networks either unlike tensorflow or pi torch and we will be giving away some credits throughout the competition to some applicate to ask some applicants who want to do that or if you have like a PC that has a discrete GPU in it or like a laptop that has a Numidia GPU you could run it on there along those lines of you know deep learning and machine learning there is a question of is it a requirement to use deep learning and machine learning in this competition and it is not you know artificial intelligence is more than just deep learning or machine learning you can do traditional computer vision methods with like open CV or Nvidia also has vision works which is an accelerated vision library that's highly optimized for GPU and some example algorithms you could run or like slam or different object detectors or trackers like optical flow algorithms things like that generally you might see a mix and match of both CV and deep learning algorithms used together also along those lines was a question of can I run multiple deep learning algorithms or neural networks at the same time or you know extrapolating that can I run computer vision algorithms and deep learning networks at the same time and yet in most cases the Nano is very much capable of running multiple at the same time if you're running a really heavy network that's already taking most the performance you would you know have to make sure you have enough performance - you know accomodate both but in general if you're doing like image classification using a resume at 18 or mobile net network which can run at you know 60 frames-per-second that should leave you more than enough room also to host other computer vision methods in the background so it's basically just about you know can you get enough performance out of both let me just take one final question here since we're at the top of the hour and again if you have more questions we're happy to and you know hang around the forums and you can post them there there's question about what programming languages can I use on the Nano and as mentioned in one of the earlier slides basically anything that has a common compiler available in Linux most common we see or C C++ and Python but you could also use things like mono Shu C sharp or rust or you can either do web servers like the PHP or you know anything that has these Linux packages available you can program in and you know there's lots of different things out there depending what you want to use and Python and C++ seem to be the the most popular so thanks again for joining us today and we really look forward you know to the competition and supporting you guys throughout and best of luck working on your projects and you know we'll see you around the forums with that I will turn it back to Monica so thanks again everyone for joining us thank you so much to Destin now is that a great presentation I hope you all post your projects to the DIY intelligence contest on Baxter and I look forward to seeing what you post so thanks again for coming to the webinar and I'll be looking for your projects on and on hackster thanks again Destin

Channel: Hackster.io

Views: 13,640

Keywords: Machine learning, AI

Id: AO2A3WLDDcI

Length: 59min 52sec (3592 seconds)

Published: Fri Nov 01 2019