Object Recognition with Orin Nano (Jetpack 6.0) Using YOLOv9, TensorRT and RealSense

in this tutorial we will use or Nano with jetpack version 6.0 and YOLO v9 to do object detection we will use pytorch in tensor RT with a real sense depth camera nowadays every autonomous robot has an object detection feature so using the newest object detection method should be useful for further Improvement of robot capabilities let's take a brief look at new features of YOLO v9 in Yolo v9 two new methods are introduced to effectively address issues related to information loss and computational efficiency the first method is programmable gradient information three modifications were made the first modification was made in the main branch the main branch is optimized for inference ensuring Simplicity of the model it is designed to bypass auxiliary components during inference to reduce computational overhead secondly auxiliary reversible Branch structure was introduced this structure ensures generation of reliable gradients and facilitates precise parameter updates it overcomes inherent information loss in deep Network layers and enables preservation and utilization of complete data for learning the third method is multi-level auxiliary information flow this method combines gradient information throughout the models layers to cope with the problem of losing information in deep supervision models this technique helps to make better predictions for objects of different sizes the second method is called generalized efficient layer aggregation Network Gillan enhances the model's ability to process and learn from data more effectively it combines the best features of CSP Net's gradient path planning with elen's inference speed optimizations in gelen two modules are used to enhance the model's ability to efficiently process and learn data the first is SBP elen module SBP stands for spatial pyramid pooling it starts with a convolutional layer followed by series of spatial pooling operations to capture multiscale contextual information the outputs are concatenated and passed through another convolutional layer to consolidate the features the second is the rep ncsp Elon 4 module this component splits the input from the initial convolutional layer into two paths processes each through a series of rep and csb and convolutional layers and then merges them back this dual path strategy facilitates efficient gradient flow and feature reuse significantly enhancing the model's learning efficiency and infer speed now let's begin with installing required libraries to run YOLO v9 first we are going to install an internet browser there is no internet browser installed by default but we will need one to access various web pages so we are going to install Firefox Firefox can be installed using snap snap is a software packaging and deployment system for a Linux operating system now we are going to install pytorch instructions for installing pytorch are written in this page scroll down to the prerequisites and installation section basically we have to execute these commands update package list using this command next we are going to install Python 3 pip and lib open blast this command is missing an install statement so add it open glass is an open source implementation of the basic linear algebra subprograms with many handcrafted optimizations for specific processor types the example below is written for Jetpack version 5 but since we have jetpack 6 we have to find a suitable pytorch version copy this part and open in a new tab move to V60 DP and then move to High torch in this tutorial we will use this version so download it the download will take some time after the download is complete go back to the installation page copy this part and execute it this command will upgrade pip to the newest version next install npy we don't need these apostrophes so remove them we have successfully installed NPI the error displayed in the terminal doesn't matter so we will leave it as it is move to the downloads folder and install install pytorch using the wheel file we previously have downloaded let's check our installation open python command line in terminal import pytorch by executing Cuda is available we can check whether we are able to use GPU now we are going to to install torch Vision move to this page open the instruction section to install torch Vision we have to execute these commands copy and execute this line here we are installing all required libraries next we have to install torch Vision version suitable for our pytorch version here we have a correspondence table of pytorch in torch Vision we have installed pytorch 2.2 but it is not here yet torch Vision 0.17.0 exists in GitHub repository so we will install version 0.17.0 copy this line and execute executed in the terminal note that we have to replace the version strings to actual version number we are going to install there is a possibility that while cloning the repository this error occurs the solution for this error is written in this page we have to set three environmental variables related to git also it is recommended to use wired connection instead of Wi-Fi now let's try to clone the repository once again this time we have cloned it successfully move to the torch Vision directory in this line we also have to specify torch Vision version we are going to install by executing this command torch Vision build will begin we have successfully installed torch Vision now we are almost ready to execute YOLO go to this page and clone the repository we have to install all required libraries move to YOLO v9 folder and open the requirements text file all required libraries can be installed at once using this file but to avoid Arch architecture specific libraries we have already installed being overwritten we need to modify this file delete npy open CV Pi torch and torch vision from the list execute this file using pip go to YOLO v9 GitHub page and download weights in this tutorial we will use YOLO v9e file move downloaded weights to YOLO v9 directory now execute the detect Pi script we have an error this error might be fixed in the future but for now open detect py script and modify line 98 the results will be generated under runs directory as we can see we have successfully executed YOLO to execute YOLO v9 using tensor RT we have to install several libraries install on NX on NX is an intermediary machine learning framework used to convert between different machine learning Frameworks install on NX simplifier this library is used to make operations with on NX more simple for YOLO v9 tensor RT deployment we will use this repository clone this repository copy this line move to YOLO v9 folder and execute the export Pi script note that we have to modify the weights file name change python to Python 3 and add a device argument to make it use GPU and on NX file will be exported copy created on NX file to configs directory which located inside of YOLO v9 tensor RT directory open the YOLO v9 yum file modify this file according to the on andx file name we have moved and the name we want to give to engine file that will be generated open a new terminal create a build folder execute the cmake command execute the make Command we have an error a similar issue is discussed on this page so we have to execute the cmake command with these options we have successfully built our project before going to the next step install Puda Puda provides scyon and python wrappers for Cuda driver and runtime apis check if open CV python headless is installed if it is installed uninstall it since it will cause problems in the program we are going to run move to the configs folder and open the YOLO v9 pong file change on NX file and engine file names to those we will use open the YOLO v9 trt file we have to modify host outputs and Cuda outputs array indices this modification is required probably because our engine file is generated using or Nano GPU instead of usual GPU for desktop computer now execute the YOLO v9 trt Pi script when executed for the first time engine file will be generated so it may take up to several tens of minutes we have successfully executed inference using tensor RT now we are going to install lib real sense go to this page clone this Repository move to the install lib real sense directory before executing the build lib real sense shell script we need to do one modification to avoid error during building process go to the install real sense folder open the build lib real sense shell script at line 135 add this argument this argument helps compiler find where python executable is located now execute the build lib real sense shell script note that memory amount is increased in the Orin Nano so we can set the jobs argument to two to accelerate building process even though it is written that the library has been installed insted in the user local lib directory actually it is installed here this problem is explained in this issue but it seems there is no solution at this moment so move to the home directory open The Bash RC file move to the bottom of this file in this line change lib to off open a new terminal execute the source commands to execute changes we made to the bash RC file now we can import the py sense to library success y now let's see the code please download the YOLO v9 tensor RT zip file from the Google Drive and extract it to your home directory open the YOLO v9 trt RSP script this code is almost the same as the YOLO v9 trt Pi script so I will explain realsense related modifications here image size format and frame rate are defined pipeline start means that we start pipeline streaming with configuration we set in the above line the alignment utility performs per pixel geometric transformation based on the depth data here we get aign frames from RGB and depth camera in these lines obtained frames are converted to a nony array here inference is done to execute the code open a new terminal and run the YOLO v9 trt RS script

Channel: robot mania

Views: 706

Keywords: robotics, ROS, python, deep learning, Nvidia

Id: aWDFtBPN2HM

Length: 18min 28sec (1108 seconds)

Published: Mon Mar 18 2024