Object detection with Jetson Nano using YOLOv7 and RealSense

in this tutorial we will use YOLO V7 with a real sense camera and Jetson Nano YOLO V7 infers faster and with greater accuracy than its predecessor so using it with depth cameras such as real sense should be valuable in robot development let's see YOLO V7 Network structure briefly this is a basic YOLO V7 Network structure the structure may vary a little depending on the model scale but this is the most common structure backbone is a convolutional neural network that pulls image pixels to form features at different granularities feature pyramids are a basic component in recognition systems for detecting objects at different scales neck combines and mixes the convolutional network layer representations before passing on to the prediction head head is the part of the network where YOLO predicts the bounding boxes locations and the classes of the bounding boxes now let's do the programming firstly go to the YOLO V7 page and clone the Repository move to YOLO V7 folder to make YOLO V7 work these python libraries should be installed on Windows you should just execute the requirements text file but for Jetson we should install some additional packages so we will install these libraries manually also note that opencv is pre-installed with Cuda support so do not install it using pip command install python3pip is a package management system used to install and manage software packages written in Python upgrade pyam library pyam was a full-featured Yama framework for the Python programming language Yama is a human readable data serialization language it is commonly used for configuration files next install tqdm tqdm is a python library that allows you to Output a smart progress bar by wrapping around any iterable a tqdm progress bar not only shows you how much time has elapsed but also shows the estimated time remaining for the iterable install cython ciphon is an optimizing static compiler for both the Python programming language in the extended scython programming language now install numpy numpy is a python Library used for working with arrays install lib jpeg lib jpeg is a widely used C library for reading and writing jpeg image files install matplotlib matplotlib is a comprehensive library for creating static animated and interactive visualizations in Python install g4tran g4tran is the gnu Fortran compiler which is part of the gnu compiler collection install lib open blast open glass is an open source implementation of the basic linear algebra subprograms with many handcrafted optimizations for specific processor types install a pack lapac is a standard software library for numerical linear algebra now we are ready to install scipy scipy is a free python Library used for scientific computing scipy contains modules for optimization linear algebra integration interpolation special functions fft signal and image processing install typing extensions this package enables use of new type system features on older python versions now we are going to install pytorch pytorch will be installed using pip but for jets and Nano we need a pip wheel that was built for arm Arch 64 architecture go to this page and download Pi torch binary from the list here are the commands required to install Pi torch in this tutorial we will use version 1.8 it seems that the newest Pi torch version working with python 3.6 is 1.8 since this example is written for pi torch 1.8 just copy these lines and execute in the terminal to make sure that we have all required packages also execute the second line we have installed psiphon and numpy previously so just install pytorch now we are going to install torch vision all you have to do is just to copy and paste commands under the torch Vision label firstly install required libraries clone torch Vision version 0.9 note that you have to replace the version strings to actual version number you want to clone move to the torch Vision directory in this line build version variable is set to 0.9 by executing this command torch Vision build will begin we have successfully installed porch vision install Seaborn Seaborn is a library for making statistical Graphics in Python it is built on top of matplotlib and closely integrated with Panda's data structures before executing YOLO V7 inference we have to download the model move to YOLO V7 directory and download the YOLO V7 tinypt file YOLO V7 tiny is a basic model optimized for Edge GPU compared to other versions The Edge optimized YOLO V7 tiny uses leaky rectified linear unit as the activation function while other models use sigmoid weighted linear unit as the activation function now execute the detect Pi script note that you have to specify weights and directory of the image also close as many other programs as possible because loading even a tiny model will consume almost all memory of the Jets and Nano and the script may be aborted if there will be lack of memory now let's see the results as you can see we have successfully executed the inference now we are going to install lib real sense go to this page clone this Repository move to install realsense SDK directory before executing build Loop real send shell script we need to do some modifications to avoid building process stopping due to memory shortage go to the install realsense SDK folder open the build lib real sent shell script at line time make J num proc change Nom proc to 1. execute build lib real sent shell script basically the installation of liberal sense is complete but we need to do some modifications to make it work with python as you can see the program shows that Library files should be in this directory but actually Pi real sense-related libraries are in another place they are in this directory so we have to modify the bash RC file go to the home directory and open the bash RC file and modify the last line Source bash RC file to apply changes immediately check whether you can import Pi real sense normally as you can see now we can use pyreal sense Library now let's see what we have added to the initial code to do inference using real sense here image size format and frame rate are defined pipeline start means that we start pipeline streaming with configuration we set in the above line the alignment utility performs per pixel geometric transformation based on the depth data here wait until a new set of frames becomes available the frame set includes time synchronized frames of each enabled stream in the pipeline here we get a line frames from RGB in-depth camera in this part frames are converted to numpy arrays here we are using the apply Color map function to produce a pseudo-color depth image in this part A boundary box is drawn around each detected object now we are ready to execute the code move to the YOLO V7 directory and just execute the detect RS Pi script we have successfully run YOLO V7 with real sense

Channel: robot mania

Views: 15,472

Keywords: python, robotics, deep learning, recognition

Id: o42BFhtVqyo

Length: 12min 36sec (756 seconds)

Published: Fri Oct 21 2022