Object tracking with YOLOv8 using Jetson Nano

in this tutorial we will make a program to track an object using camera Jets and Nano and dynamixel servos as a camera a usual USB camera or real sense can be used the big difference between simple object recognition and tracking is that even when you have several objects of the same class tracking algorithm will track only a specific object as we can see even if the program recognizes the object with the same class it was tracking the real sense keeps tracking the object it was tracking previously now let's see the theory let's get familiar with several object tracking algorithms the most basic algorithm is called sort sort stands for simple online real-time tracking it consists of four steps the first step is detection in this step an object detector detects objects that are to be tracked in this tutorial we are using YOLO V8 as a detector the second step is estimation in this step detections from the current frame are propagated to the next frame in the next frame we are estimating position of the target using a constant velocity model the third step is data Association having the target boundary box in the detected boundary box a cost Matrix is computed as intersection over Union distance between each detection and all predicted bounding boxes from the existing targets if intersection over Union of detection and Target is less than a certain threshold value then that assignment is rejected the fourth step is creation and deletion of track identities in this step creation and deletion of IDs are done unique identities are created and destroyed according to the minimum intersection over Union value tracks are terminated if they are not detected for some number of frames sort performs well in terms of tracking precision and accuracy but sort algorithm returns tracks with a high number of ID switches and fails in case of occlusion to overcome this problem deep sort algorithm was developed deep sword is an algorithm which tracks objects not only based on motion of an object with also based on appearance of the object so it can be defined as a two-branch framework with an appearance branch and a motion branch let's see features of these two branches the Deep appearance descriptor is applied to extract appearance features from detections in each frame it has a feature Bank mechanism to store the features of the last 100 frames for each tracklet as new detections come the smallest cosine distance between feature bank and feature of detection is computed the distance is used as the matching costs during the association procedure in the motion Branch the Kalman filter algorithm is used to predict positions of tracklets in the current frame it works by a two-phase process State prediction and state update in the state prediction step it predicts the current state using formulas 1 and 2. in the state update step the common gain is calculated based on covariance of estimated State PK Prime and observation noise r k then the common gain K is used to update the final state which is represented by formulas 4 and 5. Mahala nobi's distance is used to measure the spatio-temporal dissimilarity between Motion state of tracklets and newcoming detections finally matching Cascade algorithm is used to solve the association task as a series of sub-problems this algorithm gives bigger matching priority to more frequently seen objects now let's see the system architecture as an actuator we are using dynamixel servos these servos are commonly used for robots and there is a lot of reference documentation about them also we are using u2d2 as USB communication converter that enables us to control and operate dynamixel servos since u2d2 itself cannot Supply power to servos we are using u2d2 power hub as power supplier note that we need 12 volts for servos and 5 volts for jets and Nano as a camera real sense or USB camera can be used also in case you are using a monitor with USB power supply you should not draw power from Jetson Nano USB port it may cause overload followed by crash of jets and Nano now let's do the programming this tutorial is made based on this Repository thanks to the authors for sharing this great work to run the program we will need these packages in case of jets and Nano we will also need some more additional packages so we will install these libraries manually now let's install required packages install Python 3 pip pip is a package management system used to install and manage software packages written in Python upgrade piyama Library Pi yum was a full-featured Yama framework for the Python programming language yam was a human readable data serialization language it is commonly used for configuration files next install tqdm tqdm is a python library that allows you to Output a smart progress bar by wrapping around any iterable a tqdm progress bar not only shows you how much time has elapsed but also shows the estimated time remaining for the iterable install python is an optimizing static compiler for both the Python programming language in the extended cython programming language now install numpy numpy is a python Library used for working with arrays install lib jpeg lib jpeg is a widely used C library for reading and writing jpeg image files install matplotlib matplotlib is a comprehensive library for creating static animated and interactive visualizations in Python install g4tran G Fortran is the gnu Fortran compiler which is part of the gnu compiler collection install lib open blasts open glass is an open source implementation of the basic linear algebra subprograms with many handcrafted optimizations for specific processor types install a pack lapak is a standard software library for numerical linear algebra now we are ready to install scipy scipy is a free python Library used for scientific computing scipy contains modules for optimization linear algebra integration interpolation special functions fft signal and image processing install typing extensions this package enables use of new type system features on older python versions now we are going to install Pi torch pytorch will be installed using pip but for jets and Nano we need a pip wheel that was built for arm Arch 64 architecture go to this page and download Pi torch binary from the list here are the commands required to install pytorch in this tutorial we will use version 1.8 it seems that the newest Pi torch version working with python 3.6 is 1.8 since this example is written for pi torch 1.8 just copy these lines and execute in the terminal to make sure that we have all required packages also execute the second line we have installed psython and numpy previously so just install pytorch now we are going to install torch vision all you have to do is just to copy and paste commands under the torch Vision label firstly install required libraries clone torch Vision version 0.9 note that you have to replace the version strings to actual version number you want to clone move to the torch Vision directory in this line build version variable is set to 0.9 by executing this command torch Vision build will begin we have successfully installed porch vision install Seaborn Seabourn is a library for making statistical Graphics in Python it is built on top of matplotlib and closely integrated with pandas data structures now we are going to install liberalsense go to this page clone this Repository move to install realsense SDK directory execute build live real sent shell script with one job the job option is required to prevent installation process failed due to memory shortage basically the installation of lib real sense is complete but we need to do some modifications to make it work with python as you can see the program shows that Library files should be in this directory but actually Pi real sense-related libraries are in another place they are in this directory so we have to modify the bash RC file go to the home directory and open the bash RC file and modify the last line Source bash RC file to apply changes immediately check whether you can import Pi real sense normally as you can see now we can use pyrealsense Library install psutil psutil stands for python system and process utilities it is a cross-platform library for retrieving information on running processes and system utilization in Python install IPython IPython is a command shell for interactive Computing in multiple programming languages install Century SDK centuries python SDK enables automatic reporting of errors and exceptions as well as identifies performance issues in your application install easydick easydict allows to access dictionary values as attributes install G down G down enable us to download a large file from Google Drive install filter Pi this Library provides common filtering and various related optimal and non-optimal filtering software written in Python install top it is used to calculate max and parameters and model install lib Canberra gtk this is the module that takes GUI events like pressing a button from gtk plus and play some configured sound install dynamixel SDK this package provides dynamixel control functions using packet communication now let's see the code please download YOLO V8 tracking zip file from Google Drive and extract it to your home directory in this folder we have three newly created Scripts these scripts are based on this original trackpy script track real since pi and track webcam scripts are simplified versions of track using servos Pi script so I will explain the track using servos Pi script since it covers all new Scripts let's see what parts we have added compared to the original track Pi script here we are importing dynamixel SDK in this part dynamixel Servo setting is done these settings are different for each Servo model we can check these values on the robotice Emanual page for example for dynamixyl ah12a Servo address for current position will be 36. here initial angles of servos are defined and here servos are moving to their previously defined initial positions these are parameters for PID control of servos loadstream class has two new parameters resize in real sense if you are using this code with webcam the parameter should be as they are set now and if you are using it with real sense camera resize parameter should be false and real sense parameter should be true in lines below if statement we are doing tracking note that in this code the camera tracks objects which belong to cat class and has id1 in these lines Center coordinates of Target boundary box are calculated here error between boundary box center y coordinate and Image Center coordinate is calculated then we calculate the angle which Servo should move using PID control method in these lines we are doing same calculation for x-coordinate in this part servos are moving to Target angles now let us see what modifications we have made to load stream class definition of this class is located here let's open this file since we will use real sense here we are importing Pi realsense Library here is a real Sense part of the code here image size format and frame rate are defined pipeline start means that we start pipeline streaming with configuration we set in the above line the alignment utility performs per pixel geometric transformation based on the depth data here wait until a new set of frames becomes available the frame set includes time synchronized frames of each enabled stream in the pipeline here we get a line frames from RGB in-depth camera in this line a new thread is created to get video frames from real sense if the resize parameter is true when we are using a webcam the image is resized to 640x480 image size also we have created two functions to update video frames according to whether webcam or real sense is used let's run the code firstly we will run the code only with a web camera open the terminal and move to YOLO V8 tracking execute the track webcam Pi script using YOLO V8 weights now we are going to run the code which works with servos note that we should execute the CH mod command to enable the program to access the USB port to which servos are connected then execute the track using servo's pi script using YOLO V8 weights

Channel: robot mania

Views: 12,136

Keywords: robotics, ROS, deep learning, programming

Id: joAZEUbZZy8

Length: 20min 17sec (1217 seconds)

Published: Sat May 20 2023