Real Time Object Detection and Tracking with YOLO-NAS and DeepSORT on Custom Dataset

hello everyone in this video tutorial we will see how we can integrate the yolonas with deep sort object tracking algorithm and how we can do object tracking using yolonas on custom data set so in the first step we will integrate yolanas with deep sort object tracking and test it test it on pre-trained colonized reads in the next step we will train the YOLO Nas model on a custom data set and integrate deep sort object tracking with it in object tracking what we do is we assign a unique ID to each of the detected object so when the new object enters the scene that object is being assigned a unique ID and until the object remains in the scene that object is being assigned with that ID we will go into more details as we start coding so let's get started so here you can see that I have created an empty folder by the name YOLO Nas deep sort video for this project so now I will just go to on pycharm and create a new project over there so let me just open the pi jump so now here you can see that I have just opened up by jump Community Edition I will just click on file and click on new project from here and just select the new project directory from here and it will be directory in which I have just created this folder and it is inside this and the name of the folder is Yolanda's deep sort video so just select this folder and just check this if this part match okay so the location of the new virtual environment also matches and we are using base interpreter as python 3.10 so now just click on create a new project and in this window so now you can see that it's now creating a virtual environment so this might take few seconds before it's ready so this might take few more seconds okay so now you can see that we have created a new project into our directory which I just shown you and first of all we will just install all the required libraries so I will just create a new file by the name requirements.exe and in this file I will list all the required libraries over here and we just need two Library super gradients and the other is opencv python so to implement uh your uh your Nas object detection algorithm we required the super gradients package so to implement your NOS object detection algorithm we require the super gradients package plus uh to do processing on the video to display the input or output video and to use different functions of opencb in our code we require the opencb python package as well so I am just using super gradients package to run object detection using yolanas and plus uh to do a video processing and to use different professions of opencv python in the code require the opencp python package as well so just write open SIM python so we just need these two libraries super gradients and opencv python so I will just go to the terminal and just write tape install minus r requirements dot txt uh before as we was as we will be integrating object tracking using deep sort so we just don't require any package for this currently so we just required these two packages which I have listed over here one is super gradients and other is opencv python package so just write pip install minus r requirements.txt and then click on enter from here so if these packages will are not already installed this will then they will get installed and it will take some time around five to ten minutes but here you can see that it's appearing requirements already satisfied because I've already installed these packages but if you have installed those packages this will take some time next what you will do is you will just click over here and just create a new DOT Pi file over here and just write object tracking dot pi okay so now here uh in the first step I will just write the code to do object detection or a video and in the next step we will integrate object tracking using deep sort and then we will test it on videos as well and in the next uh the last step we will be training over your NOS model on a custom data set and then we will see how we can integrate your learners with deep sort object tracking on a custom data set as well so let's do first two object detection on a video the first step I will just import all the required libraries so first of all I will import the opencv python Library by writing a port CB2 in the next step I will just import a torch Library import torch then from Super gradients so to implement object detection using your Nas we require the super gradients package as I've already told you from Super getting dot training import models we also require the numpy package we will require it when we will integrate object tracking using deep sort and we will also require the import math package we also require the math package because we will use this package when calculating the confidence score so next what we will do is we just need to read a video so for to capture the video or read the video we will using cvu dot video capture function so first of all I will just create a directory over here by the name video so and in this directory I will add some videos over here so let me just go to my required folder uh just give me a minute I'm just navigating towards it so now as I told you I have created a directory by the name video into my your last deep sort video folder and inside this video directory I have ended multiple videos so these are around uh nine different videos or sorry 10 different videos which I have added in the video directory which I have created in the pycharm so let's go towards the pie chart so if I just now see over here in this video directory you can see we have tons of videos so let's read a video and let's see how we can do object detection on a video using your Nas so I will write cap is equal to CV2 Dot video capture and just uh I just will just go to the video folder and I will just read bikes dot MP4 okay then let me just calculate the frame width and frame height we need the frame width and frame height because we will be saving the output video and it's not exactly so we are just completing the frame weight and frame height now and now I will write over here device is equal to torch dot device so if you have GPU you can use the GPU but I'm in my local system I don't have my GPU so I will be using CPU if Dodge dot Q Dot is available else torch dot device so if you have GPU available I will be using GPU but in can that any case if I don't have the uh GPU available I will be using the CPU so ah in the next step in the last step we will see how we can integrate object tracking or how we can integrate uh Uranus with deep sort object tracking on custom data set but uh this we will see in the last step but currently we will be using the pre-trained your last breeds all right model is equal to models dot get so in this project I will be using yolona small model yolonas comes with three different models you will not small you will not medium you'll learn as large you're on a small is the fastest and uh but it is the fastest but it is less accurate as compared to other Uranus models but your last large is the most accurate but it is less fast as compared to other YOLO Nas models so udana small is the fastest but this is less accurate while Yolanda's large is the most accurate but it is less fast so we will using the pre-trained weights as Poco and two device thank you so here I will be initializing the count as 0 we will just doing the frame count so I've just initialized the comment here as 0 and as I just want to save my output video so I will be using cvu.video writer function to save the output video in the my output video will be saved by the name output dot Avi and I will write CV2 dot video writer Dash 4 CC m J B G and here I will be just passing the frame rate and here I will be just defining the frame width and the frame height as well okay so here I will just write y to and here I will write red form of frame is equal to cap dot read and if red results is equal to list model dot predict stream and here I will be just setting the confidence score as 0.35 and then click on enter and here I will be just calculating the bounding box coordinates over here so here in the first step you are calculating the bounding box coordinates in foreign and just converting them into the list format over here and in the next step now I will be just calculating the confidence score over here by just writing confidence is equal to result Dot prediction dot confidence and then I will just click enter and now I will be just finding the class I number label is equal to result labus is equal to result dot prediction dot label dot tool list so we have the bounding box coordinates value we have the confidence score we have the labels here as well so now we'll look through each of the boundary box confidence score and labels and we just create a bounding boxes around the detected object assign it the confidence score as well as the label all right for okay well let me save confidences levels so that's fantastic and here I will just write this so now uh we will be just calculating the coordinates so for each of the bonding box we have four coordinates X1 y1 X2 and Y two so for each of the bounding box we have the four values or four coordinates values X1 y1 X2 Y2 so X1 y1 represent the top left corner of the bounding box and X2 y to represent the bottom right corner of the bounding box I will write x 1 y 1 x 2 Y 2 V box 0 the box one V box tool B box C so here we have just got putting the coordinates of the bounding box over here and just copy this from here so currently this output is in the form of tensors we are just converting it to Output from tensors into uh you can say that into integers okay so we have converted the output from tensors into integers just change it to CLS port to make things easy further and just right over here last name is equal to ncls so basically uh we are using the pre-trained weights of Coco data set over here so Coco data set consists of 80 different classes so I will just add the class names over here so let me just copy the class names okay so these are the 80 different classes which we have in the Coco data set so I've just added those class names all over here and next thing which we can do over here is name is equal to okay so now we will have the class name and now we just need to calculate the confidence confidence equal to mat dot seal thank you so here we have calculated the confidence score over here as well so here I will just write label is equal to last name confidence so we have the label as the class name and The Confidence Code so now we will just calculate the text size over here so I will use CV2 dot get text size function for this to calculate the text size so here we have category attack size over here which you can see as well so now we will make some adjustments into the rectangle into the rectangle which will create above the bounding box foreign objects so now we will be creating a rectangle around each of the detected object or you can say that we will be creating a bounding box around each of the detected object so for this I will be using CV2 dot rectangle function so here I have just passed the coordinates of the bounding box for each of the bounding box we have four coordinates X1 y1 represents your top left corner and X2 Boy 2 represent the botting right corner and the color of the bounding box will be which I will Define over here the color of the bounding box will be sky blue so now um we have created a bounding box around each of the detected object using CB2 dot rectangle function so after creating a bounding box around each of reacted object now I will just create a rectangle above the bounding box in which I will just put a label and the confidence score or I will just add the class name and the confidence score thank you so just add the same color over here as well okay so just copy this from here and just add this over here and just change it to because we don't want our bounding box rectangle to be filled so just change it to thickness of the bounding box rectangle will be three and using CV root output text function I will be adding the text inside the bounding box foreign looks quite good so using cb2. read size function I will just resize the frame so that it fits on my screen now I will just show the output here as well so here we have our output and just a night few lines to be written over here I would put our DOT release cap dot release same V2 dot destroy it on Windows okay so let's run this script now and let's see what we get to it so let's see whether we are able to do object detection using your Nas on the video if we are able to do object detection using your Lanas on video then in the next step we will integrate of these short object tracking with your learners so if there is any error we will try to fix it up but if it okay new module name okay so the spellings were not correct it is super big it's so let's see how does it cause if there is any error we will try to fix it up so this might take few more seconds before okay there was spelling mistake now let's see how does it goes now okay so here we have also have a mistake it should be B boxes okay so now let's see if there is any error or if it works fine so here we have the error so it's the labels over here okay okay started but it just end quickly so let's see what's the error okay so here is better it should be zero okay so I hope it will work fine and we will have output in front of us so this might take few more seconds to execute so let me just navigate my screen towards the output video so here you can see that we are able to do the detections or so we have done object detection using your last use you can see over here we have detected the traffic lights we have detected numbers and we have directed a bicycle we have detected the handbag we have directed the person so the detection results look good like you can see over here we are able to do object detections using yolanas so here you can see that we have detected bicycle we have detected person we have detected a handbag we have the label like bicycle over here we have the confidence score like 0.96 which means the model is 96 percent confident that this is the bicycle here and the confidence score of the person is 0.41 this means the model is 41 confidence that 41 percent confident that this is the person so now you can see over here we have done object detection using YOLO Nas here you can see we have created a traffic light and the confidence score is 0.76 which means the model is 76 percent confident that there is a traffic light so the detection results look impressive let's go back towards the code so you can see that we have we are able to do the object detection using Yolanda asks and we are able to do uh the object detection very successfully we have created bounding box around the detected object we have assigned at each of the directed objectile label and the confidence score as well as we are also able to do a small object detection very perfectly using your loss as well okay so now let's integrate object deep sort object tracking algorithm into this object detection using your Nas algorithm so now we will just integrate the sort object tracking algorithm into it for the object tracking I will be using deep sort algorithm so you can just write over here deep sort GitHub and just go to this GitHub app over here and you can just download this deep sort file from here so I have already downloaded this file so what we basically do in object tracking we assign a unique ID each of the detected object and for here object tracking I am using deep sort algorithm so you can just download this file from here deep Dash sort where you can just download this complete folder from here or you can just uh by just cloning this repository you can just download this ZIP complete repo from here and you can just take up this folder deep sort over here okay so let me just go back towards the code so now here you can see I have just downloaded this complete folder deep sort that by torch over here and here I have all the required files uh I will just add this folder uh into I have just into the GitHub and I will just share the link with you as well I will just add this folder into the GitHub wrapper uh in my GitHub wrap point you just share with the link with you as well so you can just download this folder from here before we go ahead let me show you a we have launch three courses on odimi you'll learn us the ultimate course for object detection and tracking so this is the pro course which cover various topics and we have done object detection object tracking using your Lanas we have done around 10 different projects or 12 plus projects which you can see over here and we will go step by step how you can train your anas on custom data set plus we have implemented some real world projects which include license plate detection plastic bottles counting privacy learning Vehicles counting fire detection face mask detection real time sign language alphabets reduction and many other projects you can just check it out over here plus we have two other two other courses available as well which you can see over here we have the yellow B7 yellow V8 your NOS object detection tracking course you can see that 190 students have already enrolled and that it is the one the best seller course on the udemy platform and the rating of the course is 4.6 so the numbers speak itself uh this contain course contains 19 hours on demand video and you can see over here a pair of huge number of projects which will cover there around 22 plus real world projects 22 last real world projects which we will cover so in this course we haven't just discussed yellow V7 yellow V8 you'll be you'll learn us or in this course and it is the best seller course on udemy platform uh captions are also available also available and it contains 91 downloadable resources 19 hours on demand video you can see over here and here you can see we have all the content so we have around 20 plus sections in this course and in this course we covers the YOLO V7 we have to understand your Nas app user V8 apps as well plus we have other ports available on udemy platform as well which is uh yellow V8 object detection so the creating of this course is 4.4 and this is also the best seller course on the ODB period form and 301 students have already enrolled and this course contains 12 hours on demand video and here you can see this is the content of the course a huge content this course covers uh we will see how we can do web app development and how we can integrate YOLO V8 with flask and how we can create a complete web app as well so you can just go to these courses and you can just try out any of this course so uh for all of my YouTube fellows I am just offering 50 off in all my courses so the currently the course price is 19.99 but if you just click on the link in the description provided in this YouTube video you will get speed 50 off so if you just click on the link in the description provided you will get 50 off so what are you waiting for to check out this courses so now here you can see that this as we are implementing object tracking using deep sort so here we have the Deep sort by touch folder and here we have all the required files over here I will just add uh I have will just add the link of this folder in the description you can just uh using that link you can just download this folder so I will just add the link of this folder in the description so you can just download this folder by clicking on the link so we will be doing object tracking using deep sort algorithm what we do in object tracking we assign a unique ID to each of the detected object so first of all we will just import the required Library so that we can Implement object tracking using deep sort and integrate it with your Lonas algorithm turn right from Deep sort by Dodge the audio tails dot parser to get configuration and then I will just click enter so here I will just initialize few functions of the deep sort over here and just add this file part over here so just copy this part from here and just add this path over here okay so that's the pretty good till now so now here I will just initialize the Deep sort so it is a long line of code so I will just take it from my previous written script um so now here you can see that I am just initializing the Deep sort over here okay so here we have initialized the Deep sort so that's the pretty good okay so just copy this from here and just add this over here that's the code over here so now we will just comment out some previous quote so just comment out all this from here so let me just come I've just commented our all this from here and here I will just try to updated code first of all we will just calculate the center coordinate so uh basically uh deep sort requires the output or the input in the form of basically what we output we get from Yolanda's object detection algorithm we just pass that output into our deep sort object tracking algorithm so uh deep sort algorithm required the output in the form of Center coordinates and width and height of the bounding box here we are just calculating the center coordinates of the bounding box so now we will look at creating the bounding box width and height as well foreign into the format so that we can just pass this output received from the yolonas object detection algorithm into our deep sort algorithm so in this output from the Uranus object detection algorithm will be passed as input to the Deep sort object tracking algorithm I will be now creating four empty leads over here okay foreign and I will be just appending these values into this list so I will just append these values into this list over here so just copy this from here and just add this over here so that's no fine till now just make this change over here so that's look fine as well and just copy this from here and just add this over here oh this is my mistake and here I will write outputs is equal to and here I will write if length outputs is greater than zero and now I will be using the draw Dash boxes function uh I have not created that function so I will just create a function by the name raw Dash boxes so I will just creating the function above over here so let me just create a function by the name draw Dash boxes over here so I will write Define draw attach boxes I have already written this code for this function so I will just add this code because this video is getting very pretty long so I will just add the code for this function which I have already written so I will just explain you the code oh okay let's write that function because let's do it in front of you so I'm just calculating the bounding of coordinates over here again foreign X2 y1 Y2 now I will be using cb2.getax size function over here as well to calculate the text size foreign so I will just go down below again and just copy this from here and just copy all this from here and just add up this code over here again and let me just update this all from here as well so this is the test context color which I have defined over here thank you let me must make the color of the bounding box as sky blue and the color of the label and confidence score as real so let's look quite good already and let's see if there is an error okay this when we return image okay things look good let's see whether we are able to implement object tracking using deep sort over here as well so let's run this grip and let's see what results do we get so this might take a few seconds before it is ready foreign let's see if it works fine or there is a header let's see uh what is this error wait to do it okay so here is starting let me look into this error one thing I see is I'm not calling this draw Dash boxes function so just copy this from here and just write raw Dash boxes and right here frame V box Dash X Y X Y identities object ID okay so let's run it again and let's see what is this error if it appears again let's see how does it cost okay so we have the same error again okay let me just check it so we've just found out the error so this tree is not right just copy this and just add this in the torch Dot tensors and just make it frame and let's run it again and let's see or does it cost so if there is an error we will try to fix it up foreign so now here you can see that we have integrated object tracking with Uranus object detection algorithm but you can see here uh the rectangle is not filled like we just want this rectangle to be filled uh so I will just go back to the code and make some changes into my code so I will just go above this function okay and here I will just update this thank you I will just add minus 1 over here and this will make it work s let me just pause this and let's run it again and let's see what results do we get after do this we will just see how we can do object tracking using yolanas on any custom data set so let's run it and see what results do we get also let me just show you the output thank you so here is the output you can see that we have assigned a unique ID to each of the calculate object like one for this person six for this bicycle a 58 for this bicycle 26 for this bicycle 66 for this person um 21 for this person although you can see this person standing far away but using yolanos object detection algorithm we are able to attack the person so now here you can see that using deep sort object tracking algorithm we are able to uh detect the object and using uh so using of your nose object detection algorithm we are able to detect object and using deep sort object cracking algorithm we have assigned a unique ID to each of the detected object like you can see here 1 670 with tourist handbag so these IDs are not assigned uh like in the numeric order but these IDs are being assigned randomly like 66 81 58 26 so these IDs are being assigned randomly they are not assigned in the sequence order so now you can see over here using object detection algorithm using object your Nas object detection algorithm we detect object we create bounding boxes around each of the object plus using deep sort object tracking algorithm we assign a unique ID to each of the detected object using deep sort object tracking algorithm we track the detected object throughout all the frames until the object is in the scene and when the object leaves the scene that ID is removed so a unique ID is being assigned to each of the detected object and we track that object using that ID throughout the frames so in your using object detection using your Nas we detect object create bounding boxes around them and using deep sort object tracking algorithm we assign a unique ID to each of the detected object and that ID remains with that object until that object is in the scenes so that we can track that object you can see over 16 14 10 1 106 and these IDs are not assigned randomly are not assigned in the sequence ordered these IDs are assigned randomly so now I will just show you the collab code using that collab code we will be training the YOLO Nas model on a custom data set and then we will just add this custom model weights over here and we're doing object tracking using your Lonas on a custom data set so here you can see the data set I will be training the yolonas model on this data set the data set consists of 4680 images and we have 10 different classes in our data set so these are the drone images of board camping car car motorcycle pickup plane tractor truck van so these are the drone images of different uh vehicles and you can see over here and let me just go to the health check so the data set consists of 4681 images with 20 554 annotations out of 20 554 annotations 18 389 annotation uh belongs to the car class so the card class is overrepresented while 682 and questions belong to the pickup and 421 Integrations belong to the motorcycle class so the car class is overrepresented like it contains majority over 90 of the annotations while uh 10 of the Integrations belong to the remaining nine classes okay so I will be just training the yolonas model on this data set so here is our Google call app notebook so I will be training the yolonas model on this data set I have already trained the eulinos model on this data set um as I told you I've explained you already how you can train the Uranus model on any data set so I've already trained the unanas model on this custom data set and I have saved my model bits the best model bits already Okay so using those best model weights I will be doing uh the vehicle detection like you can see over here you can see we have checked the cars so let I've just saved those model weights so let us just go back to the uh Google our pycharm script and just add this model weights and um let's test this uh or let's integrate object tracking with yolanas on custom data set okay so I've already explained you how you can train or fine tune your Lunas model on uh like you can say on any custom data set so here is the output video of this like you can see currently we are only doing object detection using your learners on custom data set so if you just want me to I will just share this collab script file with you as well so I've already saved the best model with so let me show you the best model weights and then we'll go ahead over here you can see that this is our project folder and here I've created a directory by the name beards and inside this vs directory I have just paste put my best model bits which you can see over here so let's go back to the pycharm and let's do object tracking using deep sort on custom data set so let me first just comment this out from here and just write so we have fine tune or train the Yolanda small model on this drawn images of data set or drone images of vehicles as we have 10 different classes in our data set so now I will just add the class names I will just go back to the uh olap notebook file so you can see that here we have the 10 classes name in our data set so just add this class names over here and let's run it and let's see whether we are able to do uh object tracking using your learners and object detection object tracking using deep sort algorithm and object detection using your learners on custom data set as well so now here we are integrating Yolanda's object detection and object tracking using deep sort algorithm to do object detection and tracking on a custom data set so let's run this script now thank you here you can see that we have a weights folder and here I have my best model where it's over here foreign let's run it again one thing I have not corrected is basically I have not added the correct video path so just add me test too okay let me just run it again so now we will be doing object detection and tracking you on a custom data set which is drawn data set images of vehicles so here we are doing object detection using Google Nas and object tracking using deep sort algorithm on a custom data set so let's see this might take few seconds to execute so here is the output video you can see over here we have detected the car you can see that we have assigned a unique ID to the car like you can see five here three here twenty here 18 here so now you can see that we are able to do object detection using your nose and we are doing object tracking using deep sort algorithm on a custom data set so we are using object detection using eulinos and object tracking using deep sort algorithm on a custom data set using object detection we detect a car or it is a pickup and using object tracking we assign a unique ID to each of the detected object like you can see a 44 12 3 20 40 18 so each of the detected object is being assigned a unique ID using deep sort algorithm and for object detection we use YOLO Nas so the result so cool that is also compressive like you can see over here 54 55 40 so each of the detected object is being assigned a unique ID using deep sort object tracking algorithm so the results look super cool it's amazing and let me just go back towards the code and let's test out on some other video and let's see what results do we get and so just write test 4 dot MP4 and let's run on this video and see what results do we get so here is the output video you can see that we are able to detect the car and when you can see over here we have assigned a unique ID 9 4 to 15 so these IDs are not assigned in the sequence these IDs are being assigned randomly so you can see that we have detected the object we have assigned a unique ID to each of the detected object like 120 15 5 7 13 to this when 10 to this car and so this is what we do in object tracking we assign a unique ID to each of the detected object and for object detection I am using Yolanda so here you can see that we have tested on two different videos and the results look impressive so here we have seen that how we can do object detection and object tracking on a custom data set for object detection I am using Uranus and for object tracking I am using deep sort algorithm so that's all from this video tutorial I hope you have learned something from this video tutorial see you all in the next video till then bye bye

Channel: Muhammad Moin

Views: 3,929

Keywords: yolonas, yolo nas, yolo, object detection, object tracking, DeepSORT, pytorch, opencv, deep learning, computervision, custom dataset, training custom dataset

Id: rkTi6x5asLw

Length: 69min 41sec (4181 seconds)

Published: Wed May 31 2023