Build an AI/ML Tennis Analysis system with YOLO, PyTorch, and Key Point Extraction

hello in this video I will teach you how you can build this tennis analysis project from scratch the project detects and tracks the players as well as the tennis balls across the whole video we also detect the Court's key points to know the player's position relative to the court measure distances and even determine if the ball is in or out we will use the output of those models to know how many times a player has shot the ball and how fast he shot it we will also be measuring the player's speed and how many meters the player covered in addition we'll be training two convolutional neural networks in this project the first one will be YOLO and object detection models to detect the fast moving tennis balls to enhance the detection accuracy of the outof the-box models the other one is a CNN that is trained to estimate the Court's key points and we'll be using pyto for that so whether you're a beginner or an experienced machine learning engineer this project will totally make your resume shine let's start the project I have a project set up with one folder called input videos this folder contains one image and one video the image is a screenshot of a tennis match and the video is just a small bit of the same tennis match we are going to use those to develop and test our code our first task is to detect the players and the tennis ball with an object detector we are going to use a popular object detector model called YOLO so the way that we're going to do that is we're going to use a library called Ultra litics so um we can just pip install it first we can pip install Ultra litics and then run it it run very fast on my machine because I have it already uh but it might take a little bit more time on your end and then we're going to create a uh a a file that is going to be called YOLO inference this file we're going to be playing around with yolo a little bit we're going to see how it works and what the output is and uh yeah see how ultral litics work so to import YOLO we're just going to do from ultral litics import YOLO and uh to import a YOLO model we just do a model equal YOLO and then provide what more what model we want so here we want YOLO V8 which is the um the latest one that we can use right now so we have YOLO V 8X which is extra extra large and to run it on um um on um on an image we just do model Dot model. predict and give it the image path which is going going to be called input videos then slash image.png which is going to be this one um yeah we can also save it so we can have save equal true and yeah that is all for it so we can clear the output here and just run it it's going to take a little bit uh time um yeah it's it's going to also download the YOLO V8 model because it's not readily available on my machine but in the next run it will find it uh in my local directory so it won't download it again it will just download it the first time um so it's just going to take a little bit time and then uh run it on the image okay now we're done so you're going to find uh a new folder that was created it is called runs and then detect and then you have predict um if you open predict you're going to find the image with the detections there are multiple detections that you can find here which is multiple people or players and then you're going to have something like a tennis racket a sports ball which is right behind the tennis racket so you can uh actually see it um you have the clock and um you have uh you have also the uh the the name of the class which is a person a sports ball whatever and then followed by the confidence score the confidence score is a a number between zero and one tells you how confident the model is that this is um a person for example so the lower the number is the the less the less confident the the model is that this is going to be actually what it is and it's not going to be a false prediction but yeah this is the output so far it looks decent and U yeah the output like the output for one object consists of multiple things but let's focus on the three things that we have right now which is the bounding box the class name and then the confidence scoree the bounding box here is consistent of mainly going to be four values um and it can be in multiple uh ways to represent it so we have for example um we can represent it by the X Center and Y Center which is the X position and the Y position um of the um of the object followed by the width and the height of the uh of the uh of the rectangle and we can also represented by the um the far ends of the rectangle like the it's called X minimum y minimum then X maximum y maximum those two points uh we're going to stick by the X minimum y minimum then X maximum y maximum format um just to stay consistent then we have the uh class name here right here and then we have the confidence um yeah we can also see those uh outputs in a list format in the output format just by creating um a variable that stores the output we just didn't store it before um and we can just print it print the result and we can also print the bounding boxes because there are bounding boxes inside the results so we can have it we can have a look at it also so we can have something like this which is for box in um results of of 0 do boxes then print box uh this is going to print the boxes um then we're going to run it again it's going to take a little bit time now it's done uh first you're going to find that the runs have another folder now which is called predict two uh predict two has the latest prediction that we did and it's going to be this exactly the same thing so we won't have to look at it so looking at the result we have first the boxes which is a boxes object that we're going to look at later um then we have the key points in The Mask which is not important right now those are not the for detection uh those are for pose models and for segmentation uh we're not going to use it here so don't worry about them um we also have the uh names which is um the class names for example the ID zero stands for person the id1 stands for bicycle and so on and so forth then we have the uh original image and this is in pixel so we don't have to worry about it um and yeah this is all the main important stuff here then followed by the boxes themselves the Box themselves contains uh first of all the class um here it's zero zero you can scroll up and see that zero means person so that's the class name uh but here's the class ID so you can map it to the class name easily followed by the confidence score and then you have the is track which is going to be tracking um tracking uh across frames in a video uh we're going to tackle that um when we predict on a video so I can show you and and then we have the bounding box themselves you can see different types of formats here um but the one that we're going to use is the X yxy which is X minimum y minimum X maximum y maximum and here's the X position uh of the minimum one then the Y position then the maximum X and then the maximum Y and yeah you're going to find that we have multiple boxes and and that is it this is the uh results that we have so in order for us to uh run it on a video we can just replace the uh image path with the video path like this input video. MP4 then we're going to run it and it's going to take a little bit more time because a video consists of multiple images uh and in that case it consists of 214 images so it's going to run off those those images one at a time um so it's going to take a while so I'll be back when it finishes okay so now we're done and we have a predict three folder that we can go check out so you can press detect you can have the uh open the predict 3 and then you have the output video so let's open it and you're going to find a very similar output to the one that we have before and yeah it is just running it again and again over each uh frame in the video but the one thing to note is that if you look at the sports ball it's being detected very low like the number of times it's being detected in the whole video is just maybe 10 um and this is going to be very hard for us to analyze the the the motion of the ball and how it goes and its speed and things like that so we're going to need to uh fine-tune a detector model to detect the ball a little bit better so that for us to um utilize this output a little bit better so we can start by training and fine-tuning um aolo model on the balls on the tennis balls that are moving very past so that we can detect it a little bit better so the way that we're going to do it is that maybe we're going to uh do another uh folder called training and inside this folder we can create a notebook so we can call it um tennis ball detector detector um training. ipynb and then then you're going to have a notebook set up first before like jumping in we're going to need to find a data set so that we can utilize and I already found a data set on roof flow and roof flow has a data set of very similar images to the ones that we are using in our um in our use case where the there is an aial um photo of the tennis court and then the tennis ball is moving right here and you also have a bounding box on it uh by the way Robo flow also can give you the ability to create your own data set and it's done with very easily um but yeah we're not going to annotate it for us because it's already there and you have 428 images in the training set and 578 total images so this is is not a big data set but it's going to be good good enough for us uh for our use case um so if you haven't already you can just go create yourself a rof flow account and then um I am going to use YOLO V5 because it has the it got me the best results in detecting the balls so you can just click on it make sure that you have the show download code and um yeah you can press continue and it it gives you the code that you're going to use to download the uh data set so you just copy it and can come back here and basically um like get data set and you can paste it right here so here the API key is is going to be your key I just replaced it with a string so that not to show you my key and you can also like it also installs the robo flow so we can install it right here and I'm not going to train on this machine I'm going to train in Google collab so I'm just going to add also install ultral litics so you can install both and you can download your data set right here um so I want to show you the data set um that is being downloaded I'm going to run this uh now it's now it's being grun uh and it's downloading the uh the data set for us and yeah we're just going to wait a little bit for it to finish so now it's done and you're going to find that a new folder has been created which is called tennis ball detections -6 if you open it you're going to have have uh three folders which is training testing and validation and inside each one you're going to find images and labels and if you open the images you're going to find the images that were shown before and if you are going to open the labels you're going to find the class name which is uh going to be sorry the class ID which is going to be zero followed by the bounding boxes that we have so yeah to start training we're going to need to Wrangle the data set a little bit uh it's not going to be much uh but the training code requires us to have another folder inside this folder that is called the the same name tenis ball detection D6 and inside of it we're going to have training testing and validation so in order to do this we're just going to we can just do it manually right here uh but I'm I do do want to make the code reproducible so I'm going to do it with code and I'm going to import something called shuttle that is going to help us move copy and paste things around uh using python so the way that we do it is that we're going to write uh shuttle Das move then give it the tennis B detection then train and we want to move it here we want to move it again into another folder like this um so that way the the code for the training code won't crash it it just uh expects it to have it uh to have it in this type of format so I'm just uh abiding it uh so we can do the same for the test and for the validation you can run it and it's going to finish anytime soon okay so if you if you see the results again so you're going to find the tennis ball detections -6 then tennis ball detections -6 and then you're going to find the training and testing and validation uh folders uh to train the model it's going to be simple with alter litics so the way that we're going to do that is that um we're just going to have um write YOLO then uh mention that the task is going to be detect which is going to be uh detection because it has multiple other things like key Point extraction and uh segmentation so we're going to specify that it's U only detection and we can uh specify the mode of it which is going to be training we're not going to be detecting and we're going to specify the model itself which is going to be YOLO V5 and I'm going to choose a big one which is going to be uh this one. PT and I'm going to also specify the data set path which is going to be a location uh Slash data yaml and data set. location is from Robo flow you're going to find it here um uh so it's a data set and the data set has a location attribute so that's how you do it and you can also specify the epox to be as much as you want I'm going to keep it 100 and you can also specify the image size um I'm going to specify to be a 640 and you can run it right here um and uh yeah you will have the output uh but I'm going to run it on Google cab because it has uh free GPU so um I just moved it just move the code right here uh the same thing copy pasted it and uh run the same thing it run the model it trained it and then it produced another folder called runs you open the runs you have the tect you open the tect you have the train and then you have multiple um multiple metrics that you can open up and see but the ones that we're going to use is the weights uh you're going to have the best weights and the last weights download them both like by pressing this download button I already did um and yeah you just download them then copy them and then uh paste uh like paste them right here we can create a new folder called models and you're going to paste it right here so let me uh let me go paste them and come back so I pasted them right here and renamed them to YOLO V5 best and YOLO V5 last uh we're just going to use one of them uh so for me I think the last was doing a little bit better than the best so uh but it might not be the case for you so you can play around with both and see which one works best for you um and now we can use this model again for the inference and see how it goes so for the inference we can uh we can use this model by specifying its path so we can just do models SL YOLO s YOLO 5- last last. PT and we can also specify the confidence to be uh 0.2 I think it's uh something very close to 0.2 right now but yeah we can uh run it again on the video and we can see the results again this is going to take a little bit time to run on all the video so um I'll cut the video and come back when it's done so it's done now and we can go back to our um folder to the output folder that we have which is going to be runs detect and then you have a predict for and you can now see that the ball is being detected way more than we had in the past um but you're going to also notice that only the ball is being detected and not the players not anything else that is because we only trained on tennis balls because that was the data set so unfortunately right here we're going to have two passes one with yolo V8 to detect the players and one with yolo V five the trained one to detect the tennis balls and yeah but it's going to be doable it's just going to take a little bit more time if we look at the previous YOLO V8 output video we will find that for each frame we get a list of bounding boxes and those bounding boxes are not the same between frames as objects move from one frame to another we want to have the ability to match the player's bounding boxes to the bounding boxes in the next frame because right now we just have the X and Y positions of each we have nothing that explicitly states that this is the same object between frames this bounding box matching is called object tracking where we say that this bounding box is the same object as this bounding box in the next frame this way we can analyze the movement of the same players across the video ultral litics also gives us the ability to track instead of predict so we we can do track right here and we can return it back to yov V uh 8 and it was Yol V 8X run it it's going to take a while so again I'm going to cut the video and come back later okay so now we're done uh we can open it up again we can open the runs detect and this time you're going to find a track it's not going to be predict it's going to be a track so if you open it you're going to find that each player has a different ID so this is ID 1 and this is id2 and no matter the frame that we are in we can always say that this one is the player one and this one is player two and this one are persons with other IDs as well um this is going to be very useful for us and very helpful for us to all always say or to always detect which is which which person is which person with ease um and yeah we can also track the balls but since there's only one ball in the field in uh in in most of the times then we don't need to track the ball we can just detect it and yeah we only know that there's going to be only one so uh no need for a Tracker right here and before we proceed while we're training stuff um in the in the final output video I showed you that we also are detecting the key points multiple key points in the court uh those red key points that you see right here uh those multiple key points are going to be helpful for us to detect how much a user has um has covered in kilometers and how much um and meters and how muches the ball have traveled and it can also detect whether the ball is in or out and yeah it is very useful for us to detect those key points um so the way that we're going to do it is that we're going to also find uh a data set and I have found a repo that does exactly the same thing which is uh detecting multiple key points uh in the same are view of the tennis court and they also provided us with a data set and they also have a model for it but we're going to train our own so the uh the data set right here they have it so you can click on it and uh you can download it easily from um from the uh from the drive or we can just uh W get it um so the way that I'm did it is that I used WG to download this so um you can just copy this uh download WG function uh WG command and we can write it on our own so to train this we're going to have another um notebook that is called tennis court key points training. ipynb and we can download our data set and this is going to download it in a zip format so we will need to unzip it so we can write unzip tennis court de data set. zip and then we can start uh with our code so we can do a markdown right here and say start code uh for this code we're going to use pyour so we're going to import pytorch and we're we're going to also import um the data set from pytor so data set torch. utils do data import import data set and a data loader and we can also import um the model itself which is from torch Vision import models and we can also import Transformers transforms uh which is going to help us transform our data set a little bit so you can run this and yeah uh first we can specify the device that we're going to use so we have device equal um and you can see that it autofilled so you can see Tor device we're going to set it to Cuda if Cuda is available which is uh GPU else we're going to put CPU and let's run it and afterwards let's create our own uh torch data set so the T the data set is where most of our code is going to be here um and the other things is going to be a little bit easier so we can create a class for this data set which is called key points data set we can call it whatever you want it's going to it's going to inherit from the data set that we uh imported in the torch so it's going to inherit a lot of the functions and uh uh things like that and we can also plug it in into our um training uh so the first thing is going to make an initialized function that is going to take the um image directory as well as the data file so before we proceed I wanted to show you what data that we got by downloading uh our data set so let me move again to collab and you're going to find that I here also use coolab for the training so um you when when you download the data set and you extract it as well uh you're going to find that you have images it's going to take uh bit time a bit of time to download to open those images because we have a lot a lot more than 500 the time um so you're going to find the images are pretty similar to what we have and what we saw before so nothing interesting is here but you're going to find also a Json file and this Json file has the ground truth positions of the uh keypoints so first you're going to find the ID uh give it a minute it's just acting up because it's a lot of data so you're going to find an ID and um let an ID which is the image ID and then the key points KPS is the key points and you're going to find 14 of them those 14 are going to be X and Y positions so actually in reality the model is predicting 28 um 28 numbers it's not 14 numbers so you're going to find the X and Y positions of each one and yeah that is it so given that we can jump back to the code and create our own uh continue creating our own data set so we can um save the image directory here in the uh self we can also open up the uh data file which is open data file and we're going to read the data so uh it's predicted this wrong but we can import Json because it should it's in Json format so we can just import it as a dictionary do load and give it f and yeah that is it so in the data set you're going to find the dictionary that I just showed you on the Google collab and then we're going to uh write um a transformation um a transformation function uh this one is going to standardize the image into uh the same size and it's going to normalize it as well so you can write it as transforms equal uh transforms which is what we imported from torch and we can compose then we can give it a list of Transformations that we want so the first one is to transform it into um a pil image do to P image because then we're going to resize it to make it um 2 224 by 224 and then we we are going to uh make this into a tensor and lastly we're going to normalize it so that the values can um can be easier to train the values of the pixels can be easier to train the next function we want is to define the length of the data set so you can have the uh length of it it's just the maybe the length of the data that we have the the length of the dictionary that we read um and yeah afterwards we're going to create a function that is going to get um an item so in the training we are going to uh get items one by one or batch by batch uh and then train it on this batch and then um wait for the other one to come so the way that we're going to do this is we're going to do a function called get item that's is going to to take an index and is going to return back the image and the key points so let's start by um getting the annotations of it so we can say items. index and then um we want to get the image so we can write image uh CV2 we're going to read the image and read and then um we are going to specify the path so the path is going to be self. image directory then slash then it's going to be item do ID because if you remember we have the the ID is going to be the same name name as the image and it's followed by PNG now this will read in the image but we don't have the CV2 imported so we can import it using here Port CV2 and then we want to get the height and the width of the image so we can do that easily by that and we just can call the shape function um and CB2 reads the image in a BGR format but we want it to be in an RGB format so we can have it like this CV2 do CVT color from RGB to BGR um afterwards we want to transform the image by the Transformations that we defined above so we can have this self dot transforms and given the image um then we want to have the key points so the key points are going to be item and we saw that it is in the KPS um key and yeah uh this one is going to be um a list but uh for in order for us to to to train it we need to make it into a numpy array um so we can import numpy at the beginning then go back here and write np. array and and have it like this and also we noticed that it was a 2d array so we want it to have it only 1D so the way that we do it is we make it um flatten and flatten is just going to convert it from 1D to 2D and at the end we just want it to be a float um so it's an array of floats so we can just specify the type as np. FL 32 um now now the key points we regarding that the image has the original um size and it's not 224 by 224 so we would need to map the key points to actually show this like to actually be m map to the same um to the same um positions after the resize so the way that we're going to do that is we're going to do like very simple cross multiplication where I say if the where the width where the original width was for example the position 240 so if I made it uh 224 the width what is going to be the um what is going to be the new location so the way that I'm going to do it is that I'm going to multiply 224 by 240 divided by the width and this cross multiplication will give us a the end uh or end position so you can do that by specifying key points and you can take the second uh like um yeah so you can take the first element then the third element then The Fifth Element so on and so forth so you can have the width this this is how you can do it and the way that you're going to do it is you can multiply it by 224 [Music] um and then you divide by width and this is to adjust the x coordinates um of the key points we will need to also adjust the Y coordinates so we can start off with one not zero and um and divide by the height and not the width and yeah and now we're done so you can just return the image and the key points and your uh data set is ready so we can afterwards um we can specify the um we can initialize this uh data set we can specify the uh training and uh validation uh data sets so you can do like this uh data set equal key points let's run this um so you can have the key points data set you can specify the image directory which is images and then you can specify the Json file which is going to be data slash um and this data and this Json file of course has the annotations so it's um train. Json and if you have any doubts about those locations you can just go back here and see it uh you can see that the data has images and we have a training Json and a validation J Json and this is the training data set so you can also have the validation data set by doing this and specifying here the uh Val Json and yeah now you have your own data sets just to use them in the um in the training we need to put them into loader so you're going to do that by writing it like this and you can specify that data loader give it the training data set specify the batch size we can specify it as eight here and we can also Shuffle the same thing that we uh that we did for training we can do for validation and yeah and now we're ready so now we want to uh now our data set is ready we want to create our model so to create our model we just need um um so yeah so to create our model um we're going to initialize it first so we're going to say that it's going to be models and we're going to choose reset 50 uh and we're going to choose the pre-trained equal true now this is going to uh use and download a pre-trained convolutional neural network that was trained on other images and um this is this is going to uh download the model and have it in the U model uh um variable uh but this reset 50 was not trained for keypoint extraction so uh we will need to replace the last layer to reflect our own uh needs so this is called like fine-tuning where you get the reset 50 which is a a trained model and you utilize most of its layers again uh that was trained on another data set uh on this problem so the way that we're going to do it is that we're going to specify the last layer which is the fully connected layer the last one and we're going to say that this layer is going to be a linear layer which is just going to be a flat layer with um just the features uh the input features the the features that is going to be inputed here and the output is going to be 14 * 2 we will have 14 key points and each key Point has an X and Y which is going to be uh 28 at the end so this just replaces the last layer of the of the neural network um so now our model is done uh the next thing is that we want to move the model to the uh correct device so whether we have uh this device is in Cuda or CPU we want to move this to Cuda or CPU and the last step is actually training the model so we can so we can start training the model so first thing is that we want to define a loss function and since this is going to be a regression uh function then we're going to use uh mean squared error loss so we're going to have the Criterion set to uh torch Ms eloss uh then we're going to specify also the the optimizer and you can specify the optimizer to be whatever you want um I I chose Adam and I am going to also um choose learning rate of 1 E4 now we can have as much EPO as we want so I'm going to choose only 20 and we're going to Loop over each Epoch in range EPO and I am going to get the image and key points um again I'm I'm not running it locally because I'm going to run it on the um uh on the Google collab but the rest of the tutorial I'm going to be running locally so that uh you can see things step by step you can just say enumerate which is going to return the index and we can have the train loader and yeah the train loader is going to return back the image and the key points so we are going to have the it's images it's multiple images so you can have it the images to device we can move it to the device we can also move the key points to the device that we want we then want to flush out the optimizer so that it has um like to flush show the uh gradients um and then we want to predict uh the um we want to predict the model uh the output uh like we want to use the model to predict the key points on the images that is provided and after we predict it we want to um U like uh calculate the loss which is going to be like this we give it the outputs and we give it the key points and it's going to use the mean squared error loss to calculate our loss and at the end we want to do a backward propagation and we want to take a step an Optimizer step just to do a learning step and yeah and just for logging purposes we can log every uh 10 steps which is going to be like this which is if I is divisible by 10 then we're going to uh print out the epoch we're going to print out the I and we're also going to print out the loss so you can go ahead in your uh Google collab or if you want to train it locally train it locally um and run it so you can you can see that it ran and the loss was very high now it's decreasing a little by little and at the end of the day you're going to need to save it so after you run it uh you'll need to uh save it in and download it so so to save it we just going to run torch Dove then give it the model dot State dict and you can uh specify the output path of it so you can have the key points model. pth and you run it and save it so I'm running it in collab so you can save it and then you can download it like we downloaded the um YOLO model and then you can come paste it again um in the models directory and paste it uh now we have the other model ready so now that we're done and our models ready to be utilized we can close down all this we can save it you can download you can close all this up and now we can start uh coding our tennis analyzer so we can just specify the main and write it right here and we can have the main function and we can do a print word right now and then we can call the main so after we set up the main. pi uh we just want to start with the uh basically reading the video because we won't be able to we we don't want to run run it like that um we basically want to run it um um like frame by frame uh so yeah we're going to need to read in the video as frames and then have the ability to also save it as frames uh so let's start with that uh we can create a new folder called utils and inside the utils uh we're going to need to create a a folder a file called video TS okay uh inside the video Tils we're going to import CV2 because this is the way that we're going to read and save our videos we're going to create a function called read video it takes a video path um and then it um opens um like it it opens a connection to the video and right now we have an empty uh list with frames and we Loop uh and we're going to Loop over this uh till the frames are done so what we do is that we use c. read to read in the frame and then we have the return this return is going to be false when the when the there is no more frames to read so if there is so if it is actually false we're going to break this Loop otherwise we're going to append the frame to the list of frames and then at the end of the day we're going to release it and return back the frames um we are also going to want to have a way to save the video so that we can see what we do so we can Define save uh video that takes an output video frames and it takes in an output video path and it's going to save this as um um it's going to yeah it's going to save this as a video so yeah I'm going to utilize what uh GitHub co-pilot has suggested but I'm going to edit it a little bit so first what the format that we want is going to be MJ PG and we want to Output it as a 24 frames per second and yeah that is it I'm going I'm just going to remove that as print and yeah this is it um in order to expose those functions to the outside uh to to outside of this folder um we need to create a new file called init.py and whips I just need to create replace this with a DOT and in the init.py we just like um mention all the functions that we want to have um available outside um outside the util so what functions we want to expose outside the utils and we write it here so we can write from video utils we can import or read video and save video um and after we read and save video we can go right here write from utils import read video and save video we can also put it in Brackets and because we're going to have a lot more functions in the utils so make it neat and organize from now and yeah we can uh then uh read in the data set read in the the video sorry so we can specify the input video path input video path is equal to input videos. input video. MP4 then we can readin the video frames by using the function that we have from the utils and afterwards um we want to save the video so we can have actually we can create a folder called output videos that is going to have our output um inside of it so we can have the save video uh function that we also have in the um yours uh you can uh give it the video frames and you can also create the give it the output path so uh yeah let's run this and see if it works so there is an issue with the output format so let me go and check it real quick so yeah this issue was because I wrote MP4 at the end and it should beavi so if we run it again you shouldn't get this error and yeah so you can open it up here in the output videos and you're going to see the same video again uh outputed to us okay uh that's set up that is all set now so we can start defining our uh trackers basically so you can create a new folder called trackers inside of it you can create a file called called player tracker. py and inside this player tracker we're going to create the uh tracking for the players so uh we're going to import YOLO so from ultral litics import YOLO and we're going to specify the class which is player uh tracker we're going to specify the uh init function and it's going to take in the model path which is what model are we going to use and yeah and we can just do self. model equal YOLO model path and to detect on a frame uh we can use the same code that we did earlier so we can have Define detect frame and we can just do the same code that we have earlier so we're going to take a frame which is basically an image we have the uh results from it and then we run the model on it so model. track because we want the players to retract here and we give it the frame and we also tell it to P like the persist function we're going to give it true now persist here it tells the model that the tracks that this is not going to be um just one individual frame I'm going to give you another frame afterwards and you should be able to persist the tracks within those frames um because we're not giving the whole video at one time so we need to uh set this up so that it can remember the tracks that were done before um then uh we're just going to uh need to have this um map of lists um like this map of IDs to names so um it is in the results so you can have the results do names and it will give you the results we're going to have the output in a dictionary where the key is going to be the player ID and the output is going to be the bounding box and we're going to Loop over each bounding box in the results and we want to only choose the bounding box that are uh people that have a person inside of it we want to exclude um clocks we want to exclude rackets we want to exclude everything except people um and we're going to have another tracker for the balls so we will exclude that also here um so yeah we are going to take in the zero with option because it's only one image and we going to run results of boxes and we are going to get the track ID and this is the number that you saw on top of the uh people uh in in the track example that I gave you before we're going to make it as an integer then we going to take box uh. id. to list and this is the Tracker ID and we want to also have the bounding box so we can also do a result which is one result um we can say that box and we want it to have it in the xyxy format uh which is X minimum y minimum X y maximum y maximum and we're going to keep that consistent throughout this video and the uh we also want to have the object class name so the class ID so we can say object CLS ID as box do CLS and. to list um if I can write list correctly and we can also have the zero one um and we want to map this ID to the name so that we know what we're looking at so we can have the object class name as ID to name of object ID and we can simply say if the object uh name is equal to person then we are going to uh take it we're going to have the track ID and then put uh the value like the key as the track ID and the value as the bounding box and if it's not a person then we're not going to do anything we're not going to take this so we are going at the end to return this um and have uh and and get the results of one frame we want to also create a function that detects multiple frames so we can have detect frames and this is going to have the self and it's going to take frames it's also going to have um it's it's going to utilize the detect frame function so what we do is we have the player detections and for frame in frames and we want to do this we want to detect each frame and then put it right here and append it to the output to the output list and now we are done so you can have this player tracker uh ready for use and it will return back the list of like the list of detections for each frame and to also expose it outside this folder we need to have the init the init one uh init file and you can write from player tracker import player tracker you can put a dot right here so um afterwards we can detect from we can import it so from trackers import player tracker so we can we can initialize the player tracker first so we can have the uh player tracker as player tracker and we give it the YOLO V8 X1 as the model path and yeah now we initialized it so let's use it to get the detections so player detections is equal to player. detect frames and we give it the video frames and that should be it so here we are reading the video Read video and here we are detecting players okay so now we're ready um but the video won't have the uh detections on top of it so we will need to create a function that uh draws the bounding boxes on top of the video frames so the way that we're going to do that is that we're going to add a new function into the players uh tracker that is going to add those um drawings on top of the frames so you can go back again to the player tracker then go back here whoops and uh you can have a function called Define draw Bo boxes BB boxes bounding boxes so it takes in the video frames um and it takes in the player detections um so uh we are also going to have the output frames as an empty list let's call it output video frames and I am going to Loop over the frame and the uh output dictionary so we can have the frame then player dict IN Zip and give it the video frames and the player detections zip allows us to Loop over two lists at the same time so that we don't have to keep an index of them um and then we just want to draw the the bounding boxes so here we are going to draw the bounding boxes so we are going to Loop over each track ID and BB box in the player dect items and let's import CB2 and see how it goes so the first thing we extract the X1 y1 X2 Y2 from the bounding box and then we draw a rectangle that has the like the corners of it which is going to be X1 y1 and the other corner is going to be X2 Y2 um and we are going to specify a color um and we can make this color uh right here um so this is an an RGB format uh so uh yeah and yeah the two here means that it's not going to be filled it's just going to be the outside uh borders of it um we also want to put in some text thatth tells us this is the player and what id is it so we can just uh do something like this CB2 and then uh put text we give it the frame and then we give it the string that we want to print out so we have the player ID and then we want to put in the track ID uh we also want to have the position of it so the position is going to be int of the BB box of zero which is going to be the minimum and it's also going to be the end of the BB box of one which is going to be minimum y so this is minimum X and this is minimum Y and we want to also have a little bit of a buffer so let's do it like uh minus 10 and yeah uh we can also specify the um the font of it and the the font size and the color of it as well so we can have this and let's increase the size a little bit so to 0.9 and this is and this should be it so um and yeah you can also assign it right here like this one you can assign it frame equals frame but it actually writes on top of the frame so we don't have to do it so we can do this like we can do it like this and afterwards we just out put this again as a frame like append it to the output and then at the end of the function we return the output frames um so yeah let's also uh return back to the main um use the player tracker right here so we are going to uh be drawing some things so uh draw output and the first thing that we are are going to draw is draw player bounding boxes and we are going to draw uh the bounding boxes give it the video frames and give it the player detections uh the video frames right here we are going to uh yeah we can call it here the output uh video frames and then save it again so yeah let's run it h there is an error uh trackers do no modules called it's called like that so trackers do player tracker um ah whoops I misspelled it right here so it's tracker um you can clear the output and run it again so it is going to take a while to run this all um so yeah I am going next to create um a stub where we can save the output of this tracker uh into a file and then read it instead of predicting it again and again and it would help us in development it will help speed up the development a little bit but uh yeah this is going to be the next step so uh yeah I'm going to cut the video for now and come back when the run is over so I've come into an error right here and it seems to be that I have missed the bracket so if you did the same mistake uh just come here at the add the bracket for the int and make sure that your brackets are correct and yeah I think I'm missing one so yeah right here and so everything is good now um so yeah you can uh run it again and yeah I'll come back when it's uh when it's done running okay so now it's done so let's go back to our output videos and open this and you're going to find the output very similar to what we had uh in our uh in our ultral litics output which is going to be the player and then the ID I didn't put the confidence because I don't care about it right now I just care about the IDS and the bounding boxes so it all looks good now I just want to add another thing in the player trackers which is going to be uh basic basically uh saving the output so that we don't have when each time when we run we don't have to run the detector again we can just uh take it take the output if there is any so we can create um a folder called tracker stops tracker stops like this uh whips and we are going to be saving the output of this uh detect frames um in a pickle format so let's import pickle and right here in the detect frames uh function uh we can take two more inputs which is going to be read from stop it's going to be either true or false um if it's false then we're going to run it if it's true then we're going to uh put uh put it into a stop then we are going to take a stop path and uh this stop path is going to be none as default um so yeah if then if we have the um if we have the stop path then if we have the if we have the stop path which is going to be if stop path uh is not none uh then we are going to save it basically so we are going to open it and then uh pickle dump the player detection uh inside this path um we are also going to check if the read from stop is actually true and if it is um we are going to read the Stop and not uh basically detect each frame so we are going to check if read from stop is true um and the stop path is not equal to none um then we are going to open it then load the pickle and then return the player detections and yeah this is it right now we want to add those two in so in the detect frames we are going to give it two more things uh we are going to read from stub is equal to false right now we're going to make it false because we don't have a St and we're going to generate it after this and the other one is going to be stop path and we are going to tracker specify the output one so specify the output path which is going to be tracker stops slash then player detections do pkl which is the pickle format pickle extension so right now we're going to run it again and it is going to detect the frames but afterwards it's going to save it in the stops so that we don't have to run it again uh so let's run it and we're going to have to wait a while so I'm going to cut the video again but hopefully this is the uh last time that we're going to detect it um with the players we're going to do another one with the balls but we're also going to run it one time and then the Run should be a little bit faster afterwards so yeah see you then okay so now it's done and you you can find them in the tracker stobs you can find the player detection. pickle so all what so all we do right now is just change this to true and if we run it again then you are going to find that it takes way less time so um give it just a couple of seconds and it's done so you can go also to the output folder and every time it it overwrites the file and you can see that we have the same out it uh now we want to do the same thing uh the same like we want to track the ball so you can copy it then uh copy the player tracker and uh then you can paste it right in after pasting it you can uh rename it to a ball tracker like that and after you name the ball tracker you can uh rename also the uh the class name and we're going to change the detect frame a little bit so the detect frame right here uh basically um is checking for the if class name is equal to person and since that we have only one class then we can delete this if function so we we can delete this uh if statement and we can also delete this uh checking for the names and we can also delete this because it's not used anymore and yeah we can rename the player dict to Bal dict and let's also have it right here and H it here and let's not track it since we only have one ball so we can just predict it we can set the confidence to uh 0 uh uh like 15 uh I think that is um that was good enough uh and 0.2 is also working fine um so uh we're going to we're not going to have any tracks so we're going to have only one ball but we want to have the output consistent so we're going to have um just one which is track one and put at it the result and yeah we also want to have um this detect frames to uh to stay the same but we don't want to have it to call to be called player detections we can call it ball detections and rename each instance of this and now we're done so the last thing that we are going to need to change is the draw boxes um so the draw boxes right here is going to be ball ID and we can also call it here ball dict I don't think that we're going to change anything else than the naming um but yeah that is it uh yeah if I didn't miss anything I think this is it for yeah this is it for this class so we can go here we can also import it in the player tracker we can import ball tracker we can go back here and we can import ball tracker as well let's um um so yeah let's detect now the uh balls so we can call detect players and ball we can also have a ball tracker which is going to be ball tracker and we are going to give it this time the model path to be models and YOLO 5 Dash um slash last do pt and we are going to detect also the frames so like we did in the players we're going to do with the uh ball as well so we're going to call it ball detections we are going to read from the stop if it exists if it doesn't then we read it um then we have the uh we can call it the ball detections and save it as well and yeah afterwards in the drawing the outputs we can also draw the uh ball output like this and give it the uh ball detections and yeah that should be it uh we can now run it we'll have to wait a little bit for the ball tracker to run the first time um but yeah I'll come back when it's finished yeah before we I run into an error that tells me that this doesn't exist so you can change it to read stop from false till we have the first initial uh stop so I'm running into another error again so let me uh figure it out and come back to you afterwards uh so I'm back and I have this prectical true this is uh for only tracking so make sure to remove it and uh save it again and run it and hopefully this time it won't crash again um just give it a minute make sure that it's running and then I'm going to cut the video if it is yeah it is running so I'm going to come back when it's finished um so yeah see you then so now it's finished and we can go back again to the output video now see it and you can see that the ball is being uh tracked as well uh so let's give the ball another color than red so that we can differentiate it quite easily and we can uh do this by going to the ball trackers then go to draw boxes and give it here in the RGB section you can just give it another color and I am going to choose a yellowish color like this and this time I'm not going to uh I'm going to use the uh stubs so that I don't have to run it again so like this and you can just run it again and yeah this time it should be a very fast run so I'm not going to cut the video um yeah it's done already so you can now see the results the ball is in uh yellow you can also change the writing to yellow uh so like here you can also change it to be yellow and yeah that is it let's now detect the uh Court key points um so in order to do that let's also create another folder that is going to be called Court line detector and we are going to have um a file inside of it that is called Court line detector. py so this is going to read in the model that we have trained and then use it for prediction so in order to do that we are going to import uh torch because the model is written in torch and we are also going to import torch uh vision and uh we are going to specifically import from it the transforms uh to apply the same Transformations that we did earlier on this uh uh on the images uh we are going to also import CV2 to read in the images if we want and to do any uh thing uh any uh uh any manipulations if we want and yeah let's start uh writing the uh class so we can have the uh whs we can write the cour line detector and we can start off with the init function and we are going to take the model path as an input also um so let me just fix that and you can have the self model equal models do uh we're going to do the same one so we're going to have the uh models. reset and I did not import it so I need to import it first um and we are also going to replace the last fully connected layer with uh the one that um that is going to have the uh 28 uh key points um so FC is equal to um torch do NN do linear and then it's going to be self. model. fully connected uh in features and then have the 14 * 2 we can just write 28 if we want but I just write this to remind myself that it's 14 key points and with uh with X and Y um then to load in the model I'm just going to do the self. model. load State dict uh we are going to load in the model and I am going to map it to uh look uh the CPU because because right now I have a CPU machine and I don't have a GPU inside of it so I'm just going to map it to a CPU so now we have created the model and we have loaded the uh weights of the model inside this model so that we can have the same uh output that we uh like we can get the predictions of the train model that we have um the last thing that we are going to do in the init is going to specify the transform function which is going to be the same thing that we did so you can actually go copy and paste it so you can have it right here if you have it ready you can just copy paste it if you want uh like this and it should be uh fine um then we are going to create the predict function it's going to take in the image which is a frame um we are going to basically predict the first image and not predict it again because the camera is not moving so the key points won't change positions so we are just going to uh run it on the first frame and that's it um so what we do is that we have the image uh we just make sure that it's in RGB so we can just convert it from BGR to RGB and then we um apply the Transformations on the uh on the image so we just do the self. transform and then give it the image RGB and we want it to have uh like we want to UNS squeeze uh UNS squeeze it and yeah we we then want to run the model on it and the way that we do that is that we do it like this outputs equals self do model and then give it the image T um and you have the key points right now in the outputs so you can have the outputs dot squeeze uh then uh two list by the way UNS squeezed right here just means that if you have the image like this um you just put another list on top of it so it's it's going to be like this uh this is uh on squeeze uh this is essential because um when a model tries to predict things it it usually takes in a list of things to predict on and if you are just predicting on one thing like this one image uh you need to put it into a list and the output is also going to be a list so uh we also want to have the zero um like you want to have the zero position of it you can just write dot do zero or you just can squeeze it and by squeezing it it just removes the list so back to the key points thing where we squeezed it um we can uh move it to CPU and then we can make it into a numpy like this and um we can also have the um like like the key points right now are um are the positions for the 224 by 224 so we need to map it again to the um to the original image size and image width so the way that we're going to do that is that we're going to have the original height and original width and we can leave it like this or we can just take the first two and uh do it like that um so the way that we did it when training is that we did it with cross multiplication and this is how we are going to do it again right here so we are going to get the key points then get all the width of the key point so every uh every other element uh starting from zero um then we are going to multiply it by original. height over 2440 this is going to map it to the original width and the original height um we are also going to do the same thing but for the heights this was for the width and this one was for the heights uh this one starts from zero and this one is going to start from one every second element starting from one and yeah this should be it we can have the we can just return the key points and there you go that is how you um uh that is how you uh get the results uh from the model before returning back to the main we also want to draw those draw those predictions on top of frames so let's do that while we're here so we can just have the Define and uh called uh draw key points and we can take an image and the key points that we have just returned um we can uh loop over each uh one each um uh each key point that we have and we want to Loop over 2 by two and this is because it is going to be XY then XY then XY and one XY is one position so we are going to Loop over it from zero and to the length of key points but the next step is going to jump two so instead of I being one it's going to be two so we're going to skip one in the middle so we're going to make x equal int we just make sure that it's an INT because like when you specify a pixel position um it must be an INT and it won't be fractions it won't accept any fractions and then it's going to be the I position and the Y is going to be the I +1 position um afterwards we just want to display two things we just want to put the dot in the place and then we want to have the um key Point number on top of it so you can have the CV2 dop put text and you can write it on top of the image then you can write um the um like you can write the Str Str and then I / by two and this is integr division so it's going to be an integer at the end um and X um where minus 10 so this is going to take the X position of it and the Y position and we're going to make a buffer at 10 so we can put it on top of the um uh on top of the point so we are also going to specify the font and the U uh font uh the font sides as well so let's do that here so font we also going to uh choose the simplex one we are going to choose uh font size then we are also going to choose the color this is in a BGR format so this is or which is red um and then we are going to specify this and at the end we want to put a circle on the image in the XY position and its uh radius is going to be 5 pixels and um we are also going to make it red and then uh we are going to specify that it's going to be filled so1 means that it's going to be filled and at the end of the day we are going to return this image now we are done with drawing on one image but uh we also want to have a convenient function where we can take a list of images which is all the video frames and run all uh the um and draw in all of them and return back the results so we can have the draw key points again but on video and this takes in the video frames then the key points and it has the output video frames and yeah it Loops over the video frames and the key points uh actually uh yeah actually it did it uh like the auto to complete um the theong so you can remove this because there is only one set of key points and we're going to draw it again and again so the key points right here and we are uh going to put it right here in the key points so again always check what um what GitHub copet is uh is suggesting because it might might suggest things that are wrong and you need to fix it so uh draw key points right here is I think is working fine the draw key points it takes key points and then the frame I'm just checking it again uh so yeah it looks like it's working fine so yeah uh so before moving forward we just want to create an init file and we want to expose the C line detector again outside of this folder so we can um use it in the main function in the main. pi so we can write from Court line detector import Cod line detector we also want to have uh we want want to like we want to do the same that we did with the trackers we want to initialize it then uh detect on the frame so we can do that by writing here um cour line detection detector model we can qu model path we can specify its path first which is models Das line model. PT not sure that it's the correct name so I'm going to yeah it's not so just copy the name and paste it right here then um write the initialize the call TR detector like this and at the end of the day get the court key points and we are going to run it like this dot predict and give it the first video frame so give it only one video frame and it should return back the key points um and after that is done we can also draw it so to draw it um to draw it we can just write it right here so you can have it uh right here output video frames and you are going to do Cod line detector dot draw key points on video and then H give it the output uh the output frames right here give it the output frames and then give it the quot key points so we have the quot key points right here just give it right here and you should have the output ready I think this might not yeah I misspelled it um I misspelled it at the beginning so I'm going to um fix it uh I also noticed that here I'm detecting both on video frames I should be detecting on output video frames and yeah that should be it uh I'm making sure that here we're reading from stubs uh and this prediction shouldn't take long since we are predicting on only one frame so yeah let's try it out okay so now it's done so we can go back to the output video right here and we can see that the key points are being extracted correctly uh there is a little bit of error if you notice like for three it's a little bit shifted to the right um things like that but it is close enough for our use case uh so we'll proceed from here um yeah so let's return back to the code and let's um close down all those unnecessary tabs let's return back to the output and focus more on the ball detections so you can see that the ball is being detected way better than what we had originally but for example you can see right here the ball is not being detected at all and it's being detected like the one frame before and the one frame after so like this and this like you can see like um we can easily um we can easily estimate the position given the the the position before it was lost and the position after it was lost like this one we can estimate it to be for example the middle position um and if there were more than one position you can divide it by two divide the the the position by two and to estimate it uh luckily we can just uh use the interpolate uh function in the pandas uh library and it would uh fill out those uh those bolts uh for us so let's uh let's create that now so we can go back again to the uh tracker which is the ball ball tracker and uh we can write an interpolation function so let's uh let's start here so um we can just write here Define interpolate ball positions and it's going to take in the uh ball positions or ball uh yeah we have it called ball detections in the main so but we don't care about that right now um so what we do is that we only have one track so we don't actually need to have the track number I just need the list of uh positions that we uh that uh the bounding boxes that we get so I can just have the ball positions equal and then dot get one and if there is none and if there's uh no one uh that means that there was no detections I'm going I'm just going to return back an empty list and I'm just going to do a on line for Loop that is going to Loop over the ball position so now we just have a list of bounding boxes that is going to be empty when there is no detections the list inside is not is going to be empty when there's no detections uh we can convert this list into a data frame so that we can do the interpolation quite easily so we can import pandas as PD then we can just do pd. uh data frame and then give it the ball positions which is this list and also give it the colums uh which is going to be the column names it's going to be um X 1 then y1 then X2 Y2 and I'm going to put it in a DF ball uh positions uh data frame this is what it's called and then uh this just converts it convert the list into a panda data frame um then we just interpolate the missing values and the way that we do that is that we use the interpolate function inside of pandas so we can just do this and run interpolate and this interpolates um if something is missing uh between two frames or between mult multiple uh frames but um it's it's not going to interpolate positions at the beginning so at the beginning we want to make sure that there is no missing values uh not to crash our code so we can just duplicate the the the the the earliest uh detection we can duplicate it to the start of the frames so the way that we do it is just we backfill it so uh we do the bill function to do it um again this is just to handle the edge cases of um of not handling of not getting any detections the first frame but otherwise it should be fine um and then at the end we just want to convert this back into the same um format that we got the ball detections from so um we have B positions equal and then we can start it again we had one as in the track id1 um and then uh we can put X which is going to be from the for Loop the on line for Loop that we going to have and DF dot uh DF uh ball positions dot 2 nump I'm going to make it a uh an npire array and then I'm going to make it into a list and this is going to return back a list of dictionary where one is going to be the track ID and X is going to be the bounding box it is the same format that was uh handed to this function so I'm just going to return it back uh to the ball positions and the the reason reason that we Chang it to Panda data frame then back is that we just wanted to use those interpolate function in the pandas data frame so that we don't have to do it by hand um yeah so now we can just here we can just do it like this and we can call the interpolate function so we can have the whips ball uh detections we can call the ball detections and whoops it's it's called the ball tracker so ball tracker dot interpolate ball positions and give it the bulb detections and yeah we can run it again right now and uh see how it goes okay so let's open this again and you're going to find the ball is being detected way more so um so you like the last example I showed you here the ball was not being detected but now it is uh being detected although you can find here that the bounding box is not completely on top of the ball but it is close enough so it is uh close enough and if I run the whole video you're going to find that the ball is being detected in almost every frame uh this is going to help us way more in the analysis and makes it more smoother so uh yeah um so we're going to keep that and after we like before we continue I want wanted to I wanted to add the frame numbers on top of the frame so that we can store it and um Trace back uh how it how it functions so we can just write draw frame number on top uh on the top left corner so we just want to have the frame number on the top left corner of the video um and we can do it like this um I am going to explain the code that was written by um the uh GitHub cellet which is going to be uh looping over each one looping over each frame then putting um the frame and then the number of the frame and uh then it's specifying an exact position of it which is um X position of 10 and a y position of 30 and uh it's going to specify also the U uh the font uh type and the um font size and the color um so yeah we can also make the color uh like this uh but yeah we should have the frame number on the top uh on the top left corner of the uh video right now so let's run it again okay now it's done so let's open it and you have the frames on the uh top left corner like you see here I'm just going to minimize this a little bit so you can see the frame number um and it's in renting we can now like utilize it we we'll be able to utilize it um further on when we're uh tracing back anything that is buggy that is happening um but yeah it's going to be super useful here so let's see the output again and when you see the output we have a lot of people being detected and we just want to detect the two players in the field uh we can do that by selecting the players closest to the court and we can use the key points to to determine which players or people are closest to the court so let's do that right now so we can uh return back to the U main.py and we can see that like we have the quot key points and we have the player detections right here so we can uh basically add a function that can uh determine which IDs or person IDs that are closest to the court key points so we can uh so we can go to the player tracker and add a function that is called chosen filter uh players so we're going to choose the players and uh filter them based on on their uh proximity with the court so let's go back again to The Trackers and the players and let's create a function called Define choose and filter players this takes in the um the chord key points and then it takes in the player the player detections and yeah um we are just going to choose the players based on the first frame and not all the videoos so we can do that by just getting the player detections for the first frame and we can just make it like this we take the zero with elements of the player detections and we can also um do another functions called like chosen um chosen players equals self do choose choose players that takes in court key points and the player detections for the Press frame now let's create this function so we can have the Define choose players self then C uh key points and then the player detection frames um we can start by calculating the distances so the distances between each um uh each uh player and the court and we can do it by uh looping over over each track so track ID and we have the bounding box um in player let's call that player dict so because it's a dictionary so we can have the items um and you have the track ID and the bounding box um and then we basically want to get the middle of the box so we can do that by um like um getting the X minimum y minimum and dividing it by two to get the middle of to get the middle position and do the same for y and rather than doing it here we can also add it in the utils because um uh we are going to use that also elsewhere so in the utils you can go go ahead and create another uh file called BB box which is bounding box utils.py and uh basically I want to get the center of the box so we can have the Define um get um get Center of BB box and then I'm going to give it the box and I'm going to extract the XY positions add them both divide them by two and add them both divide them by two and return at the end the center X and the center y the only thing that was missed here is that they need to be int so that they can be integ uh like pixel positions so we can have it like this and it should be good to go um we also while we while we're here we also want to measure distance between any two given points so that uh we can measure the distance between the um uh between the point and the like the key uh the bounding boxes and the key points uh so the way that we do that is we're going to create another function that is called measure distance it's going to take point .1 and 2 um and 01 is XY and point 2 is going to be XY as well and we are going to return um this equation so this equation basically measures the distance between any two points and it does that by uh treating them as a right angle rectangle and getting the hypotenuse of it so just to visualize it you see here that um like um you're measuring the distance between a and c and the way that we do that is that we get the a squ which is the difference in X's uh sorry the difference in y plus the difference in um in x^ 2 and then you get the square root of it and this is going to be the distance between a and C uh the same thing that we are doing right here and multiply z z like the power 0.5 is just the root um and now we just want to export those functions and in it so you can have from dot BB box utils import get Center and measure distance and in the players we can also have like we can also import the this from the utils and we can import system as first and then system. pad. append um like one folder before that just because we want to go one folder back so that we can see the utils and then we say from utils import uh we want measure distance and get Center of bounding box now we are ready to use it so we can say get Center of bounding box and give it the bounding box and now we have the uh center of the bounding box and now we can assign it to the uh another variable called player Center and now we have it now we just want to calculate the distance between the court and the player and we're going to calculate the distance between the player and each key point in the court and see which one is the closest so we're going to have uh basically minimum distance um is equal to float and then Infinity then we are going to Loop over each key point and uh qu key points and we're going to Loop over two at a time because each one is going to be a one point um then we are going to get the uh Court key Point um which is going to be the X and Y positions then we are going to measure the distance between the player Center and the chord key point and then put it into distance and if the distance is less than the minimum distance we take the minimum distance and put it here um and yeah that is uh going to be it so at the end we have the distances do append and then uh we append two things we append the track ID don't know if that was written correctly track ID and then we app also the minimum distance so this way we have a tup we have a list of tups with a track ID and a minimum distance um so now we just want to choose the the track IDs that have the lowest distances between the key points so we can just uh sort them so we can sort the distances in ascending order okay and we can do that by this so distances do sort and then we sort by what value zero which is this one track ID and one which is this one so we sort the distances based on the minimum distance and then we choose the first two basically we get uh actually let's call it choose the first two tracks and you can have the uh like the chosen uh chosen players is equal to distance of zero and Zer which is going to be track ID and then we are going to get the next one which is one and zero which is going to be also track ID so this is going to be two players and we can just return back at the end of chosen players and now let's also filter out the chosen players so now we are going to Loop over the frames and we are only we are only going to give the detections related to those two chosen players so I can have like filtered player detections and I'm going to keep it as an empty uh list then I'm going to Loop over um the player detections and I am going to uh choose uh to Loop over each uh track ID in the player dict and I am only going to take the ones that are in the chosen players so this is a oneline for Loop that can be a little bit complicated but all it's doing is that it is looping over the player uh dick uh item by item and if the track ID is in the chosen players like this it is going to put the track ID and the bounding box and return back the dictionary afterwards after we have the dictionary we append it to the player D detections and then we just return back the uh output so now we have the output ready and we can go back to the main uh right here and we can uh now filter out the required uh players so we can have uh choose players and we can write player detections equal player trackers do shoes and filter players we can give it the court key points and can give it the player detections and now we have the last player detections we should now have only two players and not all the other people so let's run it and also see the output okay so we can open it again and like you see like other people are not being drone because we filtered the bounding boxes out out and only two players are being detected and this will also simplify our lives a little bit uh uh U simplifier life better here so in the uh so in the the final output video that I've showed you I've also showed you uh something like a mini cord that is being drawn um here so uh you can see that uh this is uh this is is um the court that is being done here and here's the position of the player one here's the position of player two and here's the position of the ball itself and this is what we call Mini cour I call it mini cour right here um we are going to be drawing on it and also we are going to make it um uh very similar to the original uh like um proportions of the um um of of the court so that we can measure distances and we can measure uh like the distance between the player in the ball and how much uh did the ball cover and how much did the player cover quite um uh accurately and yeah so we are going it's it's good for visualization but it's also going to help us a little bit when we are trying to determine um the distance between uh any two things uh so we can start by uh like uh drawing this and then we can move on from there so you can uh write uh tennis court uh dimensions and you have here the tennis core Dimensions that you uh see so this is called like the the the high one is called the double alley which you can see it's a 10 uh meter you also have the uh Center Line the service line the center line and uh you also have this part is called No Man's Land um so yeah we're going to be putting uh like some of those dimensions in constants and it's going to help us um understand what is the actual distance that is being uh covered by players in the ball so let's create a folder called constants constants and let's create inside it um an init file and we can start by just writing down a couple of things so we can write the single line width uh which is going to be 8.23 this is what is called the single line width I I just have the uh 8.23 which is this one and this is the double line width double ali um so it's 10.97 so we can also write it here double line width which is going to be 10.97 um there is a half quart line width um actually it's height um and it's going to be 11.88 and let me tell you which one that I'm referring to which is this one this is the half court uh height this is the U uh like the meter in meters uh you also have the service line width and it is 6.4 you also have the double Aly difference and it is 1.37 and again uh this is the double Al difference right there so between the um like the single allei and the double Ali it's 1.37 uh um those are just a couple of contents so uh so like constants that are going to help us uh map between uh pixel difference and meter difference so yeah you can just uh take them from here 5.48 and while we're here we also want to uh like determine the heights of the players themselves and you can actually actually uh like if you opened it you can see that uh you can see the names of those players and you can Google their heights quite easily and I have Googled them and I've know known them so I'm going to just write them here uh so player one height in uh meters and it's going to be 1.88 and the player uh two height in meters um it's going to be 1.91 those are also going to help us a little bit in uh mapping out uh uh like pixels to met okay so now we're done with the constants and now we want to draw the mini qu so we can make another uh folder called Mini and uh we can uh write the mini uh class mini cord dopy and yeah we can import CV2 and we can also import sis and we can uh from um we can sis. pad. append and we can also return back one folder then from utils actually let's not do that right now but we can import constants we can import the constants okay sounds good um we can def Define the class start the class which is the mini Court uh then we can Define the init function and we can give it frame this Frame is just going to be the initial frame of the video uh just to get the the size uh of it so um we can start by defining how uh like um we can start by defining how wide and how tall this this should be like the drawing style should it be uh large should it be uh small uh I'm going to set it like the ones that you see it right here uh but yeah feel free to play around with it a little bit so I'm going to set the self do drawing rectangle width uh which is going to be 250 it's going to be uh 250 pixels so uh this white rectangle is going to be 250 in width and it's going to be 450 in height and the difference between like the this padding difference um uh this uh this padding uh difference is going to be uh 20 uh I'm going to make it 20 pixels uh around so it's um there's going to be a little bit of padding between the qut itself and the white rectangle and there is also a little bit of a buffer between it and the top and the right this is 50 pixels so I'm just going to Define it right here so let's define the drawing rectangle uh height now again this is the uh White rectangle that you saw and it's going to be 450 um let's also Define the buffer between uh which is going to be this empty one uh this part and this part uh the buffer between the edges is going to be 50 and the padding itself uh which is inside of the white uh rectangle um so padding of the quart is going to be uh 20 pixels um so yeah so now that this is done uh we will just want to determine uh where should we put the um uh this rectangle like now we should uh find out the the exact X and Y coordinates we should put this rectangle on so let's create this function here so we can have uh Define set canvas background box position and it's it takes self and it takes a frame which is going to be the first frame then we do a frame. frame. copy uh just to not override the original frame um we don't need to actually here but let's keep it um we can uh Define the end of the frame which is this bit the this end bit to be uh the frame do shape 1 which is going to be the width of it minus self dot drawing like self dot sorry buffer which is going to be 50 so this is going to start uh 50 but from the end this is the end and we can also do it self Dot and Y and it's going to be self. buffer which is going to be 50 from the top and then 450 which is going to be this one so self the buffer then self do drawing rectangle height we also want to get the start and the end so the start is going to be end xus width minus the width and the uh start is going to be also the end y minus the height and now we have the positions in the uh self so don't we don't need to return it but we do need to call it so we can call set canvas background position and we can give it the frame awesome um now we just want to uh see the position of the court itself this this mini Court we we made the position of the rectangle the white rectangle now we want the position of the cord so let's also do a function for that so Define set mini Court position and uh it takes in self and actually we don't need anything else so what we do is that we're going to just add the paddings in the start and end and um like the the start X and YX so we can have the self. qut start x equal self do start X Plus self do padding C okay then we can do the same for the Y um and we can do the same for the index but instead of adding it we subtract it so that it can have uh a buffer right here and um we can also have and we can also have the endex like the end y uh so the n y minus padding qu um now the what we also want to do is that we want to define the width of the qut so the cour width um and let's call it Court drawing because it's just the drawing uh it's going to be Court end x minus uh the uh start X now we also need to call it like that in the init okay uh now we just want to uh specify those uh Court key points um the ones that you see here in Red so those Court key points are going to be a little bit manual so bear with me a little bit so we are going to write Define uh set put drawing key points okay I have misspelled it right here it takes in um nothing it just takes in the self and we can start by defining the key points so we can have the key points all at drawing key points we can have it all as zero and we can have it 28 times so this is going to create a list of zeros and it's going to have 28 zeros um so let's define 01 or 0 Z to be specific um so 0 Z is it's going to be just this this point right here which is going to be the start of the court so start X and start y so to do that we're just going to create drawing key points of zero and drawing key points of of one and it is going to be this start X and start Y and let's also put ins on top of it so that we can make sure that those can fit into pixel positions next we just want to do 01 and yeah end point1 is going to be this one which is going to be this one this is the end of X but still the the start of Y so you can have the drawing key points of two two two and three like 0o and one is for position zero 1 2 and three is for position um 1 which is going to be the end of X and the start of Y this is correct so let's move forward and let's also do it in the point two point 2 is going to be this point which is right here um and this is two2 quarts uh like with each other this is a half qu and this is a half qut and we want to be able to make it as like as proportional as possible to the actual uh meters that we have in the constants so we're going to be adding some conversion logic right here so this conversion logic is just going to be cross multiplication uh but since we're going to use it over and over uh let me put it in the the utils so in the utils you can have um a new file called conversions conversions. py and you can open it right here um and Define uh two functions which is convert pixels to distance and convert meters uh to uh pixel so we're going to have convert pixel to a distance to MERS so convert a pixel distance to meters so for example if the user moves from here to here how many meters did he move we have the amount of pixels he moved it he moved from uh like X and Y position position and then he went to X and Y position but we don't have the exact meters that he moved and we can do that uh by doing some cross multiplications and seeing okay so from this point to this point it is x m so from this point or from this pixel to this pixel it is it should be y meters we can do it with simple cross multiplication um so the way that we're going to do it is but we're going to take the pixel distance and then we are going to take the reference height in meters then reference height in pixels then we going to return it back here so let me explain it first so now we have the width of this um in pixels and we also have the width uh of this in the constants if you open it and yeah the double Ali line the this one it's in meters so you have this in pixels and meters so you can do any cross multiplications to find anything so for examp example the height in pixels like the width in pixels is let's say 20 and it's in meters it's 10.97 and uh now I want I want to give an arbitrary pixel distance and to change it to meters I'm going to take this multiply it by meters then divide it by 20 uh this is simple cross multiplication and this is um and we can we can actually uh do it in in just one line uh like this so pixel distance times meters over the pixels and Heights we also want to have the Define convert meters to pixel distance so we have also the reference height in meters and the reference height in pixels and we're also going to do some cross multiplication uh but but this time we're going to multiply by the pixel distance over the height distance and yeah so let's now expose it right here so from dot conversions import convert distance pixels and convert uh pixel distance and let's go back here um think this is should be constants constants why is it adding it SRE don't mind um then uh yeah we can add from utils um we should import convert this one and we should also convert uh pixels pixel distance to meter distance so yeah so let's continue down with the points um we have the uh drawing Point drawing key point of four and it is going to be uh the same x value as the as the start so the X didn't change so we can just um write it as int self. Court um start x uh but the Y value changed so we can have this as um so position two is going to be um so the start y plus uh two half quarts so the way that we're going to do that is we're going to uh take this start and then y then add it to convert pixels sorry we want to convert meters to pixels and then we have the Constance dot um we want to get the half quart uh height I just want to remember its name yeah so half quart line height um so this one and since it's going to be the full qut like from here to to the nut is the half qut so we want to do uh uh like two times this so we're going to multiply by two and let's not forget to add the constants do half cour and we want to give it the two references that we have right now so the reference in that that we are going to use is that we already have the cord width so we can uh simply write it right here and uh this cord twist is going to be constants do double um like uh let's also go remember the name it is the double uh double line width so we're going to give it the double line width and then we are going to give it the court uh chord drawing WID okay so now we have the point two ready um and we are going to basically converting a lot lot um of this um like meters to pixels a lot so I'm going to make it a separate helper function right here um it's going to be convert meters um to pixels it's going to take the meters and we are going to utilize this so you have this and you run it like that and instead of meters instead of having it a constant we can take it here and now we get the uh the distance in pixels when we have the meters so we can just do like this self and we can close down the bracket here and remove this um now it's a little bit easier for us uh moving forward to 3 um so point3 is going to be uh again this point which is going to be 2 plus the width of it um so let's uh let's do it uh it's going to be um drawing six so we're going to use the same x value as the first one like that and then we can um actually I forgot a t right here let's add it let's add it right here we can add the uh core drawing width uh since we already have it and we can keep the same um y value as the previous one so that we uh so we can keep the same y value as this one so uh we had two and we're going to keep the Y value to have it for three um again we're going to Loop over it again for point4 and uh 4.4 we have other uh another uh another uh 14 point that we want to do so what I'm going to do to not bore you I'm going to fill it out and then uh come back later so I've filled in the uh the all all the points right here uh the other the points and at the end I just assign the drawing key points into a a self variable so that we can have it uh here so right now we're just going to do the self and um going to call it again and right now we just want to define the lines so right here you have um apart from the points you have the lines so you need to understand which lines you want to connect with like which two points you want to con to connect a line with and for this we're going to also manually select those and let's start with it um so we can Define set cour lines and then give it the self and self. lines equal and if you look at this you have a line between zero and two and zero and one so we can put them uh right here so 0 and two and you can also have uh Z and one uh you also have a couple more uh but I don't want to go like I don't want to go through them all to not bore you you can just uh it's pretty straightforward uh so I'm just going to copy paste it and I pasted it and and I pasted the lines right here so again from 0 to two there's a line line from four to five there's a line and you can find that those are uh what we see so you can set CT lines also right like this and that is it and we are set with the uh like I think there is no more manual work right here so I think we are ready to maybe draw it um so let's draw it so we have the draw background rectangle which is the white rectangle that you saw so we can take in the frame and we can just uh first initialize the um um like uh we want to draw a white rectangle so we're going to do shapes like NP Zer let me import n p okay so now we have um the same like the the frame that we had um but we like the image that we have but with all zeros and this is going to help us um do this a little bit of transparency you can see that it's not an actual like filled color it has a little bit of transparency in it and this is the way that you do it um you do a dummy image uh which is going to be all zeros and then we going to add it to the image that is going to be the frame but with a little bit of opacity difference um so on top of this we can draw the rectangle uh which is going to be uh CV2 rectangle and then we draw on top of the shapes one and then we give it the start X start Y and then the end X and Y and we also want to make it uh uh like uh white so this is the white BGR and negative one is filled and if the numbers are confusing you you can just write CV2 do fi and it's going to be the same thing if you hover over it uh you're going to see that it's an INT which is going to be negative 1 um so we're going to have an output frame which is going to be the frame. copy and we have going to set an alpha which is going to be 0.5 so it's going to be 50% transparent um then we are going to set a mask which is going to be shapes as type bull so anything that is zero uh so it doesn't have a rectangle uh so it's going to have only the like the mask is only going to be two for the rectangle bits and we are going to get the output then the mask only the pixels that we want to have and we want to add it with weighted so that we can um add it a little bit with a transparency Factor we want to add the frame and then the alpha which is 0.5 we want to add the shapes which has the rectangle we just want to have the um the other transparency of the shapes and yeah zero is fulfilled and uh yeah so and then we get the mask um then we write output equal CV2 and then we just want to convert it so CVT color uh BGR to RGB you just want to convert it to an RGB format and then we return back the output of course this one draws on one frame only so let's draw it on all frames like Define draw mini cord then self and give it the frames we then uh have the output frames which is going to be for frame in frames and we just want to have frame equal self. draw rectangle and give it the frame and at the end we just want to append it so output frames do pen frame and return it back and this is it um we should now have the white box and afterwards we should be able to draw the key points uh so let's uh see the results till now and then we can continue so in the mini in the mini cord we just want to add an init py and this is also going to help us a little bit with exposing the function or the class outside of this folder and we are going to go here from mini Court import mini cour we also want to have um like here we want to initialize the mini court so now we initialize the mini court and and now we want to draw the mini qu um on top of the the frames uh so let me do that so draw mini qut and I will have it the output frames equal mini cord through mini cord output frames and yeah that should be it let's run it and hope that it works okay so we have an error um convert meters to pixels is not defined um I think I've misf felt something so let me check yeah I've misspelled convert meters to pixels I didn't I forgot to put the two so uh let me uh rerun it again and now I have another error which is uh I Mis also wrote the uint 8 so I'm going to fix it and uh come back to you I think is going to be here um and it's going to be you int 8 so let me run again so now it's working fine we can open this up again right here you can see that the image is a little bit weird because it's in BGR not in RGB uh but you see also the uh rectangle that is transparent uh right here um but now we need to fix the uh BGR one so I see here that we also convert it to BGR to RGB and yeah maybe we don't need to do that give me a minute yeah we didn't need to convert it again from BGR to RGB because it was RGB from the first place and now we have the um like the mini mini qut background setup okay so so now that we have the background rectangle ready we can just uh start drawing the court so to do that we can just write a new function right here uh Define draw Court uh which is going to take a self and a frame and we are going to Loop over each point so in range and uh starts from zero till the self do uh drawing key points and we Loop two at a time and uh X is going to be int um the self. drawing key points of I and Y is going to be the same but + one because it's going to be the next valuable uh variable and then we just add a circle on the frame with X and Y we are also going to set the radius to five we are also going to set the color and we are going to um make it filled which is1 now we should have the um now we should have the uh key points ready so let me return the frame and uh see the results but before seeing the results we need to call it so I'm going to call it right here self Dot and give it the frame and yeah that should be it so let's call it um there is something missing H yeah so I I wrote uh here uh like the two here in the length it should be outside the length so let's run again okay uh let's uh see the output right now so you can see the output of this is um um having the key points right here right here right here and it looks fine but the width or the the height of the um the height of the white box is not big enough um I think we can expand it a little bit it should be fine but yeah um we can also make it in red like that we can make it in red um but we can also expand the drawing rectangle height a little bit to be for example uh 500 so that it yeah it's running right now so let's see the output now the output looks fine we have the court we have everything that is ready we just want to connect the lines and this is going to be uh uh simpler so in the draw cord function uh we just want to uh now draw the lines so let's call draw lines and so for line in self. lines uh what we want to do yeah sorry four line in lines um so start point and it's going to be um this one it's going to be the uh drawing key points of line zero and this is also going to be uh the Y of it and uh we're going to have an end point also and it's going to take from a one so uh line one uh so each line is going to have like like we saw like two points zero and one and we want to just connect them both and we're going to connect them both with the CV2 line and we're going to give it the start position the end point and we are going to give it also a color let's say it's black and yeah that should be it let's uh run it and see the results so I made a small mistake earlier where I didn't multiply the position by two and this is because the the zeroth position is starts from 0 to 1 the first position uh starts from 2 and three and the second position starts from four and five so you can see that you take the position multiply it by two to get the X1 and then add one to get the Y so you need to do this the same here and if you uh run it here and and you're going to get the uh correct output uh wait for a while and you should be done any anytime soon okay uh now it's done if you look at it now you have the lines ready we just don't have the uh net line uh this is going to be an easy one it's just going to be the middle point of the cord so we can just calculate it real quick and and yeah to uh to draw the net we just get draw net so we need to get net start uh point and it's going to be self. drawing key points of zero because it's like the zero with position and um it's also going to be int which is like the start key point is going to be the zeroth position which is X which is going to be um like this x this X and it's going to be this x at the same time but it's going to be in between this one and this one so let me uh calculate it like get the positions of them both and then divided by two so we can have um self actually let's um let's use their equation but edit this to five so now we can have the start position so let's do the same for the end position let's copy paste it uh and it right here put the end position and make it start from position two instead of zero and keep the Y the same and lastly we just add the line and let's make it this color and make it start from net start and end at net end and yeah we can now run it and see the results okay now it's finished and you can see the chord is being drawn quite correctly and yeah that looks good um so let's continue with that before closing down this class uh we basically need to add a couple more functions and yeah so we set a couple of variables um so let's uh let's make functions to get them so let's get for example the start point of the Min um self and then we're going to return back the self uh self do Court start X self. court and um uh start whoops uh Court start Y and we can also get the width of the mini qu so we can Define get uh width of mini cord and we can return back the drawing width um we can also return back the key points if we want so let's get cour key points actually let's call it cour drawing key points and let's uh return the uh drawing key points and yeah I think this should be it so now we're done with drawing the mini cord and I think uh we are ready to go back here and uh continue um so right now we have the um this output and everything looks good but let's now detect the the ball hits so at this Frame exactly let's minimize this a little bit at uh around frame between 10 to 12 the ball was hit and we want to like get this ball hit frame and know which player hit the ball and the same here um so we can continue to detect the ball frames so now we just want to detect the uh bow um we we just want to detect the ball shots so we can create a temporary uh folder um that is called um basically maybe called analysis and let's make it a folder sorry about that let's delete it then let's create a folder called analysis and inside of it let's create um ball analysis. ipynb um now let's import pickle because we're going to read the pickle uh St that we have for the tracker right there and we are going to also import uh pandas we are going to import M plot lib pip plot and yeah I think that should be it um so the first thing is that we want to uh read uh the pickle file um so we are going to read it like that and let's call it uh ball positions and let's also refer to the correct path which is called tracker uh stops and ball detections do pkl let's run it and let's make sure that it threaded correctly so it it threaded correctly um we can also copy paste the The Filling of of the U the interpolation that we had so let me open up the tracker again and the ball tracker and let's copy this code and yeah let's copy this code and put it right here um this is going to be for uh aspd so yeah you just run it again and now you just have the um uh a pandas data frame that you can uh plot out um so yeah so the first thing is that uh we basically want to have the position of the ball and we can can um make um like the like um calculate the center of the ball and the way that we do that is that we can have um we can also create the mid y only like the middle y uh and it's going to be uh this one like the y1 + Y 2/ 2 and this is going to be get us the mid middle of the Y afterwards uh you can see that we have pretty smooth um pretty smooth uh detections but in some scenarios we need to remove any outliers and we can do that by having the rolling mean so we can uh add it right here we actually don't need it here but let's add it um for other use cases maybe other videos have this J action but we can have the rolling mean and this rolling mean is going to need a window so we are going to have a window of five and uh we are also going to set a minimum period of one um and we are not going to center it so Center equal false um and we are going to get the mean um afterwards uh we can basically plot it so we can plot this out so when we plot it out like this you can clearly see where the ball was hit so the ball was moving like this then it changed white positions here and it then change white positions here it changed white positions here it changed white positions here and this is like the normal behavior of the ball like if if you see that um the ball is moving in the y direction like this and after it was hit it changed Direction so the ball instead of uh being more and more like the Y coordinates is being increased uh it suddenly changed directions and now it's it's decreasing till the other player starts hitting it again and instead of decreasing the Y it starts increasing again so we want to detect when exactly in the plot that the um that this is being um like the Y coordinates is being changed um we can uh we can do that by like analyzing the difference between each point and the other point is it increasing is it decreasing and and how much so we can define a Delta y so we can have the uh ball positions Delta Y and then we have the uh define B positions rolling mean and we get the diff this difference is just going to sub subract the uh two consecutive um like rows from each other and yeah we can also try and plot it out and it should give us approximately a very similar graph so there you go it's a very similar graph and you can also understand where the ball was hit here here here here and here and yeah you can continue on with that um so yeah to understand where the ball was hit um we are going to Loop over this and we are going to determine whether the ball is increasing or decreasing the Delta Y is increasing or decreasing and we are going to get the position of the the frame number where the ball started to uh decrease when it was increasing like this change and it kept on changing for at least like 25 frames so we will check this and when it's at the bottom we will check if it's increasing again um it's going it's it's going to be a very simple code so we can have the ball positions and we can have the ball hit is all set to zero so we can have it all set to zero at the beginning and then I'm going to Loop over this data frame and like this DF um ball positions minus int and I'm also going to have the um minimum uh minimum change frames for hit and I'm going to send it to 25 so at least it's moving uh like increasing and then at least for 25 frames it's it kept decreasing so the change here kept decreasing 25 frames at a row or like maybe not in a row but um within a certain buffer uh so I can just uh have this uh and uh loop over uh like the length of the ball positions till minus minimum changes and let's put a buffer of 20% so let's multiply it by 1.2 and continue um we we want to detect if there is a negative change so there is a negative change that is happening where where the ball is here and then it decreased in value so we can have a negative position change uh where is it's going to be um uh like uh both positions Delta y um iock uh of of I iock is just getting the like the uh row number I and it's going to be if if it's a negative change then it's going to be greater than zero but the the preceding one is going to be less than zero so it was increasing and then it was decreasing so now it's going to be less than zero we also want to detect the uh positive change which is it was uh decreasing and then it was uh increasing and then it became increasing and we can call it here positive change positive position change um and if there is a negative change or a positive change um we can uh begin to count how many frames the change uh kept on going for so for change frame in uh range I + 1 till I plus int and then minimum uh let's call it like the uh oh yeah so we need to multiply this here and let's copy this put it right here and we uh yeah and let's add one to it and for for this we will need to copy paste the same thing again we will need to uh do it and detect if there's a negative ch change or a positive change uh but let's call it change in the following frame okay so now we have the change in the following frame and um yeah but instead of checking I + one we need to check the change frame and if and if there's and if there's a negative uh and if there's a negative change and there is a negative change in the following position um we just change count and make it change count plus one I change count plus equals 1 we want to do the same thing for the positive change so it's going to be the same thing and at the end we want to have if um and yeah here we just want to have if the change count is greater than the minimum change um hit minus one and if if that's the IDE if that's the scenario then we need to change the location uh like the rule the the ball hit in the row number I to one so that uh we now have the uh basically um uh the the ball hit position so let's run it you're going to find a couple of uh wearning messages don't worry about it and yeah you can run it again and you can see the ball hit a column that is uh already there we can filter it to only produce a or to only give us the ball hits that are one so let's do it like this so ball hit equal equals to one and let's turn it so you can find that there's um a hit in uh position uh frame 11 frame 58 frame 95 frame 131 and frame 182 and let's check if that's correct so here the frame number will come in handy so let's um let's move this and see so approximately frame 11 starts the hit and if we go back again here so we have a hit in frame 11 and if we continue down the line we can see that around uh frame 57 which is here 58 um another hit was being done another shot thought was being done and again right here uh around frame 95 which is another um another hit we can also see here which is around 131 another hit is being done and the last hit is at frames 182 so the logic is being um uh so the logic is is quite good and we can continue on from this so we can just um uh get those positions by just copy pasting this then saying that we want the index of it okay then we want it to make it to list uh you need to yeah you need to remove those packets from the index and now you have it as a list we can now just uh frame nums you can have that frame nums with ball hits okay so now it's all good you can uh paste it in right here and you can also try and uh finalize your code um just just going to copy all this put it all right here so that I can copy the code and paste it in the ball tracker so we can uh from here we can start by uh writing the Define get ball shot frames it's going to take in the ball positions and it is going to run this basically so again uh we need to uh convert this into a data frame so we can copy this from here uh like that and yeah I think we're done so it's it's uh it's um having it but we're it's not returning it yet so we can return it and now the uh function is done so let's uh do it right here and we can detect ball shots and we can have the ball detector ball tracker Dot what's the get ball shot frames and we give it the ball detections we can write here ball shot frames now let's print it out and uh see the results if it's the same or not so uh I think I forgot to uh copy paste something which is initialization of the uh ball hit so it's going to be ball hit and it's going to be all zeros so let me run it again so yeah right now we have the 1158 and 95 all the frames that we detected before are now being detected so all now is good we can keep printing it but uh yeah we we removing we'll remove it uh quite shortly up next is just we want to uh convert the player positions into the mini cord positions and this is going to help us draw the mini cord that's first and the second thing it's going to help us uh determine the amount of distance that is being covered by any object and uh by tracking it on the mini cord and not the actual pixels right here um so let's uh start uh by doing that uh so you can go back to the mini cord that we had and we are going to add one more function which is convert bounding boxes to actual mini mini cord coordinates uh so let me explain the logic first we are going to measure the distance between any key Point let's say key Point 2 and the player itself and now we have the distance in pixels but I want to measure how much like this distance in meters so what we have for reference is that we have the actual height of the player so we have the height in meters and we have the height in um in pixels so we can do cross multiplications to find the distance between the second key point or basically the closest keyo and the player position and then I can come here and say okay I can I have the distance in meters can can we convert it back into pixels and we already have this function so we can just call this function with meters and it will draw it in the uh position that is going to be relative to the court and yeah that is going to be the whole logic of this um so let me uh begin the code so we can have the convert convert bounding boxes to mini cour uh coordinates and we are going to take the um player uh boxes and we are going to also take the ball boxes we're going to convert them here into one function um and for convenience and then we are going to take the original uh cour key points um so yeah let's start first by let's start first by uh defining the player Heights so we can say that um player one has the constants dot uh player one uh height in meters we can have the same thing done for player two like this and then um we can Define the output boxes output player boxes and we can Define the output ball boxes and the first thing that we want to do is we want to Loop over the frames so we're going to have the uh frame number player uh bounding box and enumerate which is going to uh return back the index that we are are looping over and let's Loop over the player bounding boxes right now so uh the I so in the player bounding boxes there are multiple tracks so we are going to Loop over each track so I can have the play player ID and uh I can also do the bounding box in player. BB box. items and now I have it uh ready um the first thing that we want to do is that we want to uh get the foot position of it so the foot position is basically going to be like we are going to get the middle of the uh of the Box in terms of X and then get the maximum y so it's going to be right here um we don't want to have it to in the middle because here is actually where the uh the player is standing it's not here it's uh it's actually here so to do that we can go back to the utils and add another uh function in the uh BB box utils uh it's going to to be called a get foot position so we can have the get foot position and it takes in a bounding box and it returns in the foot position so uh we can do it like that so X1 uh Plus X2 which is the ex uh like the x and x and we can divide it by two and we return also the uh maximum key point we also want to uh expose it uh in the init and we want to uh import it so we can get the foot position of the BB box and uh this is the foot position okay uh now I want to get the closest key point so I want to measure uh like uh like we we want to get the closest uh key Point uh um in pixels basically and we don't have to search all the key points uh we can limit it to a couple so let's choose them right now um so you have this zero which is going to represent the end and we have the two is going to represent the other end and let's also take the middle of the course which is going to be 12 and 13 uh this is going to be fair enough let's not uh overdo it and search all key points so let's search only zero and two which is the extremes and two middle points which is 12 and 13 this one and this one um yeah so we will need to have um function get get closest key Point index uh that basically takes in the foot position and the uh original key points and the uh filtered like the the the key points that we want to search we said zero two and 12 and 13 and uh we return back to the uh closest key Point uh index and this closest key point we will be uh using to measure the distance between it and the player and uh yeah it's going to be the point of reference uh for us so let's also add this uh get closest key Point uh index uh to the uh BB box utils so we can have it right here get close closest key Point index and we can have the point that we want to uh measure the distance from we can also get the key points and the keyo indices that we want to have and yeah we can uh loop over them let's remove the suggested code and we can get the closest uh distance which is going to be float of infinity right now and we can get the key Point index um which is uh going to be key Point uh candidates of zero we're just going to take the candidate zero right now and overwrite it when we Loop then we are going to Loop over the key Point candidates uh indices so let's call it index in keypoint indices and let's remove all to the suggested code I don't think that it's going to work for us so we can uh write that the key point is equal to um key points of key Point index times 2 and then we want to also have it for the Y so it's going to be the same but plus one at the end then we are going to measure the distance which is going to be uh only in y distance so let's only measure y distance so point1 minus uh key Point uh 1 and if distance is smaller than the closest key Point um then what we do is that we assign the closest like the we assign the distance to the closest key point and then we we have the new index uh that we want and at the end we can just return it let's now expose this function um to the inits again so we can access it let's go back to the bounding box um uh to the mini cord and we add also this function and let's also uh yeah so it's already being utilized here then we want to get the player height in pixels so let's uh get the player height in pixels um I want to also show you something before we get the player height is that the player might be launching and is not uh straight up like like this one and this one so uh the idea is that I'm going to get the maximum height within the fif within a 50 frame to ensure that the pixels truly represent uh the total height of the player uh so let me uh try and get the uh frames that we are going to search for um so let's uh get the frame uh index uh minim and it's going to be the maximum of zero and frame number minus 20 so we're going to uh do um uh we're going to go back 20 frames and we want to also have the maximum number the maximum index which is either going to be the length of the frames or frame number uh plus 50 so we can go up to uh up to 50 frames um then we want to detect the height of the Box in pixels so we can go back to the utils and add another function um to that so we can have Define get height of uh BB box and it basically takes a bounding box and let's see the um return and it returns back the height which is the maximum y minus the minimum y we can also expose it right here and we can import it here and we can simply uh get the heights of this so uh we can say BB boxes Heights in pixels and we can simply have like uh four I in range and it's going to be frame minimum till frame maximum and we are going to get the um get the height of the bounding box and and this is going to be player bounding box of I now we have a list of heights of bounding boxes in the frames of in the in the list frames that we want um we want to get the maximum one so we can say max player height in pixels and we can get the uh maximum bounding box uh height um afterwards we want to get the um the court position so now we have the enough information to um convert this into the court position that we want uh so let's create another function that is called get mini qu position so we are going to Define another function so get mini cord coordinates it's going to take in the object position uh object position it's going to take in the closest uh key point and it's going to also take the closest keypoint index and it's also going to take the player height in pixels and let me just uh organize it into different lines so that it can be easily uh seen and so uh player height in pixels and then uh player height in meters so player we can copy this and say meters um yeah so right now we have the closest key Point index um let's also Define the um like let's get the actual point so we can say closest um key Point equal yep so it's going to be the original key point to the closest key point and then multiply it by two and we are also going to get the Y position of it okay so sounds good uh now we want to continue with the mini uh with the mini cord coordinates um we want to first uh get the distance uh between like the distance X and the distance y uh between uh the closest key point and the player so we can get it so we can say distance um from keyo X pixel pixs then we can say distance from Key Point y y pixels and then we can say it's equal to measure XY distance and you give it two points which is object position and closest key Point object position and closest key points now we don't have the function yet so we can go back to the BB to the bounding box utils we can um basically Define it and see it right here like now we get the X position the X difference and then the Y different difference uh separately and when we return them both we also want to expose it in the init and we also want to import it right here and after we get this um we want to convert this distance into meters so let's uh convert pixel distance to M and what we do is that we write distance from Key point x in met equal um convert pixel distance to meters and then we give it the pixels and we also give it the player height in meters and then the uh player height in pixels so um player height in meters right here and player height in pixels we want to do the same thing but for the Y position so let's copy paste it and get it also for the Y position like that and and now we have the meters ready so this is the amount of meters that they that the player has uh really uh like um moved and now we like uh this is the amount of meters that is between the key point and the player and now we just want to convert this into play like pixel coordinates again but for the mini qut so it's going to be for the mini size qut so we are going to say the convert to mini Court coordinates and what we do is that we write mini cord X distance in pixels self do we have a already made function for that which is convert meters to pixels and we just give it the meters so like like that we do the same thing for the Y coordinates like this uh we change it here to the Y and we change here also Y and yeah we also have we also can get the closest mini position the closest uh Mini U uh mini Point Key Point uh it's going to have the same order as the original key points so we don't have to do any calculations here we just uh need to get the position of it so we can say closest mini qu key point is equal to um self. drawing uh drawing uh key points of closest uh keyo index time 2 and we also want to have this for the Y but we are going to add one because it's a y position and yeah that is it let me just format the bracket correctly and at the end we just want to um add the distance in pixels to the mini cord position to have the last position so it's going to be mini cord player position is equal to um mini cord player position of zero times the distance in X and we want to have the same thing for the Y and yeah that is it we can return back the results like that and yeah we can now call it here so so we can call Mini uh Court player position position which is going to be calling get mini court court and we give it the foot position and then we give it the closest key point then we give it the closest key Point index then we give it the height and pixels of the player and we also give it the player height in met uh we then uh want to uh assign this so what we want to do is we want to have an output dictionary for each frame so when output player uh BB boxes uh dick and it's going to have the play ID as the key as usual and it's going to have the position as the value now we want to do the same thing but for the ball and since the ball doesn't have an exact height or the height of it is going to be very small um we are going to use the closest uh player to the ball and we are going to do the same calculation for it um so yeah so let's calculate the uh balls first so um let's get the ball uh box which is equal to ball boxes of frame number and then we have the uh ball position uh and we are going to get the uh center of the Box um let's not get the uh foot of it right now but let's get the center of it and we can get the center of the Box um and we can call it off uh BB box uh we can go here we can Define get center of the BB box and we can return and write it right here so this is going to get the center of the Box uh which is going to uh get both x's and divide them by two get both y and divide them by two and this is going to get the center of the Box um then we also add it in the init we also import it here in the um mini uh position and now we can U we can utilize it here so now we have the ball position um we can get the uh closest player ID to the ball closest player ID uh to ball and this is going to be the minimum of the player uh BB box do keys where um actually it it it's uh it did it quite good so this is what we do measure distance I think we already have the measure distance in the utils right now so we don't need to rewrite it so let me explain what the suggested code was all about um so right now it's going to measure the distance between the ball position and the uh get foot position let's let's also get the center of it let's get the center position um and the center position of the player so it's going to measure the distance between this and this and it's Al Al it's going to return the key of the minimum player bounding box and yeah so that is it so we can go here and we can yeah we can go here and we can write the the code for the ball which is if the closest uh if the closest player ID to the ball is equal equal to the player ID uh what we want to do is the to get the closest key point also so closest let's copy paste it so we need to get the closest key Point again like that and instead of uh using the foot position we're going to use the ball position and now we can just run the mini cord function again like this and instead also of the of the foot position we write the ball position and now we have those two in place and the maximum height and the player height are also being put um now the last last thing is that we want to have an output dictionary so let's call it um output ball BB box um so yeah um let's so let's just add the output to the uh to our list which is going to be output ball boxes do append and it's going to be one because it's only one one uh one uh ball and we're going to get the mini Court position right here um and at the end we want to return both the so return the output player dictionary and the output balls um actually we don't want to return the uh we don't want to return that we don't want to return the dictionary we want to return the output player boxes for the whole frames so let's add it right here and uh and let's replace it here so now we are ready so this is the uh coordinates of the um mini cord for the players and the balls so we can go back to the main and we can call it from there right here we are going to uh convert the positions uh to mini Court uh positions so uh we are going to write player uh mini cord detections and we are also going to write ball mini core detections and those are going to be equal to mini do convert to mini Court positions and we give it the court points and uh uh yeah so we give it the uh player detections and the ball detections and the court key points so we need to also give it the uh Court key points right here like that and yeah uh now we want to uh basically display this on the mini cord so we can have uh like actually let's run it first make sure that nothing crashes and then we can create another function that draws uh those positions onto to the mini cour itself okay so we have a small error uh I'm going to trace it back and come back to you so I uh so the problem was that I renamed the function here it was convert to Min cord positions it should be convert bounding box to mini cour coordinates um so let me run it again and see if we have any more errors okay we have one more error which is in line 32 of get center of the box and uh I think the box has a key zero error so let me trace it back and come back to you so I had two problems in the code one is that I was treating this as a bounding box B bounding box but it was actually a dictionary so I needed to get the one track the track number one and this will be now a bounding box and the other one is going to be the um uh this one um I was uh giving it a dictionary I needed to filter out the current player ID and this is how I did it um and yeah that is now a bounding box also so if you can rerun it again it should be working fine right now so now it's working fine um but we want to also see the uh results so we're going to make another function that is going to add the uh uh the this drawings to it and we are almost uh uh finalized with this mini cord so bear with me a little bit so to draw any points on the uh mini Court we can just call draw uh points on Mini qut and we can write the self we can get the frames and we can get the positions that we want to uh write it on and we can also specify a color we can give it um um a default color and we can Loop over the frames frame number and frame and numerate frames um we can also Loop over the positions that we want to uh uh that we want to Output so uh right now we have the position of this Frame number we have the player ID we don't need it so we can uh remove this we have the position and we can draw a circle with the frame and the position and uh then uh the radius the color and it's filled uh just to make sure that the position is in we can just uh write it like this position and we can make it as ins like that and like that just to make sure that those are ins actually um and yeah X and Y so so we can call this on our code so right here uh we can have the output frames equal mini cour Dot and then draw points on Mini qut and then we uh give it the output frames we also give it the uh positions uh that we want to uh put which is the player mini Cod detections player mini cord detections and we can do the same for the balls uh so we can call the ball uh mini detections and we can give it another color just to differentiate both of them and uh color is equal to uh 0 250 5 and then 255 that's another color um we can then uh run it okay we have another error so let me trace it back and come back to you guys so yeah we didn't return back the frames so we can just return it back here return frames and it should work fine so let's open the output and we don't see the key points yet we don't see those uh the players yet on the here let me trace it back and come back to you so if you were following along I made a small mistake when the brackets um right here in the closest mini uh key points I did uh the plus one outside the bracket it should be inside so this was the problem so you can save it and you can rerun it again again and you should find that the output has been uh on the on your output video just wait for it a little bit and yeah that is it um so you can open this here and you can see how the player is relative to the uh position here and the other player is here and the ball is also very close to to its position and if you run it you're going to see that the ball is moving quite correctly and the it's the ball is quite um that the P pill is moving correctly and the ball is a little bit jumpy because it's fast um but otherwise everything looks good and we can proceed from here so the last thing that we are going to work on is going to be um is going to be the output of the kilometers per hour and I'm going to show you the output the final output again and this is it that is the last thing that we are going to work on it's going to have the uh shot speed and the player speed and it's going to also um like count how many uh how many shots that the that the player has uh shot each player has shot and I'm not dis exping it here but we'll be calculating also um so let's uh Dive Right In this is the last bit but afterwards you're going to have a super cool um portfolio project to add to your CV so it's going to be totally worth it um yeah so let's jump into the main and I think we are not going to need any other classes maybe I'm wrong but uh I think uh we're done with those and the way that we are going to do it is that um we are going to uh loop over each shot in those frames and let me go through the logic first so uh we are going to go each shot in the frame so we have this uh this Frame number which is frame number 11 for this shot and we have the other shot which is this one um which is frame 55 uh we are going to measure the distance that the ball have covered um we are going to measure it in uh U measure it um in the in pixels first maybe in the uh mini mini qut and uh we can then switch it to meters and uh we also have the uh frame difference so we know that this Frame is um frame number 11 and it took it around um till frame 55 which is around maybe uh 40 something frames to uh go there and we know that per second we have 24 frames so we can so we can measure the time in seconds and we have the meters so we can measure the um the speed which is the kilometers per hour uh we also have like this is the player who shot the ball we also have the position of the opponent player and the amount that he covered the amount of um meters that he covered and we also have the time that he covered it in so we can also measure the player speed so we are going to Loop over the shot and take the next chot into consideration and uh measure the distance covered by the opponent player and the ball and basically um calculate the uh the kilometers per hour from here so let's uh so let's start um so for the for the shots we are going to um uh we are going to Loop over the shot which is going to be shot index in range length and we are going to have it uh shot frames which we got earlier and um we are not going to include the last one because the last one doesn't have another one uh that is going to be uh like another uh frame so we can't measure the the exact distance covered or the uh the amount of time covered uh but we are going to measure it for the uh for all other ball shots and yeah we can uh get the start frame basically which is going to be the ball shot and uh we are going to get the ball shot in Index this is the uh start frame and the end frame of the shot is going to be the next um the next bullshot so it's going to be bullshot index + one uh we can get the time difference in seconds of the bullshot which is going to be uh bullshot time in seconds which is just going to be the uh end end frame minus start frame over 24 because we have 24 frames per second um then we have the uh get distance uh covered by the ball and first we want to measure the distance in pixels so what we do is that we get the distance uh covered by ball in pixels and we are going to measure the distance we have this function in the utils I don't know if we imported it here or not so I don't think that we did so let's do that here and yeah so let's continue on and so we have the measure distance function and it will take the uh ball mini cord detections of start frame and it's going to take the ball mini cord detections of the end frame and because we only have one uh track so then we are going to just get the uh one yeah so we can we can use the convert pixels to me meters um that we have in the mini and it should return back the meters uh now to uh measure the uh speed so speed of the ball shot in kilm per hour um we have the speed of the ball which is going to be the distance covered by ball in meters over the time in seconds time 3.6 and this time 3.6 is just going to convert it to kilm per hour um now this is uh going to be the speed of the ball uh now we want to get the speed of the opponent player so we can have the opponent uh player uh uh we can have the opponent player speed and before that we can also have the um we can we can uh but before that we need to understand which uh which player shot the ball so we can do that by simply measuring the uh distance between the ball and the the two players and getting uh the player that is closest closest to the ball when the ball was shot so we can do that by simply doing that player who shot the ball we want to do that so player uh we want to get the player positions uh and the player mini cord uh detections of the start frame let's get the player that is closest to the ball so we can get the player who shot uh the ball which is going to be mini uh minimum of uh player uh positions do keys and we are going to have the Lambda uh player ID and we are going to measure the distance and the distance we are going to measure it is between the player position and the ball position so we have the player positions of uh player I and we also have the ball detections of start frame of one and this is going to return back the player ID with the minimum distance uh with that in hand we can also uh def know which uh which which player ID is the opponent which is going to be players um opponent opponent player ID is going to be opponent player ID one is going to be one if the player shot the ball is equal to zero and else it's going to be um actually it's it's going to be one if it's going if it's two and else is going to be two so let's say that the the player who shot the ball is player uh two I'm going to return one uh as in the opponent player I'm just going to return the inverse of it and if it's not one if it's not two then I'm going to return two that's the opponent player um I want to uh measure the distance covered by the opponent in pixels so distance just like what we did in the ball uh but we are going to do it for the covered by the opponent player by uh opponent in pixels and it's going to measure the distance between the uh player um it's going to player like the start frame of the of the opponent ID and the end frame of it so that is good now we want to convert it into meters so that is also going to be the same as the mini qu uh one and then we want to measure the speed so speed of opponent opponent which is equal to the distance covered by opponent over B shot time frames in second which is going to be the same uh frame time 3.6 to make it in kilm per hour uh yeah so now we have the speeds and we have uh the speed of the ball and the speed of the opponent uh we just need to put it in a type of a uh structure data structure so that we can um display it in a better way so I am going to choose to display it in a list of dictionary uh because it's going to be easier for me to change it to a pandas data frame and to um expand it to the whole um other frames as well so let's create this um data structure which is going to be just a list of dictionary uh stats uh data it's going to have a list and this list is going to be a dictionary uh the first one is going to be a dictionary it's going to have a frame number which is going to be zero at the beginning then it's going to have the player one number of shots which is going to be zero then uh it's going to have the player one uh total uh shot speeds now this uh now the number of shots is the number of times that the user that the player have shot the ball the total speed is the summation of all the speeds that the user have done and you can calculate the average by just um having this over this like this number over this number you can get the average so I'm not going to have the average right here but I'm going to calculate it later and we can have the player one last uh shot speed so this is the latest shot that he did what's the speed of it and we can also do it with zero and we can have the player one uh total distance uh total player speed um so total player speed which is going to be zero and we can also have the actually let's last player speed we can also have the last player speed on and we can do the same thing for the um player two so number of shots and the total speed of the shot the last shot uh the player speed uh the last uh player speed as well um so that should be it um so let us uh import a copy uh mechanism which is from copy import deep copy because in order to copy a dictionary you need to um uh copy it and not copy it reference so deep copy helps us uh to copy the values um and the way way that we're going to do that is that at the end of this uh we are going to uh create a a status uh like the stats the statistics so we have the current player stats which is going to be a deep copy of the the the previous player stats and we are going to overwrite some things so for examp example the frame number is going to be the sort of the frame number and then we want to have the uh player number of shots which is going to be player number of shots plus one we want to also have the total speed to be added uh we also want to have the um last shot to be added here and we want to add the opponent player so the current opponent player which is uh right here we want to add to the total speed of it and we want to add the last speed and afterwards we want to have the player stats data. append current player stats so now we have for each start frame we have this dictionary again but with the new stats and for that we will need um like um now we just have the uh like the short frames and we want to expand it to the whole list of frames so that we just can Loop over without too much logic and um basically from frame zero to the first start frame we just want to duplicate this again and again so that we have um like uh so that for each frame we have its statistics um we can also like Loop over it and um do some logic so that we can uh know which uh position to to choose from here but I'm going to uh choose to duplicate it and to make it into a pandas data frame so that it can be easier to um display so let me change it into a um first the data frame so data. DF uh I don't know if we have pandas already here so I think we do maybe we don't um and then we have it so let's import pandas now we have it as a data frame um let's also have a frames number data frame so frames DF which is going to be PD data frame which is equals to uh frame number and it's going to have a list um of range of length of the video uh this is going to be a data frame with frame number and it's going to have zero till the uh length of the frame number um I think there's a bracket missing or something like that let me figure it out yep there was a bracket missing and I just added it right here and now we can just do this and merge it so now we are merging the uh frames data frame with the player stats and I'm going to have it uh left join on the frame number and that is going to result in multiple rows going to have n because only the start frames of the B shots have value so other things will have n and I want to fill it so the way that I'm going to do it is that I'm going to do a front fill and uh basically like this which is equal to FF fill it's going to replace a nan with its previous value that is not a Nan and do this so on and so forth so that every frame we have the um we have the output of it um now I want to uh basically calculate the average of it so let's calculate also the player tats and data DF of player average uh shot speed and we are going whips we are going to um divide the total over the number of shots um I think I misspelled it right here so let's um do this and player one uh short speed over the number of shots this is the correct one um it's not it's it's it should be player one average shot speed and let's me check this player one to to shot speed and the player one number of shots that looks correct let's do the same thing for player two and let let's do the same for the player speed so this is the player speed and it's also getting the total over the it's not um it's not doing the over correctly so um it should get the other so it should have the total speed over the opponent's number of shots because the player only should only like we are only calculating the run when a user shoots and uh how the uh opponent reacts to it so this is the player one over the number of shots that were done over player two and we're going to do the same for player two and we're going to get the total of player two over the number of shots done by player one so now all all the statistics are done we are just missing the last drawing of this so we need to um draw this uh draw those tats and yeah uh and we are done we are good to go um so in the utils uh we can uh do another um another function uh another file called player stats dra utils.py and we are going to put the uh to put the uh the the drawing logic here it's not going to be logic just going to be drawing um it's uh it's going to be easy to do but uh it's going to consume a lot of lines so I thought to put it in the utils and not in the main just to keep the main clean uh so we have the numpy as NP and we also want to import CB2 and let's define the function so the function is going to be draw player stats and we are going to have the output uh video frames and we are going to have the player stats and we are going to Loop over it so Loop over the uh player stats so in player stats dot it RS and uh we have the player one shot speed as in row of player one last shot speed we have also Player Two Shot speed that is going to be last player two shot speed um we want to have the player one speed which is going to be um player one last player uh speed and we want to have it also for player two um we want to also get the average player speeds so average one player shot speed and then the uh player two we want to do it for the speed also um and average player speed um now we want to get the frame so frame is equal to the output frame of uh index should be index and now we want to have uh basically it's not that one um we want to have a black transparent box so we're going to add also the position of the box and then make it transparent and and then we're going to add those um um statistics so adding those statistics are going to be uh we are going to need the shapes again because we're adding a transparent uh uh black box which is going to be also zeros like the one that we did above above it and um we want zero like frame and we want to Al to have it u in 8 let's write it correctly this time and then we have the width is equal to 350 the height is equal to 230 we also want to um Define a start and an end X so start X is going to be um frame. shape of uh 1 minus uh 400 and we also want to have the start y um which is going to be of zero and I think I wrote shape wrong so let me fix it and uh 550 and let's let's also Define the X's so we have the end X is going to be start X Plus width and then end Y is going to be start y plus uh the height um then we are going to have the overlay which is going to be this one it's going to be the overlay start X start y and x and y and we are going to to uh put a black one this time a black box this time and it's going to be filled and we are going to Define an alpha of also 0.5 so we are going to add it weighted again and we're going to add it overlay the alpha the frame and then we have uh the rest of it um then the output video frames um then we can just have the output video frame of index is equal to the overlay is equal to the frame sorry okay now it's time to put in the text uh now the text is going to be um like the text is going to be just putting the text like the the times that we put the text here before um it's just going to be a matter of choosing the positions so I'm just going to copy paste it and walk you through it so I copy pasted it right here and uh right now we have the player one player two and then I put the text I chose the start position and the end and start start x and x uh I also chose the colors I did the same thing I put the shot speed I then wrote The Shot speed themselves and then I did the same for the player speeds and then I did the same for the average speeds uh So currently right now what I want to do is export this is expose this again so uh from dot uh uh player stats import uh draw player stats we want to also import it in the main right here like this and we want to now draw it so uh let's draw player stats and and we can have this like this so we can have it uh like this draw the like call the draw player stat function and now we just run it so there's a so there's an error here that is called uh convert pixels to meters um because the mini cord function does not have convert pixels to meters it has convert meters to pixels so we will need to um so we will need to import that from the utils convert meters to pixels and we also want to import the uh constants so what we do here is that we just want to uh convert it and use the double line width like the width of the uh the the field in meters and we also have the width of the mini cord so we are going to use those to do some to convert the um pixels the pixel like the pixels uh done into uh meters so what we're going to do is that we are going to do like this we're going to choose the meters right here then we are going to give it the reference uh uh the reference um meters which is going to be uh constant. double uh whoops it's not that was filled incorrectly so you have the width and then you have the mini quart and then you can get width of the miniart um get mini height I think I need the width of it um let me go and check if there's the width ready this function get mini cord get width of mini chord I think we have this function so yeah so we have the get width of mini chord and we will do it again right here and instead of that we can use this and just delete that let's run it again so uh player one last shot speed I think this one is in in the uh drawer because I added another S I think there shouldn't be an S like here uh but let me check the line line 139 which is yeah so it's going to be the uh drawer so yeah the speed is going to be speed not speeds um so like this so let me run it again and now it's done so you can go back to the output and see it so um there is a little bit of overlap so you can actually push it down a little bit so let's do that um let's uh move move this here uh so let's move it down like 50 pixels so let's make it start at 50 uh 500 not 550 um let's also specify the width to be a little bit more uh I think it's going um above no it's not um oh yeah because it's not um so you're going to find it to have uh oh there's some weird numbers going on so we have some weird numbers like here like 1,000 km/ hour and here is also 1,000 km/ hour uh let me trace it back and come back to you but uh let's first before tracing it back let's see if uh if this would push it down a little bit more like 500 the make it 500 and instead of 550 and let's see if there's something else that breaks but it shouldn't Let's cross our fingers and we should be able to see the box but we should still be uh seeing some weird kilm per hour that we need to fix so yeah so right now it's uh it's it's actually good but the kilometers per hour is way too much and I'm going to trace back Y and come back to you uh so give me a minute so I'm back and I know the error um I've used the convert pixels to this I've used convert uh meters to pixels uh and I should have used the convert pixels uh to meters so I I have the distance covered in pixels and I want to convert it to meters but I used the wrong function and uh you should be using this one convert pixel distance to meters and if you run it again uh you should be able to get the correct output so you can now see um so you can now see as as soon as the player hit the ball you can see the shot speed of the play player like this is 33 km/ hour and player two is going to move by uh 1.7 km/ hour and when the other player shot you can see that it was a harder shot by a 43 km per hour and the other player uh moved with 4.5 km per hour and if you see that this player printed actually so you can see that it has a 9 km per hour and and um and yeah you can also see the averages right here the average player speed the average player um uh shot speed and the actual player speed um you can do all sorts of fancy stuff like the number of shots and you can analyze how fast a player is running when he's losing the pole and you can analyze how fast um the ball like the how fast the ball is shot to make um a winning shot um you can also do a lot of other stuff like check whether the ball is inside or outside the court when it's hits the ground and the project can be very big I chose to make it simple right now uh but yeah um you can we didn't also do anything with the net so you can also add a little bit of net logic if something uh goes wrong or hits uh before the net um so yeah you can you can build on this but currently you have a very cool portfolio project right in your belt and hopefully it's going to help you u landar a job in computer vision and AI so that's it and uh have a wonderful day

Channel: Code In a Jiffy

Views: 262,514

Keywords: object detection, object detection tutorial, object detection python, machine learning, computer vision, python, yolov8, yolo, convolutional neural network, machine learning projects, pytorch, tutorial, deep learning, keypoint detection, object tracking, opencv python

Id: L23oIHZE14w

Length: 281min 25sec (16885 seconds)

Published: Sat Mar 09 2024