Build an Object Detector for Any Game Using YOLO

hi have you ever wondered how to create your own object detection system that will work with any game just following a few steps in this tutorial I'm going to show you how easily you can create your own machine learning model from scratch using tiny YOLO and open CV and perform real-time object detection that will work with any game after following this tutorial you will have a python code that can detect objects and get its coordinates in any game in real time with that you can for example create a bot to control your character using libraries like Pine put the theme control your mouse or keyboard I'm going to use the game job YouTube to illustrate all the steps and in the description you can find a link to the source code that I'm going to use in this tutorial so you can test with a game of your choice tutorial is divided into four parts in the first part we are going to create a data set using images from the game of your choice I will explain how to know if your data set is good enough for a specific scenario when we got there in the second part we are going to label the data set in other words we are going to indicate in those images where we object we want to detect are located I will also provide some tips for labeling the data set moving on to part 3 we are going to start training an object detection model using the labeled images this is where we teach the model to recognize the objects we marked in those images finally in the step 4 we are going to use the training module to detect objects in the same game from which we generated the images in this first part we are going to create an image data set in other words we are going to take pictures of the game that we want to detect objects these images will be used to train our model to recognize and detect these objects creating a data set is very important for the performance of your model so as a rule of thumb try to capture images with all the elements that your model will see when performing detections I will explain this in more details later on but for now just know that even if you want to detect let's say only birds in a forest but this first has also squirrels your data set must have images with squirrels too because in this example when you use your module to detect objects after you train it your model won't see only Birds it will also see some squirrels so in the training step you must show your model all the elements that it will see when performing detections so first of all I will download the source code you can find the link to this repository in the description after that just extract all files then inside this folder open a command line and run pip install Dash r arence.txt to install all the dependencies that we are going to use after that just run Jupiter notebook space Dot and then click on the first notebook called generate data set in this first notebook we have the code for creating a data set basically I have a class called window capture with a method called get screenshots that will be capturing images from the game as a numpy array also there's a method called generate images data set that we'll call the get screenshot method every second and save the result inside a folder called images now run the first two cells to import the dependencies and create our window capture class now open the game that you want to perform detections and get its name from the game title bar in my case the name will be Diablo 2. Now set the game window name on this variable and run the cell if you go into the project folder you will see a new folder called images and each second a new image from your current game screen is being saved there so our data set is being created I will start moving around to try to capture all the elements on this environment the number of images that you must generate will depend of the complexity of your game graphics for this 2D game I will use about 300 images so as I already generated this number of images I will just close the game to start generating the images after generating the images we are now going to label these images in other words we are going to indicate the locations in each image of the objects that we want to identify this stat is the most important in the whole process as the entire performance of your model depends on it when we were creating our data set I say that you should create images with all the elements that your model would see during detections the same rule applies here remember to label images with all the elements that your model will see when detecting objects I will now explain in more details why this is important let's consider an example scenario where you want to detect only birds in a forest so you create your data set with a variety of birds in different shapes forms colors and angles however in this verse there are not only Birds but other animals like squirrels in that case you need to include images with these other animals even if they are not going to be detected in other words in this first example with birds if you don't have any labeled image that includes a squirrel to use in the training step your training model won't know what to do when it sees a square for the first time the model won't know if it should attack that animal because the model has never seen it before maybe it will detect if the color of the square matches the color of one of the labeled birds or maybe it won't detect but the behavior of your model won't be predictable when it sees a squirrel just to be clear I'm not saying that you should label this square root as in this example you want to detect only Birds I am saying that you should have images where you only label the birds but the image also contains a squirrel like these ones on the screen essentially if you have an image with a squirrel in it but don't label the squirrel you are implicitly telling the model that it's not a bird and should not be detected now open the second notebook here we have a glass called label uchus that will help us create our labeled images and configure the YOLO files for training in the second notebook run the first two cells to impact the dependencies and declare our label will choose class now run this third cell which will create a folder called shuffled images it will Shuffle and copy the images from the images folder that we created on the previous step to this new shuffled images folder you can see in our project the new folder with the shuffled images this step is needed because with that you will cover a lot of scenarios if you don't label all of the images that you created on the previous step now we are ready to begin the labeling process I recommend using the make sense.ai website for this task here you can upload your images and start the labeling process so first I will click on get started and then upload the images from the shuffled images folder now I will click on object detection and now I will Define the labels pay attention to the order that you will insert the labels as you will need to insert these same names and in the same order in the code later on recommend ordering them of a bad clip really important so it suggests that you take note now of the names and the order that you are going to use for these labels in my case I will detect two enemies the fallen and the zombie so I am going to insert these two labels here I also suggest that you edit the label color here and choose a color that you can associate with each label if possible now click on start project here are some tips for labeling the images first you don't have to label every single image instead select the images where the object you want to detect is clearly visible is keep the images that don't contain the elements you need to detect or if they are not clearly visible for example if they are mostly hidden behind a tree and remember your labeled data set needs to include all the elements that your model will see when it performs detections second keep consistency during this part if an element you want to detect is partially visible in an image decide whether you're going to label it or not if you label partially visible objects in some images but not in others this will confuse your model and reduce its accuracy and performance and third keep in mind that the quality of your label data set which includes covering a lot of scenarios maintaining consistent labels and avoiding mistakes like mislabeling or omissions strongly affect how well your model will perform during object detection tasks so now I'll start the labeling process after you finish labeling you can click on action export annotations and select the YOLO format this action will generate a zip file with a text file for every labeled image unzip these text files and copy them to the shuffled images folder now still within the second notebook run the cell with the method create labeled images zip file to create a folder called object that will contain only the images and the text files that you just labeled also this method will zip this folder and copy the zip file into the YOLO V4 tiny folder this deep file with the labeled images will be used in the next step for training your model now in this cell enter the same names of the labels you entered on makesense.ai in my case it will be two only fallen and zombie pay close attention here if we put a different number of classes than you specified earlier during the labeling process the model will not work correctly because in this step we are editing the YOLO configuration files inside the yellow V4 tiny folder with the number of classes that you are going to use after running upload the folder YOLO before tiny to the root of your Google Drive and now we are ready to start training our model on this third part of the tutorial we are going to create and train our object detection model using the images that we labeled on the previous step so in the last step we uploaded the YOLO V4 tiny folder to our Google Drive now we are going to use Google collab to train our model go to the website collab.research.google.com click on file upload notebook and select the third notebook YOLO model training now let's begin the training of our model first you need to configure the runtime settings by navigating to runtime change runtime type here make sure to set GPU option for improving the training speed after that we need to clone the darknet Repository darknet is the framework that we are going to use for training our object detection model now run the cell to connect to Google Drive where we have stored our configuration files and label images in the folder yellow before tiny you can use the least command to verify that all the necessary files are present next run the cell to update the darknet make file to enable GPU usage and then run the make command to compile the darknet framework with GPU support after that run this cell to remove some configuration files from darknet that we won't be using then copy our configuration files into the dark net folder lastly we are going to run the process dot by Script which will partition our labeled images into separate training and test data sets finally run yourself to initiate the training process this command will use the labeled images we created earlier to drain a model how to detect objects based on our annotations after some time you can check the training folder inside the folder yellow before signing in our Google Drive the comment we just executed will produce some weights that is basically our drainage model you can start using these partial weights and jump to the last part of the guide but I recommend waiting for the model to complete its training after a few hours the command will finish its execution the weight file named YOLO before tiny custom last weights represent our fully trained model now download this file and place it in the root directory of our project in the same folder as the notebooks with that we are ready to start make detections using our YOLO model now we are on the final part of this guide and everything is ready to use our YOLO object detector basically we are going to use the model we just created to perform detections on the same game that we used to generate the data set so open the last notebook YOLO opencv detector you will find the window capture class that we use to generate an image data set and also there's the image processor class which will open our yellow model using opencv and it will display the detections in a new window and provide the detections coordinates in a list for further use in more details this class image processor has a Constructor that loads or yellow model using opencv and get the class names used during the labeling process it also has a process image method responsible for receiving an image from the window capture class and passing it to our model to perform detections there's also a get coordinates method that receives the YOLO model output and format it into a list containing the coordinates X Y width and height and the class of the detections for all the identified objects in an image additionally there's a method called Draw identified objects that takes this formatted output and display it in a new window and this will allow us to see the model's performance so run the first three cells to import the necessary libraries and declare the two classes and now start the game that you used on the first step and in this last cell all you need to do is set the game window name and awaits file name that you downloaded after finishing training the model in the previous step in my case the game window name would be true here we will create our window capture object to capture screenshots and pass these screenshots to our image processor object which has our YOLO model loaded so let's run this cell and finally see the results here we can see that we were able to identify the monsters correctly I didn't use a very large data set so it may make some mistakes sooner or later but with about 300 images we already have great results so that's it for this tutorial I hope you enjoyed and that it can help you with your own machine learning projects using python as you saw the biggest effort would be to label the images to rest is just run some python code and if you have any questions please leave a message below and if you use this guide for your own project please get in touch with me via comments or find me in linkaging I would really love to see your results too see you next time

Channel: Moises de Paulo Dias

Views: 15,330

Keywords: yolo, machine learning, object detection, deep learning

Id: RSXgyDf2ALo

Length: 22min 40sec (1360 seconds)

Published: Wed Sep 13 2023