Custom Image Segmentation using Yolo V8 + Segment Anything Model (SAM)

Hello friends welcome to another exciting video on my YouTube channel and in this video I will show you how you can create mask on an object using YOLO V8 version and then apply a segment anything model from Facebook to segment that object inside that image and then you know we will just do an add-on to you know uh remove the background and make it white so this is the original image by the way this is my son and it is in my house so this is the original image then this is the Mast image you can see that with the help of YOLO V8 and segment anything model I have first identified the object and then I have created mask over the object and then just doing some ah image manipulation I have removed the background and you can see the clarity of the image as well so let's dive deep into this and let's look how to implement the same thing I explained so here I am in my collab notebook by the way the link of this collab notebook is in the description the first thing we need to do is we need to install ultralytics because we are going to use ultralight X YOLO implementation then we need to install segment anything and then we need to download the segment anything vit hyphen H uh Pi torch model so I'll run this meanwhile I'll show you GitHub segment anything yeah so this is the official repository and you can you know get all the knowledge that I have actually acquired from this repository it is very well explained so now yeah it is downloading the model and if we go to this file section we will see that the model is being downloaded meanwhile let me upload the original image here so that we can use it as well yeah now it is done so let's import few things ultralytics Ultra electrics YOLO then segment anything model registry uh mask generator and Sam predictor this is to display the images CV2 to real image numpy to get some uh metrics matplotlib and this ultrolytic check is is a kind of a sanity check to make sure you have enough resources to run the file the first thing we will do is that with the help of YOLO we will predict on this original image so let me copy the path of that original image and let me provide it here Okay so let's see that how it comes so it will create a run folder here and it will also create predict inside that and it the the prediction will be there in the image so yeah you can see that results save 2 runs dot detect runs hyphen detect hyphen predict so let me refresh this yeah it is there so run detect predict and this is the image that YOLO has predicted so let's look into the image so I'll use this IPI widget to display the image so you can see that it has fairly accurately identified the object now what we will do is that with the help of mask generator we will generate mask on this object for that let me first get the result and we need to find the bounding boxes around the object so for that oh sorry uh my bad it is original yeah so we have results let's see the result since there is one object and it will be at the zeroth location so we'll take that we will take that and we will create the boxes so this is the boxes okay now what we will do is that we will create Sam Predator model and so we'll provide the path so this is the model and model type is vit hyphen H which is this part there are a few models if I model checkpoints yeah so there are few models in case you want to use this vit hyphen L I'm not sure it stands for large or not then you need to provide the string as a model type if you use this then you provide vit hyphen underscore B so I'm using vit underscore H so I am providing that as a model type and we will need to create a predictor so I'll create the predictor okay it can take some time next we will read the image original image okay and we will set the image to the predictor so the way it works that you need to First create a predictor once the predictor is created you set the original image and then provide the points bounding box so that the predictor dot predict method will take that boxes and try to create a mask okay so yeah the model is now predictor is created now I'll read the image and set the image to the predictor so as I was saying that when you set the image so the first step is to set the image next predictor predict method will use the set already set image and the bounding boxes to generate the mask okay and with that mask we can as you can see in the previous trial run I have created this mask so this predictor dot sat image can take a while so let me come back once it finishes [Music] okay so you can see that uh that process of setting the image ran successfully here are a few functions that we are going to use just to help us show mask show points and so box let me run that now what we will do is that we will convert the bounding box into NP array cause predictor.predict requires that as I said we have set the image now when we use predictor.predict and provide the bounding box it will generate mask so let me run that yeah you can see that so this was yeah this was our original image we first applied YOLO to identify the object then with the help of segment anything we provided the bounding box and the Sam model has identified the object and created almost accurate mask now what we can do is that we can just remove the background just make all the pixels outside of this object white and we can do this so we have done that so except the mask make all the point White and so this is uh this is uh one one cell code for the same procedure you just provide the image path here and you will get the final results here okay so I hope this is this is informative to you and you can leverage the knowledge from this video into your tasks so again I'll summarize first we predict the object using YOLO once you predict the object then we create the bounding boxes and I have used this zero because I'm using only one object but in case you have multiple object you can select Which object you want to choose and then create the Sam predictor model and set the image original image as the image and then use predicted or predict and provide the bounding box make sure you use NP array and it will generate mask over the bounding box it will generate a mask over the object inside the bounding box and finally we can you know manipulate the image so that accept the point inside mask stays same otherwise the point will be white so I hope you have gained some knowledge again and if you like my work please consider subscribing to my channel it helps thank you for watching peace

Channel: Raj Kapadia

Views: 6,734

Keywords: raj kapadia, sam, segment anything, image segmentation, computer vision tutorial, computer vision projects, segment anything model, yolov8 tutorial, yolov8, remove background yolo v8, custom image segmentation, image segmentation and object detection, image segmentation in digital image processing playlist

Id: aYP4ujUsGdk

Length: 11min 12sec (672 seconds)

Published: Wed Jun 21 2023