Segment Anything Model (SAM): Build Custom Image Segmentation Model Using YOLOv8 and SAM

hello everyone my name is arohi and welcome to my channel so guys in my today's video I'll show you how to use segment anything model for your custom requirement so guys this segment anything model provides us high quality mask for all the objects but there are no labels with it okay so if you'll see their data set over here so click on this data set and then you will be redirected to this page you can see that all these These are the images from their data set the huge data set which they have used to train their model and you can see that they this data set have different kind of mass variety of different masks right but there are no labels attached to them and even in their uh this page they have mentioned also that there are no class labels for the images or mask annotations okay so that means using this segment anything model you can only get the mask okay for every object but now what if you want to have the class label with it suppose you want to build a custom application Custom Image segmentation application where you only want to uh put a mask on a single object let's say truck okay so you are building a some sort of application and there you only want to put a mask on the truck and you also need a corresponding label like this okay so in my today's class I'll show you how we can use segment anything model to achieve this kind of result okay so guys uh before that just move to there this official site okay here you can try the demo also okay from here you can try the demo but what I want to show you is just scroll up little bit and over here see for my today's tutorial I got an idea from this particular section so that's why I'm showing you so see over here they have mentioned bounding box prompts from an object detector can enable text to object segmentation right so this this is what we are going to implement today so we need one object detection model so as I've told you so in my today's class I'm working on a track class only okay so first you need to build your object detection model which can detect drugs okay so for that I'm using YOLO we ate and using YOLO V8 you can train your custom object detection okay perform custom object detection so if you want to learn how to train your YOLO V8 model on custom data set then you can check the link I have given the link in description section from there you can see how to train your custom object detector so in my today's class I have trained my model for a truck class okay once your object detection model is ready we will use that object detection model on our image so what we will do will pass the image to our object detection model and then that object detection model will plot a bounding box around the object around the truck okay so that uh box that area where our object is actually present we will send that bounding box area to the segment anything model and segment anything model then can put a mask on it okay to segment anything model is very good in providing the mask right so we will detect the object using the object detection model and then the detected area will go to the segment anything model then segment anything model will put a mask on it because this is what uh segment anything model is good at it okay so this is what we are doing so let me show you the Practical implementation now okay so this is my Jupiter notebook okay so these are the important modules we need so those are the module names and then these three functions show mask these three are helper functions we need them to visualize the results of segment anything model okay and you can get all these things from their official uh GitHub repo it will go to segment anything and if you open their GitHub repo and from the notebooks you can get the code from here okay from this example these three functions are present there okay so you can get the code from there all right so once you have that and right now I am I will show you the output on this image okay so this is our input image this image first will go to the object detector object detector will detect the truck and that detected area will go to the um or segment anything model and it will provide a a mask on it okay so this is our input image so here you can see this is my image this is my YOLO V8 model so guys what I'm doing is I am not training a specific model OKAY a custom model because truck classes already available in uh YOLO uh pre-trained model because YOLO pre-trained models are trained on Coco dataset and Coco dataset have a truck class so I will pick the data of truck class from there only and I'm working on it okay so I'm loading the pre-trained model and then I am calling the model on this image on which image I want to work on this image class is equals to 7 this 7 is the ID of truck class okay this truck is on the seventh ID so that's why I want to work for single jars only so that's why I've written seven over here now my model I have done this process in the objects variable right and here guys you can see inside the boxes I have the bounding box on the object which is detected by our YOLO V8 model and inside this CLS you will get the class of that bounding box right so what I'm saying is this bounding box coordinates you can get using this line and the class of that bounding box area of the object you can get it by like this and then over here what I'm saying is if class is seven okay if the class is struck then I want to execute this code okay now what we are doing is firstly we are putting a rectangle on the area where object is present and the bounding box values we are getting from here and then these these few lines are used to put a class name on The Box okay so don't worry I'll show you the output then you'll understand what I'm trying to say and this is what till now we have done the detection task once you have your detection then over here we are using our segment anything model so we are loading the checkpoints and model type is this device we are provided and then we are calling the predictor and to predictor you need to provide the image on which you want to work and over here see guys now what will be the uh input to our segment anything model we have provided the image okay but if you remember this in my last class on the first lecture of segment anything I have told you that whenever you want to use segment anything model you need to put the points on the object which you want to detect right there are few things one thing is you want to put the mask on all the image all the objects which are there in the image you want to put mask on them right but for our today's class what we want is we want to work on a single class okay so that means there are two different ways either you have to put the point on the object on the center of the object where on which object you want to work or what you can do is you can give the bounding box coordinate of the object where your objectives exactly and you can pass that pounding box coordinate to your segment anything model and then segment anything model will pick those four values of those of the bounding box and will provide you the uh segmentation on it right so for our today's class we already got the bounding box coordinates from our detection model right when detection YOLO V8 performed the object detection we get bounding box coordinates we get the class names write all these things so I'm picking the bounding box coordinates from my detection results and I'm passing that to the uh segment anything model okay and then we are doing the further task so this is what I'm doing here you can see so initially you need these tips you have to do after that here you can see boxes to list what is this boxes these are the boxes which we get from here right okay so those boxes I'm providing here so now I have provided the four bounding box values of the uh which is covering the object our track okay and we are passing that and then we are calling the predict method of predictor uh segment anything model and you can see over here this is how you use it and then while at the time of plotting here you can see input box input box here is the input box variable variable which have the coordinates which I have told you which we get from detection okay the this is what we are providing right and then show mask remember there is this show box in show mask these are the functions which we have defined a work these are the helper function which will help us to plot the bounding box and to plot the mask on the objects okay so then I am calling them and then I'm plotting and I'm storing the output image with this name okay and then now let's see our output image you can see truck we are getting the label also and then we are getting the bounding box around it and we have only mask on truck instead of the entire image we are only having the mask on truck so this is how you can build your uh custom segment anything model for your custom requirement Okay so so I hope this video is helpful thank you for watching

Channel: Code With Aarohi

Views: 27,238

Keywords: SAM, Segment Anything, Meta AI, Image Segmentation, Artificial Intelligence, AI

Id: XB9zg99x2jE

Length: 11min 50sec (710 seconds)

Published: Wed Apr 12 2023