307 - Segment your images in python without training using Segment Anything Model (SAM)

hi everyone welcome back in this short tutorial I am going to talk to you about something called segment anything model in fact this was released two days ago literally on April 5th by meta Ai and many of you already left messages on my LinkedIn or Twitter asking for their specific tutorial so I'm going to walk you through the process of importing the model that free train model that these guys generously shared and segmenting your own images and I'm going to do that on my own IDE locally because I have a GPU if not you can use it on collab and again you can find more code on their website if you want but at least you'll get a good first impression about what segment anything model is capable of doing by watching this tutorial again if you are not a subscriber yet this would be a great time to hit the Subscribe button and also hit the thanks button if you're feeling extra generous okay with that let's jump in and I'm gonna quickly start by showing you a couple of web pages and again I leave all the links down in the description go ahead and look at the description down below so just go ahead and jump in and the first thing I want to show you is this page go ahead and Google search for Segment anything if you can't wait to read the description down here and here you can actually get a bit more information about what exactly it is but more importantly you can go to demo and you can upload your image and go ahead and do the stuff and in fact here what they do is let's import an image right here and there are a few things you can actually do and uh I mean you can draw a box around something and it detects the object like for example you do that and it's trying to identify the object you can also go ahead hover and click like you see how it's detecting these objects right there so you can hover and click uh but that's not fun let's go ahead and import an image into using our python code and detect all the objects okay let's go to the next step where I would like to show you this this blog and here they talk a bit more about what it actually does this gives you a first glimpse of what exactly it does right so you have an image with bunch of objects and it's going to detect all the objects and if you scroll down they do they talk here about how it works I mean basically you have an image and it's it's gets there's an encoder it gets converted into these embedded or represented by these embeddings and on top of that you can actually provide a mask or you can provide in information or mask in the form of points or box or text I just showed you the box and the point in a you know in their demo screen but you apparently I haven't tested the text part but apparently you can actually say hey in this image find the scissors in this image also on right so you can provide text and maybe I'll explore that and do another video on this topic but this is at a high level this is exactly what it is doing go ahead and read the original paper so you can get more information about exactly how it's working but this tutorial is completely focused on just using a few lines of code load the pre-trained model and try to segment your images for various objects right so that's this tutorial it is about and again the GitHub page their GitHub page is right here and it talks about a bunch of stuff everything all the resources so I don't need to walk you through all of this but I will walk you through the process of how I installed this on my system literally within 10-15 minutes right I mean completely from creating a new environment so what did I do I created a new uh conda environment for for my segmentation for this task and then I of course installed pytorch and torch Vision so how did I do that so in my case first thing first I well do I still have the yeah I just did this a few minutes ago so I still have my Cuda version where I checked what the Cuda version I have is and it's 11.0 and then I went ahead and Google searched for okay for 11.0 what pytorch version do I need and again on their website I kind of found this pip install torch 1.7 and by the way you need at least is that the did the yeah right here you need python at least 3.8 pytars 1.7 torch Vision zero 0.8 luckily for my GPU that I have and the Cuda version 11.0 uh the torch version was 1.7 is 1.7 torch equation 0.8 so I just copied this line from from the pi torch installation page and I did install these libraries and then I installed opencv python if you don't have it already or and matplotlib if they are not installed by default but more importantly after that step you go ahead and pip install this segment anything.get basically nothing but the repository I just showed you right so this this one segment anything dot get in my case uh I just downloaded this as a zip I unpacked it in fact I may still have it somewhere but I unpacked it and I did I did you know local pip installation I hope you know how to do that pip install minus E dot while you're in that local repository okay there you go and then finally you go ahead and download the default trained model and that model I believe is about two and a half gigabytes so make sure you have enough room but that is the model and if you want other models then you can follow this link that I just I I have all of those in the description down below and you can actually download this l v i t l s a m model v80b model again I have no clue what these two are but this is the default one that works on the generalized cases so I went ahead and downloaded this so that's it that's the preparation and once you're done now let's actually start using the code right so first let's go ahead and import pytorch and then let's import torch vision and in case you don't believe me let me go ahead and print the available versions by the way my python version is 3.9.16 and my pytharch is 171 and torch torch vision is 0.8.2 and Cuda of course is available so I'm all set to go if not it still works if you just work on CPU but it will obviously be slow okay next let's go ahead and import all the required libraries and now let's move down I'm reading my image using opencv it can be any library but I'm using opencv and I'm converting BGR to RGB so when I plot it you can see how it looks like obviously if you're using scikit image to read the image you don't have to convert BGR to RGB because psychic image reads images in RGB okay there you go that's the image I'm going to work with and again I downloaded this image from a quick Google search and these are all nothing but the cross section of these neurons so our goal here is to segment these neurons moving on now my my model or the checkpoint that I have again let's go ahead and open this the model right there and what is the size 2.4 gigabytes so that's pretty large but let's go ahead and Define our model right there and model type and the device equals to Cuda and let's go ahead and Define this model or segment everything or anything I guess that's what this is model this will take a few seconds to load and then let's go ahead and push that to the device okay so there you go and now let's do the next step and again I copied bulk of the code from from these guys from meta people you know who are generous enough to share it but again if you if you are creative enough if you need more stuff you know go ahead and look at other examples that they have provided but this I think uh 90 of the time this is probably what you're looking for okay and uh there are several tunable parameters now all we need to do is uh create a uh prediction right a mask generator and this mask generator let me create some room uh this mask generator is used is applied on your images to actually segment your images this is the terminology they use so I'm using exactly the same terminology or this can be segmentation Creator you know if you want to call it but this is basically what that is now you do that by applying Sam automatic Mass generator which we imported from the segment anything library right there so this method is the one that is automatic Mass generator by the way they have a few other methods you can explore them where you can click you can give the region coordinates for your objects so there are many things that you can do to provide as input but in this case let's just rely on the automatic Mass generator meaning it actually uh it actually models your I mean sorry it actually scans your image based on the settings that you provide here for example points per slide like side like how many points you want to you want this to automatically survey to actually create this generator okay so that's what we're using and our model that we're going to use is called Sam because that's exactly what we defined right here and the points per side and by the way did I put the link here right here let's click on this link one more the reason I put that link is it explains exactly what these all parameters are yeah it explains all the parameters down here so go ahead and follow the link to learn more about it but uh IOU threshold is something I played with this as obviously you can imagine so this is the threshold so the lower the threshold the more objects it's good you're letting through so it may pick up some crap from the background if you think there is a lot of junk that's being picked up go ahead and increase the threshold that's exactly what I did the default was 0.86 I changed it to 0.9 and the stability score uh threshold is again the score threshold so think of this as okay post processing after it actually did the segmentation do you want to include certain objects or filter certain objects out and that threshold I set it to 0.96 and we'll probably I'll change that to 0.9 so you can see how in fact let's go ahead and do 0.9 and 0.86 so you can see how the result looks like and we can change it so let us run this and once you define the generator now we are going to apply the generator to our image that's exactly what we're doing here and this takes some time so let's click this and pause pause the video for probably 30 seconds okay so it well it's done and now if you print the length of masks it is 284 that means 284 objects were detected in this and do I have the masks right there so if I open the masks variable right here you can see it's a dictionary and each dictionary you know you have area bounding box crop box point so all the information is right here and what is the predicted IOU in this case it's 90 and 95 percent of stability score so this is basically all the information for that specific object and how many objects do you have you have that's exactly what we printed down here right 284 objects in this image so we take all that information in terms of the area and the locations and here is a function that actually prints the the or places this information onto your original image so we can actually colorize these objects in random colors that's exactly what we are Printing and again the print or the plotting step may take a few seconds because we have a lot of objects so let's go ahead and plot it okay so it's done so now let's look at the plot right there so there you go so here is the original image and here is it and now I see a lot of junk in fact if you look at this area right there that looks like it's one large object and if I go back you see that's just a background it kind of you see why it thinks that it's it's an object so there is a lot of junk that I can clearly see I'm not sure what's going on right here if you see and right there so if you if I go back and forth so let's see if that gets improved if we change our IOU threshold to 0.9 and this to 0.96 for example okay so let's do all of these one more time okay and first thing first it we can see that okay there are 226 objects now and I forgot how many we got the last time 284 so by adjusting the threshold you can see how we got a lot fewer objects and hopefully more accurate objects more realistic objects rather than all this background and everything so this is like a sensitivity that you can adjust sorry these parameters and also go ahead and play with other parameters but you can see now how this this looks like yeah let me go ahead and expand this so you can see it in a better way now let's go ahead and compare this image with the previous one you see how I guess in the previous one it it also recognized the entire background as an object because you see how it colored it in purple but here you actually are detecting the real objects right there isn't that incredible now look at this small region right here right if I go back it was kind of weird it's now accurate and this is how the original one is okay so there you go and uh before I end this video I just want to change this to for those of you from remote sensing or not from microscopy you can actually see another example the house is not jpeg I tested this on a few images it works great on no matter what I uh no matter what image I used and I should mention what is the primary in my mind the primary I wouldn't use this yet maybe later for regular segmentation work but I may use this to assist me in annotations this can be a great annotating tool if you are tired of annotating your objects this can be a pretty good annotation annotating tool so anyway I changed this to houses and let's go ahead and run this uh instead of running the whole thing let's go ahead and run this so you can see this is the image and let's see what it does I haven't I I did test this but so I know what to expect in fact I think I included that on the landing screen I did all that in the last 30 minutes guys so I I'm I'm forgetting exactly what I've done okay so let's run all of these and have a quick look at the result and end this video okay it's done and as you can see it is getting different parts of the house as different entities right there so it's not getting the entire house one thing you can do is actually Define like boxes around each of these house so you define exactly what you're trying to segment but in this case it's segmenting the front porch or whatever that area is you see how it's segmenting this and this uh separate entities compared to the main main house like this actually looks like probably garages right there and it's also segmenting the background like the grass in many of these cases but it's actually missing so missing grass like in some some areas obviously right here so this is where you go ahead and adjust your uh adjust your settings to make sure it's uh it's segmented the way you want it to segment okay so with that let's go ahead and end this video I hope you find this to be useful I'll keep exploring this what I haven't done is how do we do for example binary segmentation using this how do we do multi-class segmentation how do we handle multi-channel uh data sets so all of that I haven't done this and in future I hope I'll do a video on that and again uh this got released only two days ago and I'm doing the video based on your request so please stay tuned for more videos related to this topic and until then keep learning and I'll see you in my next tutorial thank you

Channel: DigitalSreeni

Views: 50,370

Keywords: microscopy, python, image processing

Id: fVeW9a6wItM

Length: 16min 52sec (1012 seconds)

Published: Sat Apr 08 2023