MedSAM Segment Anything in Medical Images Universal medical image segmentation.

this video I'll be talking about at a high level about this new Med Sam what is med Sam Med samam is a foundational model which is designed for medical image segmentation uh this is from researchers spread across Canada and us uh so this is about segment anything in medical images so medical image segmentation is a critical component in clinical practice facilitating accurate diagnosis treatment planning and disease monitoring so this particular medam was a found is a foundational model and it is you know trained on a large scale medical image data set with close to 1.5 million image mask pads covering 10 Imaging modalities and over 30 cancer types so they also conducted a comprehensive evaluation of this medam model on 86 internal validation tasks and 60 external validation tasks and this demonstrated better accuracy and robustness than modality wise specialist models okay so as said before segmentation is a fundamental task in medical image analysis which involves identifying Ing and deating regions of Interest okay uh these regions of Interest could be uh organs lesions tissues right now accurate segmentation is essential for many clinical applications including disease diagnosis treatment planning monitoring of disease progression for example you want to look at the growth of a tumor okay or you want to see if the tumor has shrunk post chemotherapy or radiation therapy right so you need to segment the tumor in images so manual segmentation has been the gold standard okay but the process is time consuming labor intensive and often requires High degree of expertise so there have been lot of deep learning based models which have shown great promise in medical image segmentation but they are not generic okay one of their limitation is they are task specific nature so if there is a new modality coming in or a different kind of Imaging coming in then it kind of fails right recently uh in last year meta released a model called as segment anything okay now essentially segment anything model is a prompt segmentation model that requires points or bounding boxes to specify the segmentation targets okay it resembles conventional interactive segmentation methods and Sam had better generalization ability uh compared to existing deep learning based interactive methods for segmentation okay now the question is can Sam be extended to Medical images right so a lot of people tried on making use of you know fine-tuning Sam or segment anything model on medical images the performance was satisfactory uh right uh uh it showed satisfactory segmentation outcomes on targets or uh you know Prim targets character by distinct boundaries but there was substantial limitations in segmenting typical medical targets with Bak uh with weak boundaries or low contrast okay so Sam could not work on these kind of images that is why these researchers have come with Med Sam which significantly enhances the segmentation performance of Sam on medical images that is the idea over here okay so here here they are talking about you know their uh data set okay and this uh uh The crucial aspect of this model is that it is um it should be able to accommodate a wide range of variations in imaging conditions anatomical structures and pathological conditions so what they did was uh they collected uh this data set of close to a 1.5 million medical images okay that is what they are um showing over here so it has been trained on wide variety of images across wide variety of modalities okay uh so the data set covers a variety of anatomical structures pathological conditions Medical Imaging modalities all right so that is what they are showing over here the Magneta part over here is nothing but the mask right which denote the expert annotations and Med s um Sam um you know uh results respectively so these are the results which they are showing over here you know how accurately it kind of U masks U basically it extracts or segments this region with respect to the um you know human segmentation right and with respect to the modality distribution over here uh if you see uh these are various modalities like Mr C endoscopy ultrasound x-ray pathology fundus images dermoscopy mamography and OC these are the number of images in each category of the modality over here this is an imbalanced data set for sure but they have tried to cover different um you know clinical branches um like uh you know uh Gastrology uh lot of different uh modalities um or say anatomies are covered over here okay so the CT and MRA are 3D images so coming to that what they are seeing is is that uh they are primarily they are developing a prompt 2D segmentation model okay and for 3D images you can consider 3D images as a series of 2D slices okay um that is the idea over here and in Sam you could do free text based prompting for segmentation you could do Point based prompting you could do bounding boxes but here they are using bounding boxes as prompt okay and their architecture is same as um you know meta samam architecture so there is an image encoder a prompt encoder and a mask decoder okay so the image encoder Maps the input image into image embeddings a prompt encoder uh you know um it actually converts the bounding box prompts into you know a internal representation then there is a mask decoder which looks at this uh bounding box prompts and the image iddings to come up with your final segmentation okay so that is the idea over here so the model is similar to meta ai's Sam architecture okay so that is about the model architecture which they talk about over here so the image encoder Maps the input image into a high dimensional image embedding space the pr encoder transforms the user drawn bounding boxes into feature representation by oppositional encoding there is a mass decoder which fuses the image embeding and prompt features using cross intention and based on that it de it does the segmentation that is the idea so I'll not go into the performance and other things over here okay I'll not go into the details of that you can read through it and uh you know so going to the conclusion part over here what they say is that uh medam is strained on a meticulously assembled large data set comprising of over 1 million medical image mask Pairs and it demonstrates substantial capability in segmenting a diverse array of targets and they say the performance not only significantly exceeds that of uh stateof art segmentation model but also rivals or even surpasses specialist models for example if you had a specialist segmentation model for Mr especially segmentation model for ultrasound or something like that this kind of uh is equal or better than that okay that is the idea over here and uh you know by providing precise delineation of anatomical structures medam facilitates the computation of various quantitative measures that serve as biomarkers so they have some examples over here right like for uh 3D tumor anotation process so you can calculate uh tumor progression and things like that okay and uh what they're saying over here is that this also provides a successful Paradigm for adapting natural image Foundation models like Sam to new domains okay and this can be further find tuned on say your imbalanced U data because this is an imbalanced data set right if you look at um the main set of images are ctmri endoscopy over here but there are smaller modalities also so they can uh you know if you want to improve this model you can find tune on more images of same mography so this particular model can also be fine-tuned to other biological images like microscopy images or organal segmentation or uh in say electron microscopy images things like that can be done okay so this is a limitation that there is a modality imbalance but fine tuning can overcome that another limitation is when you have segmentation of vessel like branching structures bounding box prompt becomes ambiguous because arteries and veins share the same bounding box in KN FAS images this is an example so here uh medam has learned rich and representative medical image features so further fine-tuning can improve it that is what they are saying over here okay so this is a very interesting paper uh so if you are working in this field of medical image segmentation you can definitely look at this paper and this fundamental model and see if it can be used in your particular um Medical Imaging tasks okay so the best part is that the code is released they have already released the model over here um they have also explained over here on how you can train with your data how to pre-process data and things like that right uh they talk about that over here they also talk about how you can train it on Multi gpus they also have a collab notebook which you can actually try okay um so this is a collab notebook which they have given um so what uh here they have explained how you can actually try this model okay so I actually tried it out um over here and for this you need to install this particular uh met samamp from GitHub okay and then they have some nice functions for visualization okay they have some visualization functions over here which can show you our bounding boxes and visualizations they also have the inference part over here how to do the inference there is an inference function over here and then they talk about downloading the model and the data over here how do you can actually load the model and how you can inference how you can do um your image pre-processing and model inference and visualization of the results so this is an example which they show over here this is the input image and bounding box right and this is the medam segmentation if you if you look at this particular image there is this particular abnormality over here and there's a bounding box around it and here is a segmented abnormality okay so you can go and try out this U notebook you can make changes to this notebook to see with your own images if it is working as promised right uh so your input image should have this bounding box um so yeah so basically you need to um specify that along with the input images the ask the bounding box uh basically the bounding box uh coordinates and you know uh you can try medam segmentation so this was a short video on Med samam which is a fundamental model for Segment anything on medical images okay so they also provide you a um graphical user interface where you can load the image and specify segmentation targets by bounding boxes okay so you can actually download it on your machine and you can try it out uh there is a step byst step uh this thing your collab tutorial but you can also run it locally and um you can have a GUI on which you can do this uh you know you can uh what do you call Mark the bounding boxes and then uh you can give this image to medam and see how the results are I hope this video on medam is useful to you I'll putting the link to the paper um to the GitHub uh repository you can go and check it out if you like the video please like share subscribe to the channel see you in another video

Channel: Rithesh Sreenivasan

Views: 2,559

Keywords:

Id: XRkavmT9URY

Length: 13min 40sec (820 seconds)

Published: Tue Jan 23 2024