305 - What is Cellpose algorithm for segmentation?

hi everyone welcome back in the last video I believe I mentioned the term cell pose and I also talked about how if I find time I'd like to make a video on it well I made some time to actually uh explore cell pose and how to install it on Google collab and how to actually train your own model and this video is all about that and if you want to be informed of future such videos please do not forget to hit the Subscribe button down below and while you're there find the thanks button if you feel extra generous now jumping back into today's agenda I plan on introducing you or going through the paper original paper for cell post just to talk about what it is and how it can help you with the segmentation instant segmentation I should say and why they call it a generalized model and after that I'm gonna jump on to collab and walk you through the code well not every line of the code will install cell pose and in fact I'm going to reproduce exactly what they have on Cell post GitHub page except instead of showing you the can data that they have I'm going to show you hey here is some data I downloaded from kaggle and along with The Masks of course and here is how you can go ahead and train it okay so with that information let's go ahead and jump into our discussion today first of all what is cell pause it's a generalist algorithm for cellular segmentation they call it cellular segmentation but it can work for other images also which we'll see later on but obviously the initial paper when they published it their focus was on cellular segmentation now what does generalist actually mean well generalist means it can very precisely segment a wide range of image types it's not like only you have the fluorescence images uh or certain type of cell images you can actually dump a whole bunch of diverse looking images and this has the ability of generalizing it and uh okay now let's move on how does it actually work now here is again uh directly an excerpt from the original paper here's the link to the original paper but you can clearly see exactly their process they are not directly predicting the masks like you would in other approaches like for example NASCAR CNN you're actually predicting the regions of interest and also the masks but in this case they are converting The Masks into a different space in fact they generate an auxiliary representation for the masks and how are they doing that as you can see think of having an energy or a heat Source right in the middle of uh an object in your mask right so that's a cell and at the center of the cell you have a heat Source now let the heat radiate right so that's the energy function for each mask as they're representing that as equilibrium distribution of a heat diffusion simulation so it's basically you have a heat source and you're letting it diffuse and you kind of look at the distribution of this uh uh of this and this is basically creating this flow representation and Alternate representation of your mask and then you're looking at the horizontal and vertical gradients of your of your uh basically the heat representation right there and the important thing here is if you have the heat representation or if you have the flow Fields then it is possible for you to calculate the original mask back because what's a I mean you don't need a heat flow right I mean you need your predictions you need your final mask so this is reversible in addition to these two did I mention did I copy that here um yeah in addition to these two spatial uh gradient Maps like these heat flow Maps like horizontal and vertical they're also predicting another one that basically tells whether the pixel is inside or the cell or outside the cell so that's another one that they're predicting so all these three are combined into what you see as a flow field and once that is done then the network is actually trained on on uh these these flow Fields um so once once this is obtained now they are reversible so the masks are actually predicted by just going back and the network that they used for this training is or the one that we are all familiar with which is basically the unit framework they made some changes but uh eventually this is just a regular unit right so regular unit is extremely powerful it is a great approach for semantic segmentation type of tasks and in this case they are feeding instead of a direct mask they are actually feeding these flow fields summary of how cell pose actually works now what type of data set the more diverse the better it is and you can download the data set that they have used in training their model and as you can see from the images they range from images like these fluorescence images to these grayscale images these are like a bunch of shells not even like shells but you can see I think they also have like plant cells and a whole bunch of diverse looking images like pebbles right there uh so that's the diverse looking data set that they have and how does it compare with other approaches I mentioned mask or CNN I'm still a good fan of big fan of NASCAR CNN but I like certain approaches like cell pose and stardust because they actually work great for a certain type of applications now they have compared it again I I definitely recommend you reading the original paper it's pretty easy to read by the way and you can see this is what we're looking at compared to cell post artist and NASCAR CNN when you look at IOU uh threshold so you get like much better uh much better representation right there of uh uh your IO this is basically showing you the average Precision so you're getting higher average Precision in this case I believe it's showing you the uh what is it showing uh I'm sorry uh yeah still average Precision IU matching threshold so you generalist model specialized data specialist model specialized data so this is a specialist when it says specialist model specialized data that basically means all images look the same and you have a similar a model that's trained on images that look very similar when it says generalist model that means uh diverse type of images you know how does it work on diverse type of images So eventually when you have generalized type of data when you have a generalist model you are getting the best uh Precision using the cell pause so so there is a reason why I'm I'm making this video because I think this is useful for certain type of applications especially when it is when you're looking for a generalist model okay now finally how do the results look like again look at the diverse results and the best way for this to work is lots of diverse images just train the model on lots of diverse images you need not to annotate like thousands and thousands and thousands of images especially if you are using cell post 2.0 and cell post 2.0 this is a paper from about a year ago and as you can see it's a new package that includes an ensemble of diverse pre-trained models as well as human in the loop pipeline meaning you look at the results you're like I don't like it let me correct a few of these and feed it back to the network to actually get a better representation better trained model and the good thing is they also provide a user interface if you don't like to code you can actually use a UI to do the part and I'm not going to talk about the GUI aspects of it you can go to their GitHub page download it and test it out I'm just going to show you how you can get started on uh Google collab so cell plus 2.2 is all about okay you have cell post 1.0 which is a whole bunch of diverse looking data you train your unit model on the flow fields and you have a new image automatic segmentation you're like ah that segmentation is not good go ahead and correct it manually go ahead and add it to the training and then eventually after certain certain iterations you have a very nice trained model that actually works on on your images so that is a quick background of cell pose and let's go ahead and jump into the code and uh by the way I'll leave all the links to whatever I just showed you as part of the description and also as part of the code which is a Google collab notebook so you should be able to access whatever I just showed you as part of the presentation uh in terms of the links that refer to cell pose and here is for example the cell post 2.0 Paper that actually talks a lot about what you can do and much more obviously than what I covered and the way I plan on doing is not using their images because I just don't want to copy their code and show you hey here you go you can do it I just want to show it on a different data set so you know that okay this is how I need to get my stuff ready right so basically the you need image and mask pairs like any instance or semantic segmentation and since this is instant segmentation The Masks need to be in the format of instance segmentation which is not cocoa format or any of those basically a uh a let me open the images that I plan on using so you can see exactly what I'm talking about so here is an input for example one of the training images and here is the training mask and as you can see each object has a unique pixel number associated here so let me open this so you can see keep an eye on the bottom area right there so if I put my mouse right there this object is assigned a value of 18. that object value of 19. this one has a value of three that's value of four and there is an object here that has a value of two another object here value of one you can't see it clearly because it has a pixel value of one but you get the idea each object has a unique pixel pixel value which is basically nothing but your instant segmentation mask it's easy to create I look at my playlist on how to annotate your own images so you can find some of these over there now in this specific exercise I have uh I do not have time to annotate a whole bunch of images I downloaded something I just went to kaggle and I searched for some uh instant segmentation data sets and I I just found a data set that that's that's very challenging but that's okay it gives us a good start and I downloaded this and that's exactly what one of the images I just showed you I copied all of that into my Google Drive right here and if I go into actual train folder these are all the original data set let's can I change the view here yeah so these are all the images from the original data set as as uh you saw in the screenshot earlier but if I go back I created a data directory called SCM images and here I have train test and other images is some test images I would like to randomly segment later on once the model is trained but I have train and test and I got those two from the link I just showed you on kaggle and the train images I have images and Maps which is masks and under images you can see all the images and they look very diverse here right they are not the same they are diverse and this is a good test for for my training model and I only trained my model for 500 epochs which took about 40 minutes or so on collab but but you can uh I can tell that it's still converging so give it like 1000 to 1500 epochs just see uh how how this quality of output segmentation is but this is this is the information with this information let's jump into the code now okay now uh first thing first I don't have to tell you change the runtime type to GPU do not waste gpus if you don't need it don't use it but since because it's a shared resource all of us need it so I enabled it and here you go all the links and as of today the version that I'm using is python 3.8 1.16 and I already mounted my Google Drive because I need to access the the training data once I have that and all of this code again bulk of it is I modified certain aspects but bulk of it is directly from their GitHub page which is right here I should have showed you this so you can see how you can go ahead and download I think you can download the GUI version right here or they probably have it somewhere let me move up to right there so they have a GUI version go ahead and download it if you want but basically IP installed it and I looked at one of the notebooks and I adapted that for this tutorial that's it I'm gonna share this look at look for the link Down Under the description okay so with this information let's go ahead and run the first cell which is basically installing all the required libraries including tip file oftentimes you find yourself working with scientific images that are in tip file format maybe you have only two channels one for nuclei one for uh one for uh cytoplasm and if a bunch of other like cell posts for example that's one of the things we are installing here and this may take a few seconds so let me pause the video and continue as soon as this is done okay so this is done and uh normally when you do pip installation you restart the runtime but uh but here it's basically uh by adding this it's gonna it's gonna restart the runtime as you can see it says your session has crashed and then it kind of restarted the runtime yeah so you don't have to do that now let's run the next set of uh code and again all this is looking at first let's go ahead and run it and you can see it's looking at your notebook version what is the network we are going to use and again there are a few networks that you can use look at the documentation I'm going to use the cell pose one and we are going to import all the required libraries as part of this stage that's that's in in fact other than that I can I can't see yeah in addition to that there is a function for PDF export this is basically after the training is done all the export is captured as part of a PDF document so this is everything that you need to create that summary again I haven't written a single line in this in this block of code so there is uh in fact I didn't even look at it because if I need to write my own way of doing PDF files then I look at how to do that but in this case I'm it's just it's just a another tool for me in achieving my tasks so I don't want to waste anyone's time in talking about it neither I'm an expert at talking about that but the key Point here is that block is basically something that like a prerequisite go ahead and run it and now let's go down and make sure that we have a GPU access which we should uh this is this is Google collab sometimes you may not have the access if they are all busy but I I expect to see uh I expect to see this so Tesla T4 that's good 15 GB of RAM that's sufficient for us more than sufficient now uh this is the now we have to define the parameters right so first of all where is your training images coming from where are your masks coming from and by the way we have the markdown uh here I hope you know what that means it's basically as part of the code your for every line let's go to the end you're actually adding this so instead of meddling with the code or instead of typing stuff in the code you can actually uh quickly Define those as part of the UI that it generates as part of your notebook yeah so here the training source is my training images I dumped my training images under drive my drive collab notebooks data cell pose ah let's say SCM images and the train images and then these are the images right so the images so all I did is copy the path to it and paste the path right there same with the training Target and what is the name that I would like to give to my trained model so it can create a directory with this train model and then save all the required stuff in there so but the name I'm giving it is cellpos sem tutorial obviously that's what I'm trying to do here and the model path where do you want to save it I want to save this under sem Images slash model so if I go down here if you see under sem images I have a directory called models and within models it actually creates a directory with this name previously I trained a model called cellpose sem3 ysem3 my scm1 was 50 epochs I got crappy results the next 100 epochs are 200 epochs and scm3 was like 508 bucks that's the I should have named it better but anyway that's what that was and this is the model name and number of epochs this is where I type 500 but for this exercise let's go ahead and type 50 I'm going to stop it in the middle anyway so it doesn't matter and channel to use for training you can use grayscale you can use blue only green only red I'm using grayscale all all images I have are grayscale images and there is no secondary training Channel there is uh I'm gonna use the default Advanced parameters you can also modify those as part of the code on the left hand side if you want but uh but it makes sense to create an interface so you can easily change them on the right hand side okay so let's go ahead and run that so that should be relatively quick and it's grabbing a few random images and showing me hey there is my image there is the channel used for training and that is the training Target and looks good to me now let's go ahead and say check this box for uh use data augmentation or you can just come here and type true you can uncheck you can type true either way I want to use data augmentation so let's go ahead and check that and now let's come down to do we want to use pre-trained weights I checked this box uh for example I trained this model for 200 epochs and I'm like ah the results are not great let me add 200 more epochs then I would check this box and say hey I'm using my own model and go ahead and give the path to that model or you can just say use any of the pre-trained models when you install this uh the cell pose you can actually access the cytoplasm cytoplasm 2 and nuclei models maybe nuclei model is not a bad model to start with in this case but I I start from scratch let's see where it goes yeah that's exactly what I did for my 500 epochs so I'm not using a pre-trained model so let's go ahead and run this block of code and now coming down this is the training the network path and here wait it's basically looking at creating a local you know directory under a slash content with the model name so it can save it periodically while the model is being trained and all that stuff yeah so let's run this and this may take a little while if you're running it the first time because it's actually copying all the training data into your local instance on collab so this will take a few seconds as you can see it's not that slow but but it's still copying it to another drive so let's pause the video for again a few more minutes and then I'll continue with the next block okay so the copying is done and as you can see on the left hand side let me collapse my local my Google Drive and you can see how it created a new directory locally in that local instance right the drive is my collab Drive everything else is the local instances and as you can see there is a directory called cellpose sem tutorial there is a test folder there is a train folder and I think uh right while it's actually training it creates another folder called Model within the train folder folder where it saves uh where it saves the train model so let's go ahead and start the training process and again there is not much here it's basically doing the training process and all the information is dumped into a PDF export remember in the first block we have a PDF export of everything and that's exactly the function it's actually calling down here so let's go ahead and start the training process and have a quick look at what's going on and I'll start this and I'll show you how we can load a pre-training model and then go ahead and segment future images or some random images from the test data set here so it is you see how not all flows are present whatever the data it's giving so basically we don't have the flow map so it's actually Computing the flows for the labels remember cell pose it's actually predicting the flows so it needs to calculate this flow based on this heat thermal diffusion so that's exactly what it's doing for all the masks and in each mask it it has to do that calculate the flows for each object so it's going to take some time but again it's its computation is pretty fast nowadays so it is going to be it's going to be done pretty soon so either way let me pause the video one more time so I don't waste your time okay so it's almost getting there but while it's doing that I wanted to highlight one other aspect here as you can see how in the train folder you have both image and mask so it took your original image and corresponding mask and it copied both of them into a single train folder with the same name except with an extension of underscore image and underscore masks okay so one point to note over there and now it's actually Computing flows for the labels right there I believe for test maybe and now 136 train images with masks and a number of images per Epoch and it's looking through all the parameters that we defined like batch size and learning rates and so on and hopefully it will start raining anytime soon yep there you go you it created a directory called models within the train folder so now it started training so epoch0 whatever the time and the time is cumulative so it tells you after after Epoch 10 it took that much time uh what the loss is and it saves it into this model's directory and at the end of the 50 epochs it actually saves another copy of the model in our models folder that we defined which is nothing but let me close that or drive my drive collab notebooks and data this is a bit too deep there sem images and models and this whatever you see how it created another directory right there for tutorial so it's going to take this model the final one and copy it into this tutorial but if I open the one that I have already done you see how there is a PDF file with all the training report and there is uh the this is the actual model that I should be using okay so with that I think we can actually go ahead and close or stop of this execution because you can see how it's doing epic Epoch zero Epoch five and so on basically you'll be staring at the screen but let's go down and see what we can do now it's time to evaluate the model and there was a function I believe not here uh to evaluate this again I'm not worrying too much about every line of the code you can go through that in fact I did do that once and then I'm like okay nothing for me to change there but evaluate the model you can use the current trained model in which case it's going to look for okay what is the current train model but I'm not going to use that I am going to use the model that we trained earlier which I believe I think it's this one SCM images CM images under models let's let me confirm this one cell pause sem3 yeah that's the model I want and if using cell pose or omnipose models please indicate where you want to save the results I'm not using the default cell pose or omnipose model I'm using my own model right I'm not using any of these models I'm using my own model so there is the path so that with this information let's go ahead and run it it's going to evaluate the model using the test data set that we have provided and when I scroll all the way down that's a lot of code right so I'm glad someone else already did that so it says okay cell pause scm3 model will be evaluated so it it loaded the model now comes actually choosing the folder that contains our Quality Control Data set which is our test data set right so Source QC folder you know or test let me quickly confirm this yeah this is our test images uh Target QC which is our masks this should obviously be the test masks right there and channel grayscale nuclear Channel we don't have any and object diameter I'm gonna leave everything these default and run this and hopefully it should run let's scroll all the way down it's actually prediction on each of these images and I only have like three images in my evaluation directory uh and uh let's see how good the training uh training was done now on this image you see how the results are not great it's not it's not that great maybe this is exactly why I say hey train for another 500 epochs or something to get better results but uh anyway so this is what I got and if I go to another image you see the drop down so let's select another image on this image it actually did a pretty decent job with an intersection over Union of 94.6 in fact if you have 500 images as part of your test you'll get a nice report of everything so let's check what the third image in our training data set in our test data set and this result is super amazing I'm absolutely happy about it 95.8 percent now you may say oh I can get much better results using NASCAR CNN or something else yeah of course you can the uh if all your data set is similar but if you're looking for a generalized model this seems to be a much better approach and of course you should train enough epox okay my model was still converging if you train enough epochs then you will have the better generalized model using this approach compared to NASCAR CNN at least if you follow the paper I haven't actually in fact that's a good exercise maybe I'll do a video on comparing a generalized model on Stardust cell pose and this although it would be a repetition of what these guys already did when they published the paper but it's fun when you do things your own uh and then and then learn by your own experience that's it so now uh you can provide your future predictions now you say oh I have an image somewhere else and I want to segment it you can go ahead and do this and you only see this part of it you can just go ahead and edit the code if you want you know show the code and go ahead and edit it I'm pretty sure you know that but yeah if you want to segment future images I don't know I have no clue what image I have let's go ahead and run it to see how it looks on our on the images it never saw before or on on images on a image I literally downloaded this off a Google search so I'm not even sure what to expect out of this but you can see predicted cell pose it's basically a matter of adjusting the threshold I believe so I think I should go back and uh look at the flow threshold and mask threshold if I bring it down does it give me better result so this is another one that we should see I only trained it for 500 epochs so why not adjusting the threshold right so because it's doing an amazing job with my flow through uh flow graphs right there that means it's predicting the flow very well it's probably just a matter of the threshold right there so just just play with the threshold values let's just bring it down and see what how the output looks like so yeah when I decrease the flow threshold you see how it's actually picking only some of these I I don't know what to expect here so let's go ahead and this is where reading the documentation definitely helps and not just doing an experimentation but you can see how it started to actually now fill these things in right so if I actually yeah I want to get much better flow threshold decrease the mask threshold and that should help us in getting better better segmentation let's give this one more try yeah it's slowly filling in although these two are recognized as two different objects but I I would like to play with I would like to play with a combination of these two but either way that's that's actually not bad given that I did not train uh my I mean I did not rain on images that look like this again I want to reinforce this point if you have a whole lot of data set that uh you know with masks that look of images that look like this try mask rcnn in fact if your images look like that try Stardust that does an amazing job as long as all the images look kind of similar but for a diverse data center where you want this model to work then cell pause really works great as we just saw in this case I downloaded this off internet and that actually seems to work pretty decent and also the one of these data set it's actually doing a pretty decent job okay I should end this video it's already too long now I hope now you know what cell pose is and what is it meant to do for what type of applications you would like to use it any application as long as you want to generalize your train model and you know how to use it on Google collab and I'll share the code go ahead and look at look at the link under the description and again do not forget to hit the Subscribe button if you are not already a subscriber thank you and let's meet again in the next video

Channel: DigitalSreeni

Views: 6,786

Keywords: microscopy, python, image processing

Id: d3RrbeJKNoc

Length: 30min 3sec (1803 seconds)

Published: Wed Jul 26 2023