330 - Fine tuning Detectron2 for instance segmentation using custom data

hi everyone welcome back I hope you watched the last tutorial where I provided a quick introduction to detectron 2 framework and we have actually used a couple of pre-trained models to segment a natural scene image where we looked at individual cars that got segmented and individual humans but as scientists and researchers we have our own images to work with for example you see an image of a bunch of cells with some organelles that got segmented on the right hand side again I did that literally this is the goal for today's video where we try to achieve the right hand side and of course quantify it and get some area measurements and so on so how did we do that that's the whole point of this video going from taking the images and annotating them loading them onto our Google collab for detectron 2 based image segmentation or instant segmentation using mask or CNN as the engine that can that powers up the whole thing and if you don't know what I'm talking about please do watch my last video and if you want to be notified of future videos just hit the Subscribe button right now if you haven't already done so and while you're there find the little thanks button if you're feeling extra generous okay so like I summarized our goal is to just understand this datacron too in fact I have the next slide where we laid out the plan step one is to get our training data ready and for that there are various ways of annotating your data you can use whatever your favorite program is as long as you can get Coco style GSM format annotations and if you don't have a favorite program then we can just test this makesense.ai which is what I ended up using in generating a handful of labels and then we'll walk through the code in Google collab to train and of course to segment future images but first let's understand the image segmentation challenge right I mean we get hung up on this code all the time we're like oh let's use tensorflow let's use pie tars well if you understand the problem if you understand your images what is the image segmentation challenge maybe a simple also based thresholding is enough right so what are you trying to achieve from your segmentation and let's look at the example I'm gonna go through and you can download this data set and the link is on the top right bottom as you can see libmanlab at github.io dance cell so this is cellular segmentation even if you're not a biologist I'm a material scientist I pretend to be a biologist but but biology images are fascinating because they offer a bit of a challenge if you want to segment them although geology and uh you know remote sensing all type everything is a challenge I take my statement back there so the goal here is to segment cells and uh whatever the subclass sub or organelles you can call them and these are platelets and they were used using XI sigmat review by by this uh research group that I'm referencing at the bottom left now how do the images look like as you can see these images I should admit I downloaded this data set and I converted all 16 bits to 8-bit images I mean you don't have to but I did and for training I trained five images why only five I I lost my patience after five images and I wanted to see how the results look like they already look great but obviously any deep learning the moral training data the more robust the trained models are going to be so of all the images that I downloaded in the under I mean they have training set they have testing set they have validation set I just looked at the training set and I took image number images numbered 1 11 21 31 and 41 and I use those for training and two images numbered 5 and 35 I use them for validation in case you plan on reproducing exactly what I have done you can see exactly what I've done here all other data I just put it in the test data set so we can just test it out the real test is actually taking this training data and applying on the test data set that they have provided This research group has provided so this can be a very nice project by itself if not a master's project it can definitely be a nice side project if you're doing your Bachelors okay now if you look at this image and the corresponding uh uh classes that need to be annotated they have uh of course you have the background we don't need to annotate the background for instance segmentation but for the objects itself you have the cell which is the entire large area right that is the cell within each cell you have different organelles like mitochondria unfortunately you don't have a lot of mitochondria in these as you can see in this specific image I have one two three mitochondria in red or four actually that's pretty much it and I annotated five images in fact in five images I got like 12 mitochondria or something so definitely not enough so when I segment I probably expect lower accuracy for mitochondria but we'll wait and see and Alpha granule are these medium gray blobs that look like mitochondria but then they are not as dark so they are like medium gray and this carnicular vessels they are this lightish lighter gray blobs they're represented in green color on the screen and in addition to that there are a couple of other regions called uh the dense granules and dense core but they are even more rare they I saw them only in a couple of images so it's not worth for a tutorial to look at that but in a real life scenario obviously you're gonna you're gonna focus on those so uh what did I do I uh used make sense again we are going to use that right now and I Define these classes called cell Alpha granule mitochondria and conicular vessel and then I annotated here is the annotations this is how I kind of annotated like I for each image I just went ahead and used the polygon tool to Define around these objects again we're going to do that on one image right now and then we'll we'll understand exactly what it takes now after you're done with all of these uh images then you go back I mean you hit on export and I exported them as single file Coco Json format that's it I have done nothing other than that it so we're going to follow that in a second so let's go ahead and jump to the code in fact before jumping to the code let's jump to this make sense web page and upload our images and go ahead and annotate to get a feel for what it looks like to annotate these images for instance segmentation tasks and of course the same annotations can be used even for mask or CNN if you want because it's the same format okay so here we are this is uh make sense uh dot a I I leave all the links down as part of the description and uh first of all how do I have my data so when I downloaded the data of course uh I took uh let me go back one step let me go back and here is how I kind of organized it I have the train directory test directory validation directory and I also created two additional directories called test results because once I segment these test images I want them to be stored under test results I will also show you how not to just save these bounding box images and all colorful images but actually for scientific image purposes what is the reason why you're not doing this because you want to for example do some Downstream analysis put all these images back together into some sort of a 3D format and then extract certain metrics I'm just thinking out loud but whatever it is you may want your results to be in instance format meaning you want a binary image only for cells uh another binary image only for mitochondria and so on right so if that is the case if you want to import this into I don't know image J for example then this is a uh that that's the placeholder I'll show you how to do that so in my train I should have five images like I just showed you one two three four five and this is the Json file I downloaded after annotating them and uh same with validation I have two images and the corresponding Json file obviously for test images I don't have any ground truth so I do not have a Json file because I I don't have time to annotate 50 images for four different classes uh but yeah that's the images that I can use for testing purposes so with this again now let's go to make sense I just downloaded like five images the train images I have them accessible here but let's go ahead and hit get started down here and now click here to select or drop images so let's go ahead and drop all of these five images right here okay and uh oh now uh we need to pick a task is it a image recognition no we're not classifying the entire image we are actually doing object detection let's so let's click on object detection now let's go ahead and Define our classes what did I call them cells and another one is uh mitochondria uh and another one is I think Alpha granule yeah and the next one is carnicular carnicular vessel okay uh let's not worry if the spellings are right or wrong if you don't like a specific color I think you can cycle through them if you don't like yellow hey it's tough to see you can just click on it you'll cycle through like different colors so go ahead and do that and now you're gonna start the project okay now we are all set to go ahead and annotate them and you can pick which ones to annotate let's say we are annotating the mitochondria this is where I looked at the ground truth that uh these guys actually provided and then looking at that I'm annotating these and I found this crosshairs to be irritating so I turned the crosshairs off right there and now you only have this and now you can go ahead and zoom in and use the click and drag thing click and drag I this is how I start top left I go down and then I I go down and so on now let me turn this tool off and just with this I on the right hand side instead of point line I am going to select polygon right here and now let's go ahead and you should be a bit more careful but I think I'm not doing a bad job right there and as you can see there you go there you go and click back on the first one so it closes it up now what label do you want to provide this is a mitochondria so let's go ahead and select mitochondria right now let's do one more uh I don't know if this is mitochondria but it looks like it yeah this one is mitochondria right so I'll go back and I will annotate all mitochondria first because I don't want to switch between different classes when I am annotating that's not an efficient way so let's go ahead and Define this it automatically adds that to mitochondria unless you change it so let's say you're done with this image with all mitochondria you just click on the next one and go ahead and continue the process so let's do this one more time so I can show you and now I am going to draw click click click click and there you go and and it automatically adds that as mitochondria now you do that for all the images now I go back to the first one and now I am going to zoom well let's zoom out because I would like to do the cells so let's actually add this as a cell uh I recommend you to add sell the last because once you do that it covers the entire cell and it makes it a bit challenging for you to see the stuff that's inside because now we have classes inside another class right so let's leave this right there right there right there right there and there you go and adds a cell you know what I mean now but it calls this mitochondria right so let's change this to cells and that is a cell let's add another cell now so let's add this one and it should stick with the cells label I'm going to be sloppy right now so I can make the point but let's do that and you see how it adds this SL so and I unless you change it it's going to keep continuing with that label that's why you need to be a bit careful when you're switching from one to the other this is exactly why I recommend doing all the class one like mitochondria for example first come back and do all the other one and then come back and do all the other and once you're done you go to actions and Export annotations and this is where select Coco Json format and Export and it exports well let's click on it it exports a Json file as you can see and that is exactly the file that I placed along with my train images right here and this is exactly the Json file that's it other than that I have done nothing now you know how to get started with your annotations now let's go ahead and look at the code let me zoom in a little bit to make it easy for everyone to see I hope you can still read it but zooming it a lot makes it challenging to uh sometimes it can be too big but that's fine okay so this is Google uh collab I've already connected that to my drive as you can see on the left hand side under my drive I have collab notebooks and I have my data and I have my 3D em platelet data I have test all the same structure that I showed you in addition to this I also have a placeholder uh in models called detectron 2 underscore models so I can save any of the models under uh models right here okay so that is that I can close this part and now let's go ahead and walk through the code and runtime no need to mention change the runtime to GPU because we do need GPU in this case on a Google collab okay so now let's start by installing the electron 2 in the last video we actually did go through this exercise so this is the code that makes it easy to install detectron 2 on Google collab but if you are installing it locally then it's just my pip install and go ahead and do this git dot get okay so obviously don't do both you don't want to overwrite and you don't want to confuse the system but we are installing Pi ml and also uh detectron 2 somewhere down here uh yeah we are closing the detect Run 2 repo right here and we are running the setup just copy and paste this code so it should work right away and after that is done we are going to uh import pytorch because detectron 2 is based on Torch and we are also importing datatron 2 and let's go ahead and print out the versions of tars Techron and what Cuda version we are using and all that so it makes it easy for you if you're watching this video two years from now I don't know this is uh August of 2023 so hopefully this this video is still relevant based on when you're watching it but uh anyway uh the torch version is 2.0 Cuda 11.8 as you can see in detectron is 0.6 with that done let's now get actually into the meat of it I am importing to Techron 2 and we are importing all the required libraries and model Zoo because we are going to experiment with a pre-trained model initially and then jump into training our own custom model I promise I don't waste too much time this is for those of you who did not watch my previous video but still I just want to make sure you're we are on the same page uh and default predictor get config we will talk about these once we uh get to that specific cell but let's go ahead and import the required methods here okay so the default models are trained on natural images with a whole bunch of classes in fact let me go to edit clear all outputs so we don't see anything because obviously I did test my code it's working but I don't want you to be distracted with all the previous outputs okay so uh I already downloaded an image called kitchen.jpg and I placed it under my drive here and I am going to run it so we can see exactly what that image looks like again I zoomed in a lot so you see this image as humongous but uh you get the idea there is a whole bunch of stuff going on here including this this uh semi-transparent glasses and there's some stuff in here so let's see if a pre-trained models actually do a good job how do you do that we need to instantiate our config file our configuration and you do that using this get config method this is one of the things we imported from datatron 2 and I labeled it config CFG now go ahead and get the model the structure of the model which is NASCAR CNN R50 right there so this is the model that uh we are adding to our configuration file and then I'm setting the threshold to 0.5 and threshold is basically hey there is a let's say 60 probability that this is a chair then if the threshold is 0.5 yes it is a chair but if the there is 40 probability then this threshold is not gonna let that happen right so that's what the threshold is and uh here we just got the models we need to fill the weights in the models instead of random weights we need to fill that with the pre-trained weights that's what this step does excuse me and we are creating a predictor object using the default predictor method that gets applied onto the configuration file that we just created or to the configuration settings that we just created and we can use the predicted method on a image which is what we are doing here right our IM is nothing but this image and that's what the output is so let's go ahead and run this bunch of code and once we get that let's go ahead and look at how the outputs looks like and this outputs again is a container it's not just a it's a dictionary it's not just a numpy array and what does it contain it contain the instances the predictions and let's go ahead and print out the instances like the both the classes and the boxes so when you print that out you can see right there these are all the class numbers there's a lot of classes right there obviously and these are all the tensors meaning you know the boxes the bounding boxes that's not good enough we need to put those on images so let's do that as the next step we are creating a visualizer object by using the visualizer method and what are we visualizing the image and get the metadata out using metadata catalog method and onto our data set right there and let's change the scale to 1.0 so the image is not as humongous maybe even 0.8 let's just make this 0.8 so the image is small enough and CV2 dot IM show there you go so that's a potted plant 79 this is an oven 71 it does an amazing job well that's probably a washer not an oven this is an uh oven it's uncommon to have a washer in the kitchen but there you go that's the oven this is another that's a that's a washer this is a regular washer I think or a dryer and that seems to be a washer but it calls it an oven so it's highly confident that it's an oven but it's not it's a washer but anyway you can see how it can uh confuse that but everything else it does a good job including these cups inside uh this this cabinet and bottle refrigerator so it's working fine it's working great I have no complaints uh if I really care about getting this event and these right I'll add more data to customize the training for this one but that's not our goal uh now what if the model uh that's the natural image but what if I Supply my specific image which is uh this one right here and we just saw that and how does it segment so let's go ahead and segment it using the pre-trained models for instance again we saw this in our last tutorial it thinks this is a tie 55 this is a pizza 61 percent it's pretty confident that this is a tie and that's a donut for some reason maybe because it sees all these holes and stuff uh 50 obviously the point is pre-trained model is not good for our customized applications so we need to train our own models so now that I have proven that we need to train our own custom models let's see how we can do that first thing first we need to Define our data we need to Define our data set and I am going to register my data here and the way you do that is from the data sets within detectron go ahead and import register Coco instances because our labels are Coco labels and I am going to register Coco instances what am I going to call that the first one I'm going to call it my data set underscore train and for that my Json file is right here go ahead and look at the documentation of the register Coco instances but basically you'll see the same thing give a name for this data set and go ahead and tell where the annotations are the Json file is the location of it and go ahead and give the directory name for where the training data is and I did exactly the same for validation data that's it now that the data sets are registered now I can actually use my train data set my data is a train my dataset well to get to extract information and that's what we are trying to do here so from my tree data set train go ahead and extract the metadata information also go ahead and extract the data set dictionaries and there you go and you can see loaded five images in Coco format and it should give you all the right information right there let's do the same for our validation data because we'll need that later on yeah this part is I'm just making sure that okay everything is working fine my data set is registered fine I'm extracting the information here uh and uh let's go ahead and plot it yeah and now I have these objects data set dictionaries validation metadata dictionaries per trade metadata right there so let's go ahead and use the train metadata and the data set dictionary wherever the data set dictionary is uh yeah trained data set dicks and let's go ahead and plot two images from our training data set just to see uh how they look like it's not just plotting it's actually segmenting it using the pre-trained segmenter which is irrelevant for us anyway but you can see how it thinks uh uh sorry I I uh we're not segmenting it here we are actually visualizing I take my word right back I should have more coffee right now and all it's doing is it's a drawing the metadata right there so it has information about our training data uh annotations right there that's exactly what it's doing so what I meant to say is here is where you confirm that your your annotations and your images are kind of making sense right now so uh part of the annotations in Json file format it has the image file name with it and it has the annotations associated with it I recommend you to study the Json file and see how it's storing all the information so it kind of takes that information and puts it on here and I have no complaints you see how it's actually matching the cell and this is all my annotations this is what I have done right and this is another image where it's actually overlaying all the cells and all the annotations that I have actually performed as you can tell I got a bit lazy I did not annotate all objects I annotated most of them so I know my data is loading fine you know the everything is working okay now let's go ahead and get the training ready now how do you train it again uh you can copy and paste this but every line makes sense here so first thing first let's go ahead and Define a configuration you can use a different config name if you want but I'm just calling it CFG using get CFG method right there first thing first to this configuration I want to add an output directory so it knows where to store the models and remember I created a directory called detectron 2 models this is where I want all the models to be saved and then go ahead and download every a model called mask or CNN or 50 this is what I'm going to use with random weights for now until I tell it to update the weights it's not going to do that right and what training data set am I going to use my train data set remember my data set train this is what we just uh instantiated like this is what we did up here right there my train data set up there my train data set my data set train I keep saying my train data set my data set train that's exactly what we're gonna use to train uh you can add testing data I haven't added that maybe I should use the validation data set right here but I haven't done that but you can and a number of workers is number of you know this is basically parallel processing I just leave it to two and do you want to use random weights or do you or meaning start from scratch or do you want to add some weights I say okay weights go ahead and get the weights from here Coco instant segmentation mask or CNN so it initiates to these weights which is always a good idea to initiate with uh instead of random weights somewhere in the middle two images per batch process that learning rate if you want you can Define the learning rate how many iterations maximum iterations 1500 let's actually change this to I have trained mine on 1500 um let's change this to 1000 it's not that slow but uh but still I don't want to stay at the screen for a long time or pause the video but let's do one thousand yeah let me change this to 1000 seems to be good enough for this data set uh if you want to step uh you can add the delay in the training and batch size let's leave it to 256 uh batch size uh and number of classes in our case we have four classes this is very important do not include background as an additional class in some uh cases I think for NASCAR CNN I forgot like in some cases you define this as five because you take four classes plus one but in our case we only have four classes right so we have cells we have mitochondria Alpha granules and carnicular vessels and that's four classes that's it so four and now let's go ahead and make the directory which directory do we need to create the output directly that we just uh you know want to save this or or train models well or uh I'm losing my mind here so to try to save our weights the the training weights after the training and the trainer is default trainer config remember we have to instantiate the trainer uh with the configuration file we just defined our configuration settings we just defined once the trainer is there now uh the last line is resume or load if the resume equals to false that means you want to load a pre-trained model if it's available or start from scratch if there is no pre-trained model is available that's what this is and let's go ahead and run it it should give you a uh the full summary of exactly what's going on go through this but now let's go ahead and start the training process so the trainer.raine starts the training process and I'll leave this on for at least a few seconds and then I'll stop recording I'll continue after the training is done because it let's see how long it takes it actually gives a very good idea after the first it takes it tells me it takes about 6 minutes 54 seconds for the entire thing to be to be done estimated time of arrival in about seven minutes right so you can see the total loss and uh box and so on go ahead and look at these now you see uh iteration number 39 and we have to do 1000 such iterations so like I said let me pause and then continue as soon as this is done okay so finally it's done and it said 6 minutes 54 seconds and it took 7 Minutes 23 seconds so that's pretty close right so let's go ahead and as you can see the total loss is going down going down so it's still going down and instead of just looking at the numbers of course you can go ahead and plot it and uh first first thing first let's in fact I should have moved these up let's go ahead and move that cell up let's go ahead and look at the training curves uh in the tensor board and let's launch it of course you can do this in your local IDE right I hope you know how to do that in the local IDE but uh and no dashboards are active for current data um well let's not worry about tensorboard for now but I don't know why it didn't work it was working last night so I have I don't want to troubleshoot right now but uh go ahead and uh run these lines and you should see the tensor board or you can actually capture this and do your own plots uh it's up to you but visually you can see that it's still going down that means it can actually use additional I don't know 200 300 500 uh iterations now uh the first thing first I want to save this yaml file the the new configuration yaml file so in case I want to use it later on because if I want to load it and do future work again I may have an example coming up so let's save this now now that everything is saved and by the way the model itself is saved because I provided it the path where uh you know for it to save so it should be it should be available as part of our uh as part of our drive right there again we gave the path and let's not worry about it we're gonna load it in a minute okay so whatever that output directory is there should be a model underscore final uh path in fact let us confirm that we have the latest model so what is our output directory so I have to go back sorry give me a second so I have to go back I've got this freaking new mouse that's very loud I hope it doesn't interfere with this sound here but uh okay under models detectron 2 models right so let's go here uh my drive my drive collab notebooks models detectron 2 models and there should be the configuration yaml file that I just saved right so let's go down we save this where are we uh we are right here okay so this file right there so config.yaml is right here you can see and the model final path this one the model underscore final dot pth is what we are trying to load right here so configuration models weights go ahead and do that and I'm setting for this new configuration I'm setting a threshold to be 0.5 this is where you can adjust and I am creating a predictor object with the default predictor and configuration as the CFG as the input here we have done this before so I hope this is not a surprise to you we create a predictor object and this is the object that we apply onto our images so we get an output and that output contains the instances and other information about the segmentation which is what we are using down here for plotting and we are reading an input image one of the input images that we have and go ahead and and predict it and plot right there pretty good job with five Training images and I did take about an hour for me to annotate because we have lots of classes lots of objects but that is an amazing job as you can see individual cells right here if you think these two cells are kind of combined I don't know I can't tell oh no they're not combined it uses the same color but you can see that bounding box for this specific cell and this big bounding box for this cell with 94 but if you think half of the cell is uh you know let's go ahead and do that why not I was about to say let's change this to 0.8 very aggressive so you can see how the output looks like and after that let's go ahead and plot it and we should see I don't know some objects dropped and some you see this cell how this region is not picked up as a cell and uh things uh can be a bit weird as you can see it's not perfect right there so this is very aggressive so let's go back to 0.5 yeah yeah so let's go back to 0.5 and go ahead and run this and once this is done again we let's go down and check the average precision and recall what type of metrics because we have validation data as my data set Val now we can go ahead and check the evaluation so this is Coco evaluator and go ahead and print the inference on this data set so just copy and paste these lines you will see let's see what type of values do we get I mean obviously I would love to see hundreds or 90s right there but average Precision right there uh 41 Precision 50 percentile go ahead and look at what these are mean in the documentation but you can see uh uh about 50 60 which is okay on this one and uh for a cell average Precision is 80.64 for carnicular vessel 36.1 Alpha granule the one uh that's called mitochondria is the one that's bringing everything down because we don't have enough annotations for mitochondria if you remember we only have a few so it's also on top of that more challenging to segment mitochondria because in some cases these annotations overlap with uh I believe Alpha granules but anyway so 3.8 is too low obviously but on average I mean sales are easy I'm getting 80 so uh I love this I love this uh output because it gives you different types of metrics and you can based on the metrics you can see what to adjust you know and what to annotate more okay so I think that's okay that's uh what is this what are we looking at now uh uh yeah evaluation so all of that is fine so now let's say okay I have a good model predict and save let me delete this I was experimenting with something I don't want to give you all my experimentation stuff okay so uh predict and save the results uh best part I was experimenting I included that later so much for being prepared so load a new image and segment it so the previous one we already have like the image uh defined and that's what we got right so if I come back here this image we are not loading it uh this is something that we already have as part of our data set so we are kind of testing it on one of the ones now let's go ahead and load a new image an external image so we have a bunch of images as part of our test I'm just gonna pick a random image like test image number three and go ahead and read the image using cb2.mreed and our output is nothing but predictor on this new image and then we are visualizing and plotting so this is a absolutely completely New Image not part of our train or validation and let's see how it actually works so the cells are segmented perfectly Alpha granules it's doing an amazing job and did it pick uh are there any mitochondria in this image mitochondria is so rare in these in this data set it's uh it's tough to find it but we can segment a whole bunch of images and see if we find any mitochondria in them yeah that's the next part process multiple images in a directory and save the results in the output directory so now I have a uh the entire test directory and I want to segment every image and put it put the results in the test results directory so let's go ahead and run this just for the fun of it and that's going to be 50 images because I have 50 images as you see it's cycle cycling through and while it's doing that let's go ahead and open this to see how things are being populated here 3D em platelet there should be test results and the images are slowly showing up right here yeah you see how it's adding more images let's just take a random image download it and have a quick look at it why not and let's open the file there you go that's that's not bad right I mean is there any mitochondria now I'm fixated on this mitochondria thing um I don't see anything visually maybe you can freeze the screen and have a critical eye but uh oh there you go mitochondria that one we caught one so can I zoom in there you go that one so it's rare but it's actually getting these uh mitochondria how many of those are correctly segmented how many uh oh there is another mitochondria right here 75 okay I'm I'm okay I'm happy with this it'd be nice if we can uh color these micro each object in different colors so it makes it easy but uh anyway so this step let me go ahead and close this part I think this step you understand we are segmenting and saving these images but I hate images like those because they are not quantifiable the whole reason why we are doing this for our scientific images is we want to quantify them what good is an image that you can I mean that's a great image the one that I just showed you to print and put it on your wall showing that hey look how great my algorithm is but we want quantification so one way of doing that is segment images and save objects so in this case I want to I'm going to use the scikit image region props to actually take the labeled images and then extract region parameters from that I have covered that in many videos in the past we have done that so I'm not going to explain what exactly it is but uh code is right here so what we're trying to do is okay we have an input directory text test take all the images one by one and I'm also initiating here uh you know creating a output file called output objects.csv that captures all the information about this these objects now with that file open go ahead and uh right right what uh first of all add a First Column called first file name class name object number so for each file I'll have different classes right mitochondria self and so on and then I have different objects within within mitochondria I have mitochondria one two three four and then what is the area of it what is the centroid what is the bounding box all the all the typical stuff now open each file go ahead and read the file and go ahead and predict on it now get the mask out of it because the mask is nothing but your instances right and convert that into a numpy array and convert that into a binary like Boolean and then go ahead and get the labels and now take the label masks and get the region props now you get the properties and from the properties go ahead and get area you can add more properties if you want I just picked area and a few things and go ahead and add them to the CSV file so this is a regular python in action and regular scikit image in action right there so now it's loading each image and it's actually as you can see it's cycling through this is worth the wait so I will actually again we are processing all 50 images I should have selected only a handful of them but no point you know uh let's let's I'll pause the video but let's look at the CSV file once it's done okay so it is done and we should see a CSV file so let me go ahead and where did we save it uh is it in test results I think we saved it in test results let's scroll all the way down so output objects so let's go ahead and download it and open the CSV file I don't have Microsoft uh on this I mean Excel so but you can see how we have file name class name object number areas Android bounding box so this is the file name for example uh platelet19.png so this is a cell so how many cells do we expect in each image five six seven so there is cell number one two three four uh uh cell number five no this is a carnicular vessel right there and cell number six cell number seven eight nine ten cells in this image in this specific image 19 but then you have all others so this is everything that you want uh okay so that is one way and the other thing what I mean we are good at python so why work with Excel and all other external uh programs why not just import that Excel file or CSV file that we just saved and uh use pandas and go ahead and use Seaborn my favorite plotting uh library in Python and go ahead and plot so it depends on what questions you want to ask one question I want to ask is what's the average number of objects per image for each class in each image how many cells do I have or how many mitochondria do I have and so on and also uh what is the average area of these objects in each class these are the only two questions I thought of asking but obviously depending on what else you want to do go ahead and ask other questions but there you go so this is average number I'm zoomed in so it's tough to see zoomed in so the the code is easy for you to read but you can see uh average number of objects cells we have about 11 cells per image and when it comes to Alpha granule you have 20 plus in each image mitochondria not many two or three right there curricular vessel we have we know we have a lot of these like 30 plus of these and same thing with object areas you would expect the cells to be pretty large of course and the granules are right there again you can do your own violin plots and all that stuff now finally let's uh end this by saving the binary image because or you can call them multinery because in uh for each image original image I want to save a specific image for Alpha granules where each object is labeled one two three four five six of only Alpha granules I'll have another image of only cells so let's go ahead and run this I'm not going to waste your time uh by talking and not running this here but again our output directory here is test results instance and uh and all I'm doing is just binarizing each of these instances and uh and saving them and that's the end of this uh I'll actually talk about panoptic segmentation in a second uh but I already talked about it in the last video but uh well all of this is done this is so fast because all we are doing is uh I'm actually surprised how fast this is so let's go and see what happened here I hope it worked so test results instance right there oh so all of this is uh all of this is done let's do one thing I am going to download this and open this in image J so you can see exactly how it's going to look like so I am going back to my train I am going back to my test results instance just to have a quick look you see how I have like for this image this is all the cells image anyway it's not clear right here so let's download this test results instance download and I'm gonna pause this and continue as soon as this is downloaded and I have organized things in the background okay okay so I just downloaded these images down here and I organized them into Alpha carnicular based on uh different classes in fact as you can see if I go into these cells there are 50 items because I have 50 images and for each of those 50 images we have an image for cells and if you open these cells image right here you may not see much you're probably some something going on faint right there but the best way to actually open this is for example open this in image J the scientific program uh you know just a second let me move this down here and adjust the brightness and contrast but instead of single image let's actually import import image sequence image sequence oh God I have this recording thing uh okay so file import image sequence and test result cells yeah let's go ahead and import it and you should see the imported image right away right here now let's go ahead and adjust the brightness contrast so you can see all the cells right there you see how each of these cells uh yeah apply to all 50 images so how each of these they have unique number unique value right there these are all the cells that are detected and if in fact if I just go through this you see how we have different cells right here yeah that's pretty good and uh now you can import similarly you can import let's do one more and then uh and then get back uh what shall we look at so let's look at miter I don't know how many mitochondria we have but uh let's uh let's see if it shows up fine adjust brightness and contrast and let's adjust this to that point right there apply to everything and there you go yeah so you can see how we have these different mitochondria now why would you need this now you can reconstruct this in 3D and you can try to use another program uh that can give you 3D I don't know morphologies if you want to extract those and all that stuff so uh but uh hopefully you appreciate how you can actually save these individual uh classes individual images so the whole point here is not just putting a fancy bounding box and showing color images you're drawing scientific images which means you always are after some sort of a quantitative information and that's exactly what you can get whether it is in the CSV file capture it as object measurements or exporting them for a further Downstream analysis maybe you want to do some sort of a neighborhood analysis maybe you want to do hey segment this image but within each of the segmented images give me how many mitochondria are there and what is the average size distribution whatever you want to do in Downstream you should be able to do that okay so that comes includes how to train your own custom models uh I think I added down here something called uh interested in panoptic segmentation if you haven't watched my previous video here is a quick summary of what panoptic segmentation is which is nothing but you know this is instant segmentation right so we are drawing a bounding box panoptic segmentation is nothing but just you know segmenting the entire scene which you will see in a second uh for every pixel I am just showing this for the sake of completeness but I do not have uh I think what we have done uh up here with ourselves is pretty much what is uh at least the extent that I have used the detectron 2 model so I hope again apologies for this long video but I wanted to capture everything relevant to using your own you know training data to training your models and not just doing the models but also saving data exporting you know doing the measurements and all of that in one video otherwise I don't want you to wait two more weeks for part two of this so uh please keep asking questions as part of the comments because that gives me ideas in terms of what else to test and this is one of the ones where one of the uh commenters or subscribers actually asked about hey can we uh can we segment all overlapping objects using uh unit I'm like I know with unit you cannot do that it's not the right right framework but with NASCAR CNN of course and by digging into it I learned that detectron 2 does an amazing job so if you have like any use cases where you have all these objects open not just use case actually with data then I'd like to hear about how detectron 2 actually worked for you okay let me not extend this any further please hit the Subscribe button right now if you haven't already done so and let's meet again in the next tutorial

Channel: DigitalSreeni

Views: 19,246

Keywords: microscopy, python, image processing

Id: cEgF0YknpZw

Length: 50min 22sec (3022 seconds)

Published: Wed Aug 23 2023