334 - Training custom instance segmentation model using YOLO v8

hi everyone welcome back this is a continuation of our discussion from last week where we got a quick introduction to YOLO version 8 and we actually loaded an image and loaded a couple of pre-trained models one for object detection one for instance segmentation to segment that specific image and we saw that okay for object detection you get the bounding boxes and for instance segmentation you get both the bounding box and the pixels uh corresponding pixels actually segmented now in this video let us take it to the next level which is okay all that is great you're segmenting objects in the kitchen but I want to segment objects in my scientific image I'm going to use a electron microscope image from a public data set it does not matter you can be using your remote sensing image your aerial photography or MRI it doesn't matter anytime you have objects that need to be segmented you just load a data set and train a model segment and then perform analysis and that's exactly what the goal is for this tutorial and again I know you love this content so please go ahead and hit the Subscribe button right now and also if you're feeling extra generous try to find the thanks button and contribute based on uh based on how amazing awesome you're feeling that uh right now okay uh let's look at the plan actually uh in the last video I provided a quick introduction to YOLO V8 so please go ahead and watch it but again a quick summary of that YOLO version 8 obviously eighth version that means there are previous versions in yellow this one is better obviously I hope right I mean uh as every version goes by uh because it's better because it's faster and it also adds certain new functionality which is uh instant segmentation for example that makes it usable for scientific applications now YOLO is a AI framework that supports different types of tasks and what tasks if you ask it can support it does detection segmentation classification post detection and so on I am going to focus primarily on the instant segmentation or the segmentation part of it in this specific tutorial the usage is very similar whether you would like to do other you know like detection or tracking and they have pretty good documentation so please go ahead and look at their documentation for any of the other ones that I'm not covering here now how do we train it using custom data to get the data ready and again I have done a video video number 332 not the last one but the one before that please watch that if you start with binary images and you want to convert that to YOLO compatible format that video is for you I shared a bunch of code so you can take your images and do that or you can use some third-party services uh So the plan for now is prepare the data like I was just mentioning by converting existing annotations to YOLO compatible format this is exactly what I'm going to use today but if you don't have that you can just start with your raw images go ahead and paint them in roboflow and convert them into YOLO V8 format or if you already have a Coco style format maybe you have used makesense.ai to paint your pixels and now you have a Coco Json file you can just drag that into Robo flow and you can convert that into Euler V8 in fact I'll do a quick demonstration uh right before jumping into the code in doing that and uh the the next part is walking through the code in Google collab and what do we do we train a model this is again a complete project you're starting with raw images and this is a full project using YOLO very similar to the one I have done using detectron a few videos again I go and go ahead and watch that if you haven't watched it it was like a 50 minute video or something I hope this doesn't go that long but uh hopefully you'll find these long videos to be uh informative and not boring I hope so uh if you find it boring let me know so I can optimize the optimize the way I record these videos I'd rather not miss any details you know and then make the video like confusing and make it faster right so that's the balance I chose okay so we are going to walk through where uh we are doing training and then we are performing inference on new images and then we'll extract area calculations and all that just similar to very similar to what we have done with detectron 2 video okay now first let's uh understand the image and the data set in this specific case and you can download this and this data set is cellular segmentation of these organelles in 3D em images like these are human platelets and you can download the data the link is down there and by the way when I share this with detectron 2 I got a lot of messages about hey the data set is not available can you do this I'm like what I mean guys it's not a data set I uploaded right it's not a data set I uploaded uh and put it on my Google Drive and I don't own this so what do you do if that doesn't work you freaking Google search you cannot be data scientist if you don't know how to hunt for your data if it's not there Google search find this follow I gave enough information by the way I'm giving you a direct link to dot zip file but that may go away later on but there may be some version of these data set somewhere floating around if it's not there I can't help you because this is not my data set please understand that I legally cannot share data set that do not does not belong to me you should you shouldn't either uh the people who own the data set they share it we use it and that's why I'm putting the reference right here sorry for my one minute rant but I I the the thing that I really don't like and I want people to learn is uh uh data curation data hunting you know you go hunt for the data you should have that killer Spirit if you don't have the data the I mean what can other person do go do a quick Google search and you get 20 links and one of those links will give you the data sorry I'll I'll stop my rant but you some of you probably relate to that when when people ask questions that that can be answered using a quick Google search but other technical questions I'm not I'm not saying don't ask questions but don't ask questions that could be easily answered by one second Google search Okay uh now let me get back calm down let me get back and uh the description of the training data so this the it's a bunch of images as you can see uh they're all noisy so makes this a bit challenging and uh and let me just skip through this uh what I am using here is uh I mean this doesn't actually I should have deleted this slide I'm not just using five images for training I'm actually using a lot more images than five images because I initially started painting these and I got super lazy and I'm like okay this is not this is not what I want to do my skill lies in actually using coding to automate the task so I have written code to convert my images into YOLO uh annotations that's exactly the video I did in 332 uh so after doing that so there's no way I'm gonna paint all these pixels because I already have annotations right I mean from these guys okay the four organelles that we are going to segment here are the cells mitochondria Alpha granule and carnicular vessels so the uh the what is that color the orangish color the big blobs these are all the cells and mitochondria is in red we don't have a lot of mitochondria so the the accuracy on those would be very low and Alpha granule is this green blobs and the chronicular vessels are the blue blobs right here okay so these are the four classes that we are using for our training and with that information I think we should go ahead and jump jump into the code without wasting any uh any more time but again I would like to just show you this this uh liepman lab webpage where I actually downloaded the data from it says dens cellular segmentation for em again use that as a keyword and you'll probably find this page but when you scroll down it gives you a lot more information about this data set more than I could uh I mean I could share and the data is down there go ahead and download it it's only 180 megabytes and they have training validation and testing I'm only using the training and validation data sets so I downloaded them and I used my code in fact do I have it here I think yeah I used my code when uh to convert those input training images like these images and training masks like these binary masks into a YOLO style format so the YOLO data set format is right here where you have the training data validation data within the training data you have images and labels as text files right right here for each of these images you have a corresponding text file okay and and what else do I have I have data.yaml and the yaml file is nothing but it tells you how many classes you have and and what the class names are and what the directory paths are for your train validation and test data sets now if you don't know any of these and you just want to uh you know you don't you don't want to use my code let's say you have your input images you already have them in Coco format which means you have your input images and the Json file right there right how can you how can you generate your labels you can go to roboflow.com create a free account and any free account I believe any data you upload is going to be a open source data so usually if the data set doesn't belong to you I I get my stuff done and then delete it because I'm not the owner that's exactly one of the reasons why I did not upload the entire data set to this public repository because again I do not own the data so I'm going to process it offline and I'm using it for my for training purposes right so for tutorial purposes okay uh I I I can't stress enough on uh you know on the topic of respecting the licenses when it comes to data when it comes to code when it comes to anything that's not yours when you touch it you have to uh you have to you know credit the right people this is very important especially to those who are just started to getting into your graduate school or undergraduate school or you know doing your doctoral or push doctor uh okay uh where am I I seem to be very talkative today so uh let's create a new project well I have a project called Junk with nothing in there so let me open my uh folder I have trouble seeing here and I can just drag this let me go back and drag the entire train folder right there it it understands the Json file and it looks at all your images and as you can see uh roboflow is pretty amazing it actually automatically you know it put everything right there if you want to add more in addition to that you can always add more you can go to a class and you can use any of these and by the way it's also powered by segment anything model so it assists in uh in in annotations if you're starting from scratch okay enough about annotations but once it's all done once it's all done let's say do I have a project do I have a project okay ignore that uh uh once it's all done you can basically I don't want to save and continue let me go ahead and I think let's take this example you can export the data set there are two things you can do one download the zip so you have everything done and you can upload it or you can show download code in which case it shows you the code to download and with your API key and everything and you just plug it into the code I'll I have included that in the in the file that I'm going to share so don't worry about it okay uh there is a lot we talked about data and all that so let's I think now you understand what we have right so what we have is a data in a folder called YOLO and you have a train with images and labels and you have valid with images and labels and you have data.yaml file and by the way if you export it from Robo flow yaml file gets exported directly if you use my code from two tutorials ago from 332 I included this little bit about yaml all set enough background stuff now let's get into the actual code Let's uh connect the runtime and I chose GPU of course I added all kinds of text and I also added how you can download the data set with a direct link and all that stuff blah blah blah and I should have done a great good job because I typed so fast and where was that oh God I saw Json Json I made a spilling mistake somewhere there you go Json so okay uh should have been a bit better prepared but yeah that's all the text you have and if if you want to know more about YOLO models like you can go follow this link let's go ahead and install ultrolytics library library that we did in the last video again this is where we are we are using to to invoke our YOLO Library our YOLO methods okay so that's installed and I'm gonna import Yolo from ultrolytics and of course uh matplotlib and pill for image handling and plotting uh this is very similar to what we have done in the last video so you should not be surprised with that and again in the last video we loaded the model but in this case I'm doing this in two steps one uh from YOLO I'm going to import this model hyphen seg which means it's going to be instant segmentation Dot yaml and on top of that now when you only do this this is going to be the structure of the model with random weights I want to fill that with pre-trained weights and that's what this point is right there and this is the transfer weights so now at the end of this I should have a model that has been free trained on on some images right I mean it's not on our images so it doesn't matter as long as it's pre-trained it's trained enough on these data sets to understand to detect the edges to detect like different textures and all that on top of that I'm fine tuning it to my specific data set okay let's move on to fine tuning part and if you want you this is where you install the roboflow library if you are importing the data directly and when you do that in collab you will get at least in September 2000 2023 when you run uh install this roboflow and try to try to follow this process you will get an error error about some sort of an encoding error uh to to you know to bypass this to handle that let's go ahead and run these lines yeah so import Locale and you're actually setting your local to understand the utf-8 encoding again if you don't know what that means it doesn't matter just go ahead and run this line and then you can install your roboflow and then uh I don't know I was thinking of using roboflow dataset that's why I added this which is basically get pass import get pass and when you do this it comes up with a box saying that hey enter your token here this is where I was expecting to put my API key and so it hides from you guys that's the whole point here uh for these two lines and then this part of the code this is my library I mean I have a uh an account on roboflow or I created it called python for microscopics and within that I have a project called 3D em platelet and there I have multiple versions of that data set because in one one version I had like only I think uh four uh two classes labeled in the other version I had multiple classes labeled and blah blah blah so I want version number two and this is the download so this should be downloading your data set of course I'm not doing that because I have already let's go to the drive connected my drive to collab go to my drive collab notebooks data 3D em platelet and here I have my YOLO V8 data and here I have my train folder valid folder the exactly the same structure that you see on my desktop folders and I just uploaded them here including the data.yaml that's it okay so I have those and that's what I'm going to import uh later on so first thing first the yaml file contains all the information in terms of where my training data is and uh validation data is so let's go ahead and have a quick look at it so there you go my uh names of the four classes here are alpha cells miter and vessels how many classes do I have number of classes four and where is my test images I don't have test images now so I just left it whatever and train is in content to drive my drive collabs and blah blah blah all the way train and images right so you saw that my content is all the way down here and train and images this is where my images are and by the way you have to copy that path into your yaml file you can edit the ml file using any notepad text that's just a text file so I edited that and this is the information this is literally all the information that's contained in the yaml file it's just yaml extension okay Now define the number of classes based on yaml because my yaml has nc4 right there right so I say hey go ahead load the ml file look at this and see whatever that number is that's my number of classes so instead of manually defining this is a much better way right I mean you can use a different yaml file and not worry about changing stuff that's it we are all set to train the model so I'm training I think I trained the model yesterday when I was testing this code so under results I should have uh 200 epochs you see how I trained it three different times and it created three different directories right there so the more number of times you train the more uh you know obviously the number it's going to attach to it uh I thought of mentioning one other thing I think that that should do actually and let's go ahead and I'm defining my project right here uh this is the destination directly where all of these results are going to be stored and uh now this name 200 epochs is a subdirectory within that directory where it stores all the trained data so let's go ahead and uh run this so there you go and uh I mean I defined this obviously I'm not training it yet and this is where we are training it and we are capturing the result into a parameter called uh results or a variable called results right there and how do we train what parameters goes into this model.train obviously you need to get where the data is and data.yaml tells where the data is and what is the project name the project is going to be our project path it's going to be right there and the name of the project is 200 epochs because I'm gonna you know train it for 200 epochs and I said that patience of zero because uh I want to disable the early stopping if you put patients of 10 or 20 it looks at the best model and even if you get the best model it goes like 20 epochs just to make sure that that is the best and then it stops right so I put 0 because I want to do full 200 epochs in this case and the batch size I defined it as four and my image size is 800 by 800 so I just defined image size as 800 and let's go ahead and start this I forgot how long it takes I was testing multiple models uh so let's see it gives us an idea of how long uh it's gonna take and I'll pause the video once this thing starts and then continue as soon as this 200 epochs are done hopefully not a few hours but I'm hoping it's a few minutes but let's see uh okay so it's looking at the train it's looking at the cash created and albuminations it's actually using the data augmentation provided by albumin albuminations Library looking at the validation data set and now the atom Optimizer is what it's using my image size is 800 validation size is 800. and Epoch one out of 200 and let's see how long it actually takes so it's still doing Epoch 100 now it's doing Epoch 200 so it's taking a few seconds like uh so 200 Epoch should be done so I thought of actually stopping and showing you results with a pre you know with a with a run that I've done earlier but it's not fun let's actually look at the results from this specific run so I am going to let's see if it actually created before stopping let's see if it actually create yeah there you go it created another directory right there called 200 epochs 4 and this is where our results are going to be stored so uh I'll return in a few uh minutes and uh let's continue the tutorial okay I'm back looks like our 200 epochs are done there you go that's 200 out of 200. and what do we look for so yeah you can look at the map values if I can find them like yeah they are 50 and 50 to 95 it's like 63 72 87 uh mitochondria we don't have many mitochondria like I said so you're not gonna get good uh good accuracy right there and vessels that's not bad so overall this is pretty a pretty good given given that we we just trained it for 200 epochs and uh uh yeah so uh let's go down and see what can we do with this right so first of all this is all numbers but let's actually visually look at how the training went so you can load the tensor Board of course if you are working offline uh or working on your local IDE you hopefully know how to fire up the tensorboard to look at various aspects I throw a car even in the last video I got something very similar but when I do my training it actually works fine so maybe I have to change some settings here logs save in the folder runs let me quickly try to troubleshoot this otherwise I I don't know what's going on here so unless it's actually looking for these runs somewhere else maybe it's a path problem right here um but just uh let me just do this one test results card yellow V8 uh you know I don't want to waste your time but there is another way of actually looking at this results uh I mean we we have the data we can look at the results but yeah I mean it would be cool to actually have a quick look at okay these are the weights two weights that we just it stores the best weight it stores the last one and uh there are a bunch of images right there so you can actually look at these images instead of instead of looking at the tensorboard so let's look at one of these images so what do we have uh epochs four weights and let's look at let's look at results.png so it should be results epox Dash four right so this is the fourth training and there is the results.png so let's go ahead and plot I'm sorry guys I don't know why this tensorflow uh there's some silly mistake I'm pretty sure I'm doing here uh probably not looking at the right location is the key but it worked literally it worked yesterday so uh anyway let's look at this yeah so this is not bad in fact let me close this side so you can see the result it's still getting better but it's not bad it's converging down here so there are multiple losses as you can see segmentation loss that's the Box loss so how uh where the box is right so do I have the boxes if so where are they so this is the segmentation loss and so on classification loss is right here so how to classify that's the easy one classification is the easy problem where it is like segmentation in the box is the uh tough problem so you can see like the Precision was going up but all of a sudden it went the wrong direction but then it just came back so uh it you can I mean there's a lot to learn through this so I'll let you study that and the other one is uh epochs 200 epochs 4. let's actually look at the last batch batch batch uh what am I looking for let's look at train batch 572 train batch 572 yeah that's there so let's go ahead and plot this one but instead of third training we are looking at the training right now okay so it's it's just basically saving one of those validation images and now you can actually see how it's doing the detection and it is doing a pretty good job you can see how all the cells are correctly detected and all the organelles are correctly detected so the point here is if tensorflow doesn't work no problem go ahead and look at any of these results that it actually stores you can look at the validation label strain uh images sorry I said validation earlier this is the training image that got segmented of course it's going to do good on the training image now let's go ahead and run the inference first of all let's change this to 4 again I was obviously planning on using my previous training so I'm changing everything here so let's look at what training uh you know trained day uh weights we have here so we have the best one we have the last one it's up to you what you want to pick my last one also seems to be okay right I mean from these curves especially it was horrible right here but it came back to the same level where this was heading 2. so I'm okay with the last one sometimes if it goes crappy right here you pick the best one so I'm picking the last one or you can test both so there you go and load oh sorry not bad I just checked this but I'm loading the three let's go ahead and change this to four and load the weights from four okay now that it's loaded let's go ahead and load a test image and this is my test image and this is an image that the training never saw yeah this is just a random test image and I am going to set a confidence of 0.3 we'll we'll have to adjust it based on how the results look like so let's go ahead and look at the results okay it tells me that there are 30 Alphas 22 cells one mitochondria 72 vessels and uh it took 14 milliseconds to segment this is why it's amazing once you train it put it on a real-time camera and then let it do the job you know this is super fast that was like incredibly fast I'm super impressed with this that's one of the advantages of Elo okay now that we have the new results you know that new results can take uh input as a list of images right there but I only gave one so I'm looking at the zeroth right there go ahead and plot and let's look at the results plot the results humongous image right there because I zoomed into this but uh you see how it can be better right here so probably you can go back and change the change let's change this to 0.2 and see how that helps or hurts so there you go there you go and now let's look at this so it it will uh it did not fill in actually 0.2 the problem is if you have multi classes like this and you start adjusting the threshold for all the classes then sometimes you get over a lot more classes in certain objects and lot few classes so again it's just a threshold to see how that works for you but let's go down and check a couple things so extracting the bounding boxes how do you extract the bounding boxes so if you uh look at the first or result number zero which is the result for this specific image you can see that okay the keys are boxes and masks So within this new result you have two things the boxes and the masks and the names of these classes are alpha cells Mito and vessels and these are the numbers associated with that like zeroth one is Alpha and here is the original image okay and you can you can go into the boxes and the masks and you can extract only the boxes or The Masks again this is instant segmentation so you have both boxes and masks okay so uh with that information what can we do let's go ahead and extract masks okay so you can extract the bounding polygons these are all the polygons for this mask so masks.x y n that gives you all the polygons now if you go down do I have the example uh yeah masks.data gives you the data about the mask so go ahead and look at the so we have 144 objects detected in this specific image and it gives you the the data for each and every one of these masks so Let's uh push the mask to our CPU because all of this is in GPU right now let's push it to CPU so we can play with it you can extract some numbers so let's go ahead and push this to CPU and I'm converting that into a numpy array so if I show the image now this is just a numpy array Now show me the zeroth object that's where the zeroth object is now show me uh I don't know let's just do one right here you'll see object number one right there you got the point I don't know show me 10th object let's just do that and that object is not Conti this is something I have to look into uh I if you train more again you get this continuity and everything but this is a result of limited training and this is one thing that I realized between detectron and YOLO YOLO is fast but I feel like there's a compromise in accuracy or it is a bit more hungry when it comes to data so you need to train a lot more uh epochs with a lot more data to get the same level of accuracy so again it's uh it's it's a it's a compromise between this accuracy and uh uh and and speed again I'll let you in some cases this is okay but in some instances for example like this if I want to segment the cells uh I need to find a way to further train or optimize this okay so what class names do I have alpha cells miter vessel so I'm going to extract the bounding boxes that contain these class IDs and for each of these classes masks go ahead and print out so what am I trying to do figure out here obviously how many Alphas do I have how many mitos do I have in this image so I have 34 Alphas so I'm just showing you after you're done segmenting certain steps to extract your data so you can go ahead and do other stuff so not just detect and say done the stuff you get from watching my videos guys okay uh now extracting masks for specific class now I'm like okay just show me the masks for alphas and cells of course you're extracting those masks and now get the original image or original image is 800 by 800x3 right so I'm in this is a grayscale image in this case sorry grayscale image converted to like RGB image in this case and now display the original image so you can see exactly how my original image is with an object on top of it right here yeah so uh you can you can play with that but now let's go ahead and calculate the region properties for all the objects so all I'm doing is iterating through each object getting these objects right here and then using the scikit image measure.region probs to actually extract the region props for all of these and capturing them into a CSV file so when I actually download the CSV file you should be able to see uh actually let's look at the data frame right here and the data frame you see for class name is Alpha and the area and the perimeter so these are the three I captured and now you have a CSV file that you can play with now you can download and work with Excel and all that kind of stuff but but we we love python right so let's use C born and let's use the area information and do a swarm plot so now I want to see visualize how the area is obviously the cells they have much larger area they are pretty big so you can see all those real cells these are all probably the misidentification because our threshold is set to 0.3 0.2 actually right now if we change it to 0.4 or something only the large ones will be identified all the little ones will be gone so we have only one mitochondria right there most of the vessels they have uniform spread you can also do box plot if you're not a small plot type of person so that's a the spread is pretty tight for all of these other ones for the cells it's pretty much spread okay all of that is fine now let's say okay this is great I want to deploy this uh in real time you know for real-time applications you can export this as uh onnx format and uh I've done a couple of videos on onnx in the past but onnx is a lightweight uh let's say a model that can be deployed pretty easily so go ahead and download this and create a web app or something okay that's a relatively long video but I hope you learned something here it's a it's a full project as you can see all the way from images to annotations converting annotations to your uh into your YOLO format and then training a model and then segmenting uh your future images and extracting object parameters here and again the accuracy wise again it's not a real project I cannot just go submit and say hey give me my master as well I have a PhD but I'm saying you cannot just go and say that because the results are not great This Is Where I Leave it to you go annotate more images get more images and and train for 500 epochs and see how the results actually change and compare these results with detectron 2 and see what the Merit soft detectron do is and what the merits of yellow is compared to each other and what the drawbacks of each of these are so you understand what tool you would like to use for what type of specific job I'll do that exercise if I have time but I want to get this out right away to you guys so you can learn all about yellow I hope this video really helped you and again thanks for your questions for watching all the way to the end and do not forget to hit the Subscribe button thank you

Channel: DigitalSreeni

Views: 9,281

Keywords: microscopy, python, image processing

Id: ytlhMAF6ok0

Length: 35min 26sec (2126 seconds)

Published: Wed Nov 01 2023