Using ML.NET for Object Detection in WinForms

hey everyone in this video we're going to look at object detection and ml.net in a hot new framework winforms okay so winforms was hot probably around 15 years ago but i have gotten a few requests to do a video with it speaking of which if you have a request of your own of something you want to see feel free to put it in a comment or message me on twitter alright so i'm in visual studio 2019 here i'm going to create a new project and i'm just going to be a windows form app using the regular.net framework so it's not that net core here and i'll give it a name of mlnet one forms if you're not as old as i am and you don't remember you're not familiar with winforms you have this designer page where you can actually drag and drop all these components on here so we have a button you just drag and drop it on there and if we run it then that's what you get it's kind of one of those wysiwyg editors is what you see is what you get first thing let's add some nuget packages microsoft.ml so we get the base ml.net package version 1.5.2 and because we are dealing with object detection we need image analytics to deal with images and because the object detection model we're going to be using is from the microsoft custom vision and it's an onyx model so we will need the onyx transformer package and so real quick i'm going to go ahead and attempt to build this again we get a build error and that's because ml.net supports only 64-bit builds so we can do is go to our properties of our project to the build tab here and select x64 and that unchecks his preferred 32-bit as well now we can run this and it runs like it did before all right so i'm actually going to drag a button over like we did before and with these components here we can look at these different properties and we have the text here i'm going to say select image and resize it and also it has a name to it where we can reference this in the code behind and with these kinds of components from the ui i like to kind of prefix the names of them to what type they are so since this is a button i prefix btn and select image this is going to allow us to select an image on our machine to use and so we need a couple of other components here first is the file system watcher i'll just drag this on it doesn't show up on our designer but it comes down here also change the name here as file watcher and then we do another one called open file dialog which i will rename that to file dialog and then i'll say the file name is file name so the open file dialog is going to tell it to open that little window that shows all of our drives and we can select an image and file watcher just watches for different changes in that if we have different files all right so we want to know what happens when we click on this image so we can double click this and that creates kind of the default event for that component in this case it is a click event and when we click this we want the file dialog to show so we just call file dialog dot show dialog but this gives us a dialog result and so we can actually put an if statement so if the show dialog returns a dialog result of ok then proceed all right and before we continue let's add a few things to our solution let's add a couple of folders first we'll add one for the ml model and we'll add another one for our models or different classes that we're going to be using and for our model i'll just copy these over and this is from custom vision it would used in a few other videos and i'll link those in the description if you want to see those so i'll paste these in and then i'll make sure that these are copied over and first is the onyx model from the custom vision and then the labels text file which gives us our different labels that we're using and the model is going to be object detection and it's going to detect not only if the photo has a red or white wine in it but also where in the photo that it is and let's build out some of the immo.net code so in our constructor here for our our form and this gets executed whenever our form appears and says it's the main form it's going to always get executed so we need to create our ml context so new ml context and we need to give this some empty data because we're not training all that we're just using this for predictions and so we need to give it a new list and the data we're going to give it is a one input class and let's go ahead and create that in our models folder line input and this is going to be the kind of input schema that we expect to get and i'll paste this in here and we expect to have a image but instead of a file name of the image it's going to be an actual bitmap of the image you notice i have these image settings here so i'll create another model called image settings and that's just going to hold those two constants there so public const integer image height it's going to be 4 16 and it's also going to be the same for the image width so we have that let me go back to our code behind here we have our input next we need to do our pipeline so context transforms we need to resize the images and this comes from that image analytics package that we installed so this takes in a few parameters first is the resizing i'm going to use the image resizing estimator resizing con and we'll fill the output column name it's going to be data how we get that again if you haven't seen that original custom vision video is we have a an application called netron and so if i put this file in here this onyx file we see the first node here is called data so that is our input node and if we go all the way down our output node is called model outputs zero so we'll be using that as our upper column so the next thing is the image width which is going to be the image settings width and then the image height which would be the image settings image height and it's a little bit bigger make sure you all can see it and then the last parameter here is the input call name and that's going to be the image from the one input we can either put the string image or we can use the name of operator and put in one input dot image and then we'll append to that another context that transform where we extract the pixels of our image and here's where we use that output call name of data we're still going to be using that as the input to our onyx model and speaking of which we'll append another transform called apply onyx model we give it the model file that's going to be an ml model model.ox give it the output column name which we saw was the model outputs 0 and then the input column name is going to be data and there's our pipeline there and with that we'll use the pipeline.fit method on our empty data and since we have our it's empty data not just data and since we have our model we can create a prediction engine in our prediction engine we actually want to use down here when we get a file from our file dialog so i'm going to come up here and create a private variable i have a global variable to hold that prediction engine and so type prediction engine it has a one input then we need to create a one predictions model as well but i'm going to call this prediction engine and let's create that prediction class here one predictions and paste in this property here we give it the column name output is that once again that model outputs zero column name we can change it up in our property name i just give it a name of one type oh and i forgot we do need to on our empty data and we need to do context.data load from innumerable from that empty data into an odd data view and then we can use this data into the fit method and now we can call prediction engine or set prediction engine to context.model create prediction engine one input my predictions the fasten the model so now we have our model here for our prediction engine and down here we can use it prediction engine that predicts in here but we need to figure out what we need to put into our predict method here we can do new one input passing the image equals the file name however remember this image is a bitmap type and the file name is a string so we need to create a bitmap from the file name and we can do that using image static class from a file then pass the file dialog file name into that but the firm file returns an image class we can actually explicitly cast that into a bitmap and that's because if we go to the bitmap here it inherits from image so that that allows us to do that conversion and then we just set our image to that image all right and we need to store this prediction into a variable but this prediction has the wine predictions and it has the one type property but it's a float array and we need some way to kind of parse out this float array into a way that we can figure out what label it gets predicted on so i'm actually going to pause real quick and add some other code and when we come back i'll explain that a bit all right and we're back and we've added quite a bit of helper methods here and what these do is kind of help us determine how to fill in this bounding box class and the bounding box basically includes the x value and the y value where the bounding box starts and the height of the boundary box and the width of the boundary box and this basically can gets a red rectangle class from it determines the the box color its description that gets written into the box and a couple ways to get what color that the bounding box is going to use and these helper methods and i'll be honest here like i was in my other video not necessarily sure everything that goes on here because it's using some sigmoid and soft max and all that and some other calculations to determine all this stuff and so once again big thanks to microsoft and one of their samples for supplying all this so we have our prediction and next let's actually get our our labels and remember we have our labels file in our ml model folder so we can use file dot read all lines and that is in the ml model folder labels.txt and we need to bring in this namespace and next we actually get our bounding boxes now we can have more than one in our photo we use that parse output kind of helper method and we give it a prediction of the wand type float array and then the string of our labels and this does return a list of bounding boxes since we can have more than one give the original width of our image and the height as well now we'll do a check if our bounding boxes has more than one we get the max confidence from it using bounding boxes the max length method and we get the confidence to get the max confidence in that list of bounding boxes and from that we can get the top bounding box from that max confidence so bounding boxes that first are default and we just find out where the confidence equals that max confidence and once we have our top bounding boxes let's clear them and then just have that one bounty box to it so right now we're just only getting the the top bounding box from that list and so for each bounding box here and bounding boxes do quite quite a few things i'll just copy these in for now so we do a few calculations here where to put the bounding box on on the image and we use the graphics that are from image and we draw the bounding box rectangle and then they draw the string with that of that description so we have the bounding box information we need to do next is to put that information on our form here what we can do is use the picture box component and i'll just put this kind of overlay on top of it here give it a name pick prediction then back in our code behind after our for each loop here called pick prediction dot image fasten the image and then pick prediction that's size mode to auto size so we can get the full image in there and then pit prediction that i set to visible true and for that kind of in our constructor here i'm going to set that to false initially all right so let's run this and make sure this works correctly and we notice we get an exception here it says unable to load the onyx runtime dll if we get that we would come back to our nuget package manager and here microsoft ml.onyx and here we will just install the microsoft.ml.onyx runtime so that's installed let's start this again alright so all that ran where it loaded the model and all that so i'll select an image and i'll just select one of these red images there we go so we have our bounty box it says it's red and 78 confidence that it's red and pretty good prediction on the bounding box as well and we can do a couple of things to kind of make it a bit nicer we can add let's make this a little bit bigger we add another button i just put this to the top i'll call this select another and give the name btn select another and then in my code behind i will set that select another button visible it goes false and then once we have a prediction i just select another visible goes true what happens when we click this make it bigger there we go if we click this we'll make the visible of this button false and then we give our prediction photo as false but our select image visible equals true there we go so let's try that so select an image we'll do that same one there we had to select another now the position is kind of weird but we can worry about that later so select another let's try a white one here so we still got our select button here but we have our white wine here it was what it's like 81 percent confident that it's white and a nice bounding box there as well and we can fix this by going up here and saying our select image visible is false so we just test that real quick there we go so they hit it hides that file select button and there it goes again i hope you enjoyed how to use ml.net object detection with wind forms hope you enjoyed the video and thanks for watching

Channel: Jon Wood

Views: 22,353

Keywords: ml.net, ml.net object detection, winforms, ml.net winforms, winforms object detection

Id: BWubuclZDkc

Length: 17min 49sec (1069 seconds)

Published: Mon Nov 09 2020