Predicting on a Custom Vision ONNX Model with ML.NET

hey everyone so while back I did a video where we used Microsoft custom vision service to create this model here where it not only tells us if we have a red or white one in the photo but it's actually an object detection model so it tells us some information on where in the photo the white is but this is all own custom vision what if I want to use this model in mo Annette well we can do that and the first thing here is we need to export this model so we go to up to performance here and then export and we got different types of files that we can export here coromoto tensorflow darker file with the model in it but what I'm going to use is that onyx file here so I'll click on this and export alright then I'll click download once it's ready and it downloads the model file and some other stuff here for me all right so this download finished let's go ahead and export our extract out the files here and I see what we have is we have our onyx file here that we'll be using and we also have the labels file here that tells us red or white which is the labels in our custom vision model then it gives us some sample code and c-sharp and Python that we won't actually be be needing here so let's go to visual studio here in order to have a project set up have a model folder here a test folder with a couple of test files that I'll be using one for red one and a little for a white one then have a this bounding box class and if you notice in the program here I have some some variables and helper items already set up that we'll be using later before the model file that we have here all we need to do is take the Onix file and the labels file and we move those over to the model folder here and I'm just going to set these to copy over now that we have our model exported now we can start using it and I'm going to go ahead and download some new get dependencies so the first thing we need is ml dinette itself use the 1.4 version here since we're doing images I'm going to get the image analytics NuGet package and since we have a model file I'm gonna get the Onyx transformer package as well alright with that we can get started building our ml net a pipeline here the first thing is we need the new in little context we're just using the onyx model here we're not using any input data I'm going to create some empty data it's going to be a new list there's gonna be type of Wan inputs and let's go ahead and create this file and since we are dealing with images I'm gonna create a property it's going to be a a bitmap type and we'll call it image for in law Inez we need to add an attribute to it called the image type and this can take in the height and width of the image and one of those helper items that I mentioned earlier includes an image settings and all that just has image hide in the image width and both of these are set to 4:16 so we have our empty data now we can create an odd data view with it using context data load from innumerable and we'll just pass in the empty data into it now we can create our pipeline here so the first thing I'm gonna do is do a transform and resize the images and these this is the pipeline when you give it an image to run against the model so it's going to first resize the image I'm set resizing property and I'll set the resizing khand to fill and I'll set the output column name to data the image width using that same image settings the image height to the image settings as well the input column name I'll use the name of keyword and now use the wand input image property so that's it for the resize images next I'll append to it another transform where it extracts the image pixels then I'll just set the output column name for here set of the data as well and our last atom in the pipeline we will call the transform to apply our next model you can give to the model file which is in the model folder called modeled onyx give the output call name we don't get to actually name the upper column name whatever we want we had to be specific to whatever the output column name and this onyx file here and there's actually way we can get that name using a tool called Neutron so let's take a look at where to get that and how to use it all right I'm at the github repo here at the Metreon project and this is just a little desktop app that where you can get some information on a model here and to to get it just go to the releases and then download the release for your platform we started up it's gonna be just kind of an empty screen here we can open the model and just give it the model that we had downloaded from custom vision you know and it gives us some information about our model here we can click on the model properties and it gives us our inputs and then our outputs in fact we gave our input the same as the input here it gives us the output name as well here so we're gonna copy this the model outputs name and it will put that in our output column name and Emma done that here and then we'll specify the input call name as data alright so now that we have my pipeline we can call fit on it with our empty data and now that we have that model we can create a prediction engine saying the Wan inputs is our input then we do a 1 prediction as our output here and we're just passing the model for this one prediction I just give it a single property it's going to be a float float array I'll call it predicted labels let me give it an attribute here the column name attribute from pillow dinette and I'll call it model outputs 0 make it match what we saw in the net Ron application for the model and before we can predict on it we need to get our references to our test files here pretty quick I'll just do a variable I'm a string array with the names of those test files that we've shown and then I can loop through each of those files so we have the image name and then I'm going to use a stream specifically a new file stream that can read that following get that the file information from it so I'll give it the path and I'll give it file mode open and then the stream I'm going to set a test image variable I'm gonna do image that from stream and passing the stream here let's test image I'll set up here it's a bitmap you get an error here you can't convert it so I can do an explicit cast to a bitmap from that from stream alright so now that we have the bitmap of our test images we can create a prediction using the prediction engine dates give it a new one input and just set the image to the test image now here's where we're going to be starting using all these these extra helper methods here and I'm gonna be honest I didn't come up with all of these in more than that sample that reads from a custom vision our next model has these so so I use those as well and honestly I doubt I could come up with all this if you look at all these helper methods here and what these pretty much all do is allow you to use that bounding box class here and just gets the bounding box information from the model and it just calculates it to what we can put into our image to create the bounding box that's pretty much all this is doing so we can get our bounding boxes using the parse outputs method give it a pretty de labels and then we actually need to get the labels from that labels file up here so I'll just set this up here before the lube labels and that's just going to be filed I read all alarms now that send out labels file let me get the original width of the test image let me get the same width the height and so I'll do a check here if we have at least one bounding box I'll get the max confidence that it gets from it using the max link method here and it has that confidence properly on it now get the top bounding box with that Max confidence used to first your default link method it's gonna be the first one it's gonna be first bounding box where the confidence equals that nice confidence now I'm gonna clear all these bounding boxes and then add the top one so I only have one bounding box that's gonna be the one with the max confidence because I can give you multiple bounding boxes for each of the predictions that it gives you so it can have multiple predictions for where in the image the one is but we're just gonna take in the top one and here is where we're gonna be using a bunch of that a bunch of those helper methods in fact I'll just I'll just copy these over because it's a good bit of math here and I misspelled the original height and width here but basically it's just getting the X & Y width and height of the bounding boxes so we know where to draw the rectangle onto the image and we're using the graphics API here to draw that rectangle and we're gonna draw the string of the description which is pretty much give us the confidence percentage and then we're gonna save that test image to the file system and give the name and then that's what we can use to look at the file that has the information from our model all right so let's run this and see what we get here or so that ran so let's go to our bin folder and our test I guess we got our two predicted files here's the red one and hopefully y'all can see this okay I get our a nice bounding box on it and it says it's a red wine and a 76 percent confident on it white wine is a little bit different at volume box didn't get the entire one but it's pretty close and it did predicted to be whites in this fifty nine percent confident alright everyone that's just how you can use your custom vision model within in-law dinette if you look at the pipeline is actually pretty simple just dealing with the images the resize extract them and then we just apply the Onix model and then you have to use that metrion tool to get the output column names in there noting that the the biggest part here is gonna be doing this hopefully then it gets easier in the future where we don't have to do all this to get our bounding boxes but for now this is what we had to do so I hope you enjoyed the video and thanks for watching

Channel: Jon Wood

Views: 8,363

Keywords: ml.net, custom vision, microsoft ml.net, microsoft custom vision, custom vision service, custom vision object detection, onnx, onnx model, onnx ml.net, .net, c#, deep learning, deep learning c#, deep learning ml.net, custom vision ml.net, machine learning

Id: KDxbXlwPPqk

Length: 13min 39sec (819 seconds)

Published: Mon Jan 27 2020