Easy Step by Step Neural Net Image Classification using C#, Visual Studio and ML.Net Model Builder

hey everybody so today we're going to talk about how to do machine learning with net specifically with c sharp and microsoft's done a great job in integrating tools to allow you to do supervised machine learning within visual studio and that's what we're going to do and we'll have a fun little project here we're going to do to work on and so if you'll just follow along step by step and in subsequent videos we'll look at some different problems that we might solve using this technology as well just a quick review about machine learning supervised learning specifically and you can go watch the presentation on this on the other video if you'd like but we always start with defining some objectives some reason that some problem that we need to solve in this case we're going to go after image classification and i'll talk a little bit more about that here in just a minute but basically we're trying to identify for example a picture of a flower and a picture of a car which one's the car and which one's the flower and this is a pretty classic machine learning problem but it's also pretty common and you can have a lot of fun with it and i hope that you find that as you work through this today so the first thing we have to do is go gather some representative data to train our model with and we'll prepare that data and then we'll train the model now microsoft makes this really really easy to do really really easy to do it's all integrated into visual studio they take a lot of the heavy lifting away the problem that we're going to look at today is one that i've done with python training my own model and i can tell you this is a lot easier to do it this way and then we'll evaluate the results and you'll see that and then if that's not good enough we'll go back and get some more data i'll probably skip that step just for time but you you do need to understand that you can usually increase the model performance by getting more representative data but that's not always the case and then you're going to deploy it and use it and we'll actually see running in a program where it can do a prediction so we'll be able to give it two images and it'll be able to tell the difference between those two or it may not be able to depending on how well we did here and then we'll just if we were running this for real we'd monitor the model performance so that's enough about the powerpoint enough death by powerpoint so we will just go ahead and jump right in the place to go is um dot net dot microsoft dot com slash app slash machine learning dash ai slash ml dash dotnet or in double quotes you can put ml.net and it'll bring you to this website and what we're gonna do is we're going to download and install the model builder which is a pre-release product but it looks awesome so i hope that you have fun working with this and so that there's some great videos up here that talk you through the different use cases and and what the technology is i'm just going to go right ahead and jump right in and download it so this is a visual studio extension and so it'll take it a few seconds here and i'll probably pause the video so that you don't have to sit here and wait for the paint to dry on this download so let me uh let me do that just well there you go for you that was a fast download for me it took a little bit longer but let's go ahead and just we'll just open this and that will cause it to go ahead and find the instance of visual studio i'm using visual studio 2019 community edition you do need to have a relatively recent version of visual studio i'd recommend that you just keep up with that especially if you're if you're learning so again we'll pause for just a moment and let it do its thing okay and it's telling me that if i i'm gonna go ahead and install this and there's a cross-platform development uh this this will um the coding part the command line interface to this will actually work on linux and the mac but the model built-in building plug-in is only in visual studio so um unfortunately that's not on all the other platforms yet now at least this plugin is not available or this extension so again we'll pause and come right back okay all done so it's installed that extension in visual studio and i'll show you how to access it here in just a minute you will notice this project is uh active it is um i've been out there for a couple of years but it is an active project which is awesome i do recommend you go off and watch the videos and learn a little bit that way what i like to do they show me the videos they tell me about how wonderful it is and i like to go try it out on my own problems so that's what we're going to do today so image classification remember we're going to have two images and what we're going to do is we're going to do stop signs and yield signs okay so stop traffic stop sign and a traffic yield sign and we're gonna see if we can build something that can tell the difference between those two signs so uh let's let's jump in let's fire up visual studio and get started so let me close this out again there's the there's the url and this is the website again go out watch their videos all right and we are going to create a new project and we are gonna do console we're just gonna do a simple c sharp um a simple c sharp console app is what we're going to do right here and we are going to put that in a particular location and rather than watch me pick folders hang on we'll call this stop yield now there's nothing particular about this thing i picked you could do this i've done it before with chihuahuas and blueberry muffins stop and yield signs two different years of cars two different types of cars there's all kinds of things you can think of that you might try so pick your own thing that you might do you could do different faces so pictures of one person or another you could do male female there's all kinds of things you could do so uh try it out let's go ahead and just create this and it's not that actually that important that we create a console app we're not going to do anything with this you'll see that the tool is going to actually build its own console app and we'll be running that so what i'm going to do is right click on the solution and select add and let's see okay not on the solution we want to do that here here at this level uh with the project so you you select add and then machine learning and this is where the cool part comes in up here it says would you like to enable it yes we're going to enable the ml.net model builder so we're going to go ahead and click that and once we do that then we're going to have to come back and do this again remember this is pre-release stuff so hopefully they get some of those kinks worked out so let's go ahead and do that all right so uh the videos that are up there are a little bit older than what's being released right now but we are going to do image classification but you can see there's lots of different things here and we're going to do some different videos on all of these because this is a fun way an easy way to get into machine learning and have it actually do something for you without having to know all the ins and outs of things like neural nets we're going to use a neural net so that we're going to be using a neural net for image classification and so that's great so we'll pick this okay so first thing you're going to run into is when you're doing anything with images and machine learning it's always good to use a gpu and i have one unfortunately you have to have certain drivers installed so you can click this little link here and it'll tell you about that my drivers are all set up for anaconda and python and unfortunately they're different versions and what this requires hopefully again that's something they'll address as they release this and there there doesn't seem to be an easy way to tweak that other than uninstalling and installing drivers and if you've done anything with python and anaconda and trying to get everything configured on a windows machine to use gpu you know that once you get it working you really don't want to mess around with it because it might not work anymore so anyway i'm going to use a local cpu environment that's going to be a little slower my experience with the gpu is it's about 10 times as fast and i've just got a rtx 2060 an nvidia geforce rtx 2060 which is not a high-end gpu but it is a gpu that you can actually use and so if you're going to do a lot of machine learning especially with images gpus are always useful all right so we're just going to use the local cpu instead and we'll just hit next step microsoft does a great job of making this kind of like a wizard that you just go through and select different things all right the first thing that wants to know remember in our machine learning model we got to pick our data well the problem is is we got to go prepare our data first which means we got to go select it so let's go do that so over here now i have a little chrome extension that lets me go and download images so i'm going to say stop signs here and i am going to download some of these and i've got this image downloader extension i'm not advertising that if you want to use it great it's a it's an easy way to go get data and and we're getting the images at thumbnail size so this and i'm doing that mainly for speed reasons but basically i'm going to go through here and select all the images that i want i'm going to get rid of things that i don't think are representative like this or yellow i'm going to get rid of some of those and so i'm going to go through and work through that list all and for you that's just going to take a couple of seconds well turns out when i hit the resume button on camtasia guess what my extension goes away my little uh plugin for image selection so you can fast forward over this part if you don't want to sit here and watch me select these but i'll try to chatter a little bit while i'm doing this remember that you're looking for representative stop signs things that you would actually see out there and while these some of these are that you really don't want to train a model to recognize this as a stop sign now you could argue that maybe you you need to do that depends on your use case but i am going to go through here and eliminate a lot of these especially like the do not enter signs the stop signs that have additional text on them we're just going to get rid of these and remember we're trying to get representative data certainly orange cones are not that yeah what do we do with that well it's got a slow and a stop i really don't want it learning that so we want it keying up on the stop sign and you could argue that you might ought to keep this in here because you might see both of those like that but that's up to you so this is the advantage of doing this with machine learning is you can play around with this and microsoft's made it really easy to do so let's keep going and again apologize for the time it takes to do this uh you will find in machine learning that most of your time in your project is spent with the data either preparing the data or analyzing data so get used to it it's not like regular software development where you spend most your time in code you're going to spend most of your time dealing with data like this now i'm not before when i did this uh with python and anaconda i had to make sure that all my images were the same size i think microsoft taken care of a lot of that for you don't know all the details on this yet because i'm just new to it myself but in any case we're going to call this stop signs it's going to end up over in the old download folder so i'm going to go ahead and hit that and it's going to warn me and yeah whatever go ahead and then so there it is there's all my stop sign images and now i need yield signs right because we're going to do both so you get to watch me sit here and go through those as well and you can get more images than what i picked up here but this is a good little set for this example just to show you how this all works so let's do it let's go ahead and grab our yield signs now so we don't want that don't want that certainly don't want that not liking that too much nope uh you can see that it's kind of arbitrary it's like well wait what decision process are you making well i want things that are representative as what of what yield signs might look like and also i know enough about neural nets to know that it's going to pick up on certain things like maybe that text or maybe this shape or the color and so part of this is you just get some experience in in working with the data and working with the models and it all tends to turn out okay for you if you're willing to spend some time on it lots of yield signs here so none of those and nope and nope nope nope not even that'd be really confusing to the neural network if you told them both of those were yield signs say wait a minute you told me something different so let's just keep on going here all right i'm going i guess i'm going with the more red and white yield signs i'm not really interested in the old old style yellow ones even though as those may those may appear in places again you you get to decide you get to decide how you want to train the model so remember that the model um i'm not loving that one either nope these these are really the pictures i really want because you can kind of imagine a self-driving car that was trying to pick out yield or stop and you'll see here that this probably isn't good enough to go trust your life in a self-driving car on that's why we don't have them out on the road yet it's going to take a while to get it get everything going in such a way that we can really feel pretty safe all right so we got rid of all those and we're going to download those into a folder called yield signs now the other thing we need to do is we need to hold out some of this data or we'd have to go get some additional data and because we're grabbing it through a google search it's a lot easier to just hopefully it did it it's a lot easier to just deal with that right up front so let me see oops let's go over here in our downloads folder and we got yield signs and there they are there's our yield signs wonderful it looks like we got some stuff in here we don't want i don't know how those slip through so i'm gonna go delete those i might have to go through these again sorry yeah these don't look too good i'm gonna clean up these are all pretty good okay not don't think so nope nope and there's some look pretty good all right and then what about our stop signs stop signs over here oh man we don't want it learning that kind of stuff so i don't know how that's slipping through i think maybe uh i think maybe that extension's doing some wonky stuff so again you gotta you gotta clean your data up so we're doing data preparation right now we're just doing a review here making sure this all really looks the way we want it to we don't want slow get rid of that one and these are kind of you know a little bit tricky because you don't really see stop signs out in the wild so to speak they'll look like this they tend to look like this right all right you see anything else in there we need to get rid of well these look a little round but hopefully it'll be able to figure out the difference there and i'm probably not going to do that let's delete that one all right good so um so let's hold out some what we're going to do now is we're going to move these folders over so i'm going to grab these two folders and we are going to copy these and put them over here in this other this other directory and we'll put them in the ml folder here and let's just paste those in and then what i'm going to do is i'm going to copy them again and the reason is is because i need to have training data and test data both and so um we are going to do that so i'm going to go ahead and paste those in and we'll call this stop signs this is going to be our test data so we're going to we're going to hold out some of the data to test the out the the machine learning model once we develop it and we don't we can't test it with what we trained it on because it's already seen all of that so you gotta if you want to test see how well your model is doing you got to give it things that it hasn't seen yet that are things that you want to classify in this case stop signs and yield signs and so what i'm going to do is i'm i'm just going to hold on to i'm just going to hold on to about 10 of these so i'm going to i'm going to just i'm just going to arbitrarily pick some well i'm not arbitrary i don't like that one do we so i'm not gonna i'm gonna i'm gonna mainly pick the ones that look like they're the ones out on the street okay so here so here here here so one two three four five six seven eight nine ten and we are going to get rid of all the rest of these so is that good so we're gonna hold out ten and get rid of the rest this is remember our test data so we're going to we're going to validate that we've got good good data and we'll do a similar thing with the yield sign so let's grab the uh let's grab one two three four five six seven eight nine ten i don't like that one get rid of that anyway and let's pull all these out and delete those and then let's go clean up some more of our data again you're like hey can't we just move on well your model is going to learn what you give it and so you better give it the right thing to learn because if you don't then well you're going to get some wonky results so let's look at our stop signs again real quick don't like this one whack that don't like this one yeah i'm sure i'm going to miss some in here and then we're going to delete the last 10 because those are the ones that are in our 6 7 8 9 10. these are the ones that are over in our test set so we don't we don't want to train on those so we're going to delete those get those out of there and okay well not this gun i don't know why i don't know maybe you see street signs like that i don't i see them you know like this so anyway all right let's go back to the yield signs and make sure that we delete our we need to delete our our test samples one two three four five six seven eight nine ten bye and i think i deleted that one so i'm going to get rid of that one too i'm not doing a very good job of being very methodical with the data which i really need to be but i'm trying to kind of rush through this a little bit so that we can get to the to how this actually works so again just cleaning up our training set i don't really like that one i don't like that one all right those you visually look at those those all look like yield signs don't they we don't need that in there whatever that is a little craft let's delete that darn it i didn't want to go away all right and we don't need stars obviously my my filtering thing here just did not work so go away i want that so maybe you'll find a better utility for for dealing with uh with your data having good data tools is really essential to uh what you're doing so again let's do one final check um again just to make sure we're good here so we got yield signs those are the ones we want to train on that's our training set probably should have called a train let's just be clear and do that so training and then stop signs let's open that up make sure we're good on this set looks pretty good pretty good some of those you'll never see out in the wild but i don't know if i love that one it's like that arbitrary right it learns your biases is essentially what happens right so you you say you don't like this one and then you find out that out there there's a bunch of them like that then you got a problem okay here's our test set okay where i think we're good with that and here's our stop signs test yeah we're good with that 10 of those all right great wow spent a lot of time prepping the data uh well not actually a whole lot you will spend days and days in a lot of cases and we're doing a small amount i just did a run with this where i was doing about 4 000 images and it it took a while took a while to work through that much data that's why your gpu is always uh useful to have so all right so here we are on the data screen so what it wants it says select the folder which contains all of your images this folder should organize your photos into separate labeled subfolders all right so we got to fix that so let's go back over here so what it wants is it wants um yeah so what it really wants is this non-stop yield so we're gonna go let's create a new folder stop yield data and we are gonna put the training data underneath that now what it wants is it wants you to do this let me just stop it i want you to do this where this folder name is the label that you want to use and we want this to be called stop so we'll just call it stop sign or stop sign and we'll call this one yield sign now normally you might have to go through and do a lot of labeling and put things in a file and like in a csv file with a pointer to the they they really made this a lot easier it's like hey if you're doing image classification put all the things that are alike that you want in one category in one folder and you can have as many folders as you want so we could do stop signs yield signs do not enter signs whatever we wanted we just need to have the signs in it in a separate folder and remember to set up your your test data as well okay so we're get we've got this sitting in stop yield data and it's going to the labels our stop sign and yield sign that's what it's going to use so pretty simple we go out here and just say okay so let's go do it let's go get it and in boy why do i bury these so deep just make it hard on myself so it's stop yield data there it is and it has immediately said okay you've got 172 stop signs and you got 58 yield signs that's probably not a real good idea we probably have probably ought to have something a little bit similar and i'm seeing stuff in here i do not like with those stop signs what is going on let's go check i don't see those in here maybe i didn't go far enough down and there's some more garbage in there well i don't know i don't see those so oh there's there's that one let's get rid of that that was good this could cause all kinds of grief here so i might have to reselect the folder remember we're dealing with we're dealing with pre-release software so i want to i want to catch that other one this is where you got to really spend your time with the data and i know i'm rushing through this real quick so where's that where's that stop maybe i know i'm missing it don't need to stop right here pilgrim get rid of that isn't this fun watching me mess with data that's your life and machine learning that's what it's all about all right whatever we're going to go with it the way it is and hopefully let's do this i'm just going to reselect it because i'm worried that it because i deleted a few files out of there that it's gonna get it's gonna get confused so don't know if that if the programmer took that into account or not right i might not have if i were writing the code so let's do stop yield data again and okay whatever we'll we'll see if it can handle it okay now so all i did was hit next right so i got i just hit next step here and now we just hit start training now this is the cool part about this this model builder it's going to go out and try several different models and it's going to see if it it can figure out which ones are the best and it's going to do all the parameters for you the hyper parameter setting and then it's going to come back and say this is the best one and here's the results and i can't tell you how wonderful that is that is just awesome that you can do that because otherwise you're going to have to do all that you're going to sit here and write code that would set up the different models and the different parameters and try different things out and uh yeah it takes a lot of of work to do that so this is taking a lot away and some people may feel like well yeah but it's also taking a lot of control away from me that i want to hang on to and you'll have to judge for yourself as to whether that's really true or not so i'm going to go ahead and start training and you'll see some stuff going on down here in the console and off we go right so it's off training so it has selected a machine learning model and it is training that model with the data that we provided and so um what do we have 250 samples or something like that 214 apparently so and it said okay this one got that see it's it's trying different things and it's just going to keep doing that until it settles on what it thinks is the best one so now we're going to do an uh multi uh class classifier and here we go again just gonna try that one out let's see what it comes up with pretty cool about like watching paint dry isn't it 208 okay it's thinking about that one and it's saying well i got a 91 accuracy on that so i'm gonna go try some other things here so off it goes again so rather than uh i'll just pause this and come back and it'll be done all right here we are so we got done it got through all the different iterations and it tells you up here that it it took 396 seconds to do all of that the different permutations with all the different parameters and it came back and it said i can give you 98.99 accuracy which is awesome uh using the image classification which it turns out was the first iteration that it did but it had to go try all the other ones anyway all right so that's good that's awesome all right so let's go on to the next step by the way here it tells you the model best model is uh dnn neural network with resnet50 resnet50 is a common uh image classification model that's out there and it told you how many uh it trained in it and it did the one model and tried all the different parameters is what it was doing all right next step okay so here we go we're gonna go try it out so let's pick an image and we will go pick an image out of our test set so we jump over here and off we go into the test set let's try yield signs well we'll throw it at that one and it's thinking about it using the model that it just trained and it said it's a 97 chance that's a yield sign and three percent chance it's a stop sign okay that's great let's try another one this is a little bit more challenging we got other stuff going on in this image but it says i'm 81 percent sure that that's a yield sign try another image and give it a softball yeah i gotta get this one 100 percent yield sign it says and then finally let's do this one it's got the other little sign on it 95 all right let's go try our stop signs and see what it see what it does with those stop science test oh let's do the ones with the railroad crossing and two stop signs see what it thinks about that 99 stop sign and this has got some other stuff on it pretty good works pretty good now we've given it a pretty simple problem to solve right but you can see that it's not a hundred percent on everyone that's machine learning the model it's it's built in uh it's it's built up based on the training data that you gave it what to do and here we go how about this one so far we didn't see it predicting a yield sign when it wasn't so that's great that's awesome all right last step uh so our last step is deploy this thing and actually make it work so it's real easy it's one button click add to solution and it's going to come out here and write some code for you basically it's going to create this this console app and here's the model this takes care of loading the model and everything and this is the this is just a program that's going to run and it's going to actually do a prediction and so what i'm what i'm going to do is go change this to one of the test data items so let's grab this one here and let's grab the path first i'm sure there's an easier way to do this somebody needs to do a tricks and tips windows copying files video or something like that all right let's go get that image i wanted to do this one here so do a little renaming on that and do control a control c drop that in right there so this is your predictor it's going to use this image and right here it's going to go do the predict and so where's this consume model code that's what's sitting over here in this other uh class this other project that it created for you and then it's going to tell you the results and so what we're going to do is change the solution prop the solution properties so that it we can start this console app right here so the startup is not going to be stop yield it's going to be this one and we hit ok and then we run it and now in c sharp we are running the prediction here it goes and there it is okay so um the the way this works is it's showing you both values just like it was on the previous screen and so it says i'm 91 percent confident that that's a stop sign so the predicted label value is stop sign with 91 and then 8 that it might be a yield sign and then you could just change you can change that image to whatever you'd like right you can put whichever one in there that you'd like and see what happens so i hope you have enjoyed i hope you've been able to follow along on this and get this working it is just really neat that microsoft has provided this capability with ml.net i'm looking forward to using it some more and sharing some additional learnings with some of the other uh use cases that are out there that it can do as well so thanks for watching please subscribe like the video i always appreciate your comments and uh check out ml.net for microsoft thanks

Channel: Jon Jensen

Views: 13,898

Keywords: ML.Net Model Builder, Image Classification, Machine Learning, Visual Studio, Visual Studio Machine Learning, C#, C# Machine Learning, C# Neural Net, Neural Net

Id: T94YH7_mx3E

Length: 37min 40sec (2260 seconds)

Published: Sun Aug 16 2020