OpenCV in C# For Beginners - Introduction to Template Matching

you can do some very cool stuff with template matching in opencv and in this video i'm going to show you one possible usage that i think could make a pretty cool app so let's dig in hi my name is chris and welcome back to my opencv in c-sharp series this series is meant to introduce and give you all the tools you need to use opencv to its fullest extent as well as some foundational building blocks on your computer vision journey in this video as mentioned we're going to talk about template matching so what is template matching well it's a way to find the location of a smaller image inside of a larger image you have your smaller template then that template is applied to the image sliding it over each pixel and seeing how similar the two slices are to each other you could theoretically use this to find say a ring from sonic you know on the actual video game screen so if you had a smaller template that was a picture of that ring from sonic you took that smaller template and you had an image of you know the game screen in sonic you could then use template matching to identify the rings on the screen right so that's that's essentially what template matching is you can you can do it for pretty much any type of object or or template because the system does not care what what you're looking for or anything it's here's my little template image that is just this matrix of pixels and i need to find a similar matrix of pixels in this larger image so it really is as simple as that um it's not extremely sophisticated so there's a lot of limitations to it but for our purposes it's actually going to do something very cool and let's get into that with starting the example i teased it a little bit in the intro to the video but basically the little example i'm going to show you here is an idea i had for an app a long time ago that i never ended up building but um i think i think it's a it's definitely at the time was a very interesting idea basically what it is is you have a multiple choice test you know picture so someone who filled in on multiple choice tests and then you take templates of the possible answers for instance so let's say all possible answers of a all possible answers to b whatever and you uh identify on the test that you know the student took what they answered through template matching you could use this theoretically to you know just have a an automated scantron type system but with just a phone app for instance so we'll show that but that that's the kind of example i want to go because i think that's a that's a decent example of usage of template matching because like i said it's kind of limited in what it can do there's there's some cool stuff you can do for sure and so i did want to actually show it in the series but um there's there's more advanced ways to do this kind of stuff but i'm just going to show you this so um the the project uh the the example we're gonna build it requires two images uh and i'm gonna move those into here so our two images are um our answered test so i just made this um i just made this one day and um and actually you can tell this is originally my from my app idea because i put qr codes in here for orientation um and then uh also we have a template so let's say we're gonna search for anywhere that the user would have answered a right so we can go in here and oh that one that one's like i mean they answered a but it's it's not perfect so we'll template matching work there we'll find out um but it's just going to go through and find everywhere where it answered a so that's the example idea let's get started on that all right so let's get started in our example um the first thing we're going to do is we're going to import imgu as always cv um and we'll also need info.cv.structure okay and then we are going to create some mats it's just what i name these this should be img greater page answer.jpg i could have named that shorter for ease of typing but here we are um and then our template speaking of interesting names to type all right okay so we have our mats okay so just to talk about this real fast template matching uh your the size of your objects matter a lot so if you rescale one of these objects the template will no longer probably be found in the image unless you know you're looking for something very generic um if you do rescale stuff you should do it you should do it to both the template and the original and you should do it at the same ratio so actually um let's do that let's try a resize uh and this we may end up not liking this but that's okay that's just part of um part of what's going on here so i do want to show you a new thing that i don't think i've showed before if you for resize if you give it 0 0 as the size and then you see there's two more there's two more parameters here after the size that's called double fx and double fy these are basically scaling factors so let's say we wanted to reduce our our image by 30 percent so we could give it a scaling factor of 0.7 and from there it knows because we gave it a size of 0 0 it knows that okay we're going to use these scaling factors so we'll do the width by .7 we'll do the height 5.7 okay and then from there we're going to do the same thing to our to this okay yeah and and like i said making sure that we have the same ratio um going both ways all right so then from here uh we need one more mat we need uh a template i'll put matt uh and i'll show what that is in a moment uh and then we're going to call our the the star of our show um it's called match template it's another cv invoke function and it has some arguments here our input is going to be our answer pick our template is going to be our a was answered and then our result is going to be in template output and then it takes a uh another argument that is let me bring that back that is a template matching type of method and this is important um because depending on which ones you pick it actually impacts how you use the data later on so in our case we're going to use one called mgu.cv.cvn dot template matching type dot uh and here's the options so if you pick one of these two squared uh differentials dimple matching types um this actually changes the outputs to where i believe you're looking for a min value from your output instead of a max value um and then if you use the coefficient ones uh or you know these others you're going to need the max value is going to be your found image so we're going to use this c co eff normed one um and you can see what it does there that's the actual math behind the matching um so you know you can look further into that if you wish it's you know i don't really necessarily need to go through the explanation here uh you can look at the documentation see the differences some of them work better than others uh and that's something that you do probably want to understand a little bit when you get into template matching because it changes you know depending on what you're trying to do it could affect things so just know that all right so from here uh we're gonna make a double min val and this is going to be our min and max values in the template that comes out and there's also there's two other things that come out of this there is a min location and a max location and those are system drawing dot points and then copy and paste that and it's going to be maxlock technically we're not really going to use these but i wanted to show that um i wanted to show another function that will give you some data from your template um and actually right now now that i think about it let's actually just go ahead and show that template um because i want you to see what kind of data you get out of it oh i need to name it okay and then we'll do we'll go ahead and do our our weight key okay so let's execute this let me double check i am okay all right so i'm in main let me execute this and we'll see what we get oh i mistyped um i mistyped the name it is a b c d i wrote dc sorry about that all right let's try this again we'll kill it run it okay this is our output so we have some data in the image that is for instance the black ink they picked up in our template and the reason for that is the template itself is has that in it so anywhere where you see stuff like that it's going to be it's going to varying degrees match right you know the qr code obviously doesn't look like our template but it kind of matched in there so i just wanted to show this real quick and we will look further at this in a moment so that that kind of that that gives us our our template output uh let me delete that and now let's look at the actual values you're going to get out of this kind of stuff so we're going to do a cv invoke function called min max loc finds the minimum and maximum element values and their positions okay so we're going to do that in template output and we're going to pass min val maxwell min lok and maxlock by reference if you don't know what that means basically we're going to instead pass instead of passing the value of these which these are value types these two are so by default they would you would pass a value and anything you did to that value in the function would not update the original value right but the way they wrote this uh this static class here is they just want you to pass a reference to that value it means a pointer and so anything they do in this function to that pointer to that variable it's going to actually update it back where you have it so we're passing a reference to min val and that means anything done in here is going to actually update min val which is a little different than you might be used to so we're going to pass min val max val we're going to pass a reference to min lope and we're going to pass a reference to maxlock okay and then that's going to basically take our template output store the minimum value the maximum value the minimum location and the maximum location and i just wanted to show this function we don't technically need it but i wanted to show this function because these values here you're going to be able to do some good stuff with those so that's important to see next we're going to do our threshold function and basically what this is going to do is it's going to filter out you remember all the noise we saw in our template it's going to filter out anything that doesn't meet a specific threshold so we're going to do that template output and we're going to store the out the the output of threshold in back into template output and then it's asking for this threshold value this is something that you're just gonna have to play with to determine um in my case i use 0.85 and that gave me a pretty good result so i'm gonna do that but this is that's a that's a parameter that you would tweak um max value i'll just do one and then again it's it's uh or sorry not again but we have a threshold type so the threshold type we're going to use and again these are these are things you can play with um i'm going to use one called 2-0 uh they they all do different things um and again i don't necessarily want to go into every single one of them but uh just know that that can impact some of this too so but we're going to use 2-0 for now to get the effect we want okay so now we've filtered out anything that doesn't match basically a 0.58 or 0.85 intensity on the original template so basically this matrix of our matches to our template uh has now we filtered out everything that wasn't a great match and we've only got things that are good matches left okay cool so why don't we just go ahead and show that real quick i want you to see what that looks like okay and then we'll run this oh i have the old one open all right okay so you'll see we filtered out all the other stuff and what we're left with these little dots and these little dots are the starting locations of where our template matched the best and if you remember what our image looked like i can pull it up here you see that it matched it pretty much picked up exactly where we answered a so we matched our template very cool okay hopefully that all makes sense um so now we've got our data for where the template matched in the picture itself and we've got we've we've filtered out all the bad matches so now we've only got good matches so question is what do we do with it and depending on the type of app you're building or the type of you know whatever uh that can change in our case i'm actually just going to show you how to draw triangles or sorry triangles rectangles um over where the template matched uh just to kind of show you visually where where we found it if you were for instance if this were a real application you know a real app to grade a student's test where what you might do is you might take that information you might match it to a positional template match of the numbers for instance of each question and then you may say well this student answered a for number one turns out one was actually c so count that one wrong you know in the program itself so you've got the data now you just need to do something with it and again in our case we're i'm just going to uh draw some shapes on top of it to show you visually so i'm going to show you how to do that real quick we're going to create a new var called matches and just double checking in case you're new to c-sharp var is an implicit type it basically just means you figure it out to the compiler and we're going to and save us some typing here and we're going to do two image and gray bytes and so what this is saying is we're going to take our template that we now have our good data in we're going to create an image of types gray byte and so basically we're putting our grayscale image in here and we suppose we have this image object and the reason we're doing that is because it's easy to iterate over so um we're going to iterate through matches which is our you know our image of our threshold image and we're going to iterate through it and finding the places where it matches the intensity we want so if you remember how to iterate through an image like this is we're going to do dot rows so we're just going to do um you know typical double for loop to iterate through this kind of two-dimensional array just as an example okay and then the thing we're going to do we're going to be looking for if uh matches the i j dot intensity is greater than some value let's say um 0.8 and i believe this is a direct correlation to this so we we only should have values that are over 0.85 so 0.8 is fine you could also just say 0.4 you know it's and i don't know if this is inclusive or exclusive so just to be safe you could do that but just for right now we'll just 0.8 and that should work perfectly fine so if if that pixel we're in the intensity of that pixel matches there's greater than 0.8 we want to do the following so we're going to have a system.drawing.point and this is this is going to be our location i'll just call it lope um drawing dot point and uh we're going to do this point at j i so that's the first thing we're gonna this is our location of where we're currently at and then we're gonna do a system.drawing.com rectangle box and we're going to do a new system dot drawing no rectangle and the arguments for rectangle are going to be a point which is loc and size and so the size we're going to use is actually our template size so if you remember our template was in the mat a was answered and we can do dot size for that map um so now this rectangle is going to be at location loc and it's going to be size of our template and remember loc is our current location in the image as we're entering through it okay and then from there there's a cv invoke function called rectangle and it and we're going to draw our rectangle on our original picture that we're looking at and we're going to draw a box that's the rectangle we're using and uh we're going to give it a color and the way they're doing color and here is with an mcv scaler object so we're going to do a new dot cv dot structure mcv scaler and we're going to make them green you can do whatever color you want and then the next is a thickness of the rectangle so in our case we're going to do a two just to make it a little easier to see you can do whatever thickness you want there really funny to do like a million and it just covers the whole thing um all right so yeah that's that's the code now we just need to show this so i can actually show you what that looks like so i'm just gonna do an invoke dot i'm show uh on our answered pick which again now has the triangle or i just keep saying triangles the rectangle is drawn on it um and then invoked.i'm show i'm also going to show the original template so you can see the kind of correlation or not the original template sorry the the threshold matrix uh of of where our template was detected okay i believe i got rid of yeah okay all right so let me run this all right and so you see um it actually found i believe every instance of where a was answered so if you think about it um how does it know that this is a but this is not and really what it is is our template has a b shape here a c shape here a d shape here which does not match the f g or h shape though i will say it does match it it's just a lower threshold than what we filtered out so the the threshold that we use to create this is pretty important and depending on what you're doing you know matters if you find a match or not if our threshold was 0.6 for instance all of these would be filled in as well anywhere they answered e would be filled in here or here etc but because we set our threshold correctly we're getting correct answers of a a a a a and even matched this one which honestly i was not sure if it would or not but it actually does so that's um pretty good that's better than a scantron probably does so all right so in this video i gave you another opencv functionality use case that i bet you could come up with some cool apps with if you want more opencv tutorials or just more program content in general let me know in the comments about you know what you'd like to see because i'm open for suggestions i'm kind of just doing things i'm interested in but if i get enough interest in something else you know i'm totally down to do that and uh kind of speaking of if you want more programming content uh if you haven't seen it yet you may want to check out my dev level up series where i create little bite-sized chunks of information to kind of fill in some gaps that devs might miss along their learning journey especially if they're early on you know if you're early on learning c sharp um you probably didn't dig into that stuff at least the ones i've made so far so i would highly recommend clicking that and taking a look at those they're really quick you can get through them really fast and hopefully learn something so if that sounds good to you click right here otherwise i will see you on the next one thanks bye

Channel: Programming With Chris

Views: 3,983

Keywords: C#, Computer Vision Programming, Computer Vision Tutorial, Computer Vision tutorial, Emgu, Image Processing, Image Processing in .Net, Image Processing in C#, Image Processing in C# Tutorial, Introduction to Computer Vision, Introduction to Computer Vision and Image Processing, Object-Oriented Programming, OpenCV, OpenCV Tutorial, OpenCV for Beginners, OpenCV in C#, Programming, Template Matching, computer vision, csharp, image processing, intro to OpenCV, opencv, opencv tutorial

Id: AAV3a_ngSUc

Length: 24min 43sec (1483 seconds)

Published: Fri Jul 15 2022