AI for Everyone LESSON 14: Understanding Contours in OpenCV

hello guys this is paul mcquarter with toptechboy.com and we're here today with episode number 14 in our incredible new tutorial series where you're going to learn artificial intelligence or you're going to die trying what i'm going to need you to do is for yourself a nice tall glass of iced coffee that is straight up black coffee poured over ice no sugar no sweeteners none needed and as you're pouring your coffee as always i want to give a shout out to you guys who are helping me out over at patreon it is your support and your encouragement that keeps this great content coming you guys that are not helping out yet take a look down in the description there is a link over to my patreon account think about hopping on over there and hooking a brother up but enough of this shameless self-promotion let's talk about what i am going to teach you today and what i'm going to teach you is i'm going to teach you how to track an object an o i i o i object of interest i'm going to teach to track an ooi an object of interest based on color now you say wait a minute we did that last week no really what we did last week was we just created the mask and then we applied the mask to the original image and basically we blacked out everything except the object of interest what we're going to go to today is we're actually on that mass going to find the object of interest and know where it is and in terms of kind of like an x value and a y value of where the object of interest is so we're not just visually seeing it moving around but we train opencv for a particular object and then opencv knows where in the frame that object is okay so that is what we are going to do today and what that is called is the the the command that does that is contours basically you look at that mask and you create a contour around your object of interest now you know one of the things that i'm always emphasizing on this channel is get it figured out on paper before you try to do it in code and that's what i'm going to do today because if i just jump in and start programming you're not going to understand the data structure for the contour and you're just going to be copying me but you're not going to really be understanding it if i take a few minutes and i draw it for you then you'll understand that data structure and it'll make a whole lot more sense when i go in and start coding so what i'm going to do is i'm going to switch over to our most excellent sketch sketchbook view and what i'm going to do is i'm going to come up here and then what we are going to do is draw a frame now you could imagine that is that frame that you are grabbing from the camera and then what we are going to do is we are going to let's say that we trained it on red and over here ooh let me go backwards let me turn that off okay okay so what we do is let's say that we have a red object and we've trained it on red and just for fun over here i'll put a smaller red object okay so now we've got two objects of interest and let's say that we do all the things that we had done before create the the ranges and after creating the ranges create the mask now we've got a mask where anywhere that there's red we have a white dot against a black background okay that's what we did in the earlier lessons well with this uh with this command cv2 dot find contours it returns to you a set of x y values that define the perimeter of your object of interest and so you could look at it this way it would give you a point here here here all the way around like this and then it would give you tuples of the position x1 comma y1 x2 comma y2 x3 comma y3 and so forth and so on and so what the find contours will return a contour and that contour is an array and it is the point x1 comma y1 comma x2 comma y2 comma x3 comma y3 dot dot dot and then you close that contour okay you close that contour so it's an array of x y coordinates is a contour but the command find contours doesn't just find one contour it finds all the contours so it would also find this one which would be a different set of x y points so again a different x1 comma y1 a different x2 comma y2 and a different x3 comma y3 and so that would then be your second contour x1 y1 comma x2 comma y2 comma x3 comma y3 dot dot dot and this is your second contour all right now if it's going to return more than one contour it has to create a data structure and that data structure is it puts them in an array and i'm going to change the color i'm going to change the color here and so you have an array of contours you start the array of contours here and then you have to close it here and there has to be a comma between it and so contours commands returns an array of arrays of x y points an array of contours of x y points so it's sort of like a a data structure that is three deep so you could kind of think of it as it re returns the first contour comma it returns the second contour comma it returns the third contour and then it ends the array of contours and then each one of these is like x1 comma y1 tuple comma x2 comma y2 and so forth so i hope that makes sense but the reason this is really important is let's say that i have something bright like this okay if i have something bright like this and there's only one contour and let's say it returns that one contour as x1 comma y1 comma x2 comma y2 comma x3 comma y3 dot dot dot it's not going to it's not going to return this it's going to return an array of contours which means even when there's only one it is still going to put the outside brackets it will return an array of arrays even if the number of arrays inside is 1. i hope that makes sense i hope that makes sense because in some of the commands if you just put in a contour and it's expecting contours the program would crash if you just have one contour you just end up with one you have to put the bracket around it so that the data structure here will match and you didn't see that did you okay so that the data structures will match all right enough of this chit chat let's jump in and what i'm going to need you to do is fire up your most excellent visual studio code and uh then we're gonna jump in and i'm gonna show you how to code this i'm sorry i'm a little got a little bit of a scratchy void this this is actually the fourth time i've tried to make this video i've had some bad luck today and so i've been sitting here talking for the last four hours trying to make this video but i have a feeling that fourth time here is gonna be a charm but i've just got a little bit of a dry throat after all this talking today okay you are going to fire up your most excellent visual studio code and then we're going to come down and i do believe that this is going to be program oh open cv dash 19 dot p y and the dot p y is kind of important now we don't want to write this whole thing from scratch because last week the program that we ended up with does all the masking and training and the track bars and all that so we want to take advantage of that what the work we've already done if you have that great if you don't you can go to the most excellent www.toptechboy.com and you want to come over here to this happy little search bar and you want to search on tracking two colors in opencv it'll bring you to this lesson this is the code that we ended up with last week you click on the two little ah this is really going to do that to me let's see if i reload it sometimes that little button doesn't work and it breaks at the worst possible time all right let's see if i can just snoop give me a second i promise you we'll get this thing figured out man this has been quite ornery today isn't it there it is so now we're going to right mouse click copy we will come back over here and we will right mouse click paste and then of course we always wonder can we copy and paste without breaking something so we're going to go ahead and run this just to make sure that we ended up at the same we're at the same place we ended up with last week you can see the mask down below and then over here down below you can see the composited image and look at that boom it looks like things are working as we left them last week so we will quit out of this but now instead of just showing the object we want to outline it we want to trace around it and actually track it okay so in order to do that what we're going to need to do is we're going to need to find that perimeter we're going to need to be able to find that perimeter find that contour that defines the perimeter of the object that we are looking at and since we want to trace around it we need to do it on every single frame which means we need to do it inside of the while loop we need to do it inside of the while loop and we need this is very important we need to find the contour not on the original image not on the frame on the mask that we created we want to find the contour on the mask that we created and so we need to find the contour after we find the mask and remember in this particular example we're tracking on two different ranges of hue and that way we can sort of straddle that red that goes you know like above zero and between 175 and 179 that that red range we want to be able to track that we need to do two different hue ranges and this software does two different hue ranges and then my mask is the composite of the original mask and the second mask and so my mask is able to basically have both colors in the range if you watched last week's lesson that will make perfect sense so now i have my mask now what do i need to do i need to find the contour so i am going to say that contours why do i say contours well i could have called it my cat boxes if i wanted to but i'm going to call it my contours and plural because why because that cv2.fine contours returns an array plural of contours it also returns a second variable which we don't use so i just call it junk but we have to catch it if it's throwing it at us we got to catch it so i catch both parameters you know both data structures that this command is going to return so this is going to be equal to cv2 dot find contour cv2 dot find contours well where do we want to look for the contour we want to look for the contour not on the frame but on the mask okay and so what this does is you know the the mask is a white spot which is our object of interest against a black background and so what the command find contours is looking for it's looking for white spots on a mask so i want to look at my mask all right and now i got to deal with the data i got to kind of tell it how to structure the data as it comes in because you could imagine that you might have a red object and inside the red object is a blue object and inside the blue object is a red object in that case i've got a contour a red object inside of a red object and so how do you want to deal with those complicated data structures well you want to deal with them with the command retr is in retrieve like how do you actually want to retrieve this data and you could do like a composite you could do a flood fill you could just list them all or you could have a tree structure there's all these data structures but all i want is the outside one if there's one inside of another i don't want to deal with it just give me the outside um you know give me the perimeter of the outside one so what i'm going to say is give me external like that okay so give me the external one and that gives me the simplest data structure possible which makes it easier for me to deal with okay now i also want to tell it cv2 dot chain and that's kind of like what are we going to do with all of this data cv2 dot chain and then what i want to do is i don't want every single pixel all the way around that's too much data i want it to just give me kind of an approximation with fewer pixels so i'm going to say chain approx and then simple like that chain approx simple let's make sure you can see that you can't quite see that can you okay like that chain approx simple like that does that make sense okay so my mask is where i look for the contour when i return the contours i only want the outside ones i don't want ones that are inside of other ones and then don't give me every single data point just give me a nice outline you know a nice simple outline so at this point i should have my contours if i'm thinking about this right well if i have them what do i want to do well i want to draw them i don't want to draw them on the mask i want to draw them back on our original frame and that way i can outline my object of interest with a drawn in curve around it how would i do that i would say cv2 dot draw contours i want to draw the contours where do i want to draw it well i got the contours from my mask but i want to draw them there i want to draw them back on frame my original image i want to draw them on the original image okay what do i want to draw contours all right why do i have to tell it that well i might have given it lots of different contour commands and i might have lots of different arrays of contours i've got to tell it which one well the one i just found which i call contours if i called it my cat boxes i would put my cat boxes in here okay now i need to tell it that's an array of contours which one in the array do you want to draw i don't know i don't know draw them all how would i say draw them all minus one okay now if you set a zero here it would draw the first one if you said a one here it would draw the second one but you see the first one might just be a little random pixel over here a little bit of digital noise i don't know if the most important one is first so i'm just gonna i'm just gonna for right now draw them all and then we'll sort it out later okay and then what color do you want it you put color as a it's a tuple let's put it blue that would be 250 remember it's blue green red so 255 full on for the blue off for the bg off for the green and off for the red okay now how thick do you want that how heavy do you want that contour i'll make it a three which is pretty heavy okay if i'm thinking right guys if i'm thinking right this should trace out my contour let's see good news is the program didn't crash and then boom look at that okay do you see how i have traced out my object of interest what do we like we like that i have traced out my object of interest what do we not like we don't like that the noise down in the mask each one of those little pixels is showing up as its own contour so if you look at my hand it's it's putting all those dots on my hand because each one is like an individual contour we got like way way way too many contours now there's several ways that you could handle this one way is you could sort your arrays of contour in the order of what their area is and then only print the first one only draw the first one the biggest one okay another way you could try to do it is just try to tune your way out of the problem okay i'm going to try to tune my way out of the problem by adjust adjusting these track bars okay and in this case i was able to tune my way out of the problem but in this case i've got a bright yellow object of interest against a bright green background and i am wearing a lavender shirt and so it's pretty easy to tune in on yellow in most real world applications you're not going to have things that easy so you cannot solve it simply by tuning it carefully and so we've got to get a little bit more clever and what we want to do is we only want to trace the contour if it's a certain size and if it's smaller than that we're just going to assume it's noise and we're not going to trace that one okay so if we're going to do that how would we go about doing it well i want to start by commenting out this because we're going to do it in a more elegant way now and what we're going to do is we're going to step through the contours so i'm going to say 4 contour single in contours so with this for loop the first time through contour is going to be the first contour in contours the next time through is going to be the second contour in contours it's going to step through the contours all right and then what i want to do is i want to find the area i want to find the area of the contour so i'm going to say the area is equal to cv 2 dot contour area now what do i want to find the area of this contour this command is not expecting the full array it's it's expecting the contour you're interested in so just one so i'm passing it what singular just won the contour now i've got the area of that contour if the error area is greater than or equal to let's say 100 pixels then it's a keeper it's a real one okay if it's a real one what will i want to do i will want to see the two dot draw contours i want to draw them on what on frame i want to draw what contour i want to draw that contour okay and then this time i don't want to do them all right i just want to do one the first one which is the what the zeroth one okay i want to do the zeroth one all right now what color let's keep it blue let's go 255 comma zero comma zero inside that tuple and now how heavy do i want the line i still want the trace to be a a heaviness of three now this is where you have to pay attention this draw contours command it is expecting what it is expecting what the array of contours and i am passing it a single contour there is a shape mismatch in the data because it wants an array of arrays and i'm just giving it an array well the easiest way to do that i could just come here and say my contour is equal to and then i'll put contour inside of those outside arrays and so i've just put the array in an array and there's one one array in the array but yet still it's in there so i could do it that way and then i would put here my contour now that would work but there's a little bit of a shortcut you can do and that is i can put that contour in an array by just putting the array brackets around it and now i am passing draw contours an array of contours how many contours are there there's one okay and therefore the first element is the zero element and this should work so let's take a look at this unless of course the silio typos good news is it didn't crash boom look at that okay now what i'm going to want to do is i'm going to want to deliberately come in here and kind of detune this you can already see there's all that noise around my hand but it did not draw it and let's see if i can get a little more noise there okay that it's getting big enough that it's drawing but okay do you see in the mask there is all that noise but in the image up here there is not anything except for the object of interest okay now you see sometimes it is drawing something that shouldn't be there and so the way i could deal with that is i could come up here and say well let's only draw it if it is 200 okay so i can kind of get rid of that noise that way okay so in the picture down there you still see the noise but here you don't see the noise you only see the object of interest pretty darn cool huh all right but you see i really don't like tracing around it i want a bounding box around it so instead of drawing an outline around it of the perimeter i want to create a bounding box and so if i am going to create a bounding box i'm going to come back over here i should get rid of this yeah that will give you more things to see here that are useful right so i'm going to comment this out and then i'm going to come down here and what am i going to do i'm going to find the box that contains the contour and i do that by using the command cv to and i'll come i've got to put the put this equal to something but i'll just show you the command it's cv to dot down bound being rectangle like that i want to find the bounding rectangle of what contour singular and you got to keep track bounding rectangle wants one contour okay uh that other one which was what that other one uh the draw contours it wants the whole array of contours so you got to keep track what wants a single contour versus what wants the whole package of contours well bounding rectangle you just pass it a contour and then it will return to you it will return to you uh the information you need to draw a box around it and so what does it return it returns four numbers the x coordinate and the y coordinate of the upper left corner of the bounding rectangle and then the width of the bounding rectangle and the height of the bounding rectangle so if it is sending us four parameters we have to catch what four parameters so i'm going to catch it in x in y and then comes with and then comes height okay so now i've got a variable for each one of those parameters that it's gonna it's gonna throw back to me okay so now i know what i need to draw a box around my contour and how would i do that with our old friend cv2 rectangle you should already know this and i want to put it on what i want to put it on the frame because that's what i am working on and then i've got to tell it in a tuple the coordinate of the upper left corner well that i already have it is x comma y make sure that you're putting it inside those parentheses as a tuple and then i need the bottom right the upper left and the bottom right well what would the bottom right corner be well it would be x plus the width and then y plus the height like that and now that should box up that object of interest and then i commented out my draw contour so this one should just be a box but i need a little bit more information here because in fact it is going to want to color let's make it a red so it's blue green red and it's in a tuple so the first one blue is zero the second one green is zero and the third one red is full strength 255 like that okay i think this is going to run okay good news is it didn't crash now let's look boom look at that huh object of interest tracking with the box what i don't like there is i did not give it a i didn't give it a heaviness and i want a bigger box so i should just put comma three i think it used the default of one and so this should be a bigger a bigger or a heavier box there you go look at that you see i am tracking that object shazam okay that is pretty cool okay so you see i think that is really neat and so now we can actually kind of formally track an object and we're tracking it based on color and we train that color using track bar so this is really starting to kind of look like artificial intelligence right because we train it on a color and then we track an object based on the color then there's all kinds of types of interesting and useful things that you can do so real quick there's three things i need to do i need to give you the homework assignment for next week i need to look i need to give you the solution to last week's person of mystery contest and then i need to give you a new personal mystery contest for next week okay so let's see if we can get through this what i want you to do for next week is i actually want you to do something with the with the data that you're getting from the tracking right it's just don't just draw it actually interact with your computer you can use this to actually input to your computer your computer can get input based on what it sees happening in the real world and so what i will do is rather than try to tell you in words what i will do is i will run the program and then you can see exactly what i want you to do so give me just a second here i believe that will do it [Music] okay so you see me you see a mask down below me right okay let's see if i can do this here we go okay i need to tune it a little better let me tune it a little better okay that looks good okay so now the object of interest is in the center of the frame and the frame is in the center of my screen as i move it this way the frame moves that way as i move it this way the frame moves this way let's see here let me make sure that you guys can see that okay yeah you can see that okay so i can put it up in this corner i can put it down in this corner i can put it up the funny thing is it's mirror image from what what i'm looking at so everything's going the backwards direction okay but you see if i put it in the center it's in the center of the screen and wherever i move it it puts the frame in that area i think that is just pretty darn neat i think that is just pretty darn neat because you're actually now your computer is doing something based on what your program sees you could use this to like adjust the volume or you could adjust the color balance or you could do different things that if you combined it with like mouse clicks or other things you could sort of start developing a gesture-based input system for your computer that could do all types of different things and so i just did this assignment just so that you could kind of get the concept of i can get data from the screen and then my program or my computer can do things based on that data okay i am making this video before i have released the uh the the video that had the uh personal mystery contest so i don't know if any of you guys got this or not but the person mystery last week was who was this and this is in fact william shockley and he is quite famous because he invented the transistor he was like one of three people but really he was the one uh he he was the primary person behind inventing it and remember the whole idea behind a digital computer is everything is zeros and ones and all math can be put in terms of zeros and ones and so whether you're talking about data or calculation everything is zeros and ones and the way you could make a computer is all you need is a switch and if the switch is on you count it as a one if the switch is off you count it as a zero so if you want to make a computer you just have to have a whole bunch of switches well back in the day they made those switches with vacuum tubes and vacuum tubes were very big they were very expensive they used a huge amount of power and they broke down all the time they would burn out all the time those things use so much power one of those vacuum tubes you could actually bring a piece of bread in and you could make toast over that vacuum tube vacuum tube would be about this big so computers didn't have much computation power because you couldn't have very many switches used a whole lot of power and broke down all the time well what mr shockley dr shockley figured out was he came up with a way to make a switch in silicon and uh it was very small it was very low cost it was very low power and it would basically run forever without without breaking down and with that then you could make the transistor smaller and smaller and smaller you could pack a bazillion of them onto a chip and that is sort of what enabled the computer revolution sort of the silicon revolution was enabled when it was determined by mr shockley that you could build a transistor a switch out of silicon so he is probably kind of like of of all the people that have lived on you know all the all the living people inventors he probably has more profoundly impacted the world than just about uh than just about anyone else and so that is the little story of the transistor that is shockley okay so now we're gonna go to the person of mystery for next week who is going to be this guy who is he and why is he important okay so guys i hope you're having as much fun taking these lessons as i am making them also on the homework i love it when you do the homework and then you do a screen capture and upload it to youtube and then down in my comments below provide a link over to your homework solution i go over there and i watch your solutions i really enjoy watching your solutions then when you upload it to youtube make sure in the description to put a link back to this video so anybody that watches it will have context as to what you're trying to do okay guys if you like this video give us a thumbs up be sure to tune in later this week for the solution to this homework assignment if you have not already subscribed to the channel and i just really i'm really having a lot of fun with this hope you guys are enjoying it too paul mcquarter with toptechboy.com i will talk to you guys later [Music]

Channel: Paul McWhorter

Views: 7,176

Keywords:

Id: T98-5YaB3qY

Length: 36min 8sec (2168 seconds)

Published: Mon Oct 04 2021