AI for Everyone LESSON 10: Processing Mouse Clicks and Events in OpenCV

hello guys this is paul mcquarter with toptechboy.com and we're here today with episode number 10 in our incredible new tutorial series where you're going to learn artificial intelligence or you're going to die trying what i'm going to need you to do is pour yourself a nice large glass of iced coffee that straight up black coffee poured over ice no sugar no sweeteners none needed and as you're pouring your coffee as always i want to give a shout out to you guys who are helping me out over at patreon it is your support and your encouragement that keeps this great content coming you guys that are not helping out yet look down in the description there is a link over to my patreon account think about hopping on over there and hooking a brother up but enough of this shameless self-promotion let's jump in and let's talk about what i am going to teach you today and what i'm going to teach you how to do is i'm going to teach you how to process mouse events or mouse clicks in opencv because as you're doing artificial intelligence and as you are interacting with an image from your camera you're going to need to be able to process a mouse click so that you can interact with your active frame so that is what i'm going to show you today and always as always i will be giving you a little bit of a homework at the end of the program so what i will need you to do is call up i will need you to call up your most excellent visual studio code and then we're going to come over here and we are going to start a new program and so i will come down and click and i will come up here and say create a new file and this is going to be open [Music] cv-11 i do i do believe dot p y and the dot p y is kind of important and boom now we have a fresh new python program just waiting to be written now we don't want to go in and just start from scratch every week so we're going to start with that core program that we always start with that basically sets up and launches our webcam and you can find that if you don't already have it you can find it at the most excellent www.toptechboy.com and you can use this happy little search bar here and you can search on the lesson which is called faster launch of webcam if you search on faster launch a webcam you will come to this and this has a little snippet of code you can hit i better get out of your way huh there's the happy little search box you are going to search on faster launch of webcam you'll come here click on these two little page icons it'll select all the code right mouse click and copy and you have snagged the code come back over here right mouse click and paste and now you have that core program that should launch the webcam just to make sure that the universe is in proper order and that we can in fact copy and paste without breaking something let's go ahead and run this let's see here come over here and run this takes a second okay boom there it is all right we can copy and paste and have things still work all right what you have to learn how to do today is you have to learn how to get input or understand that the user is trying to get the program to do something with a mouse click so you've got to sort of sit if you're the program sit and listen for a mouse click and then if you see a mouse click depending on what the mouse click is doing something interesting or relevant and so how do we go about doing that well what you will need to do is you will need to come over to your program here and you've got to kind of create a listener that listens for the mouse click and so uh i'm trying to think i think i will do that step right before the while loop so you come right before you go into the while loop and what we are going to do is we are going to do a cv 2 dot and then it's going to be set mouse call back like that make sure you look at the capitalization the capitalization is very important all right and now we have to tell it what window we have to tell it what window that it should be listening on because we could have more than one window open more than one video window open and we have to tell opencv which window should be you'd be listening for a mouse click on and so what i'm going to do is i'm going to do our main window which is my webcam right down here my webcam that will be the window that we will be looking for a mouse click over so my webcam all right now the other thing we're going to have to do is the other thing that we are going to have to do is we're going to have to define a function like when you hear a mouse click when there has been a mouse event where something happens with the mouse you have to have a function that you run to when that happens so really all set mouse click callback does is send the program to a function when something happens with the mouse and so what i'm going to do is i'm going to have a function and i am going to call that function mouse click so now we have a couple of problems right now one is we haven't defined the function mouse click we will do that and the other problem is is that when we do this command we have not yet created the window my webcam so if we just ran it now this program is going to crash for two reasons one is if you mouse clicked there's not a function mouse click to go to and the second thing is is that it's going to be looking for a window called my webcam and that window does not exist so let's go ahead and make that window just so that we don't have a crash when we try to run this and so what we are going to do is we are going to do the cv2 and we are going to do the command named name not two dots named window like that named window and then what do we want to call the window well we would just want to call it my web webcam all right and so what this is doing is this is creating the window up here that we are going to be using down here but what's the difference we created here and then when we use this command the set mouse callback it will not panic because it doesn't know what my webcam is because that window has been created in the line before i hope that makes sense all right now we've got to actually process the mouse click and therefore we need a function called mouse click so i'm going to come up here and i'm going to define that if you have been taking all my lessons you should understand functions so i'm going to define the function which function mouse click like that okay now when this set mouse callback calls the function mouse click it passes it some parameters so we have to capture those parameters between the parentheses in mouse click and one of the things that it is going to pass it is an event and that event is what has actually happened with the mouse maybe you press the left must mouse button down maybe you have let the left mouse button up maybe you have pressed the right mouse button down or let it up or maybe you have moved the wheel on the mouse there's all these different events on the mouse that can happen and each one of them has a number and what this set mouse callback does is it passes a number to the function mouse click to let mouse click know what the mouse is actually done if that makes sense and as we go on it should make more sense okay so it's going to pass it event that is what did the mouse actually do then it's also going to pass the position of where the cursor was when the mouse event occurred and that is going to be some x value and it's going to be some y value like how many pixels over in x and how many pixels down in y and so i'm just going to grab that as x position and i'm going to grab that as y position now these are numbers that are being passed to mouse click i could have called event anything that i wanted to xpos ypots those are just variables that i'm making up to catch that data that is coming from the set mouse callback call all right now there's two more parameters i care a lot about what the event is and i care a lot about x position in y position but it passes two more things that i'm not going to use but i have to have variables there to catch them and the first thing is just flags okay i'm not even going to talk about flags because we're not going to use it and then some parameters and so i just put in a variable for flag and a variable for parameters and that way the program doesn't crash but we're not going to be using those and so we are not going to worry about it okay now what we have to see is you are only going to end up in this function if somebody has done something with the mouse and so then i want to do certain things based on what was done with the mouse and so let's just see if we can pick up a click like the left button click let's see if i can just sense the left button click so let's say that if the left button is clicked then it's going to come up and you're going to be inside this function so i'm just going to do something just going to say if if event if event equal equal cv2 cv2 dot event and then which event do i want to look for l for left b u t t o n and then d d o w n so what that means is if the event was the left button was pressed down then you will be inside of this if statement well then what would i want to do inside of this if statement well i'm just going to print and i'm going to say mouse event was and it's going to print out a number but then we can see like what is the number associated with the left mouse click and so we're going to print event like that and then also let's print at position and then comma x position comma y position so this will give us the x and y value of where the mouse was when it was clicked okay does that sound good i think that sounds pretty good and then let's just do the same thing for the mouse up and if you think about it when i press the button down eventually i'm going to have to let it up and when i let it up that creates a new mouse event and so this then we could do that by coming down here and pasting and now we're just going to kind of edit this and i'm going to say if cv2 event left button instead of left button down l button up like that and then i'm going to say the mouse event was event and then where it was when that happened and so this should detect the left mounts button being pressed down or the left mouse button being allowed to come up okay i hope this actually runs let's give it a try and see what happens okay we've got a big window and i think you can see my cursor and what i need you to do is look down here right look down here in the bottom of my visual studio code under the 4.5.3 and it should print out when i actually do a mouse click so i'm going to press the mouse button down and it says mouse event 1 so left button down is 1 and it was at x position 389 389 pixels over from the left and 137 pixels down from the right well let me go way way way down here and now i'm going to let it up and then i get the second mouse event was that left button popping up and that was at exposition 1123 over and 691 down so you see that is pretty cool now let's come up here i'm going to come way way up more towards the upper left and click and you see that was at three pixels over and six pixels down so that you can see is very much up towards the top left of the picture and then 993 505 is more towards the bottom right this is very powerful guys do you see how now we can interact with the picture interact with our video by seeing what the mouse has done and where it was when that happened so let's just say one of the things that i might want to do one of the things that i might want to do is just put a circle put a circle on my video where i've clicked my mouse button put a circle where i've clicked my mouse button now what you might think about doing is you might think about right here doing like a doing like a circle a cv2 dot circle command right here but the problem is you've got to do it you've got to do it down here in your while loop because you can't do it just one time when there's a mouse of it because then that circle would only be there for one frame and then the next time through this while loop it's going to disappear so you have to do your circle down here in the while loop so every frame it grabs and every frame it shows your circle is going to be on there now what you might think about doing you might think about doing something like right here saying something like if event equal equal like what was the mouse down event was one if event equal equal one something like that and then come in here and do something here's the problem event is a local variable it is a variable associated with this function mouse click the rest of the program doesn't know that this variable event even exists it doesn't know that it exists and so if we just ran this program now you would get an error that it doesn't know what event is and so what we need up here is we need to create a variable inside this function that is a global variable that everybody all the other parts of python will know about it so what we're going to do is we are going to create a global variable and that global variable we'll just call evt for event if that makes sense and then what we can do in here is we can say we can say e v t is equal to event and so now we're setting the global variable to the local variable and then it should be known down here it should be known down here all right and so what we would do is we would now make this evt because that is the global variable so now what's going to happen what's going to happen if this cv2 set mouse callback is going to be sitting there and looking at this window my webcam and seeing if the mouse has any activity if there's any activity on the mouse it will call the function mouse click then up here it mouse click it's going to look it's going to look at the event what happened that made you jump over to this function there was some event well if that event was left button down then we are going to do this if that event is uh left button up we are going to do this okay but while we are in here we are going to set evt we are going to set ev2 to event and the same thing here we'll just do this for later get it ready evt here is equal to e v e and t like that event okay so when the mouse button goes down we know it when the mouse button comes up we know it okay now what do we want to do if evt is 1. now why did i set it to 1 because that 1 is the integer value that the callback passes when the left mouse button is pressed down and that's why we printed it so we know what it is so what are we going to do here if that happens then evt is going to be equal to 1 we'll end up here well then what do we want to do we want to put a circle cv2 dot circle and then what do we want well we want to put it where on frame that's our image and then where do we want to center it well we want to center it where we mouse clicked it and so we're just going to call that point pnt now what's the problem i don't know what point is what i know is x position and y position inside of this function so now i've got to create that point and so what i can do here is i can say pnt is equal to what it is equal to the tuple x position y position and so pnt is a tuple with two values the first value is x position the second value is y position it's like a little array you can think of it as now in order for the main program to know what pnt is in order for this program to know what pnt is what do i need to do well i need to make it global up here so now i'm going to say global and i am going to make pnt a global variable like that okay now if we come down here you can see that it's no longer giving you a little squiggly warning it actually knows what pnt is so now we've got to go ahead and finish this circle command so we are going to come in and we are going to put frame and then pnt and then it wants a radius well let's make it 25 pixels just for fun and then it wants a color let's make it blue so remember the colors are blue comma green color red and so we'll make blue 255. we'll make green zero we will make red zero and now we have our color in and then we want our thickness of our circle the line on our circle we'll make it a two okay and now that should make a circle when that left button goes down when that left button goes down it should put a circle right at that point on our picture if we are thinking about this right so let's run this thing i'm so excited denied denied okay so evt is not defined what is this nonsense ah okay so why did this crash because i set it as a global variable well think when i run the program the first time when i run the program the first time has the mouse been clicked no it hasn't so will i be inside this function no i won't and so i've never been inside this function so the variable evt has never been set because the mouse has never been clicked so we better seed it with a value here so we better just say evt equals 0 just so the program doesn't crash and so the first time through it's going to say is evt equal equal 1 well it's not but at least it's going to know what evt is and so it's not going to crash here i hope that makes sense let's see here kill that run it again okay now let's come up and let's come up here and let's click boom who's your huckleberry we get a circle until i let the button off and then the circle disappears so watch this i'm going to go i'm going to go mouse down and i get a circle and it stays there as long as the left button is down when the left button comes up the circle goes away now why is that why is that well let's come over and look the first time the first time we clicked the mouse we call the function mouse click when it comes up here what was the event well the event was left button down and because left button was down we snag x position in y position we put it in the variable point and this e v t this e v t is set to one okay and then we come down here and as long as the button is down that evt stays at one okay and as long as it's at one it keeps putting the circle on it now as soon as i let it up there is a new mouse event there is a new mouse event and so the s mouse callback calls mouse click a second time because there's been a second mouse event it comes up to mouse click and this time the event is l button up this time the uh event is left button up and now evt is going to be left button up which is the number four and therefore when we come down here evt is no longer one evt is no longer one okay it's no longer one and therefore on the next time through it's not going to put the circle on the frame it's not that it's erasing it it's just you're putting that circle on there every time through the while loop and if evt changes you don't go into this if statement you don't get your circle and so how could we do this well we could say if evt is equal equal 1 or if evt equal equal 4 because then when we let the mouse button up it's going to be 4 and then it's going to keep still keep putting the circle on there does that make sense let's try this [Music] okay now i'm going to left mouse button down we get the circle left mouse button up and the circle stays okay now if i press it over here what's going to happen that that one is going to go away because i have a new mouse click and i have a new x position and a new y position and it's going to show up over here now watch this down boom it's over there up it stays there all right pretty cool huh so wherever i go like that it's going to move it around what if i wanted to get rid of it all together what if i wanted to get rid of it all together well let's make a new let's let's look for a right mouse click so i could say if event equal equal cv 2 dot cv2 dot event and now it's going to be r r button let's do an r button up okay and so when i press down on the right button when i press down on the right button nothing's going to happen but when i let the right button up that's when the circle we want to disappear so i'm going to say right button up now when that happens then i'm just still go i'm not going to even have to set the i will though point is equal to x position y position just because we might want to use that later and now evt is equal to event now the problem is right now i don't know what the number is that goes with that right button up so i better print it just so i'll know what it is so i'll say print and i'll just say right butt ton up and what is that going to be well we're going to find out because we're going to print event like that okay and then we'll go in and we'll do what we want to do but we got to know what that number is first so i'm going to go like that okay there it is all right there's our circle okay there's our circle now what i want to look is down here in the print down here in the print area when i do a right mouse button down it doesn't print out anything but then right mouse button up you can see right button up is five and that's what i needed to know and so then what i can say here is i don't even think i need to do an if statement because once it comes in here it's going to set evt to 5 and because evt because evt is 5 it's not a 1 and it's not a 4 i will not come back in here so as soon as that right mouse button is released evt is going to become 5. because of that it won't end up in the if statement and therefore it won't put the circle on there remember you have to put the circle on every frame when you don't put the circle on again it doesn't show up so i think that is all we are going to have to do and so let's run this and see what happens okay so i come over here get my circle get my circle get my circle get my circle now i'm going to right mouse click down nothing happens the circle stays there because i'm looking for the up i release and the circle goes away boom okay so do you see how we can process mouse clicks and you can look you can play around these same things see what happens if you move the wheel see what happens left down left up right down right up do the different things with the mouse and then also you can see that it also is always wherever the the cursor was when that happened it's going to give you that x value and y value of those things so that is pretty pretty darn neat there i think that is just pretty darn neat okay so we have learned how to interact with our opencv video window using the mouse now we can let those mouse clicks then fire off and make other things happen that we would be interested in okay so this is your homework assignment i want you to create a region of interest using your mouse i want you to create a region of interest using your mouse so on your left mouse button that you will position your cursor when you press that key down that is going to be the corner of your region of interest then i want you to drag and then let the mouse up and that's going to be like the bottom right so the first click your click down is the upper left of your image and then over here is going to be the lower right and then when you let it up that will be the lower right corner so now you have defined a region of interest and then i want that region of interest to show up as a separate region of interest window over here so you kind of click down click up and then you grab that and you move it over here so now you've got a second window based on you've got a second window based on where you clicked so you're defining that region of interest with the mouse then what i want is is that if you right mouse click i want you to kill that window i want you to kill that window that you just generated all right so if you left mouse click down that's the start of the region of interest left mouse button up that's the end of the region of interest and then right mouse button down you kill that region of interest window does that make sense i hope that makes sense okay guys i hope you are having as much fun taking these lessons as i am making them and i need you to leave a comment down below if you were successful i need you to leave the comment i am legend if you are not able to do this homework on your own i want you to leave the comment i fold it up like a cheap walmart lawn chair and if you guys do fold up that's okay because you really learn when you really really really try to do something and you can't do it and you're just hopelessly stuck and then when you see me do it you'll you'll actually understand it and learn if you just watch me do it without trying to do it yourself you don't learn anything because it just seems easy oh yeah that's obvious i would have done it that way but you got to kind of struggle with it on your own before you watch me do it so i'll come in on the next lesson and i'll show you my solution to using these mouse clicks to generate a region of interest okay guys if you enjoyed this video give us a thumbs up be sure to leave a comment down below the thumbs up in the comment help me with the old youtube juice and help these videos be shown to more people and then also if you've not already be sure and subscribe to the channel maybe share these videos with someone else okay this is paul mcquarter with toptechboy.com i will talk to you guys later

Channel: Paul McWhorter

Views: 2,873

Keywords:

Id: HmuGQ-XFPgc

Length: 33min 43sec (2023 seconds)

Published: Tue Sep 07 2021