AI on the Jetson Nano LESSON 17: Detecting and Processing Mouse Click Events

hello guys this is pulmicort from top tech boy calm and we are here today with lesson number 17 in our incredible new tutorial series where you are learning artificial intelligence on the Jetson Nano I will need you to pour yourself a nice enormous mug of iced coffee and I will need you guys to get out your Jetson nano gear hey want to take a second and thank you guys that are helping me out over on patreon you're a huge encouragement to me you guys that are not helping yet look down in the description there's a link to my patreon account think about hopping over there and hooking a brother up but enough of this shameless self-promotion let's jump in and start talking about what we are going to learn today what we're gonna learn is we're gonna learn how to process mouse clicks or Mouse events in OpenCV now as we really begin to go in and develop artificial intelligence algorithms we're going to need to be able to interact with the user and we're gonna need to be able to interact with the window that we are working in and one of the best ways to do that is using the mouse so we need to know where the mouse is and we need to know whether it has been clicked or not and that's what we are going to learn today so I will get out of your way I'll need you to come over to your Jetson Nano and open up a visual studio okay you remember that we are working in a folder called PI Pro which is on our desktop and then we are working in a folder called OpenCV so i need you to click on the opencv folder to select it then come up and select at your main folder this little page with the plus sign on it to create a new file and then we are going to call this open CV I'm going to call it 6 and then - I'm gonna say mouse click and then don't forget the dot PI the dot PI tells Visual Studio that it's a Python program kind of important click inner boom we now have the program environment up and we've got a blank program ready to write now what we do is we start at that base program that we're using on all of these lessons and that base program fires up our camera so let's go ahead and get that you might have it saved over here somewhere but in case you don't we will come to a Chrome browser we will go to the most excellent wwth boy calm calm all right and then you want to search on probably the simplest thing to search on would be PI camera and if you search the site on PI camera you see this post here you can click on it and you'll get that code that's the starting point for firing up the camera and rather than retyping this we will just come in and click on those two little pages like that that selects the program right mouse click and copy we now have it so we can come over and close this and now come back to visual studio and paste it in there all right couple of little bookkeeping things that we need to do that since I am running the Raspberry Pi camera i will uncomment out this which sets the parameters in this which fires the camera if you are using a webcam you would comment out instead this line and then you would either put a 0 or a 1 here if 0 doesn't work put a 1 but one of those two should work for a webcam so now we should be ready to run the camera there is one other little thing that I really like I like my window to be in a certain position so I am going to come over here and after I show the window I am going to do a CV 2 dot move window I hope that it recognizes this yeah move window okay click enter and then it is going to need some parameters what window do I want to move I want to move a nano cam and where do I want to move it to zero comma zero and that should put it up in the corner so we keep deliberate control of where our windows are let's go ahead and see if this works just to make sure everything is copasetic before we move on right mouse click run it pops up and there it is where we told it to be everything's good cue to quit on this particular program okay now we are ready or I hope you're ready to start processing a mouse click ok so this is a little involved so I'm going to kind of be very deliberate in explaining it so you understand what it is that is going on okay so we are going to come over and come up to the top of the program and sort of after this flip equal to business we need to set up a mouse callback and that's basically you set up a listener to listen for something happening with the mouse and if something happens with the mouse then it calls a program okay but you've got to tell it where to listen well what do we want to listen our window is called nano cam but at this point in the program Python Python doesn't know anything about that window called nano cam so we got to do a little command here which is C v2 dot named window and then the named window is single quote nano cam all right now what you got to see is this is the same window that we use down here but we just got to kind of define it before we reference it and we're not in the while loop yet so we got to kind of define it here in order to be able to do the most excellent see v2 dot set mouse call back okay and so this is going to be our listener listening for activity on the mouse and there's some parameters that we have to have here okay it's pretty simple but the parameter that we have to have is what window are we going to be listening in you guessed it Nano cam since we're referencing it here that's why we had to kind of define it in the line of code before and then it's relatively simple we just have to now tell it that you're listening for a mouse event if there's any Mouse event what function do you want to run well we have to define the function but I'm just going to call the function click CL i ck so when this command is run a listener is set up and any time a mouse event is detected what does it do it calls a special function called click okay does that make sense well if it calls click we darn well better define what click is okay so I will come up here and then what I will need to do is I will need to define that function and I think I will just do it right up here at the very top okay we will do it well we got to import CVT with cv2 first of course okay we've got to import c v2 of course and let me come over here okay just got to get a little bookkeeping kitten taken care of here okay so we're going to import c v2 and now what we are going to do is we are going to define a defined EEF define a function what is that function click write because that's what we said it was going to be now because this is a callback function these parameters have to match what c v2 c v2 is expecting so the first is event that is which mouse event occurred right it could be a left button click right button click scroll move you know it could be a lot of different things but it is an event and then where was the x position of the mouse when that occurred and where is the Y position of the mouse when that occurred and that's the X and the y in the window which window nano cam so remember nano cam starts at zero zero and then goes to 640 480 so it's kind of that parameter space that that XY is in all right now there's a couple of parameters we're not going to use but they've got to be there and that is flags don't worry about it and params don't worry about it now a function starts kind of like a for loop or an if statement you put a colon there now you get the indent in anything that is indented is indeed going to be inside of that inside of that function okay so now we need to just kind of see if we can process the event okay so what event are we interested in let's say that it is a left button click so I'm going to say if event equal equal C v2 dot what evie ENT underscore left button down okay so you see the left button down so now if anything happens with the mouse you're going to end up in this function okay but once you're in this function if the thing that happened with the mouse was a left mouse click now I'm going to be inside this if statement if I put my colon there okay so I put my colon now if you're in here what do we want to do well let's just kind of see if this works if this whole structure works before we go too much further so I'm just going to say print and what am I going to print I'm gonna print the the mouse event laws and then I will print what the event so we'll see what the mouse event was and then I am going to print what the X and then I will print the string comma and then I will print why so this if this is working right we're gonna process our first Mouse event and if it's a met left mouse click it should print out what the event was and where it was are you ready please hold your breath okay let's see we are going to run Python file okay and nothing happened what did I do wrong here let me try again right mouse click run Python file in terminal boom okay now I need you to be looking down here in where the printout is occurring I'm going to come over with the mouse I'm going to do a left mouse click in boom the mouse event was won and it was at the x position 160 in the Y position 130 if I come up here in the corner it should be a lot closer to 0 0 yes it 1 2 down here should be about 640 480 yes ok now what does this Mouse event was 1 well what you can see is the parameter event underscore L button down is just a constant and it's the constant one so I could have just said if event equal equal 1 if you remember if you memorize taught all your mouse events were but it's easier to use the names but you can see we just processed our first mouse event and we are just a few minutes into this program so that is really really cool so this is the problem though you really don't want to be looking down here you really want to be interacting with your image now what does this freeze up as I'm using my mouse here do you see how this freeze up because it's actually going into that function because moving the mouse is a mouse event but it's not a click so it sort of freezes up because it's staying inside of that function just so that you know that it's trying to process a mouse event okay now what you might think you would really like to interact with your image and so what you would really like is maybe you would like to put a little circle here where you click so you could add a circle where you click to your image and that would be kind of a neat thing to do so you might think that you would come over here and that you would do one of those see v2 dot circle commands this is the problem that I want you to think about the program flow the main action in the program is cut happening in this while loop you grab a frame you show a frame you grab a frame you show a frame and then if there's a mouse click you run up here you do this and then you come back grab a frame show frame grab a frame show frame so if I put a circle at this point in this if statement what is the problem immediately you would chop back here grab the next frame show the next frame you would not even see the circle being shown so what I need to do is I need to I need to draw that circle here all right but here's the problem okay here is the problem these variables inside a function in Python are local variables so the function click knows what x and y in event is but down here in the main program in the while loop it doesn't know those variables all right so what I need to do is I've got to get these variables out of the function and into the main program and I do that by making the variables the variables that I'm going to use I've got to make them global so I'm going to create a global variable and I'm gonna call it pnt ok and that is gonna be a point like X Y ok what is that P in T okay and then global I'm gonna have a variable in that global variable is going to be EVT which is going to end up being the event now I need to come down here and I've created the variables now if I come down here what I need to do is I need to set them P and T is going to be equal to the tuple X comma Y so I just took those kind of secret variables and why and I just exposed them to the whole world or at least the whole the whole Python program alright and then I also need to set EVT equal to I'm just going to set it equal to event okay or I could set it one because if you're in here it's going to be one but let's set it to EVT so I've made those two variables and global variables now there's still one little problem you have to look down here and if I have made the if I made the statement if when do I want to draw the circle will if EVT equal equal equal what one alright now this is the kind of quirky thing that you've got to see though if I've never had a mouse event it is not going to know what EVT is and the program is going to crash okay so I've got to come up here and I've got to define EVT out here and I'm just going to set it to minus 1 why because minus 1 is not 1 and that means before there is a mouse click I will never get inside this if statement okay make sure you understand this okay make sure you understand this all right now if there is a mouse event and if it is a less left button click then EVT is going to be 1 and then what am I going to want to do well I am going to want to just start by just simply drawing a circle so let's just come in here and let's say C v2 dot Circle Circle C V T dot circle and then where do I want to put it I want to put it on frame because that is the image we read it in as frame and then where do I want to put it I want to put it at point point is that tuple that now the world knows which is X comma y which is where the mouse was when it was clicked okay and then let's have a radius of five okay and then let's have the color 0 for blue zero for green and 255 for red so this is going to be a red five pixels wide and then we will want it to be solid so we'll put it at minus one right a couple of lessons ago we went over all this so you should know this okay so now let's right mouse click and let's run this puppy ah is it running what's wrong maybe I didn't quit it before so let me q4 quit okay there it is that's the new one all right so now the moment of truth I am going to left button down oh oh wait a minute I want it somewhere else that's kind of creepy I don't like it on my eye or let's say I wanted to mark the position of the microphone I could mark it right and also if you look down here what you can see is you can actually see where that microphone is right it's at 4:42 85 and you put a dot on it all right so you can you guys see that let me move that up a little bit to make sure that you can see that you see how we're printing out where that mouse event is down here okay so let's do another click so you see how it's updating that X&Y position all right I think that's pretty cool but again I really don't want to have to be looking down in my Python terminal in my console to see what where that is really where do we want that we want it print it out beside the dots so how would we do that well we would quit this and we would come back in here again we've got to do it here so that every time through it will show and so what I am going to need to do is I am going to need to do a put text but I have to first define a font and my font is going to be C v2 dot we'll call it font underscore Hershey and then what I want is plane so I will just come down there and select it so now I have a font okay and now I am going to need to create a string right I want to print the X comma Y I want to print point but the problem is point P NT is a tuple and it's kind of like a tuple of an integer comma integer and this put text doesn't want an integer it wants a string so what am I going to have to do I'm going to have to create a my string which is my STR my string and that is going to be this STR the string value of what pnt so it's going to take that tuple and it's gonna make it a string I think that will work quite nicely and now I need to do a what CV to dot put text and I got a sake where am I gonna put it I'm gonna put it on the frame okay and then I am going to need to put what my I need to close this over here so it's out of your way okay my st AR so that's what I'm going to print is my string and then where do I want to put it pnt the tuple PT that says where we're going to put it okay got to tell it what font well I just set it up font and then how big do I want it to be I'm going to make it a size one that's pretty small and then again I think I will make it red now I'm going to make it blue the dots going to be red in the font or the text is going to be blue so I will put 255 comma zero comma zero sort of as a tuple I forgot my comma okay and now I will put it a weight of two all right I think that should put it that should put the location on the frame so now we're gonna run okay hold your breath left mouse click boom that was at 107 128 in that matches what is going on down here and I do need indeed move this window management is the big challenges we're doing this kind of stuff okay I think that's gonna work and now I need to peek over here to get this and then I'm gonna quit all right so now I should have everything ready to go let's try running it again right mouse click run Python file in terminal okay so I click here and wherever I click it puts a red dot and it shows me what the it shows me the font that is a little bit light that's a little bit small I'm gonna do an experiment and I really shouldn't do this but I wonder if it will take float point values for the font size so let's come back over here and I do need to kill it I think okay quit right mouse click run Python file in terminal all right hey that works I think that looks good I can see it boom that is great that is really working and that is working well okay now the thing that I want you to think about though is that's one of the things that you might want but what if like I wanted to lift up my cell phone and I wanted to see the position of the four corners okay what if I wanted to see the position of the four corners in that case I need to not just move the dot around I need to keep the old ones okay I need to keep the old ones now how are we gonna do that you got to think about this a little while this is a little trickier than what you think because I can't just do another one because it only knows one XY coordinate point okay so wherever a point is it'll put it there but that last one I put that's lost because I read a new frame I show a new frame whatever is not between those two things as so how would I do that hmm I need to create an array all right so we're gonna come back to the top of the program okay we're going to come back to the top of the program and we are going to create an array all right so we are going to create an array in where would I do that well here I made this EVT up here okay and so what I need to also do I need to mmm I need to create an array of pnts okay an array of tuples and I will call that array cord for coordinate and I'm going to create it as just an empty a right now this is the crazy thing in in Python with OpenCV the variables inside the function are local unless you make them global but it seems like the the variables are local but it seems like the arrays are global and so I don't need to come down now and make coordinate global because it just seems like it's born global at least that's the way that it seems to me okay so we're gonna come down here and we're gonna do these same things okay but what we need to do now is we need to put we create pnt that's the XY value of the last mouth click but we need to now do what with it we need to add it to this array coordinate so what are we going to do we are going to append to that array what are we going to append we are going to append P in T alright so now every time you lap left mouse click that array is going to get larger does that make sense hey I wonder maybe instead of printing out pnt just for fun we should print okay we should print see Ord and that way as we click we should see the list getting longer and longer and longer right that might be kind of cool and so now we will be keeping track of that okay at this point the problem with this if event is equal equal one the problem with that is that would just still print one thing one circle so we need to print lots of circles how many however many times we click the mouse so you can't just do print circle print circle print circle you've got to do it a specific number of times we would do that with a for loop okay and so what I am going to do here is I am going to create a for loop alright so think about that for just a little bit okay so instead of doing all of this nonsense in an if statement what I am going to do is eliminate that tab over and I'm gonna say 4 okay now we want to do all the points pnts I know this is not that original point that original point is PMT this is a new variable but for points P and TS which is going to be a tuple for points in the list what is the list I want to step through see Ord okay now if there's no values in CEO Ord you won't do anything in here so you don't need the if statement you only get in the for loop if there's something to do so the variable tuple variable points is going to step through the cord list the cord array does that make sense and then what am I going to want to do each time through I am going to want to see v2 dot I'm gonna want to see v2 dot a circle and then where do I want the circle frame okay and then now where do I want it pnts okay because this pnts will always have the tuple for that position in the list so the first won't be the first tuple then the next then the next then the next this is that's new variable P and T s that is stepping through the array coordinate does that make sense I hope it does okay and then how big do I want it to be five and then what color I want zero comma zero comma 255 all right and then I want comma minus one for a solid circle so I think what this is going to do now this isn't going to put the text on it yet but let's just see if we can add a lot of dots okay so I will come in run Python file in terminal I get this holding our breath click and I got an error pienty is not defined let's look up there see what happened so I appended PMT ah I see I must have erased that where I define what P and T is P and T is equal to X comma Y that's a rookie mistake I hope you guys found that now I am appending the pnt let's try it right mouse click run Python file in terminal okay we're gonna do a right mouse click or a left mouse click I'm sorry are you ready boom moment of truth boom moment of truth look at that all right our little scheme worked our little scheme worked okay so we can just go crazy right also do you see down here how the list is getting longer and longer that array is getting longer and longer as I add dots to it now do you see what our problem is like do I really want to have to kill the program to get rid of that mess I just created not really so let's try to do a little bookkeeping where we can go in and clear it if we want so we're gonna say Q now I want you to come down here and think about this this C v2 dot wait key here it's looking to see if a key is pressed well if the key that is pressed is Q okay what are you going to do you're gonna break okay but what we want we want to look for a different key if C v2 dot wait mmm let's see is this I'm gonna do it a little different because I don't want to keep making it wait so what I'm going to do is I'm going to come here and I'm going to say key event is equal to C v2 dot wait key one like that so I just I'm gonna do the read and now on the if statement so I'm only going to look for that event one time and then I'm gonna say if key key event equal equal Q I'm gonna break out of it right or if key event in that had better be a equal equal so our D the value of now I'm not looking for a Q but I'm looking for a lot a c c is for clear so I'm gonna put a C and if the mount if the button or the key that is pushed is C what do I want to do how do I clear out all of that mess that we created well all that you've got to see here is is that it's gonna erase itself you just don't want to put it back in there so what is it that you really want to clear coordinate is equal to them it gets rid of all the values out of coordinate okay now the other thing that you got to see though is that would be a problem that if you hit C and cleared out all of those values I think that would take care of it I really think that would take care of it so if you see it's gonna clear it won't go back in that for loop I think this might work so let's just take a quick look at this right mouse click run Python file in terminal all right so let's see all right moment of truth press C boom they went away okay so you see if I sit here and I make a mess it's like the measles or the chickenpox would that be measles or chicken pox you know I had both measles and chickenpox and I lived through it I lived through it I had measles and chickenpox I understand it's like they act like these things are deadly diseases nowadays okay so now look at that now I put a C and it clears it well that is really great okay so that's looking really good we need to do a little bit more now what we would really want though remember we would want to like pick up the phone and then show the phone okay and then like put the coordinates around so we really want to print the coordinates of where things are and so we need to do a little bit more I will now that I've cleared it I will cue out of this cue out of this program so up here and in this we need to besides putting a dot we need to go back and put that that font back in so we will say font is equal to C v2 dot F Ont and then we're gonna do Hershey plane you can play around with these different fonts and see what you really like I kind of like a simple one and then what I need to do is I need to create my string again in my string is going to be the string okay not a P and T not a P and T but what has the right value right now of that X and why it's pnts so I want the string value of pnts all right and now that I have that I need to do a cv2 dot c v2 dot put put text so I'm going to put the text back on there where am I going to put it on frame okay what am I going to put my string where am I going to put it at the tuple pnts which should be updated every time through the list because it is the index on the for loop and then what do I want I need to put the font which I just defined that should be good we decided 1.5 was pretty good and then I want the font to be blue so I will put 255 comma zero comma zero I'm gonna get further out of your way here to 5500 and then what do I want I want a heaviness of about two that kind of makes it a heavier line could it really be that simple could it really be that simple did I close all my parentheses out let's try it run Python file in terminal all right moment of truth here is my phone okay and then I'm going to come in and corner Corner corner corner look at that and I've got those coordinates you can see now I can hit clicks clear and it all clears up you can see that I need to probably adjust things a little bit like the font is a little big - you know it's kind of running into each other when I'm doing something like that but that is pretty cool let's look up here you can see yeah that's close to zero zero you can see that that's close to the other corner so this is just pretty darn amazing so I'm gonna say c4 clear all right guys you see what power this is is like if I wanted to do some image manipulation I might need to know where this button is I could come and click and I could find the value of where that button is and then I could go in and do my coding knowing where that little button is or knowing where that button is okay or I could like find all the buttons and identify all the buttons and it would show where it is all right now there is one more thing that we're going to do that would be kind of useful and that is I don't just need to know where it is like what if I was doing something to recognize like I'm trying to recognize an object and I'm going to recognize it on color like this little red thing here I might need to know what are the parameters what is the RGB values of that so I would want to be able to come up and click on it and get not just its position but get its RGB values all right and so let me show you how to do that we're going to quit out of this and we're gonna come back over here alright so we're gonna kind of now how many Mouse events are we going to be looking for it now instead of one mouse event we are going to be looking for two mouse events we still use that same listener that we already created okay we still use that same function right we still have this this set mouse callback that's our listener and it still calls click but now we're going to look for a left mouse click is going to do the position in the dots as we've already done but the right mouse button we want to grab what the RGB values are okay so here what I will do is I will put another if statement a new if statement if event equal equal what am I looking for see v2 dot event and this time I will be looking for our butt and down our button down okay now if our button is down what I'm gonna do is I'm gonna see where it was so I'm just gonna do a print and I'll print the X comma Y again this will just put it down in the little console in case that is useful all right now I need to get the blue value of that pixel all right so I'm gonna get the blue value and it is going to be equal to I'm getting the volute blue value of which pixel that pixel is in what image it is in image frame and what you got to see is guys you got to see frame is not a picture really frame is just a huge matrix it's a lot of rows and a lot of columns but it's a two dimensional matrix where you have rows and you have columns and that x and y is telling you what row you're on and what column you're on it is telling you your row and your column okay so frame blue just the simple number blue is going to look inside that array frame okay and which pixel does it want to look at it wants to look at the row in the column now what you gotta see though is this is the confusing part row is which variable row is why okay and column is X yeah I just got it I don't want to slow down too much but I just gotta explain this because this is really kind of confusing when we're dealing with a coordinate axis we always think of X comma Y right we think of X comma y in a coordinate system so the first number is X and the second number is why but when we're dealing with a matrice a matrix you always think of row column row comes first are by sea row column well in this crazy world the row is the which value the Y value and that's why we put Y first so this is saying take the array frame and go X rose down and go why columns over and that is one pixel and then you look at that pixel and at that pixel there is a tuple that tuple has got B comma G comma R so I've got one pixel and that pixel is not a number it's a tuple and that tuple has B G R so if I want the B value which position you say the first position but remember in programming you start counting at 0 so the first number is the 0th number and now I have blue okay now I want green and so that's going to be equal to frame frame all right in same thing Y comma X comma what it's the next one over which is the one which is the second number in the tuple and then you guessed it red is equal to frame that array what Y comma X comma 2 guys you really need to understand this a picture a frame is a matrix row and column ok rows and columns it's a two-dimensional matrix rows and columns at each position is a tuple the tuple has three numbers G or B comma G comma R and then I can pull those out so now I know the color of that pixel where I clicked and for right now just for fun let's just print it so I'm going to say print and then what am I going to print I'm going to print blue comma green comma red and then I'm going to take this other huge print statement out down here so it doesn't mess me up remember how we were printing coordinate here somewhere maybe it was up here yeah printed the whole array I don't want to mess things up with that alright alright so what I'm gonna do here is I'm just going to be priming blue green red just to see if I'm grabbing the pixel okay so let's see what happens here so I'm gonna right mouse click run Python file in terminal alright the a moment of truth okay so now let's just make sure our other programs still working so you see I can uh yeah I can still do that right I can still get those but now I'm going to right mouse click and if I right mouse click here which number would I expect to be the biggest I would expect the middle number to be the biggest because it's blue green red and this is kind of green and look at that okay do you see down here it picked off it picked off let's see if I can highlight that blue is 73 green is 118 and red is 0 and so in fact that really looks like green okay that looks like green now let's see if I can do this let's see if I can grab this red here and remember I'm right mouse clicking look at that red is the biggest I wish I had blue oh I do have blue look I can click on the 4 because that's really on the frame and I'm gonna right mouse click and look at that 255 255 0 so you see I am reading the color of an individual pixel and if you like use Photoshop this is a lot like the eyedropper tool right it's like the eyedropper you're going out in your grabbing in real time live the color of a pixel okay but I want something a little more exciting than that okay I want something a little more exciting than that so what I think I am going to do is I'm going to create a new window we haven't done that before we're going to create just a new window and I'm going to call that or a new image ok new frame but instead of reading the frame from the camera I'm just going to create the frame and I'm going to do that by saying and you know what darn it to do this I'm we're gonna have to go back to our old friend numpy so I'm gonna say import numpy as NP okay then that will allow me to create a matrix a matrix that is two-dimensional by doing image is equal to MP dot zeroes that's the function MP zeros and now I've got to give it some sizes so how big do I want this image to be well I want it to be 250 rows by 250 columns and then how many numbers do I need at each pixel 3 so I would need a pic I need a tuple with three numbers in it okay now actually I need another set of parentheses around this so we go here and here okay so now what is this thing it's saying numpy go out and create a matrix that I'm gonna call image and I want it 250 rows by 250 columns in each position have a tuple with three numbers in it that is what I just did there now I've got to kind of tell it the format for the numbers don't worry about this just do it it's in P dot u you int be careful you NT eight and that's just giving it the format okay you know we could do a whole lesson on this but sometimes just do it okay so that's going to create a new window or that's going to create a new array in that new array is going to be called IMG now we want to do something down in this if statement with the IMG now IMG isn't going to update every time in the while loop so we can actually make the action happen here because it's only going to change when there is a mouse event but what do we want it to do okay well we printed it already but instead of just printing it I want to create a string okay in that string is going to be what will we call that string how about call it color string and what what do we want color string to be we want color string to be equal to the string value of blue remember that is an integer that we read from the frame array okay and then I want to what do I want to do I want to add a comma so I will add append and then the string comma and then I want to add the string of green so I'm turning green to a string and adding it to my string and then I want to add another comma which is a string and then I want to add string of red so now that should just create a string that is just kind of blue comma green comma red because if I'm going to display it I need that okay and I believe that will work if I didn't make a mistake you guys are probably yelling at me if I made a mistake there but I think that will work I now have color strength now what I want is I want that box IMG that IMG was just an empty array so it would probably if we tried to show that IMG matrix it would probably just show black because we didn't put any numbers in there so we need to actually put some numbers in there so I'm going to say IMG and this is a matrix so I can address it like this but what I'm going to do is I'm just going to put the colon and that just saying take the whole enchilada take the whole matrix image from the first row for column or the zeroth row 0 column all the way to the in row in column take all of those points that's what that cult colon means it takes everything and then what do I want to set it to well I want to set it to the color what blue green red and so now whatever that pixel was that I grabbed its gonna make the whole box the whole frame the whole window that we're working in that color uh-huh okay so now that is going to be blue green red and now it is that color so what do we want to do now let's thing okay well now I need to do a put text right I need to do a put text so I'm going to make a new P a new FNT I got to give it a font in that font is going to be C V 2 dot font underscore Hershey and we kind of like the plane right alright now I want this is the thing I want to print so that window that's going to pop up is going to be the color that I picked but I also want to print on there what color it is but if I grabbed a red pixel and it makes the Box red if the font color is red you're not going to see it we'll make it white well what if I grabbed white so I sort of want the color that I print the text end to be the opposite color of what I grabbed and so we're going to create a new color are our new R is going to be equal to 255 - red okay and then green is going to be equal to 255 - green and then B is going to be equal to 255 - blue Cece I'm creating a new RGB in the only thing that this RGB is going to be used for is the color of the text that I'm putting so I'm guaranteed that it will be kind of opposite of whatever the background is does that make sense does that make sense ask me if it doesn't make sense all right so now I'm going to create a two pole TP is going to be equal to what the comma G comma R so now that is the new tuple for the opposite color of the background and now I think I am ready to do my put text see v2 dot put text and that would not be good put text all right and now where do I want to put it on IMG and then what do I want to put there while color string that is what I am going to print and did I do that really color string yep that looks good and then where do I want to put it mmm it's gonna be it on a fixed value I'm gonna put it like at the point 10 comma 25 and then I want to put F I want to put we've got to come out of that tuple come out of the - hole comma and then I got to tell it the font which is FNT how big I'm gonna put it as one and then I need to put what is the thing I'm gonna put there or what is the color the color is going to be TP because I just set that up there right TP is going to be Beach comma G comma R and so this is that opposite color that I just created and then I'm going to put a weight of two man what is that how many million mistakes did I make here I don't know but we're gonna run this thing anyway and we're gonna see okay I almost ran into that okay see see see v2 dot put text image color string didn't put it it to the point 1025 F n t1 is how big into is how heavy so now let's right mouse click and let's run this thing and see what happens I'm kind of encouraged I'm kind of encouraged that it didn't crash okay let's make sure our old functionality still works our old functionality was putting some points on here that looks good are you ready to use the color picker alright I am going to now right mouse click scalar value for argument color is not numeric it's one of these I can tell you what it's going to be is I used a string when I should have used a variable or I used a variable where I should have used a string and so this is in line 29 and so this is right here okay maybe this red and green and blue okay we grab blue green and red from frame and it probably grabbed it as a string and so you know what I think we need to do I think we need to int these things end always man this is this is kind of strange because sometimes you get a number and it says 3 comma 5 or something like that but it's the string 3 comma 5 and in this case it went out and let's say it grabbed 3 but it was the character 3 or the carrot the string 2 5 7 not the number 2 5 7 so we're gonna have to float those things I mean int them I didn't mean to say float okay all right so now it's the number 255 minus the integer value of the string blue I think this should work all right so now we are going to right mouse click run Python file in terminal and I didn't quit it at the time before so we've got a little housekeeping to keep the track of here I will quit it'll pop back up once it quit okay now let's make sure we didn't break the other thing all right we're still getting our pixel locations we are going to do a color grab right mouse click okay all right what did we do we we went to all of this trouble to create to create IMG but what did we never do so I'm going to be a good boy and quit what did we not do we went to all of this work but what did we not do we never did a CV to dot I am show and what do I want to I am show my color which was our new which is going to be our new window and then what are we going to show IMG cuz we've done everything that we needed to to IMG and now we're going to show it let's see if this works right mouse click run Python file in terminal alright make sure our original thing works left mouse click left mouse click left mouse click left mouse click clear left mouse click left mouse click left mouse click left mouse click we are gonna try to color grab right here right mouse click hold your breath ah look at that it gives me the blue green red value uh-huh let me move this up where you can see it gives me the blue green red value and then it shows me right where I grabbed so let's see if I could grab this this is a little redder look at that look at that microphone color green screen background almost white okay you see the water you get let's all look at something really dark this should be a very low number low number low number yeah okay let's see if I can really get something here my phone turns on when I pick it up okay it's hard to get something really black wrong mouse click okay look at that that's very low number okay see for clear alright so we made ourselves a little color picker now let's say that we were trying you see we're starting to edge into getting ready to do some artificial intelligence what if I wanted to track an item on color so I could introduce into the scene an item of a specific color like let's say my key ring here and if I'm going to track my key ring I need to know what color it is so I could come over here and I could see what its position is and then I could right mouse click and I could look and see what color it is and then I could try to track on that color so you see this is very very very helpful okay guys this has been really I've had a lot of fun with this you guys this is an important one so you might have followed along with me and you might have gotten your program to work but you've really got to understand what we're doing and so if there is a program if there is a lesson that you need to go back now and kind of watch again and then start with a blank sheet of paper and see if you can do these things because this is really really important we've got to be able to interact with our screen we've got to be able to interact with our frame and with our window and we've got to be able to do it with the mouse and so this is a really really really important one it's one you need to be able to kind of understand and if you don't understand it do the lesson again try to do it on your own ask questions down below okay now what do we have coming up next the next thing we're going to do the other way that we can interact with things is to have slider bars so up next in the next lesson we're gonna put slider bars an issue slide the slider bar you can read what the slider bar is and so you can use the slider bar and you can use the mouse and then you become very interactive with your window with your frame and we will need that then in future lessons it so that's the next thing that we'll be doing and then after that lesson we'll really start getting in and start doing some you know some tracking of objects okay and so we're really we're starting to get into some real business here alright guys if you like this video think about giving me a thumbs up subscribe to the channel when you subscribe make sure you ring the bell so that you get notifications for the future lessons and as really really appreciate it when you help me out on patreon this equipment I'm always needing to update my software this this Wirecast software is really expensive and you know I've got issues with USB bandwidth it's just when you help me I can deliver a better product for you ok you guys leave your questions down below hope you enjoyed this lesson and I will look forward to seeing you next week Palmer quarter from top tech boy calm I will talk to you guys later

Channel: Paul McWhorter

Views: 11,156

Keywords: Jetson Nano, AI, Artificial Intelligence, Tutorial

Id: -ONaL_AYk6I

Length: 63min 27sec (3807 seconds)

Published: Sat Dec 21 2019