Unity + OpenCV Interactive Webcam Video Tutorial

how's it going everyone today we're going to be making this fun little uh unity plus opencv project uh so what we've got here is a live webcam feed and we're using opencv that's open computer vision that's a library we're going to use it to uh detect objects in the scene in real time and then use the uh unity physics to make things bounce off of it so here's the setup basically i've taken just a normal webcam uh and i've got it kind of suspended over just a blank whiteboard and it's that way the camera just has a blank white background to look at so here's a little detail of how i jerry rigged it i've literally just used some baling wire and stuck it to a clip light so to get going we're going to start a a new unity project so i'm over here in unity hubs i'm going to say new project i'm going to make this a urp that's the universal rendering pipeline and uh it's just nice for this sort of thing i'm going to put this here and we're going to create the project and then that's going to take forever all right so here we go we've got the basic urp sample scene in here and the first thing i want to do is bring in the the opencv library that we're going to use so to do that i'm going to head over to the unity asset store and search for opencv there's a couple options on this one um so there's this a paid version here and honestly i haven't tried it i'm using the opencv plus unity one it's free um it does say that it's no longer uh supported and i don't know if that means they're not doing development um any case it's using opencv 3 and for me it seems to be working just fine so i'm going to say open in unity that'll bring up the package manager over here and then the first time you do this it'll probably say download and then you can say import and this will this will take a second but uh we'll bring in the library and then this will be all the uh all the stuff that we need to actually use all the opencv fanciness okay so it finished uh importing and the first thing to note here is that there's some red text down here uh so that's that's an error message from the console and you can see here it's got all these error messages saying unsafe code may only appear if compiling with slash unsafe so what that means is um that the the opencv library uses what's called unsafe code which does sound a little bit terrifying don't win it when you get right down to it what it means is that it's it's using basically c plus libraries in the background to do unmanaged code which it kind of needs to do to be able to do the computer vision stuff uh quickly you know perform it so what we're going to do is go into project settings go to the player tab go into other settings and if we scroll down far enough we should come here where it says allow unsafe code and if we check that it'll act like it's not doing anything but here in a second it'll uh it'll pop up and start recompiling um so and now all our error messages are going so next we're just going to get a kind of a basic scene set up just so we have something to work with so i'm going to go up here to file and go to new scene we'll just do a basic built in scene all right and over here well actually so we're going to save it first just to get that out of the way say scenes and i just like to call my see main but call whatever you want all right so now we've got a scene we can uh right click here and we're gonna add a ui raw image okay so this this raw image this is the thing that we're going to actually be displaying the the video feed on and you can see it added a canvas a raw image and an event system this is using unity's uh uh ui uh stuff so it comes with a little bit extra things just to get it set up in a spot that we can work with it kind of nice i'm going to change the render mode over here into world space i'm going to zero out its position and set its uh width and height down to 10. and i do this just so that we get something that's a little bit more manageable uh in size all right so other than that our raw image here is going to come out huge so [Music] i'm just going to initially change its position back to zero and change its scale down to point zero one five and i just discovered these uh these values by playing with it um but this give us an image and we can put some crap on it all right so now to actually uh we need to do a little bit of coding here so let's look down in our project view here when we imported opencv uh it brought in this new folder and made some stuff but for our purpose here we're going to go over to the scripts folder right click go to create and make a new c sharp script and i'm going to call this thing uh contour uh finder i don't know it doesn't really matter i'm just going to call it something so once you do that unity will rebuild and i'm going to add add that to our raw image here right all right so i'm going to double click on that and we're going to bring up a visual studio or whatever code editor you happen to like okay so let's just get rid of this bit of code here so normally our scripts inherit from mono behavior but for this one we're going to inherit from web camera which uh if you hover over it you see we got this red squiggles and it says it doesn't know what the world we're talking about so if we right click on it i'm sorry yeah right click on it or hover over it sorry uh and then use this little drop down over here we can say using opencv sharp demo so basically this web camera class actually comes from the the demo code and if we hit f12 on that we can actually see it inherits from monobehavior so it's just adding some extra stuff for us to make this a little bit easier um so we can see that moves the red squiggly over here and it says contour finder does not implement inherited abstract member web camera process texture blah blah it just means that that that's an abstract class and it wants us to implement some stuff so again we can hover over this thing and say implement abstract class that gets rid of the error message and now we've got something to work with right okay so basically what it is is this web camera thing basically it'll take care of setting up the camera and it's just asking us to do the processing that we want to do so gonna delete that and we're gonna do just a real basic setup here just to get the camera going so the first thing we're gonna do is make a mat which is a material and again this is a get the right squiggly and says we can't be found so i'm gonna click on that and say using opencv so matt is kind of open cv's image texture a lot of stuff revolves around that like we're going to do pretty much all our operations on this uh this map right all right so first thing we gotta do is say image equals opencv sharp dot unity texture to matte and we're gonna use the input all right so we're passed in the input from the uh the web camera so that gives us a material that we can work with um so normally we're going to do cool processing stuff on it and then we're gonna output it so we've got an input and this uh this thing's responsibility is to get something into the output and return a boolean right so we're going to say if output is equal to null then we're going to do this we're going to set output as equal to that's a texture so basically what this does is this image that we we made up here it's going to convert it from a matte to a texture and texture being unity's uh version of a texture and we're just setting the output to that um so we're only doing this once though uh just when the output's null otherwise we want to we want to do this a little bit differently that same method the map to texture actually has um an overload a second one here where we can specify the texture and it'll just instead of making a new image or sorry a new texture a new object in memory it'll just take this stuff from the input image here and dump it into that this is kind of important from the standpoint that if you just do this every time it'll work like yeah you'll see the image on the screen everything will seem fine but the problem is you're allocating a new object in memory every frame uh and if you if you bring up your task manager and you look at your performance you'll see that your memory is slowly starting to creep up and until you run out um so with this it just means we initialize it once and then from that point on we take the image and just shove it into the already existing uh output okay cool wrong on so we're gonna save this oh yeah sorry one last thing is we have to return true and that's us that's just this functions way of telling the web camera we did indeed do something so um yeah you just you have to return true from this function to get it to update the image if you return false it'll just act like nothing happened and not update the image okay so we can save this we've got no more error messages and we've got some basic code that just takes the input image converts it to something opencv can work with and then outputs it back right so if we go back into our tutorial here we'll get our compiling thing and if we hit play hypothetically what we should see is um well sorry let me drag this game view down into this corner just i don't have to look at it um hypothetically what we should have seen is the webcam view here but again we've got some red error messages down here and you may or may not get this error message uh for me it says could not find specified video device uh so i can see that that's actually coming from that web camera class that we're using so i'm gonna double click it and that'll bring us into the code where it started having problems um to me it looks like uh this thing is trying to find a a front-facing camera which is all cool um but for me i it didn't work on my computer so i looked at this code a little bit and i just decided to set camera index to zero right here inside the uh the if statement basically what this will do is just say find the first uh webcam that's attached and we'll just use that okay so save that go back into unity and we'll see if that fixes our problem here oh let's see we do have an error message here it says variable surface of conduit finder has not been assigned yes all right so what that's complaining about is our contour finder uh it inherited the surface uh property and we need to set it and it's just asking where does it want to put the uh the image so we're going to take the raw image and drag that i'm sorry the raw image from the hierarchy and drag that into there and now now if we hit play we should see this let's see if that works all right so if we zoom way out uh we'll see that we've got uh we've got our image right that's our webcam feed so it's way too big though uh so i'm gonna take this and scale it down to something like this uh i just can't do 0.015 to be honest i thought i already did this but all right actually i'm going to exit play mode and then set the scale to 0.015 that way it saves it okay cool cool all right so here we are we have a basic scene so in the scene we just have just a raw image and it's taking in our webcam feed which is just cool so that's what we need to get to get ourselves to a point where we can actually do some of the fancy image processing all right so the next thing we want to do is we want to start processing this image right so i'm going to go back to visual studio and go to our contour finder and what we need to actually do here is we need to get ourselves to a point where we have a solid black and white image that we can use to kind of trace out uh the stuff that we want to interact in the scene so to do that we're going to add a little bit of code here and just to get ahead of the game i'm going to add some um variables that i know we're going to need so i'm going to add a flip mode and this is a this is an enumeration an enum that lets us now that's not a fun way to basically let us flip the image back and forth if we need it when we do our black wine image we're going to end up doing a threshold where we say everything below this value is black and everything above it is white but i'm setting this up as a serialized field so that we can edit it in the inspector and that will be useful later now see and then we're going to use this as a little checkbox just so we can decide if we want to see our our processed image you know which is kind of ugly but it's it's useful for debugging um or if we just want to see the uh the original image all right so since we saved that if we look at our raw image in the inspector you can see that we got a couple new fields here we got image flip which basically lets us flip it over the x-axis the y-axis or both so we'll just leave that alone for now the threshold we just set up a basic value but the thing is we can change it right we can slide it around and that'll be useful here in a second other than that we just have a checkbox for showing the processed image right okay so now to actually do something with it so the first thing we want to do is uh cv2 and this is the computer vision library and it's uh it's got most of the stuff we need so we're going to say cv2.flip that's what we're going to use to flip the image so you can see on the parameters it's wanting the source and output and the flip code so we're just going to give it the image that we already have we're going to output to the exact same image so it's going to read it and write it to the same spot and then we're going to use this image flip variable that we made all right um let's see so next we're going to convert color we're going to say take it from the image and stick it into the uh no sorry i actually need to add one more variable up here so process image this is going to be the uh the image that we'll do most of our work on we're going to have to do a few steps to convert to do our processing so we're just going to use this one image and reuse it um all right cool so convert color the next thing wants to know is conversion code we're going to change it from bgr2 gray and what this means is the image that comes in uh is actually in bgr format uh which this just refers to like the uh the image format like the order that the bytes are actually in um it's not super important to know just know that that's how it comes in and then we're converting it to a grayscale image because we're trying to get to that black and white thing uh so next up we're gonna do a threshold and we're going to take the process image we're going to write it back out to process image and then for our threshold we're going to use our threshold variable that comes in from the editor the max value we're going to set it 255 and that's just that way the threshold values actually come out as full white and lastly our threshold type we're going to set to binary invert so the next thing we want to do is uh where we're outputting the uh the texture i want to use this uh show processing image flag so i'm gonna say show i'll see image i'm gonna use a ternary operator and i'm going to say well if we want to show processing image we'll use the process image otherwise we'll use the image so we'll do the same thing here so this expression here the this part here where it says show processing image question mark and this and that if you're not familiar with it this is a kind of like a little compact if else statement so we're saying if this thing is true and you can write whatever expression you want there so long as it evaluates to a boolean value right so if this is true then return this and if it's not true then go to the other side of the colon and return that so for our case if we're going to show the processing image then we'll use the process image otherwise we'll use the original image okay so what we've done here is we've added a little bit of coding here just um to flip the image based on the flag convert it to grayscale and then run a threshold on it so i'm going to save that we're going to head back over into unity and we'll hit play okay cool so you can see now um i'm going to go ahead and change my camera view to orthographic and just head in and then also the grid is kind of annoying at this point so i'm just going to change the opacity on the grid all the way down okay so you can see now we've got a great scale image right um and so let's take a look at these options we put in image flip so if i flip uh on the y or on the x or the x y that basically just lets us move it around to what we need the next one was threshold and you can see if i move this down less and less of the image is turning white and if i go too far up then everything will turn white so the trick here is to find something where the background isn't turning white but your object that you want to track is more or less turning white it doesn't have to be 100 perfect but the closer the better and next up we just have this uh flag here so we can toggle between seeing the uh the process image versus the original image right so that'll just be handy for us okay so we've got an image and it's kind of ready for us to start trying to find the the actual objects in it so that's what we're going to do next we're going to add the actual magic here all right so after the threshold here we're going to do cv2 find contours and this this is the magic so this is a function that will look at the image and it will find the uh the outlines of things so we're going to give it the process image and it wants to know uh spot to output our contours and if we look at the second overload of this function we can see that it wants to output a point array of arrays so you can see the two square brackets here or sets of square brackets that's um an array of arrays so what that means is basically we're going to get for every shape that's in the scene we'll get an array of points uh and those points will be kind of the points along the outside so we do need a variable to to store that stuff in so i'm gonna come up here and say private point one array two array and give it uh that is variable uh so that way uh it has some place to put it and i'm putting that the variable up here that way we don't have to read um reallocate space for it all the time all right so the next thing it wants is a an output array or sorry a hierarchy hierarchy index um to be honest i don't really know what this is for uh maybe this is for holes or something but we're just going to make a variable for it so that it will be happy we'll give it to it the next thing wants to know is the retrieval mode i don't actually know what this is i just found it from the example we'll use the retrieval mode counter contour approximation mode again i just uh found this in the example okay so now we've got an array of points uh but the thing is there is a ton of points like this is the really tight uh outline of it and that's it's going to end up being way too much detail for us so what we're going to do is loop over the array of points so we're going to say for each contour in contours uh and we're going to use another approximation function we're going to say point give it an array and say points we're going to use this function called approx polydp and we'll give it the contour that we're looping over here and it wants this uh variable called epsilon uh and i'm gonna loosely just call this curve accuracy and this is basically how dense uh should the points be on it so that one that's something i'm going to want to discover so i'm going to make a new variable up here called private flute uh curve accuracy and i'm just going to give it a default value of say 10 again i just kind of discovered that value from playing around with it but this is something we can adjust and we'll see it in use here in a second so the next thing is we want to measure the area of the shape because uh when we look at our image and we get the threshold thing we might have some noise we might have some like really tiny little areas where it's just i don't know just a handful of pixels and for those things we just we don't want to we don't want to bother with those there's no point in including those in our simulation so i'm going to use this contour area i'm going to measure out the contour and from there i want to say if area is greater than some sort of minimum area then i want to do some stuff with it right so again since that's going to be a kind of a value that we have to discover experimentally i'm just going to put it up here and say min area and then give it a default value of uh 5 000. all right so now now we've got a spot we can say well we've we found all of our contours we simplified them and if the area is um over a certain minimum threshold now we want to do something with it and what we want to do uh just to start off with is we just kind of want to draw light back onto the image just so that we can see what in the world we're working with so i'm going to make a new function here i'm going to call it uh draw contour and i'm gonna ask for the image that uh i want to draw onto something called a scalar which in our case is a color uh thickness and basically what it is is this function here i wanted to take these points and draw it on to this image using this color and this thickness of line so i'm going to add this over here so i'm going to draw on the image and we're going to draw it on the process image um we're going to just for a moment we're just going to say new scaler and we're going to use a gray color so scalar here is just a set of three values the values range from 0 to 255 so 127 is right in the middle of that so this is a gray color the reason i'm picking gray is because our process image is a grayscale color and we've gotta use grayscale values all right so the next thing up is it's asking for the uh the thickness so yeah we're just gonna say two and then it wants to know the points so we'll pass those in and it's these points from the uh the approximation thing so this is the simplified thing all right so to actually implement this we're going to say do a little loop that actually loops over each point in the array we're going to say cv2 dot line and what this will do is it literally draws a line onto the actual picture um there's actually a whole bunch of uh functions in here uh you can draw circles and rectangles and draw text and all that sort of thing which is actually pretty handy but for our purposes we're going to draw a line so it's going to ask us where we want what picture do we want to draw the line onto and where we want to draw all the points from so our first point of our line is points i minus one so whatever point we're actually on and we're going to go to the previous point and points i and then give it the color and the thickness uh which actually reminds me i need to fix this so instead of starting at zero we're going to start at one and the reason is i'm basically going to draw a point from wherever we are at i previous to that and then connect the two points so we have to start off at one so that we don't you know go to an index of negative one or whatever um next up we just need to close the close lines we're going to do one more line i'm going to say points dot length minus 1 points [Music] okay so what this is is this one will connect every point zero to one one to two three to four four five so on but the last line we just want to close the shape so this will go from the last point in the array to the first point in the array cool all right so let's take a look at that and see if it's done what we wanted to do so save everything hit play here all right so you can see we've got uh we've got outlines so you see the the gray outline around my hand you know whatever so that's that's cool that's uh that's our thing so the curve accuracy here uh you can see right now when we have a value of 10 here we've got a kind of a point here and here and it's pretty low poly if i move that down to say five uh you can see we've got point here and here and here so it's a little bit tighter fitting and we can turn this down to say one and we had a really tight fitting uh thing which is cool um when it comes to our colliders the less less points we have the better this is going to be so i'm just going to leave this at 10 and this minimum area thing you can see we've got all these little white uh dudes in there so if i turn this down to one um it's hard to see but we're we're tracing out all these individual little things and you can turn this up i found the values get too big to really slide through it so basically we're trying to turn this up until we get to a point where we don't have uh traces around things that we don't really care about but yeah um cool so at this point we now the computer now has the knowledge of the the objects in the scene and kind of where they exist in a two-dimensional space so next up we actually get into some of the fun things where we're gonna add the uh the physics okay so let's make a basic physics thing so we're just gonna we're gonna skip out of the code for a moment we're just gonna set up kind of a basic physics thing in uh unity uh so i'm gonna right click over here in the hierarchy and i'm gonna add add a little sphere all right so and so we've got a sphere in the scene all right so right now uh it's not going to do much of anything uh what we want to do is get this into the the 2d physics that unity's got so to do that i'm going to take off the sphere collider by uh hitting the little three dots and saying remove component the reason i'm doing that is the sphere collider is a 3d uh collider and we actually need uh uh one of the 2d ones so i go to physics 2d uh we can see we've got a circle collider 2d and then the big thing is that this is in the 2d system so that will give us a collider and then also i want to add a uh rigid body cool all right so uh if we hit play on this now we should see that this ball just kind of falls down whoo gravity's taking and it's just going down um so obviously we want to give it something to bounce off of right you know that's the whole point of this so we're going to add a new game object we're going to call this our collider and i'm going to reset its position two zero zero zero uh and while i'm at it i'm gonna set this thing's position to zero all right so back on our collider here for the uh the physics the 2d physics what we want is a polygon collider so we can see that while we've got this thing selected we get this little green outline and that's the shape of our collider here right if we deselect it it'll disappear because it's just kind of a helpful indicator for us but in case it looks like when our ball drops it probably should hit that so let's go ahead and hit play it bounced off it uh which is cool that's uh that's actually just what we want one thing though is i want my my ball to be a bit bouncier to me that that seemed a little flat uh so i'm gonna introduce uh a physics material all right so in my assets folder down here for the project i'm going to right click i'm going to say create 2d physics material and i'll just call this ball and if we look at that ball in our inspector we've got two properties friction and bounciness so i'm just going to turn that up to 0.8 and then this material i'm going to apply it to our sphere so there's two spots we can put it on the collider but i think for the 2d stuff it makes better sense to put it on the rigid body so i'm going to hit this little circle on the side and select the ball that we just got so now um if we hit play what i expect to see is the ball bounce quite a bit more off of the uh the thing when it hits woo cool wrong one so that's kind of our basic system right we've got we've got a uh a rigid body inside the system and we've got a collider something for it to hit now the reason it looks like this uh this pentagon is because of the points so this polygon collider you can have multiple paths and each path can have multiple points so we've got one path which is say one shape and we've got five points and if we move say the the point around you can see how it changes the shape so for us the trick is going to be instead of us manually putting these shapes in we want our our actual code to go and set these properties all right so we're going to go ahead and bounce back into our code over here we're going to add a new variable for that polygon finder right or sorry the polygon collider um let's see what do we call this thing so this this will give us a reference to that uh that collider so that we can actually do some stuff with it so i'm gonna go ahead and save this flip back over into unity and we'll see that on our uh our contour finder here we've got a new spot for to assign what collider should work with i might hit the little button and i'm doing that now because i just too often forget to assign it then i go hit play and then i get all referenced exceptions then it's just a bad day all right cool so our business here is now that we've got our polygon collider we know our shape we've got all our points now we just need to actually assign it to that that thing so the first thing i want to do is before we get into our loop before we're actually looping over all the shapes we found i want to say hey polygon collider your path count is zero we're just going to assume that there's nothing in the scene there's nothing there for us to work with then we're going to get into our loop we're going to loop over each object that we find and we're going to approximate it and measure its area and if it's if it meets our basic requirements we're going to do is we're going to add it to that polygon colliders paths so we're going to say path count plus plus it is okay you've got one more path than what you had before initially you had none our first one we find now you've got one and if we keep finding them we'll we'll add to it right all right so polygon collider set path and it's going to know which path do we want to set so we're just going to say hey the last one uh this and i'm saying minus one because it's actually an index and in code indexes start at zero all right so the next thing it wants is vector two uh now you might be inclined to say well we've got our points here um and this and these are the points that we want but these are something called a point and it wants something called a vector two all right so here's the thing vector 2 is unity's representation of a 2d point and point is open cv's representation of a 2d point it's basically the same thing but they are two different classes so if you just try to pass in points it will yell you it's going to say i can't convert from a point to a vector 2 what are you talking about all right so we're going to help it out we're going to make ourselves a little helper function here we're going to say make a little function it's going to return this array of vector 2's and it's going to take in an array of points so basically we're going to put in the opencv points and we're going to get out unity points so this isn't anything complicated but it's just something we've got to do so one thing we want to do is up here we're going to say make ourselves a list of vector twos um and we're going to initialize it to a new list of vector twos and the size of this is going to be the exact same size as the points array right so the notion here is if we've got 20 points in this array then we want our vector list to have 20 points then we're just going to do a loop so we're looping over every point in the pointer and we'll say in our vector list and slot i is a new vector two and we're going to use points i dot x dot y okay so nothing too complicated here all we've really done is we've made a new list that's the same length as that and then we've looped over all the points and just made a new vector too and shoved them in there okay so now we can go back up to our set path and we can make this guy happy so we can say two vector two points and this will just uh you know convert it over to uh a format that it knows what it's talking about all right so that's cool that wasn't too bad let's see if uh let's see if that worked uh yeah i saved it okay um so i'm gonna just hover over or i'm gonna click on the collider so i can see what shape it's got and then i'm gonna hit play and hopefully what we're kind of expecting is that these these shapes would have shown up outlined in green but don't know where to be seen um so here's the problem uh if we zoom way out we can see oh there's our shapes so like that's me putting my hand in the scene um so as i say it's freaking huge by comparison and the reason is the the points that we're getting those are like actually the pixel values so like this is say 600 pixels up and 400 pixels wide and i'm in unity terms that's that's 600 meters up and 400 meters out so as it's just huge and all we need to do is just change the scale to fix it so i'm just going to scale it way down and then hit f to to fit it and i'm going to keep scaling this down until it's you know something that i can deal with okay so the next thing i've noticed for me my uh this this looks mirrored um so my uh my little pliers they're kind of pointing down to the right and this guy's pointing up and to the right um so we could do this different ways but for for this i'm just going to make the collider match so for the scale where it says y i'm just going to put a negative value in there and now we can see that our pliers are facing the same direction they're offset they're not in the right spot but at least they're pointing the right way all right so next up i'm going to change the position i'm going to move it so that the position of the thing is literally right in the top right corner and we can see our green outline is pretty close to our gray outline it's not quite right but it's getting close so next up i'm just going to change the scale until it lines up so it's hard to see but we've got our little green outline so this now is instead of drawing on the picture now we've got a collider that's actually in the uh the the world space of the scene so now this is this is truly the magical spot where we've gotten unity to actually be aware of the uh those shapes and terms that it knows what to do with all right so the thing to know here is we're in play mode and we went and changed the transform on this object if we exit out play mode unity will do its normal thing and it'll just erase everything we just did so to get around that i'm going to right click on the transform and i'm going to say copy component right so i'm going to put all those values on the clipboard and you see when we exit all the values went back to where they were so that's lame we're going to paste component values and that gets us our values back so that way if i hit play again we should get right back to where we were and look at that now our ball actually responds and i can dump the ball out which i mean come on how cool is that uh so at this point this is really the crux of the project we've we've done all the stuff we really need to do to to conquer the big problem we've made our uh we've imported opencv we've processed our image and we've made our collider that responds uh to the actual image um so but you know it's not that fun we just had the one ball and now it's gone and you know now we're lonely right um so what we're gonna do is to make it actually uh get closer to what i showed at the beginning we're gonna add a uh an emitter uh sort of thing we're just gonna make a million balls kind of come down from the the top and bounce around the scene and then once they get down to the bottom we're just gonna kill them off so that they don't spend all day uh following right so um so here's how we're gonna do that i'm going to make a a new 3d object i'm going to make it a quad i'm going to call this thing the emitter it's facing the opposite direction for me so i'm gonna rotate 180 degrees just so i can see it i might zero its position out just so i'm kind of starting in a known spot and i'm gonna move it up here kind of above the scene i'm gonna scale it out um so basically wherever we have our a gray box here we're gonna pick when we emit uh points we're just kind of gonna pick a random spot kind of inside there um the one thing i am going to want to do is i'm going to set the z scale to zero because i don't want it to uh having depth okay cool so let's make ourselves a new script and we're going to call this script emitter and unity is going to recompile and i'm going to add that script that i just made onto our emitter object and then i'm going to double click it and then i'm going to open up the script right so so uh we don't really have to go into too much detail on this but here's here's what we got we need some uh we need some parameters for this thing to work with uh the first thing is we need a game object to spawn right we want to make a million of these little uh particles falling down it needs to know what that object is so we'll give it a reference for a prefab we also need to know how fast we want to spawn these things in at let's see next up we want to know uh what the maximum amount of particles is uh we're going to start off with a low number and we're just going to say the most particles we can have in our scene at any one time is three which is a ridiculously small number but when you're making emitters and gonna make a bunch of objects it's better to start small and make sure your stuff is actually working before you uh go big right uh the last thing here is uh i want to make a size range so for me i want to make those balls you know some smaller some bigger so a vector 2 just gives me two numbers i say okay between the small value and the big value other than that the way i'm going to do my object or my my emitter here is i'm going to have a pool of uh objects and basically the notion here is that uh we could every time we want to spawn a new particle we could um call instantiate in unity and that would make a brand new object and that would work the problem is that when you do that you're you're making a new object in memory and when you're making lots of objects um eventually that makes unity unhappy uh you've got garbage collection stuff where you've allocated memory and then like when you get done with it tries to get rid of it and it all kind of just ends up being a pain we know that we're going to always want a bunch of uh particles so what we're going to do is use a pool of objects we're just going to say well i want uh you know our max number of particles i just want to make those and have them around and then i'll just turn them on and off when i need them all right so i'm going to make a new function called initialize pool and then we're going to say pool equals new game object array uh max particles so that's uh this variable and we're just going to make an array that is this big so for okay so we've got three particles so it's an array it's three big so we've got three slots next up we're just going to fill them so we're going to make ngi zero is less than max particles and here we're going to actually instantiate it uh instantiate means actually make the object like take the uh the prefab and make it and put it in the scene all right so at this point this this particle would be in the scene we're going to set the active to false because we don't actually want to see it we're just going to make it and turn it off so just so it's ready for us to use it when we want it and other than that we're going to put it into our pool so we're going to say pool i equals particle so all we're really doing is looping over every slot uh in our pool making a new particle turning it off and that that's it that's the whole thing uh so in our start function here we're going to initialize the pool just so that we've got it and just so you can see what's going on with that um i'm gonna flip back over here to unity so here's our emitter and we've got our default values for some of the stuff but it needs to know what prefab and we want to use this sphere but the sphere is not a prefab so i'm going to create a new folder called prefabs and i'm going to drag the sphere down into there and then this makes the sphere into a prefab uh and now i no longer need it in the scene and in the emitter i want to say what what object do we want to make we want to make this object um yeah so if we save that and hit play what we'll see is it should make all three of those objects so they're there spheres one two and three uh it made them turn them off we can't see them but they're there right uh if we turn one on there it is all right uh one more thing we should do is uh the emitter has this uh size range that we put in so i want to say how big the uh the spheres are just kind of arranged so i'm gonna look at my sphere and say i'm looking at the the scale here say i want to be kind of from now about this down to about that so down here is 0.25 and i don't know about this big let's just say it's 0.6 so our size range here we'll just say it was 0.25 to 0.6 that way when we spawn these things we're gonna go somewhere in between these two values okay now i really can't get rid of that sphere all right so uh what we wanna do is we wanna have a thing that fires every so often uh basically the spawn rate um and makes makes a new one right so we're not going to use the update function so we're going to get rid of it so that you know unity doesn't really have to bother calling that function we're going to make a new one called spawn and i'm going to call that from the start function so when we start it's going to call the spawn function and what we're going to do is spawn the particle into the scene and then and here's kind of our trick we're going to call the same function spawn after this delay so what we're doing is we're calling it once and then every time you call this function it ends up calling itself again after this amount of time and this is in seconds so um however often spawn rate is so if this was 0.1 this would be every 0.1 seconds we'd fire this and the reason i do this as opposed to says you could do invoke repeating like this uh invoke repeating would do pretty much the same thing uh the only difference is that if you change the spawn rate after the game is started then this it won't change like you called it invoke repeating and it's done uh you could deal with it in various ways but basically this way whenever you're you're spawning the thing and you get to to start it we'll look at whatever the spawn rate is right at that moment and that way it's easier to update sorry tangent all right so the next thing we want to do um is we want to look at our pool of particles and we want to find one that is not already doing it's not already active in the scene and we want to turn it on and get it get it going right so for that we're going to say for each particle in the pool we're going to check to see if that particle is active self all right which is the same if it's not already active then boom this is our guy this is the one that we want to mess with and that's all we're going to do so we're going to break and now now that we've found a dead particle one that is just sitting around being a slacker doesn't have anything to do we're going to use it right so we're going to say hey particle transform position so this is us telling it where where it should actually be and we're going to pick a uh a value here a random value so inside random inside units here this will give us a vector 3. so just you know a position in the world that is somewhere inside or on a sphere with a radius of 1. i'm multiplying that by 0.5 because i want this to be a value that's somewhere between negative point five and point five because our this uh this uh rectangle up here when before you've scaled it this side is negative point five and this side is point five um yeah yeah i'm sorry uh we're just going to go ahead and do that and we'll say transformed uh transform point and basically transform point we'll take it from this object's local space and move it into world space so the reason i'm doing that is because this thing is stretched out and it's moved up and all that stuff if i pick a random value that's somewhere inside its units here it'll just pick values in there um sorry i don't need to build labor at that point um on that i'm gonna set the things uh local scale for this one i'm gonna say a random range between uh cool so local scale that is a vector three and it's a vector three because we've got x y and z uh for us we actually want our sphere to to scale uniformly so all those are the the same value um so what we're going to do is we make we pick our random number and we multiply it by one and by doing that we basically just convert this into a vector where all the values are the same uh cool so now we've we've moved a particle into kind of a random spot we've set it scale to something random last thing we need to do is just turn it on all right so let's take a look at that so if we hit play what i'm hoping to see is a few particles spawn out of here and let's just let's slow the uh the emitter down the spawn rate to one per second just so we can see that a little bit more clearly so we hit play one two three all right so we've got our particles but they're just kind of they just stay in the scene and they they're just there forever right um those two that fell off they fell off and they're gone and they're they're going down and down down down right if we look at it you can see the the y value here is just going down all right so that's no good our our scene is boring because the objects left us and they're not coming back all right so what we want to do is say something along the lines of hey once they get outside of the screen just kill them they're they're of no more use to us so we're gonna use basically a similar trick where we get to define this so new 3d object quad we're going to say this is our kill zone and i'm going to rotate it around set its position back to zero and what i want to do is move this thing down here kind of off screen scale it up so it's nice and big and what i'm looking to say is that anything that goes into this zone we're just going to kill that particle now to do that we need a collider on it and because we're working with 2d stuff we need a 2d glider so i'm going to get rid of its mesh collider because that's a 3d one and i'm just going to say a box collider 2d cool so i made a little green box around it and then this is important we're going to say it's a trigger zone so objects don't bounce off of it rather we just get a signal when they go into it all right so we need another little script here we're going to call this thing killzone and we're going to add that component to our kill zone and don't worry we are almost done i promise uh okay so for our kill zone it's really really super simple we're just gonna get rid of these functions and if we do private void on trigger enter 2d make sure you get the 2d that part's important what we want to do is this will tell us any object that's any of the 2d objects that come into it it'll tell us about it so we're going to say the thing that collided with it we're going to take that things game object and set it to inactive that's it that's that's the whole kill zone uh so let's take a look at what that does so this object comes down and then as soon as it gets to here it turns off and because it's turned off that means when the uh the emitter goes to turn something on there's something there for it to find ah so that's cool uh so let's crank it up man let's go back to our emitter and let's say we want to spawn pretty often and instead of having three let's let's have a hundred cool all right so now we've got now we got something we can work with here it's cool like this thing actually will fill up with this stuff or we can bring an object in and they bounce off uh and that's that's super fun and the cool thing is the the unity physics system is actually um pretty robust so i'm gonna change these values around just a little bit so now now we've got a thousand particles uh let's see what that looks like oh all right well so that's not that's not spawning fast enough let's really yeah there we go [Laughter] um and just for fun let's turn let's not look at the processed image let's turn that off turn off our gizmos look at this yeah well i mean that's it that's all i wanted to show i know i uh kind of droned on and on about it but this is kind of cool we've got uh we've got a little unity project where we've got a webcam coming in we've got computer vision thing like actually looking at the scene and making uh making our colliders so that we can see have it actually interact with our world and then the physics system doing what the physics system does best and it's just fun cool all right with that guys i'm going to say goodbye and good luck

Channel: Matt Bell

Views: 20,171

Keywords: opencv, unity, live, realtime, interactive, physics, tracking, particles

Id: ZV5eejYG6NI

Length: 65min 6sec (3906 seconds)

Published: Fri Apr 29 2022