C# Detecting Moving Objects in Streaming Video w/ EmguCV Part 1: Motion Detection Basics

okay so welcome back um today we are going to start looking at how we can develop this application here and the purpose of this application is it will detect in a incoming video either a video file or streaming video as you can see here on the left this is live video it can detect objects in motion and we're going to look in this first video at some of the basic concepts how can you possibly from streaming video coming in how can you possibly detect objects in motion and what I've got here is on the left we've got video coming in from a smartphone mounted on a window looking outside and this is streaming video over Wi-Fi and we've done videos before on how to several us up and how to build this application to display the streaming video and in this video we're going to take the next step and once we've got the streaming video we're going to have to process it in this application and on the right you will see the results of the processing done on the streaming video and on the right we should see as we have moving objects we should see the objects identified and this application will draw a rectangle around them as they move through the video so now here we have an example of someone walking their dog and you can see the process version shows this rectangle that our application is drawn because it has recognized a moving object in the streaming video and here we have another example of a car driving by and the application has successfully identified a moving object and drawn a rectangle around it and here we've got another car hasn't identifyingly identifies it draws a rectangle and as it comes by you can see we've got a rectangle around that moving object so now in this video we're going to talk about some of the basic concepts how we can take a streaming video and somehow identify moving objects well if you think about it this streaming video on the left is basically video frames coming in sequentially and they're coming in at maybe 30 or in our case 60 frames every second so how would you identify moving objects well if you think about it if you can compare sequential frames like compare one frame with the next frame the difference would be that the frame is going to change the pixels in the frame are going to change as the object moves because really this is just a bunch of pixels in images and you've got sequential images and those pixel values will change as a object moves through so let's take a little closer look at what that might look like so here what I've done is I've gone into a image editing application it's called Affinity photo it's kind of like Photoshop and I've brought in two frames of a video a streaming video and here's one frame and what I can do is I can overlay another frame and this frame has a car moving through the scene so with and without the car so what is the difference if we were going to detect that this car had been moving through what is the difference between these frames that we can make a note of to determine that there is actually a moving object well here is the background and here is the frame with the car so what's the difference well the difference is the pixels that Define this area where the car is and we can show that by we can scroll in we can zoom in to see what is really making up these images and as you can see these little squares these are called pixels and each image is comprised of probably millions of these little pixels and each one of these pixels has a color an r g and B value so if we were going to detect that there is now a car in this streaming video what we could do is we could look at this background where nothing is moving compare the pixels in that background image with this subsequent image and you can see that for example these pixels were once light gray and now they've become dark so we can say hey we've got a bunch of pixels here that have changed colors and that tells us that something has happened in this streaming video there is a moving object so if somehow you could take your streaming video and you could look at all of the pixels in one frame and compare it to all of the pixels in the next frame then you could somehow determine that there was a moving object and what this application is basically going to do is what you see here it's going to be black everywhere where there's no moving objects and it's going to be white or a shade of gray where there are moving objects and it's basically going to do that by comparing subsequent images you see we have this moving car and all the other pixels are black because there has been no movement or no change in pixel color between subsequent frames now what you do see here is you see these little noisy dots now what is that well if you look over in the image there's no motion but if you look here there's quite a bit of motion going on so what is that all about well over here on the left what's happening is these leaves are moving because of the wind and it's changing the color of those pixels and that noise appears all over the place here you can see when the cloud comes over or when the sun comes out the pixel colors change so one of the challenges we're going to have is we're going to have to not only compare changes in pixel colors with subsequent frames we're also going to have to determine what's noise what's irrelevant and what is actually in this case we have a car why is it detecting these vehicles but it's not drawing rectangles around the rest well we're also going to have to add in future videos we're going to have to add some logic to ignore some of the noise and only detect what looks like an object like a person or a vehicle now that's going to be a bit of a challenge because we have all of these moving clouds we have the moving leaves we're gonna have to add some logic to remove that noisy data and we'll look at that in future videos now you can see here that we also had when the car went by a lot of the pixels changed so there's other things you need to be concerned about like your camera your exposure your automatic exposure if a bright object comes into the field of view you can see that it might cause your automatic exposure on your camera to change which you're going to change a lot of the pixels so it's not a real simple matter of just looking at changing pixel colors you also have to consider other stuff noise of moving leaves Auto exposure moving clouds the sun coming out all of those have to be considered if you really want to just make sure that you're only detecting moving images like these vehicles so we're going to take a look at that in this and future videos so now let's take a look at a very simple example of how we might do this basic detection of objects in Motion in streaming video so let's say we've got a three by three pixel image in fact it's going to be like 1920 by 1080 or 640 by 480 but we're looking at a very simple example of a three by three pixel image and we've got two frames two subsequent frames in streaming video and we're going to grab one frame and we're going to put it into an array or a matrix and as you can imagine each pixel is going to have a value in fact it's going to have three values an R value a red value a green value and a blue value but we're just going to have very simplified example of one array of colors and these are the values of those pixels then we'll take the next frame and each of those nine elements is going to have a value and we're going to say that the initial frame and the subsequent frame the next frame are going to have identical values so what our application is going to somehow do is it's going to go through all of those values and compare for example this first pixel value of 10 with the next frame the same pixel value is 10 and if there's no difference in those values it's going to make the output pixel we're going to call it a mask it's going to make the output pixel black because there's no change in those values and it's going to do so for all of these since both frames have identical colors for the pixels so now we're going to do what's called a background subtractor we're going to use this Frame 1 as the background and we're going to subtract Frame 2 from that to get a difference in this case the differences are all zero which means it's going to all be black so it's called a background subtractor and we're going to use that in emgucv or whatever application we're going to use that concept is going to apply where we're going to make look at the difference in the pixel values of subsequent frames now one of the functions here is to figure out what is the background image and the logic in this background subtractor is going to determine what is a image with no changing pixels so some of the Logics is going to be to figure out if in subsequent frames maybe this Frame and the next frame or this Frame the next frame and the next frame you can stretch it over as many frames as you want there's been no change in the pixel values for a bunch of frames you can say okay well this is our reference this is our background and we're going to use that particular image that particular frame as the background and we're going to subtract future frames from that background image so part of the logic is to figure out what's the background image so now we've got our background image because it hasn't changed in a bunch of frames so let's say we get a new frame and all the values are the same except for hey we've got one pixel instead of being a value of 12 it's about value of 50. so what we're going to do is we're going to say okay all of the differences are zero except for this pixel has a difference of 50 minus 12 or 38. so in that pixel we're going to put a white value instead of black or some shade of gray depending on the logic we use to determine what the pixel color what the shade of gray of the pixel color is going to be but you can see that we now have a change in the pixel color that is identified in our foreground we're going to call it the foreground mask which says okay we had this pixel changed in color and the amount of change might Define how much white or how much gray it is so that is what's called a background subtractor it comes out with an image a grayscale image which we can call a foreground mask which defines how much change in pixel colors there has been with subsequent frames so it gives you an indication of the change and again we said that change can be due to uh for example the clouds or sun changing the colors moving Shadows wind and trees Reflections and glare so for example if a car goes by and you get a big reflection it might cause some glare that causes all the pixels to change Birds going through camera vibrations the auto exposure of the camera it might automatically change What's called the iris or change the exposure so that suddenly all the pixels get brighter or all of them get dark so there's a lot of things we're gonna have to look at in future videos how to get rid of this irrelevant stuff but the basic concept behind motion detection is what we show here it's called the background subtractor so in future videos we're going to make the next step and look at how to deal with these noise factors and also how to determine I've got some indication of a moving object how do we determine if it's something we care about if it's a vehicle or if it's a human or whatever how do we separate that out from other moving things to put a rectangle around just the things that we care about so that's about it for this video if you like any of these videos please hit the like button subscribe hit the Bell notifications but most of all please let others know that we're here so we get some views really appreciate it otherwise take care and have a really good day thanks

Channel: EETechStuff

Views: 2,846

Keywords:

Id: 0xW0ZnfQIhg

Length: 13min 51sec (831 seconds)

Published: Tue Aug 01 2023