C# Detecting Moving Objects in Streaming Video w/ EmguCV Part 4: C# Code

okay so welcome back today we're going to take the next step in our discussion of how you can detect moving objects in streaming video we've been talking about that in the previous videos some of the building blocks of that I encourage you to take a look at that in this video we're going to put it all together and build this c-sharp Visual Studio Windows forms application using emgu CV and this application is going to allow you to take what you see here on the left which is streaming video live streaming video from a camera looking out a window and it's we're going to be processing each frame of this video and the result is going to be what you see over here on the right and we're going to be utilized when we talked in the previous video which is what's called a background subtractor which gives you a series of images like you see here representing a change in the streaming video coming in and as we discussed before the way you detect motion is you detect a change in the colors between subsequent images in the Stream So this on the right is going to be what's called our background subtractor which gives you the results showing you pixels representing any changes in this streaming video and you can see here we've got some leaves rustling and we're getting an indication of that with these little dots so let's take a look at this um application and see some of the basic components now here on the left we've got our streaming video we developed that previously we've also got a text box down here that's going to give us a little feedback in which I use in most of my c-sharp applications some debugging feedback and whatever we also have over here on the right the option to either give this background subtractor images or the image itself so I can select these radio buttons I can have the image itself or the mask and it will change it in real time now if I've got the image what I can do is I can show what are called The Limit lines and this is something I've added you may not need it but this allows you to select an area of the image that you don't want to be included in the determination if the if there's any moving objects so for example in the upper right corner here I've got some trees and I really don't care about the motion of those trees I don't want to be distracted by any motion up here so these limit lines are user adjustable and they allow us to say okay I only want to detect moving objects that are to the left of this vertical line and below this horizontal line so we can move it we can we can decide where we want to what we want to include in our evaluation those are called limit lines we also have some readouts of the pixel position horizontally and vertically of these limit lines we also have the option of we can go to the mask here and we can see all this motion we can say well I don't want all this noise so I'm going to hit the noise filter and you can see it filters out a lot of the motion of these leaves I can also do what's called a threshold which says there has to be a certain amount of motion before we're going to show it and this boxes check mark is going to say okay if you actually detect a vehicle or a important object we want you to draw a rectangle or a box around the object as you just saw with the car going by and you can turn that on and off and you can see it's doing a little bit of that now with the noise and you can see we've got a lot of stuff going on here that doesn't look very obvious in the streaming video but it's clouds coming over it's leaves moving it's sun coming out Little Critters running around so there's a lot of stuff going on that you may want to just ignore like you can see all of this being detected as an object when in fact it's just moving Shadows so that's why you might want to put on the threshold and noise filter and adjust those so you don't falsely detect stuff so we also have um we're updating the UI to show for example the size of the image coming from the camera in this case it's 640 by 480. and what we're doing is we're resizing it to put it in these boxes we're making it 1920 by 1080. we're also querying the camera to say okay what FPS are you set at you can see it set at 60 and we're once we do the processing after we've done all of this background subtraction we're also calculating each time we receive a frame we're figuring out how many seconds between receiving those frames and using that to calculate what's the frames per second of this processed video so we've got the camera set at 60 but after the processing it drops down to 30. so we've got that we've got the selections we've got start stop and we've also got a capture um where we can at any point in time say we want to capture the streaming video and the resulting mask here we can hit capture and it will send both images to files to PNG files so we'll grab this Frame and grab the mask and save both of those as files that we can evaluate later so that's the basic concept behind this application again we're going to do basic capturing that we've done before and then we're going to use emgucv and opencv to do this processing on the video to allow it to detect moving objects so let's go into the application and see how this all works so here we are in our c-sharp Visual Studio Windows forms application if you haven't done programming before I encourage you to look at my beginner series for how to do a simple c-sharp Windows forms application it's wonderful way to do science engineering and UI applications really great what we've got here is the basic format of most of my applications um we've got some using statements we'll talk about we've got my docs where we explain what this application does a bunch of parameters we'll talk about in our form one we do the standard initialized component initialize the UI with some default values and then we have a new emgu CV video capture device using this camera index I did multiple videos on emgu CV and how to set up the basic video capturing but we'll go through this a little bit more detail and then I've got some methods and event handlers and as you can imagine there's a bunch of event handlers for the buttons and then a bunch of methods to do the image processing and so on so the first thing we need to do is as we showed before and I'm not going to go through detail you have to install the mgucv so tools nuget package manager manage new get package solution installed I've got emgu.cv dot CV Dot bitmap and Dot cv.runtime.windows again we showed you how to do this previously so I encourage you to look at that we also need to set this for an x64 not any CPU so make sure you look at the video where we did that and we got to use some using statements so as we did before using emgu.cv and then emgu.cv.cv enum and those are going to be used for some of the image processing filtering tools uh we need emgu.cv.structure which is when we draw our limit lines it's going to allow us to draw some lines [Music] emgu.cv.util needed for the vector of vector of point we'll talk about that and then the using system system.drawing because we're going to use the bitmap class using system.threading.tasks we're going to launch the image processing and it's is its own thread to make things a little bit more easy and then system Windows dot forms so now we've got our basic application and what we can do is start out with designing the UI and as you can imagine we're going to have a couple picture boxes we got one here that shows the raw streaming video from the camera over Wi-Fi and then we've got one on the right that shows the processed image which is generally a grayscale foreground mask showing the areas of motion um we've also got some track bars I did a video recently on how to use track bars and you may not need these but these we're going to use to draw the limit lines where we can say only identify moving objects in a certain area and ignore the other areas so it we got a couple track bars as with most of my applications we've got a text box down here to give some user feedback to make sure everything's going okay we've got some labels this tells us what the dimensions of the frame coming from the camera what is the camera frames per second and then the processed and calculated frames per second and then we've got a couple radio buttons to Define this picture box on the right we want it to be showing an image the raw image or the foreground mast and those two buttons are inside a group box and we talked about how to do that just put in a group box and drag and drop the radio buttons and that will allow the radio buttons to only have one selected at a time so then we've got uh one two three four check boxes um this first one is if we want to show the limit lines that the track bars control uh this one if we want to do a threshold calculation in the grayscale foreground mask this one if we want to filter out any noisy parts of the image and then this one if we want the software to identify certain moving objects and put a rectangular boxes around them uh and other than that all we've got is these four buttons that we're going to have to have button event handlers for we've got the start which starts the capture and the processing and of course the stop we've also got a button here that when you click this it will instantly grab whatever frame is being displayed in our picture box and also the process version of that and it will save those two images those two captured frames as images that you can look at later and evaluate and we'll also do some nice stuff like give it a the title of the image will be the date time stamp so it will tell you exactly when this was captured and then we've got the button exit so really um I assume you know how to drag and drop picture boxes and track bars and these check boxes and everything we've talked about that previously so now we um have the UI all set up we also need to make sure we have our picture boxes set to zoom mode so that the frame will fit inside but otherwise I encourage you to look at previous videos for more detail on how to set this up and here is our main application so we've got a using statements our docs which basically says this app captures streaming video from either a camera or a video file and processes it presently it detects moving objects in the streaming video by using emgu CV slash opencv background subtraction functionality we talked about that previously it also allows the user the option is applying noise filtering threshold filtering and using track bars to Define movable areas to ignore it sets up an emgu CV video capture device uses the image captured event to grab and process each image in the Stream also launches a test.run to perform the motion detection on a separate thread so we've got that we've got a bunch of parameters I'll scroll through these you can pause and just type those in but basically we've got some Bulls which Define um do we want to stream the video yes or no so we can pause it or we can continue streaming uh and for the right hand picture box we want to show the image or do we want to show a mask so we're defaulting to show an image uh do we want to filter noise we're defaulting to node do we want a threshold applied we're faulting to no and then here we are defining our mats and we talked about that emgucv Matrix which defines each frame and we're calling it public matte and we're calling the raw frame coming in as frame uh we have a Mac called Smooth frame which is basically a gaussian blurred version of that we talked about and then we have the foreground mask mat which is the grayscale image showing the detected motion um we have the camera index when you set up the video capture device you have to tell it what is the camera index if you've got multiple cameras um we are we've got two doubles that we're defining to be able to calculate the frames per second and uh we'll show you how to use these we're going to do a little bit of filtering but this is just for frames per second calculation um here we've got some definitions about the bounding boxes or the rectangles we're going to apply and also the base frame width and height that is coming in from the camera what's the what's the dimensions of the images coming in from the camera so we're going to show you we can use this minimum Bounty box area to Define that we only want things to be detected as object if they're greater than a certain size so we'll talk about that here we've got our image path when we capture the the image and the associated grayscale foreground mask here is the path that we want to save it and here we've got some date time and time span for calculating the frames per second every time we get a image captured event we're going to calculate the frames per second we're also going to define a systemdrawing.size we're going to call it frame size which says we want to take whatever image is coming from our camera it might be 640 by 480 or something and we want to increase it to 1920 by 1080 and we're going to use this frame size to do it and here we are defining our emgu CV video capture device we're going to call it capture and then we're going to instantiate it later on and here is the most important thing our emgucvi background subtractor and we're calling it background subtractor which does the motion detection we talked about previously so that is all of the parameters so I said we've got the standard initialize component we've got a method that we're going to just initialize the UI to some some defaults and we're actually instantiating our video capture using that camera index and we're going to call it capture so the first thing that's going to happen we start up our application we can do this button start so we've got this event handler and all this is going to do is say okay we are setting that Boolean we had before stream video now we're going to say it's true we're also going to subscribe to the video capture event the event that comes with every video capture device it sends out an event whenever it grabs an image from your camera and we've got the video capture device we called It capture and we're subscribing this capture underscore image grabbed events that you can see down here below it we're subscribing that event handler to this video capture image grabbed event so every time we get a an image coming from the camera we will go into this event and then we're doing capture.start to start up the capture now we can also as we talked about previously we can set some parameters we talked we've got another video talking about camera parameters very complex field but here we know that we can set the frames per second of the camera to 60 frames rather than the default 30. it's going to be different on your camera but here I'm just showing you can use an emzu.cvcveenom.cap prop or capture property dot FPS and we're setting it to 60. that's because when you do the processing it's going to slow things down and your 30 frames if it's set at 30 it's going to get down to like 15 or so so you want to crank that up a bit so we get better response um we also are instantiating new background subtractor and we're using background subtractor mog2 now we're not going to go into the details of that but for now you can use that as the method for background subtracting and here we're going to enable timer one to set that to true so it can update the UI every 100 milliseconds or 300 milliseconds or whatever you want so that's basically it we've got the capture being started and all of the processing now is going to go to this capture underscore image grabbed event handler and to the timer one tick so every time it grabs an image it's going to go to this and this is going to basically just grab the raw image from the camera display it on the left hand picture box and then call the motion detection methods and start it off in another thread but that's all this image grabbed event handler is going to do so we're going to do a try catch and we're going to try to retrieve a frame but we're going to start out for calculating it frames per second we're going to say now is date time dot now and when we're done with this we will do the last which will allow us to calculate frames per second so if stream video is true if we've got the Boolean set where we want to stream the video the first thing we're going to do is Define our new mat called frame and that's going to represent each captured frame from the camera and we're going to do a capture.retrieve and we talked about this previously if we have an image grabbed event that means it has already grabbed an image so all we have to do is do a retrieve we don't have to grab it we just retrieve an existing frame so we're doing capture.retrieve and we're assigning that to this Frame and now we have grabbed the frame and we're going to say whatever width and height are coming in that frame from the camera that's our base frame width and height that we can display on the UI to say hey the width and height coming from the camera is like this and then we can resize that frame so we're doing a CV invoke we're basically invoking an opencv command using this emgu CV invoke to resize and we're taking this incoming frame we're making the outgoing frame the same thing and we are doing a frame size which is what we defined in the parameters and basically going to take this if it's 640 by 480 it's going to resize it to 1920 by 1080. and then we're going to display that frame in the picture box it's just the raw image it's a resized frame but it's basically the same frame and to do that we have to use the 2-bit map function of the matte and it will put that into the picture box so once that's in the picture box now we can go and process the image so we're going to call Process image and we're looked at the bit but it's basically going to launch another process to do some image processing depending on what the user selected so then when that's done we figure out a time span is the difference between last and now and that is giving us for each captured frame it tells us how often are we capturing the frame what is and then we can calculate frames per second and then we're going to do a frame dot dispose because we're done with this Frame we get a failure it's just going to give us a message box that application failed actually we can just make this as empty and it will just ignore that frame because we don't really care but that's basically all it's going to do and now process image is going to be a method we'll see that's going to do the actual image processing in a different thread now we also have this timer one tick and that's just going to do a bunch of stuff to update the UI and here is we have our track bar our horizontal and vertical track bar this is just going to tell us what the text is there's a label that's associated with that track bar it's going to tell us what the actual values of those track bars are in terms of pixels or lines and columns in that frame so we're converting the track bar horizontal value to string and setting that as the text for the track bar basically label up here and down here to give you real-time feedback and then we can calculate frames per second if the time span is greater than zero there's actually time between received frames we can go through this calculation I'm not going to go through this in detail it's a little bit Overkill but we're just saying that whatever the time span between frames is multiply times .001 because that's in milliseconds so 1 over that is frames per second and then we can do what we talked about in previous videos which we can apply a low pass filter so we don't keep seeing these frame per second calculations uh jumping around 60 times a second we can filter it out by saying we're taking 90 a new our frames per second is going to be 90 of the old frames per second plus only 10 percent of the new so it's going to kind of stick with the old one and only update it a little bit depending on the new and then display that in the processed FPS and then set FBS old equals FPS so let's again you don't need to do this we're just showing um what the frames per second calculated is um and this is just saying if the user says I want to show an image in the right hand picture box it's that radio box image is checked then we're saying show image equals true and show mask equals false else if radio mask is checked show image is false and radio mask equals true and then if the user wants us to do a noise filter on this image if that is checked then filter noise equals true else filter noise equals false if the user wants a threshold calculation done threshold equals true else threshold equals false and then we can also ask the device what frames per second it's set at and we can do this using the capture.get and we can feed it this capture property.fps which we showed in the other video that you can access you can ask the camera some of its settings and that's going to come back as a double and that will tell us what the FPS is set on the camera not the process but what the camera is set at and we're basically saying camera FPS is device SPS that we just calculated and put that into the label um and then track bar horizontal minimum equals zero track bar horizontal maximum is whatever the base frame width is and then track bar vertical minimum of zero track bar vertical maximum is base frame height so now we can do some calculations where we're we're using these track bars to say hey I don't want you to make a bounding box that's outside of the range shown by these track bars so the back the bounding box right hand limit is going to be whatever the track bar vertical value is because we only want it to be to the left of that track bar only objects to the left on the road will be shown same thing with the bottom limit is the base frame height minus the minus the track bar horizontal value again you don't need this we're just doing it if you want to um discard any areas of your image from having motion detection and then the last thing is we're updating the camera frame base frame width times base frame height so we're just updating the dimensions of the images coming from the camera and that's it for this timer again you may not need this um we're just showing you this one option to update the UI without having to worry about going into this 60 frames a second image capture and then the other important thing is this button capture where the user says Hey immediately grab whatever image is being displayed and whatever foreground mask is displayed this method grabs the present frame and mask images to a file with a file name that reflects the hour minute and second it was saved so we're coming up with this time stamp which is going to be the name of the file and it's daytime.now.our tostring with a D2 We want two numbers with an underscore plus daytime now dot minute to string with D2 Plus an underscore plus date time now dot second to string with a D2 so the name of the file is going to be the hour minute and second and then we're going to save it using frame dot save again it's another method in the Mac class and it's going to be image path that we defined above plus this time stamp plus frame dot PNG so it's going to save to that path with the timestamp plus frame PNG and then the mask is going to be the exact same thing foregroundmash.save image path plus timestamp plus Mass DOT PNG so they'll both have the same time stamp but one will be framed.png and one will be mask and then if we do that it will update the text box one and say we sent the images to whatever that image path is so that's it for the event handlers that we have to worry about again button stop all we're going to do is going to say no stop streaming video stream video equals false capture image grad we're going to unsubscribe from that event we're going to stop the capture we're going to disable the timer and we're all done and then button exit just as an application exit so that's it for the events and as expected they're just going to call these methods when we start up the application we're going to do this initialize UI which is just going to set defaults for the track bars horizontal and vertical value you can do that whatever you want we're going to set the check boxes as some default values and then the radio box we're going to default it to have the image being shown so that's basically just this uh initialize UI and then the important thing is going to be this process image and motion detection as we said when we we get a new image in the image capture it's going to call this process image after it displays the raw image and all this is going to do is going to do it's an async method called process image and it's going to launch a task now we did a video on async and await and multi-threaded applications and it's going to in parallel run this motion detection method you see below here and then allow you to access to go back and access the UI while this is going off on a separate thread so we can also add more methods here to do in parallel if we want later on but here we're just going to launch this motion detection so motion detection is going to be the method that does all the work and it detects motion and if the user wants it it applies thresholds filters and so on so this is going to be very simple it's just going to be one big try and catch and the try in our case we're going to re re-grab the existing frame now in the future we may look at this and and modify it but for now to make it easy we're going to recapture whatever frame is there and call it frame if frame is empty we're going to return we're not going to go through this and then we're going to do what we talked about before we're going to process the frame and we're going to define a new mat a new image called Smooth frame and that is going to do a CV invoke basically a opencv it's going to invoke an opencv method called gaussian blur and it's going to take our input frame and give us an output frame called Smooth frame and it's going to do a gaussian blur using a system drawing dot size of a three by three pixel square that it's going to blur inside that square so to start our our motion detection we're going to do a gaussian blur and then we're going to do the really really important part of this which is going to we're going to Define our foreground Mass which is going to be the grayscale image that shows all the detected motion so foreground mask is a new mat and we're going to use this back eye background subtractor we we defined in the parameters background subtractor we're going to apply and the input is going to be this blurred smooth frame and the output is going to be the foreground mask which is going to show all of the detected motion so at this point we have a foreground mass that has all this the detected motion and if you want you could enter right there it's not going to actually detect objects it's not going to have the filtering but really it's basically giving you your your background image your foreground mask however if the user wants to do some more processing like apply a threshold and noise filter if threshold Boolean is true we're going to call the threshold method if filter noise is true we're going to cause a call the noise filter if check boxes if we want the application to detect objects that we've defined like cars and people we're going to call this bounding boxes method and if we want to display the limit lines we're going to call this limit lines method so we've got a bunch of methods here that we're going to show that basically do the different processing threshold noise filter bounding boxes and limit lines and then what we're going to do is when all that's done we're going to do a resize we're going to resize the smooth frame to this 1920 by 1080 resize the foreground Mass to 1920 by 1080 and resize the input frame to 1920 by 1080 and then we're going to call a method to draw the mat so that is the the main Workhorse where we're going to actually do the processing and draw the images so the only thing that's left now is these methods that do the subsets of that so the limit lines if we're going to draw the limit lines that are controlled by the track bar as you can imagine we're going to have to say here's our frame and we're going to have to define the point if it's a horizontal line it's going to be from the left hand side of the image to the right hand side of the image we want to draw a line so we're going to use a CV invoke the line method that draws a line in this Frame from point zero which is the bounding box bottom limit to point at frame dot width which is the bounding box bottom limit and then we're going to use this MCV scalar which tells us what color we want the line to be and here's the vertical line same thing we're going to draw a vertical line in this Frame of this color so we won't go into that but that's basically if you want to do the limit lines then the threshold is very simple we're going to do a CV invoke of the threshold mesh method which takes the foreground mask as input foreground mask as output and then is going to remove anything below this range of 225 pixel value is going to be 0 to 255 so it's going to remove everything anything below 225 and above 255 and it's going to do a binary threshold so you can put these numbers whatever you want I just chose some and then we've got the noise filter and we talked about that we're going to use an opening operation which first erodes or decreases the white region of the image hopefully to remove any noise then does it dilate to re-expand the white region and hopefully this will move any random noise like our our moving leaves and that kind of thing we're going to do a CV invoke morphology ex the input is foreground Mass the output is foreground mask and we're going to do this open method and I'll let you look into the details of how this works the last thing is going to be bounding boxes and this is going to be the way that we detect objects of a certain size that that represent either humans or vehicles or something and draw bounding boxes around it now um maybe in the future I'll get into more details about this for now I'm just going to allow you to look at this and if you want to pause the video and type it in we're going to use a vector vector of points and those will be called Contours and then we're going to do a fine Contours in our foreground mask and then for each Contour that we've detected we're going to draw the bounding box and we're going to show the image or show the mask and again I'll let you take a look at this and pause the video and type it in but this is basically going to draw the bounding boxes around the moving objects then the last thing we have is for picture box number in our case we're going to do box number two is the right hand side it's going to draw a matte in the right hand in pixel box number two it's going to be a try catch if box number equals two which is the picture box on the right hand side if the user is selected show image picturebox2.images frame 2.2 bitmap otherwise if they said show mask picture box two dot images foreground mask a bitmap and that's about it so that's how you can make this motion detection application in C sharp Visual Studio Windows forms in the future you may get into a little bit more detail about this we're also going to look at instead of having all these methods here we might look at a generic image processor class so that we can expand this and make it a lot easier but for now um this works pretty well there is one issue that you may encounter when you run this after 10 20 minutes it may crash and there's a reason for that we're going to get into we're gonna have to do some modifications but we'll look at that later but for now that's about it for this one if you like any of these videos please hit the like button subscribe hit the Bell notifications but most of all please let others know that we're here so we get some views really appreciate it otherwise take care have a really good day thanks

Channel: EETechStuff

Views: 1,932

Keywords:

Id: _GuJffSTjBg

Length: 37min 0sec (2220 seconds)

Published: Tue Aug 22 2023