Optical-Flow using Lucas Kanade for Motion Tracking

hi I'm Aparna and this project is about object tracking using the lucas-kanade II tracker for computing optical flow so how I'm going to go about explaining this project is first I'm going to start with some basic concepts and terms I will give a brief introduction to motion tracking and the challenges behind it I'll then talk about the methodology used in this project for motion tracking this will involve a brief overview of the steps involved in the algorithm or it will also involve a demonstration of the code in MATLAB so what is motion tracking motion tracking involves recognizing a feature of interest and/or tracking it through sequence of frames so salient feature here refers to any feature that is of interest to us in an image now there are various important applications like facial detection and surveillance purposes traffic estimation to figure out where there's excessive traffic all of these applications use motion tracking and another way to phrase this would be that motion tracking is when an object is monitored for spatial and temporal changes through a sequence of frames it is always important to know what are the important challenges involved while designing a particular algorithm for an application so some are important challenges that we could face while designing a motion tracking algorithm is the rapid movement of object across frames that would make it difficult to track sometimes the object would change its orientation across different frames and sometimes the illumination is not the same through all the frames and another challenge could be that when you're tracking complex features like facial expressions another challenge could be when you are trying to track an object in a background that interferes with the object that is being tracked now that we looked at the challenges we understand that there is a need to address these challenges and therefore there is a need for a robust and efficient object tracking algorithm that can possibly address these challenges in this project I am going to attempt to explore one of the techniques for object tracking this algorithm would involve two primary steps the first step would involve or using a suitable feature detection algorithm that would extract and detect an interesting feature in an image I'm using Harris corner detector in this particular project and the next stage would be to use the Lucas Kennedy algorithm for computing optical flow which can then be used to track the interest point so what is the hardest corner detect algorithm all about so this algorithm is about detecting corners which can be accounted as interest points or salient features in an image now corner simply put is just an intersection of two edges and this algorithm that is a high-risk corner detector gives a mathematical representation for this concept so this equation here is the equation for the Harris kernel matrix now using another formula we can find the corn Earnest value of this particular matrix R is the cornice value that I spoke about in the previous slide where m is the matrix that we calculated in the previous line and this formula represents what we call a corniness value which indicates whether or not a particular point represents a corner so mathematically if the cornice value that is are computed by this formula represents a value which is greater than a given threshold then we can categorize that particular point as here I'm going to go over the algorithm a little more in detail which would help me when I'm giving you the demonstration in MATLAB so the very first step in this algorithm would be to find the image X and the y derivatives which would represent the image gradient in the X and the y direction now once we calculate this in step two we find the product of the derivatives which is I X square over square and I XY now as you can see in this formula the matrix that the hardest matrix is represented by this formula for which you need to find the summation of the product of derivatives which would be the next stage once we calculate this matrix we then need to find the cornice using this equation now I'd like to say add a little note about this window function now this function in the simplest case can be understood as a single pixel or a single pixel window which is run across the entire image this is in the simplest case now once we've computed this matrix M XY we can find the corniness value using this formula we will find the determinant of this matrix and subtract it with K which is a constant and find the trace of this matrix when this corniness value exceeds a given threshold one can conclude that it is definitely a corner so here I'm going to show you the implementation of the Harris Khan algorithm also first idea the image use the private filter to calculate the image Experion and the image by gradient this is done by Calvin convolving the image with the appropriate filter and I then use God the Gaussian filters to smoothen the image after which I find I X square I wa square as I mentioned in step one and two now using the formula that we already discussed I calculate the coordinates value now the size of the filter that I have chosen a size five but you can pick whatever size that you want and this size is with reference to the maximum filter which is used to ensure that I don't get the redundant cornice values so this filter would check for its neighborhood and just return the maximum coordinates in a neighborhood and only if a particular corner nest value is the maximum within its neighborhood and if it is greater than a given threshold I indicate that it's a corner and I get the row and the column and in the indices of that particular column so let me just run this with a particular image and show you so this is the output obtained now that we've completed step one of the algorithm which was to use the Harris corner detector to detect salient features in an image which was the detection stage we will now move on to the tracking stage now for this tracking stage we use the Lucas Kennedy algorithm which helps calculate the optical flow vectors now what is optical flow optical flow is really the motion of features across frames due to the relative motion between the scene and the camera now when a pixel gets displaced which happens when there is motion or these vectors indicate the direction of the displacement now this computation can help us in motion tracking consider an image I of X comma undergoes a displacement that can be represented by u comma V we can then represent this image once it undergoes this displacement by this equation where H of X comma Y is the image after displacement U and V represents the displacement of the pixel now the idea is to use this equation to find the motion or the displacement at every pixel which is u comma V now how do we use this equation to solve for u and V now I using Taylor's expansion and introducing some important constraints Lucas and Kennedy found that this equation could indeed be solved to find U and V at every pixel so some important constraints that they introduce is one is the brightness consistency constraint where he assumes that all the pixels have constant consistent brightness through all the frames another important constraint that they introduced is a spatial coherence constrain where he assumes that um all the neighbors of a pixel have the same displacement the same U and V values now using these constraints they found this equation where which could be solved to find U and V which is the displacement here I'm going to go over the algorithm in detail which can be used to compute a optical flow using the equations that I explained in the previous line now the steps involved um are as follows that is the first step would be to find the image x and y derivatives we then find the image difference between the two images now note that the two images that I talked about is two consecutive frames in a sequence of frames the next step would be to smoothen these components that is the image IX component the image iy and the difference image now linear equation for each pixel should be solved and we should calculate the eigenvalues and depending on these eigenvalues we would solve them using the Kramer's rule and then plot the optical flow vectors obtained now this would be clearer when I explain and show you the demonstration in MATLAB here I'm going to talk about the course to find registration so the nature of the locus conadi algorithm is to track very small changes or motions across frames now in cases where the speed of the motion is too high or the displacement is too high we could use the eye trait of approach where we represent the image in Gaussian pyramids of different levels or that is yeah as I said it's helpful for our tracking or larger movement now the idea is to this the solution to be able to track a larger movement is to reduce the resolution of the image which is what we do with the pyramidal representation so here I'm gonna be talking about the implementation for motion tracking using Harris corner and the optical flow using the lucas-kanade ii method so this follows the it--it of approach where I construct a three-level causing pyramid for the input images so image 1 and image 2 are the first and the second frames in a sequence of frames and I construct three levels for each of these images so if you go to the implementation of the Gaussian pyramid script here all it does is it reduces the resolution of the input image by three levels so it just it's a very simple implementation that uses sampling to reduce the resolution of the image and I write these are reduced images two separate files that can then be read by the next script so the first step is to find the Gaussian implementations of my input image and then I call the optical flow code so now this optical flow what it does is in steps that I already mentioned before so first I read the two frames and then these are all some simple initializations that are needed for the next few steps I find the image X and the image Y derivative by constructing some kernels now there are many ways to find image x and y derivatives that can be explored I'm using a Gaussian X kernel and a gaussian why kernel applying using the filter to function in MATLAB to apply this filter and get the image x and y derivatives I then compute the difference between the two image frames or was a step two that I spoke about in the previous slide once the difference is computed I smooth in all the obtained images or using a Gaussian kernel now once this morning is done I apply the Harris Corner algorithm or to obtain the rows and corner column indices of the corners now the number of rows and the columns will obviously be the same and hence up I'm iterating over either you can use either the row length of the column length so I I trade over all the hard Harris corner and this is obtained returned by this algorithm now for each of these corner points I compute and calculate this matrix it's initially initialized to zero all zeros and then I keep getting the sum of the pixel and it's neighborhood computations for these products so this would give me this matrix which using which I can use the Kramer's rule to obtain the values of U and Y so let me quickly show you what this would look like let me run the motion tracking algorithm so this is the output obtained now these are the corners detected for the three levels of the Gaussian pyramid for the input image and these are the corresponding optical flow images now as you can see for each of these corners I've found out what the optical flow looks like and the this graph is quite accurate in in indicating the motion of the two objects that mainly of interest now these are the lower resolution images and if you find if you look at optical flow vectors you'll find that they still represent quite accurate information especially this you'd be surprised to see that it does represent information that is accurate to an extent so what we can take from this is that a lower resolution could be definitely be used for to find optical flow vectors especially in cases where the speed of the motion is too high when a higher resolution image would probably not give a satisfactory results so in conclusion I evaluated a motion tracking technique that involved combining both the highest corner detector and the lucas-kanade II optical saw method and I found that this combination technique is more efficient than a technique that would involve finding the opticals for flow vectors for all the pixels in an image or the efficiency improvement obtained was 65.5% when I compared a technique that did not track only corners and and found the optical flow vectors for all the pixels in an image so in applications like embedded applications where the execution time is critical it would probably be desirable to have an algorithm that would track only those feature points that are of interest to us another important conclusion is that for images where the speed of the motion of the object is very high we could use the eye trait of approach where decomposing the image into role lower resolution images and then tracking those images can provide sufficient results for us in case of higher speeds of motion thank you

Channel: Aparna Narayanan

Views: 27,378

Rating: 4.878788 out of 5

Keywords: YouTube Editor

Id: 1r8E9uAcn4E

Length: 18min 14sec (1094 seconds)

Published: Sat Dec 10 2016