Eye motion tracking - Opencv with Python

welcome everyone to this new video tutorial we're going to see today eye movement detection I have a video which is this one where I recording my eye with my eye I'm working in different directions and our goal of today will be to create a script that can detect where the eye is looking fit looking looking in the left side right side top or bottom how can we do that let's first take a close look at the eye and let's analyze the image to see how we can work [Music] we can see first thing we add the pupil which is a circle black circle second we have another circle which is the iris even if it's not a complete fourth circle as this line is almost straight especially if the eye is half closed the pupil will not result as a circle and last part we have the sclera which is the white part we have on one side and second side what could use different approaches one could be detect only the black part or second one detect the circles the iris so choosing the the color of the the iris detect through the half methods have transfer metals detect what are the circles in the video or finally detect the white part so the as we know when the eye is looking on only one side let's see if I look there in this part there will be eyes and pupil and all the other side will be wide with the sclera if a look in the other side would be the opposite so this this is a quick explanation of how this how we can approach in different ways the the method to the detection method let's see which one is the best we import seaweed so we import numpy as impede we load the video cap is equal to sieve it so that we video capture here we need to type the name of the file and which is I recording dot f lv and you can download this video from the link in the description after this let's run the while loop while true red frame and cut that Reed we show the frames see v2 dot in show frame and then frame and then wait key event key is equal to C v2 dot white key 30 so we're going to say wait 30 milliseconds between each frame to show about showing each frame and then if the key is equal to 27 which is the S key on the keyboard and we can break the loop and finally we can write C which will destroy all windows let's run the script if everything is correct we should see this video and everything is correct we should select from this starting point top left and right bottom and cut it from the original frame so we have our I equals two frame and then from the frame we take from the position off from the row from 20 69 to 69 to 795 and from 537 to 1416 this means we are taking from the top we start from this point from 20 69 to 795 we starting from the left from 537 so from here so here we can't 437 pixels to 1015 so from here to here would be 1416 how do I know these numbers simply I look at them before and I found the area before making this video not to lose too much time with just this part let's show the arrow I only because it's the only part we are interested in right now see to let him show ROI and ROI and let's see what we get what we see is the eye and that's what we're going to work on now we have the eye first step that we can do is to make the image simpler by making it in gray frame format we don't need all the colors right now so frame will be equal to cv to dot CVT color we convert the frame C to that color underscore BG r to gray actually instead of converting the frame as we're working with the arrow I let's convert the ROI only so gray are I will be equals to suit city color the ROI and we showed off the ROI N and the greater I seated in show gray eye and then great are I we run the script and we have both of them let's focus on the gray format as I said we have different methods for this to find the movement of the eye one of them was for example to find the circles and using half transform I use that method and I can tell you that it doesn't work properly because there are too many circles in this image also this can be a small circle there are some circles that we cannot see but this the algorithm of algorithm sees and so it's not a solving method another one can be to detect the white part which could be a more solid method but what we are going to use is to detect the pupil let's analyze the image what has the pupil that nothing else has in this image the pupil is a black circle we can use a threshold and Siri and say to show only the the part of the image that are more black than some gradient so for example under score threshold is equal to cv - dot - all of the gray are I let's take all the values up to five and then 2055 seaweed so dot Thresh binary let's show and let's show the threshold save it to the team shop threshold and then threshold actually it should be the opposite the white part is the part that we are allowing so let's use Thresh binary inverse okay no it's matter this is the result of the the eye looking we can see that with just this simple approach we selected mas c/d iris the the pupil sorry except that we have some unnecessary noise that we need to remove so let's make this step first let's remove this noise we can we can blur the image to remove some noise so great awry we'd be equals to see if it so that Gaussian blur of the error in the kernel size it's 7 by 7 the greater the kernel the more blood will be the image the less noise we will have but also the less precise will be the detection so we need to find some balance let's try save 7 by 7 let's see what result we get we can see that we don't have any more those small dots that we had before and that was the goal of the Gaussian blur so we can say that the blur in this image it's okay once with this let's now draw the contours so from this mask from this threshold we can draw the contours on the region image and we can see if what we are detecting and later we can still improved the script of course before during the contours we need to find the contours underscore contours contours underscore again CV to that find contours we're going to find the contours on the threshold the mod is CV to that red three and metal c2 dot C I don't remember no chain aprox okay see it's not chained up rocks simple we detect the contours let's print you should know this already I'm printing it for the for the ones that are beginner and never use contours before contours it's an array inside of an array where we have we have all the points here actually the contours of the boundary of this object we have all the boundaries of Deeds this white part once we have the boundaries we can draw them on the original image so it will be for cone for CNT in contours see v2 dot draw contours we're going to draw the contours on the arrow I the cantos will be inside brackets CNT minus 1 which is to draw all the contours this function draw an array of contours you can you can define the index if you put 0 it will draw only the first one if you put 1 it will be it will draw the second one and so on we - 1 we're going to draw all of them after this which color with let's say red BGR it's zero blow zero green and 255 of red and the thickness let's make them three pixels we are done with the contours let's run a sweep and see what we get here we get the contours we can see a good proper detection of the iris at least when it's in the center position let's see when the eye move still still the detection it's quite ok I accept it it's missing some part there is one problem that still we have a lot of noise let's take a look at this image let's analyze what can we do and what was the difference between did the pupil that the corner of the pupil and the contours that are above we can see for example that the pupil the contours of the pupil are the area is much bigger than the area of all the rest you can see this bigger than all the rest we can use one thing which should be calculate the area and let's take only the biggest area of all the image once we take the biggest area on this one all the rest would be an accessory and we're going to skip it so it will be contours we have the phone on the function to calculate the area which is c v2 dot contours area but we need to we need to sort all the contours starting from the one with the biggest area to the smallest one so it we will have contours it will be sorted again contours key equals lambda are not going to introduce about lambda function this is only the method to sort the contours lambda is equal to x and then c c v2 dot contour area of x and then reverse equal to true this function here takes all the contours put it in order from the one with the biggest area to the one with the smallest area and then we are going to print them we will print first the one with the biggest area second one with the smaller smaller area except that we break after the first we draw the first contour biggest area we break the loop and we don't draw anything else let's see the result we see we are drawing only the contour with biggest area let's see what we get only the pupil still the pupil in this part there is some problem as we don't attack the pupil anymore but the part above the pupil this Sun falls detection we can fix this somehow that's for example trying to change the threshold so the threshold instead of five let's say three let's see if this solves our problem it's probably a bit better than before there is some process some small mistake when I'm looking that side but except that it works kind of okay we can still improve that using different some different approach but anyway even even if doesn't detect exactly the pupil it can still detect the position where the eye is looking so it it still works okay we are almost at the end one last thing we can define the center point and draw a circle here and it would be it will look much better to draw the center point I use some Orthodox method the world you can say ie I take a rectangle from the contour so it will be X Y width and I'd equals to C 2 dot bounding rekt of CNT so from the contours we get a rectangle not let's not draw the contours any more and more let's just draw a rectangle suited in a rectangle on the arrow I point 1 is equal to x and y and point 2 is equal to X plus the width and y plus the height the color let's make it blue 255 0 and 0 and the thickness to rectangle okay here we have the rectangle and now we can draw we can detect the center point and have some line that moves on the screen so we will have some more clear direction let's let's draw now horizontal and vertical lines the horizontal lines will tell us if we are moving our eye to the top the line will go up or to the bottom we go down vertical line will tell us if you can going to one side or to the other side we can draw the line just by getting the the center point so once we have this point we can draw straight vertical and straight horizontal line let's do it this way so see we through that line of the arrow eye and then point one point one x and y let's see what will be the point one the point one we start from the top of the image from the from the zero position on the y axis and from the left will be the x position so it will be first point X will be the center point of of the pupil X plus with divided by two and then zero from the top of the image and then the second point will start the X will be still the same X plus with divided by two so the bottom of the image to the bottom of the image so we need to count how many pixels the image has from the top to the bottom that we have the opencv function shape so let's see from the ROI we take Rose which is the ID columns which is the width channels which is we don't need no arrow I dot shape and for the height we put Rose the color will be 2 let's say green so 0 blue 2050 fine of green and 0 red and the thickness let's say - let's run the script and we get an error that error integer argument expected god flaw we get this error because we are giving the position not an integer number the pixel can be only integer if for example the width of the image is 500 and we 500 one for example and we divided by 2 we will get 250 point 0.5 we need either 2050 order or 251 so we need to convert this in integer and this one also in integer and we should solve the problem and we get the line as expected now we need the say same line for the for horizontal line see we 2.9 ROI in this case the X start from 0 so we will follow the same approach as before but inverse so the X will be 0 and the Y will be would be y plus height but integer integers of height divided by two and the second point will be columns calls to the white plus integers of height divided by two color green zero twenty fifty five and zero and the thickness to let's change the weight ke to first so let's leave it with normal speed and let's run the script and let's see what we get and this is the detection and I think it's okay we can for example so improve it more saying when he's moving to the left let's print on the screen left when it's moving to the right let's print the screen right so for example on the first frame we take the center position when the eye is Center so when we move we can calculate the exact position it's not necessary that I do that right now we can improve one more thing for example there is this type errors since the beginning because when the video finishes the script the loop starts again and trying to do all these things but without any frame we can either stop the video on making it loop again and we're going to stop it so if right in this case we have Brattain frame when there is when there are normal frames read we will return false results so if red is false we break the loop and that's the end so we will solve this problem and that's all this is all for this video if you have any question just put it on the comments

Channel: Pysource

Views: 113,042

Keywords: opencv, eye, motion, movement, pupil, detection

Id: kbdbZFT9NQI

Length: 26min 43sec (1603 seconds)

Published: Fri Jan 04 2019