Essential and Fundamental Matrices

hi in this tutorial I'm gonna talk about the essential matrix in the fundamental matrix and computer vision how to find them in the properties so sectional matrix and the fundamental but before we start let's review some stuff together to see if you're on the same page okay imagine we have two cameras and this is my first camera this is the image plane of the first camera this is my second camera [Music] and this is the optical center of the camera on the left and this is the optical center of the camera on the right and this is a point in arbitrary point in the world i denoted with capital P okay so the Rays of the light from the point P will go through the optical center in the first camera and hit the camera image plane in this point for instance and also they will go through the snake on camera and hit them in a point like this so when we look at this camera when you look at the image plane we can't say for sure if this point was actually here here and call it p1 p2 p3 we can't say for sure it was here here or here because we lost one dimension so this point could be on any of them and if I project them I will get a another point here in this the second camera plane so the let's just review the names this line that connect these two optical center is called base line and the point that this line hit the image plane is called a peephole and as you can see these ray of lights which go through a p2 p3 and PM which could be any of this point on this line they will form a line and this camera so for this point P we have a line which is associated with that and the second camera because we don't know where is the line where is the point exactly on this line and for every point on this line will get a point here which eventually they will form a line and always one side of this line will go through the air people you can see geometrically if I choose another point here eventually one had one one side of the line will go through this pole this point which is called a peephole so this is some geometrical constraint we later on we will visit that algebraically and we will see that this is this constraint is true also algebraically okay so this is called a peephole this is called a epipolar line for point P and this plane was called a peephole our plane if I move the P up up and down I'll get another plane but it always go through these tentacles okay that's the first thing that we need to review the other thing is the equation of line in 2d is something like a X plus dy plus C is equal to zero I can rewrite it as a BC multiple by XY one equal to zero so this can be seen this a B and C can be seen as they are the component of the unit vector in the direction of normal between the origin and the line and the every point on that line I want to start with that as a distance D from the origin for instance if you have a lot and if you have a line X equal to Y and write an X -1 equal to 0 this is a line and a and B are 1 and minus 1 so it is gonna be the unit vector and so I can write down like n of X and of Y and minus D so here did this ABC could be seen as the this vector a and B 1 and minus 1 and the distance from origin is 0 which is minus D so we are looking for some duality between line and point because you can see here we have lost one dimensionality one one dimension we here we had a line and here we had a point so there's a duality between line and points the real world and in the image plane ok the other thing that we need to review before going further is this notation is the cross product product between every two vector is 0 this is something is pretty simple and the cross product between one Waechter and the other vector can be written as in a matrix form with this notation so here we have two vectors and this is a cross product and we here we have a matrix a which is something like this a sorry 0 minus a 3 a 2 a 3 0 minus a 1 minus a 2 and a 1 0 by E 1 e 2 3 so I just turned the cross product into a matrix multiplication if you do the cross product you will see exactly the same result and the rank of this matrix is 2 this is something we have to remember we're gonna need it soon and what is the rank of the matrix rank of matrix is the maximum number of independent rows or columns and 4 by 3 by trees at most 3 and here the rank of this matrix is 2 and if you multiply this matrix by any other matrix the ranks gonna be the they need a new rank to the rank of the matrix gonna be 2 again because obviously a small is right ok now you're ready to talk about the central matrix so essential matrix is a matrix that relates to cameras the points of the frame of two cameras together first if this is your first camera and this is your second camera and this is the optical center we're looking for a matrix that relates this frame which represent the camera frame in one the right camera to the left camera so if this is if I call this frame XL this one X R we know that for any two frames in 3d we can translate and transfer we can transform one frame to another one by rotation and translation so X of L is equal to 3 by 3 rotation multiplied by X R plus T so if I cross product both side by T I'll get R plus and I just said this is 0 okay now if I dot product both side by X L sar so the result of this cross dr. cross product as a vector which is perpendicular to the bows to the XL and T and if I multiply dot product by a whipped by XL which is perpendicular to that the result going to be 0 so these are going to be 0 and this side gonna be I can write this dot product as a matrix just by and here I can again turn this cross product matrix operation that I just introduced earlier in the previous page okay so I can this is three by three matrix now of rank two this is three by three and these are two two points three one so I can call these E and rewrite it as this next no II X R transpose sorry so this is the definition of essential match that as you can see relates the frame of right right camera to the left camera so that was the essential matrix now we are introducing fundamental matrix fundamental matrix is a matrix that relates something in the image plane from one camera to another camera okay so let's write fundamental matrix okay we know that if you have a point in world coordinate system like X Y Z I'll tell you if I multiply it by four by four matrix which translate the point transform a point from the work order into the camera coordinate I can get the point in camera coordinate so is there any rotation and translation this will give me the this is P in world coordinate and this is simply for what for translation which give me the point in the camera frame and if I multiply this by extrinsic matrix sort of intrinsic matrix I will get the point and the peaks on the image plane based on the pixel and you remember it was f of s multiple by year we had same origin the origin of X origin of Y and here we have F over S which basically gave me how many pixel per meter or per millimeter that I have F of Y or s and here was the skew parameter which for modern camera we usually put it 0 and the rest is against zero so if I have a point in world coordinate this will give me the camera coordinate and this will give me in the pixel based in unit of pixel on the image plane okay so I can write down the point in the image plane is equal to K which I this guy and multiplication of this by this point in the world coordinate multiple by is four by four matrix with a point in the camera coordinate so I can write the inverse and I can write it like point in camera is equal to K minus 1 and the image plane P okay so one point in the camera coordinate is equal to K inverse multiplied by point in the image plane so if I just plug these P for right and left I will get K minus 1 P okay for for the level just put it L here because the camera might not be exactly identical P of image for the left transpose multiple by essential matrix multiplied by K of right interesting metric to the right camera multiplied by point of image plane in the right equal to zero so if I just extend this transpose you know that for photometric is gonna be from right to left so it's gonna be point in the left h k inverse in the l e K R inverse and P R is equal to 0 so I can rename this as if call it I sorta have a tea here because he read the transpose P L transpose multiple by F multiple by point in the right camera equal to zero so this is a fundamental matrix which relies something from the left image plane to the right image plane and the essential metric was relating basically these two freight to each other okay so now let's see how can we find the fundamental matrix earlier I said this operator has rank - and basically our fundamental matrix has a rank two here it's a three by three matrix but it is the rank two so if I want to write it down it's something like this one two three [Music] so basically a four-for-one given point if I write the situation it's going to be you [Music] we won multiple by training by three or sorry should be 3 by 1 sorry 1 by 3 u b1 multiple by F multiple by corresponding point you preen we print 1 is equal to 0 so now we're looking for this F which is 3 by 3 and has it running - we can divide everything by F 3 3 so it just give for just for scaling so we have actually eight unknowns so if I find the equation for 8 unknown points and rewrite it I will get a big matrix like this you print one you one you print one we won I'm just multiplying this by this this by this and this by this and then by this and I have to write the equation you prin beep ring 1/u 1/v print 1 V 1 we print 1 u 1 V 1 1 if I write it 8 times 4 basically 4 points I will and rewrite everything like this you get f11 f12 to F 3 3 basically I have a non-homogeneous equation I cannot use the least square to solve it I have to use SVD singular value decomposition to find the F so if I use the SVD I can find the F and remember beside is 0 you cannot use the least square if I use F if I use SVD I will find F but as I said earlier the F has rank 2 and what we have here is the F of rank 3 which we have to somehow impose this constraint to the F so we could get the correct F because the F that we get right now has rank 3 and if you plot the lines and point you will get some wrong lines so how do we impose that we compute Lee we compute the SVD for F again I mean this is this is another SVD we use SVD to rewrite the F the first time we use to find it the second time you ready to decomposed to you the V transpose so this matrix is a 3 by 3 matrix which the value and the this diagonal are s T and 0 are the eigen value of F and they were sorted like this descendant Li so R is bigger than s bigger than T so we set the t to the 0 and we call this a new d d hat or as i mean these are 0 already then you sit is to 0 as well so we compute a new F with D hat and call them f hat and this is the F or the fundamental matrix that we needed this is a correct one with imposed constraint okay now how to as I said earlier geometrically revisit that we saw that that for for every point that we had in one camera plane we had a line for for for this given point here because we didn't know that this line gonna be here here here or here we actually get a line which is called epipolar line so here if you have a look in the equation of fundamental matrix P transpose F P prime is zero this is capital P this is a P 1 into and is a rate projection and here this is L and L prime because this we can write it also for this camera and get the other line here so if if you remember from the first slide I said we can write the equation of any line in 2d like the weekend we can see it as a normal which is perpendicular to this point and when I say perpendicular point I mean the to this ray of light which goes through this point to the optical center okay so PT L is equal to 0 is the equation of the I consider the equation of the line and because these two are equal I can rewrite it as L equal to FP Prime and basically it says for a given point a given point P Prime here I will have a line and the this is L prime is 0 for a for a given point here in this camera frame I will have a line L and the other camera also other way around we could write an L prime is equal FP so for for this given point here we have a line this is something with geometrically saw that and your LG of product Li we can reach the game okay and the other thing is about the fundamental matrix is as I said it has rank two are the about the AP polls so as we geometrically again we saw that every point will every epipolar line will go through the AP poll so how can we find it so because this has to be true p t transpose f e prime because for every line it will go through that and it has to be true for every line so regardless of the point so this value should be 0 so if I set F P prime equal to 0 I can find the AP poll so by setting this to the 0 you can find also the applicants thank you for your attention

Channel: Behnam Asadi

Views: 12,158

Keywords: Essential mtrix, Fundamental Matrix

Id: ggzJsYuCOeY

Length: 28min 0sec (1680 seconds)

Published: Sun Mar 04 2018