Photogrammetry I - 16b - DLT & Camera Calibration (2015)

okay so in the second part of the course today we look into camera calibration but you know not in using a known 2d pattern so assume you have a plane on this plane you have this checkerboard pattern so just black and white spheres why check our board because you can very easily identify those points we have a large number of those points they all lie on the plane and we typically can it seem to know the size of this checkerboard so we know the size of the of the whole checkerboard and therefore also the locations of those points on that plane so it's just the 2d pattern could also be a different pattern but often when you use such a checkerboard patterns because they're really easy to use but the key thing is we have a known 2d structure a 2d pattern and we know its size and its structure and we can identify points on this plane and then we know where those points on the plane are in reality yes but we what we actually try will do is we will explore so the question was if we could also use the shirt of the guy holding the checkerboard we can't because we will make an assumption that the points lie all on a plane and this is not a plane oh typically not but otherwise that would be possible so if the guy would have a perfect square body and that you should put foot what fit very well may be but in this case I think it's better to rely on the checkerboard okay so we can do is we can identify all those points in here and we can now use a certain trick you say okay I know where those points are on the plane if I set my world coordinate frame in a smart way I can ensure that I know the world coordinate of all those intersections here how can I do that which trick can I apply to not two for every image apply to this image and own reference system an old-world reference frame or object reference frame so they know the location of all points in the world any idea so I know where those points are on the checkerboard because the checkerboard has known size the structure what I can do noise okay so let's simply use this point over here and that's our reference frame so that this is the x and y direction and that is orthogonal to the plane then I know that the set coordinate of all points is 0 and the XY coordinate depends on the point on the checkerboard and I know the checkup what I know the 2d pattern so what I do is identify those points take one of our points say okay that's my coordinate system and then I know they all have a Z coordinate which is equal to 0 and give this given I know the size of that checkerboard I can also compute actually their XY location on the checkerboard it's kind of just a very simple trick of taking a 2d pattern and saying ok for every image of the checkerboard gets an own Walt reference system and simply the checkerboard to find the reference that system the coordinate system the reference frame ok let's see how this simplifies our math I would do that if it wouldn't simplify all math ok again my calibration matrix K my matrix containing the extrinsic parameters the point in the real world on the checkerboard and where this pixel is generated in the image so what I yes all points on one plane this is something where the DLT would fail when we now put them explicitly on the plane and will last you know how we can exploit that fact so we said the Z coordinates of all points are 0 because that's how we find our reference frame so this guy is 0 for all points so let's exploit that because this element is multiplied with this vector so all those elements or if I multiply and sorry this vector with this vector this vector with this vector and this the resist nectar those are the elements which will be multiplied with that equals zero so they will simply be eliminated so they don't matter whatever values they take they don't matter it's okay perfect let's simply delete that simply delete this column over you and then our matrix simplifies oops the Zetas got over here the third column has gone over here now I say okay I'm still of no genius coordinates I have something in 2d year I have a transformation which is linear and I have to do over here so perfect what I actually have is I have a mock Rafi in the projective space two dimensional spittoon well this weird projection is gone it's three by four matrix is now gone by just setting the coordinate system in a smart way so and I have this matrix over here it's no no genius matrix so how many degrees of freedom does this matrix has why nine perfect eight so nine is intuitive choice because the 3x3 matrix for nine elements but it's no genius matrix like this scaling factor this one over here so it boils down to the setting eight degrees of freedom okay so let's have a closer look to those elements in here I can describe this as a matrix K my calibration matrix and a matrix which now contain contains two columns of my rotation matrix and my translation vector T occurs the third column was eliminated can actually writing this in this way so I can actually write the system exactly in this way so I have my two image coordinates I have the three column vectors of the homography which is just those two guys multiplied with each other corresponds to the XY one I get that for one single image like this for sorry for one single point like this but I can do this for every point on my checker board so we do that for all my points on my checker board I simply have this system over here and for I different points on my checker board now one estimate this home ography over here how do I do that how do I proceed the matrix over here at some points over here which we measure and we have some points over here which I know show my control points on the checkerboard if you press the rewind key in your brain and go back for thirty minutes maybe but exactly the same problem in the DLT the only difference was that we had one vector more in here right just no unknowns P was at 12 elements in this matrix is nine elements who cares you just go unlock ously to step two and three or one two and three of computing the DLT so what we have in the end okay instead of estimating the 3.4 projection matrix we estimate a 3x3 homography and the rest is identical just everywhere where that coordinate is the set coordinates can go away because we don't have that anymore and we don't have the third column of my rotation matrix anymore so again again define my weird vector a I don't multiply it with MSP just with H because H is now the staked vector and so it subject is defined the same way except that P here before and okay now you know where the cross is coming from there a few elements in this vectors a which are not relevant anymore those which correspond to the Z coordinate they simply die and the rest stays the same we again have this we set up our matrix M we compute the SVD and here we are so if exactly the same thing before just all just kind of the same slide as before just these vectors a are kind of shrunk now we're crossed off the coordinate so I do exactly the same what I did before I do the DLT like estimation except that my vector P is now smaller three degrees at three unknowns less 20 of nine instead of 12 but the rest basically stays the same so what we get if we do that we solve the system of linear equations by minimizing the squared error using the SVD and obtain our solution we have a degrees of freedom that we have in there so we identified correctly technical normalization appropriately into account as before we measure to the locations of points in my image so I get two equations per point so need at least four points on the checkerboard in order to estimate the scenography okay so at least four points are needed in order to do that typically it's like that in this case we have to find our coordinate system so six degrees of freedom go into the coordinate systems so two degrees of freedom are left that contributes to my calibration matrix so we'll need actually in the end as we will see we will need multiple images in order to estimate all the intrinsic parameters okay okay let's go back and say okay about your now I need to do is we need to compute from matrix H so we computed H we want to estimate all the individual parameters one of the things we can't do with a trick with a QR decomposition now because this matrix is a lot of rotation matrix anymore so we can't just apply the QR decomposition as we did before it simply doesn't work in here because this is not a rotation matrix anymore well it's not a bit more tricky now so we needed a little bit invest more brainpower into this part what we need to do is now we will define some constraints some of the things we know about this matrix in order to set up a new system of linear equations solve that in order to estimate those parameters right okay so we start at this point we know all the elements in H so H 1 H 2 H 3 are known that's what I got from the previous step from the DLT like estimation and now my next steps is turning that into my matrix K which I actually want to have and maybe also the extrinsic parameters or parts of them find so far everyone knows what to do good ok so what we do now is as I said we we generate now certain constrains we know about those elements especially those elements which sit in here to to exploit those constraints in order to estimate the different recalibration the our matrix K with the intrinsic matrix of the intrinsic so what do we know the first thing we know is given by a we apply this how is our how are our and H related things if they are one this one this K to the power of minus 1 times H 1 just get that if I multiply from the left hand side plus K to the power of minus 1 so this matrix goes away k to the power of minus 1 H 1 I could do the same thing for our 2 in H 2 so it's clear how obtain those equations just by multiplying this line with K to the power of minus 1 from the left ok so what do I know about R 1 and R 2 where 2 r1 and r2 come from what do they come from what are those lilies yes yeah so these are the elements of the rover rotation matrix so what do we know about if you take two columns out of rotation matrix what do we know about those two vectors nearly there so rotation matrix take out two of the columns of the rotation matrix what do I know about this rotation matrix of those two vectors they are formal right the first constraint they are orthogonal was the second thing I know about their length length one now two constraints I want no exploit so r1 times r2 is the dot product r2 should be zero because they are formal and the norm of those two should be one okay so these are exactly my two constraints that I want to exploit now let's exploit those constraints so this is our one this is our two so if we transpose this guy multiply it with this one it must be zero right so we can exploit the first constraint this is my constraint so they are zero everyone happy with this constraint this constraint just results from the fact that our was a rotation matrix and I could relate the elements of the hallmark rafi with the inverse of the calibration matrix which I don't know at the moment but I want to gain knowledge about it and the columns of my rotation matrix so those guys I know and I want to know those so getting closer the next thing I can't exploit is that they both have the same length so let me say simply take R transpose R 1 transposed R 1 is one is the same as our two transposed or two because they're the same length or length one the same things the second series so this is our one transposed r1 equals or two transposed or two what I can do is kind kind of subtract this element from both sides so I get this one - this one equals zero it's exactly what I've done here so this one stays the same and this one over here the negative part equals zero okay so what I have now is I have two constraints and what I now want to do is I want to exploit those constraints on estimating my matrix okay okay so gold is now we have our MOG rafi already we want to estimate my matrix K because this this contains the intrinsic those elements are interested in its please transpose of the inverse so I'm yeah so this is Kate - the power K to the power of minus T is the transpose of the inverse of K okay so now just to copy paste take those two and copied on the next slide and add that in here said okay in all these equations I have this Viet matrix K transpose of the inverse times the inverse so let's simply define a new matrix B say that's B that's B that's B so just simplify that this is what now what I want to solve and that's fine if I do that because if I want to SC if I know if I know B from this matrix I can actually throw a Leske decomposition this time recover the structure of K and the reason is so this matrix the transpose times selves is a symmetric positive-definite matrix even positive definite not only positive semi-definite because I know the main diagonal of K has Valtor nonzero so I can directly and I know so I know that this matrix P has nine elements but only six of them are distinct because the three elements of those elements over here right if I know B I can compute the Telesca decomposition basically because this is a transposed so that a is the inverse the transpose of the inverse well then what I get out of years gives me the transpose of the inverse I need to transpose it invert it and I get my matrix K ok so what I need to do is I need to determine these parameters these six parameters from these equations so take this block take those equations and actually solve them so this is my on vector of unknowns I have only six elements not nine as before even twelve s before me six elements and what I can do is I can construct now a system of linear equations in the form V times B equals zero where I set up define those V's by using the elements of my heirloom ography and some vector V that I will define in a second exploiting the constrains that I have the first constraint resulting from the orthogonal basis and the second one resulting from the norm one so the matrix we is basically two constraints so this so the first one was that this guy okay so maybe that was I took quick I guess it was a bit bit of a bit of fast so I had my two constraints so these are the equations so this was supposed to be 0 this had the index 1 and this at the index 2 and this is supposed to be 0 index 1 1 and X 2 2 these were the constraints which they resulted from the rotation matrix then I was here I said ok and this is a general mapping which resulted from from the month from from the homography form the multiplication of the homography originally so I said if this was 1 & 2 so if I equals 1 and J equals 2 this must be 0 so we want 2 times B must be 0 this was the first constraint it's resulted from the earth that they are orthogonal and then the second one was if I had this element was 1 1 minus 2 2 should also be 0 so V 1 1 B times V 2 2 B should be 0 there was a second constraint said ok I have two perfect constraints that are now can exploit and I simply build up a matrix H it says V 1 2 and V 1 1 minus V 2 2 all transpose times B should be 0 there V is defined exactly in this plane this just comes from the multiplication of the homography so what I have now I built up I defined this vector V and with this vector V I can very easily define for a single image those two constraints so I have one image I can do these multiple images and can stack n images on top of each other so and every image I estimate the Ahmad Rafi and for every image those values will be different because those consist of the homography so I broke this up in a chewy with 2 having 2 n elements in here because every image contributes those two constraints another system of linear equations which I in the end can solve and obtain the solution for B once I know B I can recover K so what is needed I need to have at least three images like this of course the more images the better it to be get yet in every of those images need to be able to identify at least four points I need these four points per plane to compute the matrix H to compute the Amalfi so how this plane is mapped up to the image each plane gives us two equations if we're exactly the equations I had before as a result of that as this matrix B has six degrees of freedom six elements I need to have at least three views because three times two equations gives me the six degrees of freedom so I need four points per image three images of that type and then I can solve this system of linear equations which gives me B and once I B I can easily compute K just by chill SK decomposition okay exactly the same things happen at before what I want to have I want to solve the system of linear equations as we have again the trivial solution B equals B 0 which I'm not interested in because this would be a calibration matrix which is 0 I impose the additional constraint of norm 1 and then so what I'm basically doing I'm minimizing this equation which is again the eigen vector problem very very similar to what we did in the DLT so you see singular value decomposition take the eigen vector which corresponds to the smallest singular value of the smallest eigen value and then I obtain the solution for B which minimizes this expression that's it now we have our calibration matrix we're done with the chancel with a checkerboard pattern at least three images the cadet identify all points actually can compute the intrinsic parameters of course not to get a robust estimator should take more images it shouldn't see more than four points per image but it's kind of the minimum that I need now I can't even go a step further and say okay now I have all that done can we also estimate the nonlinear errors so last week we talked about nonlinear as like Radio Distortion for example they are simply a nonlinear mapping which is added after applying the calibration of the direct linear transform so the mapping of the direct linear transform we have the additional other distortion parameters and if we know them we can correct for example make straight lines straight again which are mapped as non straight lines due to the nonlinearities so there's an example model I want to go to the details this is basically what we we had the radial distortion model which has the individual parameters and RS again distance from the principal point I don't want to go in all the details so kind of you can make those models complex and more complex we basically do for do first the projection onto the image and then apply it with correction the key thing is we have some nonlinear parameters in there and once we have the calibration matrix we can again apply another standard least squares approach to that and say okay what we want to minimize is adjust the parameters of my camera calibration and I take my the ones I computed as my initial values before I have now this nonlinear parameters and my rotation matrix and so where the camera is and where the center of projection is and I simply say what I want to minimize wrong font I'm sorry this should be homogeneous vector so over it goes over images over images I and points J and this is kind of the nonlinear mapping so where's the estimated the the appoint expected to be given I know where it is given the position of the cameras respect to the world reference frame the nonlinear parameters in the linear parameters and then I go with a standard approach obtain a quadratic function i linearize it obtain a quadratic function compute the first derivative set it to zero and iterate and in this way are typically obtained I refine my calibration matrix K and I get put nonlinear parameters and the complexity of this function over here simply results from how complex I make this one lower here this just one symbolism one example they could be more complex they could be easier that's just kind of the radial distortion one time ventral distortion allows kind of standard things to compensate especially for wide-angle lenses and if I do that these are kind of the results so this is my before calibration I take my images and then I can actually correct for that and get my calibrated images out what we can see over here for example that this guy this line which is here clearly not it's not a straight line is a nice straight line over here also we can nicely see that here that those lines are now also mapped into straight lines just example how this was done for simple webcam just holding a webcam in front of the checkerboard I can estimate the parameters also the nonlinear ones through a standard least squares approach after estimated the linear calibration matrix over those five parameters so to sum that up what we have done in the beginning in the first lecture today on the first hour today look at the deal team where we don't use any checkerboard pattern we just need to know low locations of some random points in the scene which should not lie on a plane the second part we said okay now let's simply simplify this process of estimating where those points are in the real world just put everything on a checkerboard something we can print out on a printer glue on a flat surface and hold it in front of the camera by smartly setting the coordinate system we could actually simplify those equations and estimate the parameters of the the calibration parameters of this camera the only thing we assumed was a pure pinhole camera model at the first round the second round we can add nonlinear distortions and estimate the parameters of these nonlinear distortions with a second least squares approach with this approach we can actually do camera calibration or just printing out a piece of paper with the checkerboard gluing it on a flat surface holding in front of the camera estimating the intersection of the red and the bird white and the black cross had those points and we are done this is kind of the standard metal of calibration toolbox actually works in this way you can use to estimate the intrinsic parameters of your camera that's it basically from my side what is an outlook for next week what happens next week is we assume to have this calibration matrix so we have done this calibration now I have calibrated my camera and now I take images of the scene and I want to estimate where the camera are given I know a few points in the scene and it turns on we need three or four points in the scene in order to estimate where the cameras if we fix the calibration parameters that's what we are going to do next week the first part for the DLT is both grants script but one of the best references for and the Sun this is the original paper from 1998 on performing this camera calibration which we have seen in the second part of the course today there anything any question I can answer all done okay yes please yeah okay so the question was if we once calibrate a camera if the camera keeps its intrinsic surah this depends on the quality of the camera extremely cheap cameras this can change also if you have a good camera you drop it on the ground you may need to recalibrate that the more expensive the cameras typically are the better also the mechanics and the lenses are in there the less often you need to recalibrate them if you have a zoom lens you need to recalibrate it every time because the camera constant changes it changes the view zoom but if you fixed soon decent camera can calibrate it once unless you expose the Twiggs she extreme temperature changes for example so if you calibrated at room temperature and go out at minus 20 degrees just due to the temperature the parameters of the camera changes that's something that you can experience an area cameras to do aerial photography the cameras are calibrated indoors let's say 20 degrees Celsius taking the airplane fly very high it can easily go down to whatever minus 30 minus 40 degrees Celsius and then the camera parameters change therefore for those applications may actually need to estimate or re-estimate those parameters very often even during a flight unit which you need to adapt the camera calibration but if you're it's a standard indoor settings and you don't drop your cameras on the ground the camera is lot the most cheapest everything was a fixed focus lens you calibrated once you're happy with it any further questions and this is not the case thank you very much for attention and the last lecture of this summer term for the photogrammetry course will be next week thank you very much

Channel: Cyrill Stachniss

Views: 31,223

Rating: 4.9101124 out of 5

Keywords: robotics, photogrammetry

Id: Ou9Uj75DJX0

Length: 32min 40sec (1960 seconds)

Published: Thu Jul 09 2015