CVFX Lecture 24: Structured light scanning

so today's lecture should be kind of fun this is a lecture about different ways of doing what's called structured light scanning so last time we talked about lidar scanning and time-of-flight cameras and especially for lidar that's really what you'd use if you're scanning something that's as big as a building or maybe a car or something like that but you know that's not the right technology if you're scanning something that is on your tabletop right and we know that there are lots of things especially for visual effects that we want to scan there more at the person scale you want to skin you know a prop or a you know something that the special effects Department is created or you want to scan an actor in costume for later use making some sort of digital double and so the key idea for most of that kind of type of scanning is called structured light and so the idea is that what we're going to do is we are going to shine light into the scene and we're going to look at how that light deforms from the perspective of the camera and we use that deformation as the basis for determining the 3d shape okay and this picture kind of sketches how it works so what you see here is on the left hand side you've got some sort of a lasers like the light stripe projector this could be a laser that is spitting out a very thin red line it could be even a slide projector like an old-school kodak slide projector with one white line that lets light come through or you could use a LCD or a DLP projector to basically just have a single sharp 1 pixel wide line but you project into the scene there are lots of ways you could do this typically I think for really high quality scanning of this type we're usually going to find different types of lasers stripes and so how does the deformation of the stripe give us some sense of what the 3d shape is so you can kind of tell that you know when the stripe is following a surface that is kind of smoothly varying you get smooth deformations in the image of the stripe when it goes across something where the surface normal changes you know where the first derivative is not continuous then you get kind of a kink in the stripe and then when the stripe actually falls on to physically different surfaces you get a discontinuity in the stripe okay and it turns out that fundamentally you know figuring out how the image of the stripe corresponds to the 3d position is just basically a matter of calibration not that different from the type of camera calibration that we talked about that in Chapter six I guess right so here's a better picture of that so if you think about the laser stripe and then think about what the camera sees right so the laser stripe can only fall somewhere on a certain 3d plane in space so basically all the possible points the laser stripe to be visible are only kind of in this sheet in 3d space okay we know that's a plane in 3d and the image plane is also a 3d or there's also a plane right so we have as a projective transformation or homography between the possible points on the laser plane and the possible points on the image plane and so if we know the relationship between those two planes that if I see a point in one I can infer a point in the other and so what I get is kind of what I'm doing is I'm going to say okay I saw my laser stripe point hit here and now I can put that back with the projective transformation into the coordinates of this laser plane in 3d where I've kind of drawn the XY axes to coincide with this laser plane and so I guess let me switch over to my piece of paper here so structured light scanning and so the first method we're talking about is basically with a laser stripe and the fundamental idea is to compute the projective transformation between the plane of the laser in 3d and the image plane okay well we can't talked about how to do that type of projective transformation estimation back in Chapter five when we were doing dense corresponds right this was like the first section of dense corresponds chapter was if I know that the two images are images at the same plane how do i estimate these eight parameters that match up the two images of all I need a supply are four corresponding points and so you know it turns out that that's not really very hard so here's one example of how you could do it it's not the only way that you would do it but for example suppose that I you know have a calibration object that is made up of a bunch of planar faces all of which are fundamentally the same so like it's like I have a thing that's built up out of the same checkerboard and I make this kind of 3d object okay so now when the camera sees that object so suppose I turn the laser stripe off and all I see is this boxes of checker boards right so from the cameras perspective that's like saying okay well actually you know that is equivalent to showing the camera the checkerboard in a bunch of different configurations I'm just seeing all the configurations in one image right if I assume that all the faces are identical this is exactly all I need to calibrate the camera because I assume that I know what the corresponding well so this case I actually would know all the corresponding points in 3d on this calibration object cuz I built the object right so I could do resection or I could do kind of plane based camera calibration if I didn't know my calibration object very well so now I know exactly where the camera is with respect to this object and now I turn the laser stripe on and I see where that stripe hits the object and now I know okay from the camera calibration I know exactly where each of these points are in the image plane and I also know that all these points on the red stripe have to be on the same three-dimensional plan because that's the only thing that laser can hit right so laser can only hit things on this plane and as long as I have four correspondences between the points on this laser plane and the points in the image that's all I need to estimate the mapping from any point over here to any point on the plane right and so that's you know kind of a simple way of doing calibration you can do this kind of laser strike calibration with lots of different types of calibration objects this is only one kind of sketch but that's the basic idea is the first you have to show the system some sort of an object with known geometry and you take pictures of it and you shine the laser stripe on it and then you kind of can tell how to match it up right and so basically any sort of get most of these kind of laser stripes getting systems will give you some sort of a calibration object in the box with the scanner that you use before you start to scan anything right so this is the basic idea but on the other hand it only gives me the relationship between the laser plane and the image for this one position of the laser and the one position of the camera and so fundamentally this means that I can get the 3d shape of anything this happens to be sitting on this plane but I mean what I want to do is I want to move this stripe over the surface of the object and that means that fundamentally I have to do is I have to calibrate the system for every possible position of that laser plan right so that makes it more complicated and so what you usually have in a real-world system is you have the camera and you have the laser strike projector tightly coupled to each other bolted together right so it's basically like one thing that has a handle one thing is predicting the laser another thing is projecting is taking the picture and I assume that I always know the rigid relationship between the camera and the laser stripe projector within this gadget but the thing I don't know is as I move it around where should I put those strike positions back into 3d space right so for that I need to have some sort of fairly accurate knowledge of where that strike should be and so here's an example we just actually had a demo of this device the other day in the Katz robotics lab so this is called a Romer arm and so what you see here is that this is the part here that has the camera and laser stripe projector in it and this is a robotic arm it's like a passive so it's not directly controlling things but as you move it around this arm knows exactly where it has been put in 3d space so for any position of the scanning head the super precise calibration of the robot arm tells you where you should put back that given stripe and I have a video of that so you can kind of see how it works so this is again called a Romer arm and so here you can see that this arm has to generally be mounted you know very rigidly like you can't shake the arm around you have to usually bolt to the table and hopefully that table should be something like an optical table or even a granite table so that the table shakes you're not going to lose anything but you here you can see this the person is using this you know kind of fluidly moveable scanner it's always attached to the arm and it's you know this is exactly a stripe this is a flying dot but the same idea so you're as you're moving the stripe across the surface you know exactly where to put that stripe back down in 3d and so a very common use of this kind of thing as it's being shown here is for things like cat modeling and things like well so certainly you could do something like reverse engineering with this technology where you you buy a thing and then you want to make a CAD model of it by scaling it the other thing that's very common is that you have to figure out is the part that I have in front of me conformal to the model that I have of the part because if if the part is a little bit thicker in one region that it's not going to work for my industrial application so what you won't you often do is say okay you know here's the part as manufactured I scan it now I compare it to my CAD model I have to register it to my cat ba we'll talk about that process next week and then I can say okay well in this region you're within tolerance but over here you're five millimeters too thick right and so this is a very common thing to do in industrial inspection for example so here is you know more visual effects application of these laser scanners so here is a thing called the cyberware scanner so the actor will stand on this cross and then there are these four big yellow platforms that move down the scan tree and so what you get effectively I'll show you a picture in a second it's kind of like instead of just a single laser strike one perspective you get kind of a laser circle or slice that goes around the person as the stripe moves up and down so it kind of looks cool so it's like you have this kind of tomographic image of the stripe moving up and down the person and so these yellow things have the laser stripe projectors and they have the cameras inside of them and then these things are all super calibrated against each other so that I know how to put that whole ring of data back together into 3d and this is a kind of a similar system specifically for head scanning so you set the actor in the chair you mount this you know kind of opening at the head level and then instead of the thing moving up and down the rail this thing actually swivels around the platform and kind of does the 360-degree acquisition of the actor's heck right so again both of these things in order to work have to be extremely well calibrated you know there's before you do any scam you put some sort of like calibration cylinder block in the middle of this platform you scan it with the scanner and then you calibrate it then you put your actor into it so here's a very young Harry Potter you know standing in the scanner not actually being scanned apparently but it's standing there here's from the first Iron Man movie apparently Robert Downey jr. and the and the crew liked the look of the scanner so much they almost put it to the movies this is like they dressed the set you know they put them in there you know because you could imagine this is exactly what Iron Man would use to build his armor right is that you want to have this really conformal 3d body map but this didn't make it actually into the movie but here you get a sense of the stripes and how they're projected onto the actor and you can see again the deformation of the stripe gives you some clue about the shape of the object and another application on visual effects was a very ambitious effort you know this is already probably 15 years ago for digitally scanning all of the Michelangelo statues in Florence this is the digital Michelangelo project and so this was a extremely ambitious effort where they went into these galleries after hours and they basically had to set up this extremely heavy precise gantry after all tourists left and they scanned until the tourists came in in the morning and you could imagine you could see how close they are to these priceless statues right so you got this you know I don't know how heavy this thing was but it wasn't light right it connected if it just got tipped over the wrong way and you knock the arm off the David it would be really bad right so it was extremely impressive that they even got permission to do this whole thing in the first place and at the time you know laser stripe scanning was was it was known how to do it but they had a lot of challenges in putting all this stuff back together because I mean unlike in a carefully controlled movie said they couldn't afford to like spend three hours a night really carefully calibrating the scanner I mean they had to get in there scan as much as they could and get out and so after the fact they had lots of challenges and trying to put back together these 3d scans and again with laser scanning you know even though you you can have a co-located camera but it's not like you get RGB colored points directly from the scan because in some sense the thing that you're scanning is kind of washed out by the red light right so I mean you can turn the laser on turn the laser off to get kind of co-located RGB images but in terms of the 3d points you really do get a set again the same kind of like angry cloud of bees I talked about last time where you just got a whole bunch of 3d data and I figure how do I put all that data back together into the same frame of reference so that can be tricky and the ambition of this project as I recall was to scan the statue at a sufficient precision to be able to disturb discern the chisel strokes of Michelangelo all right so they really want to scan this at super high precision because you can see here the stripe is not very wide right so you guys on how long it must have taken them to scan the statues at that kind of precision and of course you know statues are really not made obviously for for making it easy for you to scan right and so like they had lots of issues about okay well how can we possibly place the scanner to get into these nooks and crannies the statues that are not easily visible right maybe you can see them but can you mount your heavy gantry in a way that the laser can probe up into the statue so it was extremely challenging I think that they think that they augment AddThis with handheld laser scanners that collect the data in places that are hard to reach but then you have to figure out how to how to register those scans against the other scams right so there's another technology called the polymer scanner for example that's a scanner that again projection laser stripe but it has there are different versions of it but the one I'm thinking of has basically kind of something it's equivalent to a pen like a magnetic motion capture register system it's almost like a combination of a GPS or IMU or something like that for indoor scanners so basically you put this block in a place where you know where it is and then as you move the scanner around in 3d the combination of the scanner magnetically talking to the block and some sort of inertial measurements gives you a sense of where you put these stripes back and so if you watch I mean people get scammed this all time I where there's a there was a time when they scan Stevensville bear with a laser scanner and then they gave her when the scan to do fun things with right so right so that's basically the fundamental idea behind moving this laser stripe around again the caveat right now is that you have to know very carefully where your stripes should follow in 3d space to put all these stripes back together so it's a little bit tricky to do like a homebrew laser scanner because you have to do all the super precise 3d you know calibration but these are the kinds of scans that you would get in a you know Hollywood production right so you'll put the actor inside the cyberware scanner you'll scan them and then you get very accurate you know one of the reasons you do this for sure is things like making digital doubles it's also good just for general you know costume reference about how is fabric hanging on the person that day how do we make up that person you know how much did this makeup apply and stick out from their face because you can do 3d measurements of that kind of thing now so you know they do this for you know I think it's almost routine these days where you know the scanner is just sitting off to the side is set when you're not on set and you're in costume you might as well just stand there for a minute and get scanned and then they put that data into a file and maybe they use it maybe they don't but it's up to burdensome to sit there and be scanned so here are various scans you know this guy from Scott Pilgrim this is a guy from Thor one of the frost giants and this is the kind of level of detail that you would get from these kinds of scans again you're getting things that are basically you know millimeter accurate at least maybe maybe more because you can I was like buzz just a second you know you can localize that laser stripe the center of that stripe pretty much in detail and as you sweep the stripe across you can kind of get you know very high quality scans and there's not really noisy to speak of in the same way that you get with anything that was image based I don't know precisely whether it this and this were taken with the very same scanner it could be that this is you know from the head scanner and this is from the body scanner I'm not totally sure these are all courtesy of gentle giant Studios which does all sorts of free scanning for Hollywood I'm going to show you some pictures from my visit there at the end of the lecture here's the predator again you know you can see here that laser stripe scanning has some of the same problems as light our scanning does in the sense that you know the laser strike or the the projected you know laser strike still has to be visible from the perspective of the camera and so here you know I'm not sure what you'd call these I guess the the predator dreadlocks you know I mean there's all sorts of nooks and crannies and maybe that's really dark and so maybe the camera didn't see the stripe in this region and so you just don't get any returns here so again the best case scenario for laser stripe scanning is pretty similar to the best case scenario for lidar scanning you know get some nice matte white you know not super shiny thing right this is a guy from sucker punch again you know the thing that's really remarkable at these scan is that is the level of detail that you can get I mean it's really kind of cool to see all the folds of clothing all the wrinkles in this guy's face and you can really pick up some some nice detail it doesn't really work very well with hair as you can see like you know here you know you can see that hair just draw those it works oh well you'll see with our scanner here that you're not going to get too much great hair either and you also do it for things like props and so again the nice thing about props is that you can you know have to wait for the person to you know stay stationary you can scan this crap as many times in the many ways as you want so you got swords and jet packs and the Loki helmet and so on right so pretty much any prop that you'll see in Hollywood has probably been scanned in again just like well nothing's happening the stuff is sitting around on the set you know just scan everything you get your hands on and you know maybe the effects companies use it and maybe they don't okay so like I said you can do what's called space-time analysis and actually before I do that let me just talk about one other things so here this is a picture of a slightly different kind of structured light scanner but this one has a projector of a pattern or a stripe in the middle that has two cameras on either side once you've got two cameras then life becomes a lot easier in the sense that you know you can calibrate these two cameras together as if they were a stereo rig and then you can calibrate them against the laser stripe and so this is a concentration you have here where if you have two calibrated cameras and you say okay well I know where these cameras are in 3d space and I can calculate the polar geometry between the camera so that's what these diagonal lines are so if I know the polar geometry I know that having correspondences between these two cameras if I know where they are in 3d space I can triangulate through them to get a 3d point right and so the laser stripe or the pattern in Britain to the scene is just a means to an end to finding a good correspondence between either poor or lines so it's like to say okay well if I shine a laser stripe that's kind of horizontal onto the scene then you know if I didn't have the F poor lines I would say okay I know that you know the laser strike here probably corresponds to the laser strike here and I know exactly which point corresponds to which by cutting that laser strike with that you have polar lines so now I have for every position along the along every pair of conjugate point lines I know exactly where I am and since I know where the cameras are I can triangulate that point in 3d space I have the same kind of issue as you know as I you know as I move that whole rigid rig around I still need to know where do I put those points back but this is kind of nicer in the sense that the 3d with it that the structured light is really only there for putting some texture on the scene that gives me some unambiguous correspondence on the uploaded lines and so let me let me talk a little more thing and then I'll show an example of that let me see what my videos look like here so I have it's not exactly this let me first talk about one of the things so the other thing I talk about is called space-time analysis and so it is that if I have everything really nicely calibrated what I can do is instead of just trying to see the image of the stripe and find the center of it what I could imagine doing is I slowly move my stripe across the surface of the object and every image of the stripe probably has kind of like this Gaussian you know pattern right so I mean if you look at a laser strike that's not like it's just a binary thing you know it's it fades off at the edges and strongest in the middle right so you imagine that if I slowly move the stripe across the surface of something I can look at over time what is the intensity the stripe as it passes across my pixel and then I could very accurately localize where the center of that stripe was right so that kind of gives me sub pixel resolution that I can get by spaces called space time analysis right because it is I'm doing some sort of temporal motion of the stripe and that gives you much more accurate localization of the center of this tract and so here's an example of how that works so this is the work Li Jiang back at the University of Washington and so here we guys see is you've got a pair of stereo cameras and you've got this pattern of stripes being projected onto it and again these striped patterns are fundamentally random right because you don't really care what you're projecting you just want to be able to get matches along the stripes and so this is the kind of thing where you can get reconstruction from from scary I think that the other some other pictures like this where again you got two cameras and you've got this moving pattern of whites and you can actually do this I think I don't want to speak for this I know when this is real time or not this may not be real time so I don't want to say that it is I think there's some more pictures over here okay the same kind of idea here where you've got you know this is a case where you're moving the light stripe across the surface slowly and you get this nice image of an object so we're going to talk in a second about what's the deal with these colored stripes but here are some examples of this is this is kinda what I was talking about is that you move the light pattern slowly over the surface and you use that to very accurately triangulate where the stripe goes so let me just talk for a second about you know some of the challenges of stripe scanning so one of the challenges is okay well couldn't I do better by you know good night you better by instead of having to move the single stripe over the surface of the other why don't I just project like lots of stripes on the object at once right so let's say I have a device that projects 50 red laser stripes on to the object and then I get 50 slices the object all at once right why wouldn't I want to do that instead of having to laborious ly move the camera move this laser thing over the surface of the object well one of the reasons is that turns out to be very difficult to disambiguate which stripe is which if they all look the same so here's an example of saying okay suppose I've got a striped projector that sends out these five stripes and here's what the camera looks at and so the cameras saying them out in the order one two three four five here you know the camera sees them in the order one four three five and so what happened we lost stripe number two and strikes three and four switched places because this object was closer to the camera right so if all these stripes look like red lines on the object there's no way to disambiguate which stripe is whipped right you need some sort of way to code the stripes in order to be able to unedig us at all this is strike number one this is strike number two number three all right and so that brings me into the next idea so one of the it is that you can use is called time multiplexing okay and so it is instead of just looking at one image of one stripe what you do is you shine a bunch of patterns onto the object that are all a little bit different and then you ask my sin you say okay how did the time progression of these patterns uniquely give me a position in space and so here's kind of an example what I mean so here is a 3d object and what I've done is shown eight different patterns onto the object right and these are kind of like binary patterns that increase in how often the white and black alternate and so here at this red dot what I could do is I could say okay well at this red dot the pattern of dark and light or on off and on that I see looks like that see I see black white black white black black black white okay and so the idea is that if I arrange these patterns in a certain way there's only going to be so so basically you can see there are eight possible on/off positions corresponding to the 256 finest stripes that I've got and if you go back and think about it the way that I constructed these patterns says that you know for each one is turn of who six stripes the odd off pattern of these eight images is unique right and so that's kind of clever where basically I just say okay and I don't have to show 256 patterns to get to six stripes I have to show log two of 256 so I only to show eight patterns to get 256 and I only have to show nine patterns to get five hair and twelve and so on at some point you become limited by how finely you can project your stripe right so for example I'm doing this with a standard off-the-shelf projector at some point you run up against the width resolution of the projector and also how well you could even tell different stripes apart you know the black to white transition from the perspective of your camera right so there is a kind of a physical limit on how well this could work right I mean even if you have the highest resolution camera in the world just because of the way that the light diffuses from the edges of the projector you're probably not going to be able to get like 2,000 stripes on the surface of an object right another kind of minors note about this so these these patterns that are uniquely decodable are called grade codes and so this is a sketch of what those black and white patterns looks like I think the homework problem I assigned is basically how do you generate these grade codes you know it's not very hard to do they're basically just alternating patterns of black and white that are shifted by half a period kind of and so it is that another thing is that each of these patterns if you think about if you if you go back and figure it out each stripe and its neighbor only differ by one bit of difference and you're going to prove that in homework and that makes it easy to kind of tell which pattern in my app is just looking for which bit has flipped as you move along the sequence again if you get stripes that are lost or mixed up doesn't matter because you have this uniquely decodable position of the stripe right so now every on/off parent I see uniquely tells me which strike was I at and so I can now undo any sort of weird mixing and matching and bosses that I got from the projection so one thing that's a little tricky in practice is that you might think okay well this is really easy to tell if for the image the object am i in a on stripe or a man off stripe but actually you know that's not necessarily so easy so for example it depends on the color of the object itself right so in this case you know let's look at these pixels red and blue and I believe herbs like red and green so here the intensity at the red pixel when it's on is 137 and when it's off is 3 okay and that's because this lies on this kind of a darker patch on the surface of this jug whereas on the green dot this is on a bright surface and so when when this is on I get 226 and when it's off I got 18 so the on position is like almost 100 levels brighter than the on position at the red dot and so that means that what I should really be doing is making a per image location threshold for what constitutes Allah because if I use the same threshold across the entire image for unlove it may be that for example if I have a really dark region of the image then I would always think it was off because it would never get brighter then you would get for a bright region so the smart thing to do is to basically make a perfect soul fresh hold for what that should be and the easiest way to do that is fundamentally to you know you could collect two extra images so if your staff you're scanning something like a prop you could spend all day showing patterns on the prop there's no downside so one thing that you could do is you could say okay I'm going to take an all on picture like this picture on the left and all off picture like picture on the right and then I'm going to choose the per pixel threshold as the average between those two images right and that gives me the perfect so thrilled that I need once I look at the stripe panel right so I know what on looks like I know what off looks like for every pattern or for every pixel the other thing that I could do is actually use arguably a lot simpler would be instead of that what I could do is for every pattern I show the pattern and I show its opposite right and I just can tell whether it's on or off by which one is started right so here in this case it's like say okay here's one of the patterns from the grade code earlier it's like say okay let's take this guy and you know I show that and I show its inverse I say okay well the first image at the red dot had a brighter intensity than at the dark dot so I know this was the on image and this was the off image right so you know again they're not actually necessarily measuring any thresholds I'm just saying which one is lighter or darker but this one actually requires me to show twice as many patterns as I'm showing every pattern and its opposite whereas the other method the require me to show all the patterns plus it all on and all off right but again when you've got all the time in the world to scan you can try all sorts of things right this will not work for a impatient Hollywood actor for a couple reasons you know what is the impatience and the other one is the fact that you cannot tolerate motion of the object while you're shining these stripes on Rex I mean it's very difficult for somebody to stay still as you shine you know sixteen light patterns on them and you'll find that if you you know then you'll get these weird three deformations that come from you know even if you've got the most patient actor in the world even just subtle things like breathing is going to change the 3d surface of the object and that may give you some really weird results right so I mean actually to be fair that's also true for even just a laser described scanning right so if you're if you're scanning somebody and they shift or move as the scanner is moving up and down well then you're going to get these little waves or jigs that like it you could literally see sometimes the breathing profile that comes from at the stripe moving down the person's chest and so there usually is you know a little bit of cleanup that you have to do on these scans to remove these kinds of jiggles and so as I'll talk about in a second you know when you hire somebody to do these scans you're not just hiring them to physically drag the scanner out there and do it you're also hiring them to touch up the scan in a way that is going to be important and useful for the visual effects artists later on right so yeah okay so another kind of a neat idea that it's kind of related to these on-off patterns is this is this is a paper from many years ago but I always like this this halt halt and ressentiment scaper it's called stripe boundary codes and so the idea is that you know instead of trying to find the center of each stripe what I could also do is I could say okay I encode the position of a stripe by looking at what happened either side of it and so for example here the idea is I would show these four patterns of stripes on the object right so I'm going to show you four patterns and what I do is I say okay you know let's look at the let's look at the line between the first and the second stripes and so the idea is that here the change from one to two basically I had a zero zero right here I have a zero one here I have zero zero here I have zero one right and that is that this pattern of four on-off pairs only occurs once if I think about going across the entire strength right and so it turns out that I can encode 110 patterns of on-off looking at the boundary between stripes not the center of this right because if I look at the center of the stripes there are lots of stripes that are like black black black black right I can probably find another black black black black somewhere in this pattern but I'm looking at here is the difference between you know black black black black on the left and black white black white on the right and there's only one of those right and so instead of having to shut so this gives me 110 patterns log 210 is closer to seven that is to four right so that is I've saved some patterns by thinking about you know adjacent stripes instead of just the stripe itself and then you know the X next obvious thing is well couldn't I do even better if I allowed myself colorful stripes right and that's definitely also true and so if I allowed myself to do red and green and blue stripes instead of black and white stripes then I could show you even less patterns and in fact Kimbo our department heads you know PhD thesis had to do with colored stripes structured light scan right that was one of the first color structures like papers and so here's kind of idea of how that kind of thing works where again what you do is you shine a unique pattern of color onto the object and so the cool thing about this is that all I'm doing is I'm shining this one colorful pattern to the object right there's no need to shine a bunch of different pairings on the object show this colorful pattern and the idea is that every three stripes is unique right so there is no other Bluegreen blue pattern anywhere in the rest of the sequence right so when I see blue green blue in my image I know oh that was strike number five right so this unique alteration of colors tells me the strike index it's a very clever idea right and so this kind of thing is actually based on a mathematical theory and I'm not Dutch so I'm going screwed up called a deberían sequence and so the idea here is you know what you're doing is you're trying to construct a set of numbers such that within this set you never get a repeat and so here's is like saying okay so if I give you four symbols 0 1 2 3 and I insist that no pattern of 3 numbers were peers more than once this is such a pattern right so if you look at this there's no other 0 0 0 anywhere in the sequence and not only that every possible 3 symbol pattern occurs exactly once in the sequence right and so here and that's even true cyclically right so here it kind of wraps around so the last one here is 3 3 3 and then if I have another repeat of this pattern so if I look at this wrapping around 3 3 and then back to 0 there's no 3 3 0 anywhere in this sequence until I get to the act right and so idea is that what I have is for example I have 4 symbols and I require no three of them to repeat then I have a 4 to the third power of 64 bit but not bit forward to 64 length sequence of symbols and that can be converted into a colored stripe code like this and so you know one of the homework problems I think is to find me a simpler such sequence like 3 patterns and or you know 3 symbols and no threa that repeats you should be able to kind of monkey around and find a way to get these 27 numbers in such a way that they go in order and so that's actually a really neat mathematical idea and in fact I read a you know there's a guide in a magic trick based on this to bring sequences which was really clever so basically he memorized the order of transitory one of these Debruyne sequences into the order of the cards of deck right and so then he would say anybody okay well you know take a card out of the deck and then you could cut it as mate you know you can cut it as many times as you want and then you just have ripples through the deck you can tell you what was it so take the card out put it back in you know then you look at this and you can find the one thing that was out of the deberían sequence because he knows that you know their cards have to occur in a certain order right and so it makes it seem like it's impossible you could somehow do that's what I like to do is memorize the sequence of the cars and keep this in the back of your mind so and so how would you use this in practice well the idea is that you shined this colored stripe pattern on the object and then you take the known pattern that you know you projected and then you look at what you actually received and you got to match them up right so then you to actually imagine using dynamic programming or something like that and here this image pass adjusts the fact that actually may not be totally easy to match them up because the image of the stripes is never going to be as beautiful as the solid stripes and projected right so here you know you can say okay well I know that the order here is like black pink blue and here's the black here's the pink here's the blue but it's definitely not such a great you know beautiful color right there's lots of weird variation inside here and again that comes from the fact that you're projecting this color stripe pattern onto a possibly colorful object and so when I shine a red light onto a green surface I'm going to get something that looks like a muddy crappy color not red right and so that makes it a little bit tricky to figure out how to tell what colors actually been predicted on the object so that's a problem that you don't have when you have something like black-and-white patterns you know even if you don't get precisely black or precisely white it doesn't really matter because all you have to know is whether it's on or off or the color sequence suddenly it becomes more of an issue to figure out what color do I really put on that surface and then again you have the same issues here with if the stripes are missing or switched places then well if they're missing you probably okay but if they switch places you can't just simply use dynamic programming you probably have to decompose the the correspondence into simpler pieces so if you remember back to the part in dense correspondence where I talked about how do you correspond so long at poor lines when you may have switches through this double mail illusion that's the same kind of problem in you're faced with here right and so you know again you may have to do multiple steps of dynamic programming in order to disambiguate this um there's also a way that is kind of like a hybrid between so here all so far we've been talking about is basically hard-edged stripes right you know just step function like projections another kind of way of doing it let me just delete just go back and talk about this for one second so one thing I forgot to say is what is the advantage of these colored stripes there possibly lots of problems with the advantages that I can actually just show this one pattern at once and I get but it with the numbers this pattern I believe has I'm trying to find the number 125 unique stripes right so basically what I'm doing is I shine this color pattern on and I get 125 3d positions right and since I only have to shine one stripe on the surface to get the 125 numbers I can basically do that at frame rates right so what I do is I have a moving object the color powder is always the same and I can basically reconstruct the 3d shape in real time right maybe the reconstruction is an exactly real time but I can record the pattern moving in real time and reconstruct it later right and so this is what you would call like one shot scanning where instead of having to shine multiple patterns of light on stationary object you have only one pattern of light you get the 3d immediately right and so that's the advantage is that you can basically do you know scanning of things that are moving right you have to worry about how to put the stars back together because there's no ambiguity you just get one nice picture of a moving object the same kind of thing you can do with what are called these fringe patterns and so here what you got is instead of sharp edged objects you have what looks like you know if you're if you were to plot these from the side you would see like a cosine wave and the cosine wave is shifted by three different phase shifts and if you show the cosine wave in you know succession what you get are basically three continuous numbers not just black/white patterns and you can use that to undo what was the 3d position of the point the really cool thing is that you know for this you need to project three patterns onto the object right and so one thing that that someone realized is that the way that your high quality color you know DLP projector projects a 3d image is not by having those only one lens on your projector right so what's happening inside the projector is this thing called a color wheel that is spinning extremely rapidly right and so instead of getting all the RGB at once what you're really getting is a frame where white light is being shot through a red filter then that filter spins around you shooting through a green filter and so what you're getting is the projector is sending out our DP RGB RGB RGB at so high frame rate 120 Hertz right and so the color wheel is spinning really fast and so the observation here was well you could do really really fast structured light if you just removed the color wheel from the DLP projector and then you kind of told this is the R this is the G this is the B the pet the projector is going to show those things for you in quick succession and then suddenly you get extremely high frame rate structured light by using the mechanics of the projector and so that was a really slick idea I mean you have to be a little bit handy to take park the projector and remove the color wheel and you know that's not something for the faint of heart but once you do it you have this kind of unique you know opportunity to get very high frame rate structuring light and so that's a pretty cool idea um okay so let's talk a little bit about like real world stuff right so personally I paused asked any questions about the theory okay so um let's talk a lot about you know what people are doing this refer light these days and so one very common structure of light things this is so you guys know about the Microsoft Kinect we talked about the new version of the Kinect last time which is really a time of flight technology but the Kinect version one was actually a structured light technology and so it wasn't projecting stripes into the scene it was projecting basically kind of like a random 2d dot pattern or a pseudo-random Tunney dot pattern in the infrared world right so if you were to look at the living room bathed in infrared light by the Kinect you would see lots of little dots and the idea is that these dots are not totally random because every point in space if you look at the local neighborhood of these dots you know it has a unique neighborhood of dot it's kind of similar to thinking about the unique neighborhood of color stripes that define a particular position this is kind of done in a 2d world where you say okay if I see this dot and I look at all my dot neighbors I know where that projection was from the perspective of the of the scanner or the from the projector and that gives me a 3d triangulation because there's an image of those dots so actually that's the way the connecting version one looked and actually this is not from that this is a picture of what's coming off of the new google tango cell phone right so that just was announced beginning the semester in January right that's a cell phone it basically is doing real time depth sensing and someone you know looked at in the infrared and this is what they saw so basically the new tango phone is fundamentally using what used to be inside the first version of the connect any so you can think about this as a structured light technology right so I didn't really talk about here but you know you can kind of take all the stuff I talked about and extend it into 2d right so instead of just having a single vertical stripe you could do like checker boards of colored stripes in in space and give you you know you get even more localization ability so this is kind of the dot pattern that you see from a device like that um so here a couple devices that and one of them is over here so I want to buy something that basically did kind of quick dirty you know 3d scanning and so here's one thing I considered this was called the ghost scan from Korea forum and so what you see here is it's got a projector of a pattern and this is not really a laser stripe it's really much more about like a white black and white checkerboard that you see being flickered onto the scene and so that's probably a proprietary pattern but the idea is the same as like the dot pattern where you know that if I see a certain region of checkerboard I know what what corresponds to that in my projection so seeing a certain point out of checkerboard pattern tells me how I should reconstruct that point in 3d and so here's an example of the scanner that I brought today this is the Arctic Evo scanner and so we'll do a demo of that in just a second but the idea is the same so here you can see that it's got you know it's got a camera I guess he's got one or two cameras and it's got a projector of light and actually in this one you know it's predicting some sort of light pattern but you can't even really perceive what the you know pattern is I mean it looks just like white light basically at a very high stroke rate so if any of you have any sort of you know epilepsy you know seizure problems you don't want to get scammed by this thing so I'll do this in just a second um I also want to mention that so my friend Sophie Khan is an artist and so she Mayo makes art out of 3d and one of the things that she's really interested in is kind of the kinds of glitches and the mistakes that the scanner makes and so instead of trying to make these beautiful scans that you would use for visual effects later on she's kind of interested in you know weird scanning artifacts and chunks of 3d that you get and things that you don't get so I encourage you to go take a look at her work and then associate she scans people and objects and then she 3d prints them and you know it's a really striking effect so definitely cool to look at um okay so let's do some scanning so let me unplug my one thing plug in the other thing okay so again what we're going to see here is the art tech Evie a scanner that can probably close this thing okay so again if you're a flashing light person you know you're going to be freaked out and so what you see here basically is this kind of histogram bar thing at the left is telling Austin kind of the sweet spot where he should be for skin so if he goes you know if this collection of returns goes too far outside the range then he's not going to get any result and so what you're seeing here is basically real-time images from the scanner that kind of lets you get a sense of am I getting good returns so here we're saying this bucket and you can see it looks pretty good I guess we're not getting is the bucket and see is is kind of coming back as no return now he's actually starting to scan the bucket and as he moves around the cameras are being used to in real-time register the you know the scans together and so you get is basically a real-time 3d plus texture scan so it's pretty slick and then of course once you're done with the scanning you can take this thing and turn into you know a 3d model you can see the points that came back from and there's all sorts in once you're at this point you can definitely do all sorts of things like you know clean up the mesh you know remove stuff like this on the ground plan that you don't want but the texture mapping is you know pretty reasonable so someone want to be scanned all right come on up let's go over person maybe over here yeah close your eyes maybe that'll be good yeah that's right and try and stay as still as possible so this is just a pretty good yep good and actually you can see here that basically you know her coat is good your skirt is good but the hair is not cutting through very well and that's something that you know what you need to do is fix up fix that is something like you can see this is tracking is lost all right so this is fire so what basically happens is we got a partial scan actually can kind of look at this while we're here so how do I move the I want to kind of do the hand you know to drag it as their a both button clean I see okay you know so you can see that you know even though actually the the density on the face is not like amazing once you text your map it looks pretty good right and so you can see we didn't get a lot of results on your hair and but you know the rest of the of the fabric you know fabric is really pretty solid for this kind of thing and again even though the 3d may not be necessarily great once you texture it up you know it actually looks pretty good and so we can you know we can try and scan you some more and then it is that once you've got a bunch of scans you can register them together and to a into a full model so it's not like if you don't get it all the first time you're totally stuck so I don't renounce gets worked out you googled a little bit right so now we talked to it would be tough to register your non-rigid dis but that's the idea model on my computer through that's long as the after class yeah so afterwards we did do some scamming people in the lab right I'll try to get anyone else like it scammed all right come on get skin I'm just always the first of all in tears that's true so so integration here is going to be so before we start we can kind of predict that like I don't know what's gonna happen on the keep the glasses on let's go yeah and the rest you looks pretty good here may pair may not work you yeah it looks good alright so let's try to get here don't move but you can open your eyes if you like Oh lost tracking so part of it is just we're scanning in kind of this weird strange space yeah I mean I think that if we were had a little more range of motion on the scanner we would do better so like again this this what I felt came out pretty well so like actually you know even through the glasses we were getting pretty good results I'm not sure where the 3d points actually are and so you know it's not bad yeah when the light hits it so baby it maybe some of this stuff is coming from the side or instead of getting the head-on thing but you can see again to really get all the nooks and crannies is tricky right so like getting under the chin you know getting all the folds of your jacket may be tricky you know I think for the most part looks pretty good and so you know those of you that took like CAD classes right would be able to reverse-engineer this into a nice mesh right clean up all holes and then once you've got that then you could you know do the water-type you know mesh and you could print it right so one thing we're talking about would be a lot of fun to make like you know we'll have action figures right where do you everyone stands that you can 3d print the bell right so once you've got them you can do whatever you want with them okay so let me just say so I think we do a first gang later if you want but let me just finish up by saying that you know so let me talk a little about about my trip to gentle giant studios in California so this is a company that does basically 3d scanning for all sorts of movies or they've scan any actor in a set that you can imagine but in addition to that they also have lots of highly trained classical sculptors and digital artists and so this is so the other great thing about this company is that fundamentally they not only make scans for movies and so on for visual effects but those same scans are used for you know video games toys like all the toys that you would see as movie tie-ins come these 3ds games used to be when I was a kid you get this pitiful action figure that didn't look anything like the actual person right now that you actually think is you get look pretty good because they come from the same 3d scans that were taken you know on set fundamentally and so here this woman it's hard to tell but this is a you know a character from the first Star Wars right so this this guy here is one of the guys who gets forced choked out you know this guy and so you know he's no longer around to make a great action figure of right so this woman is is very carefully sculpting you know this this person from clay or from something like that for the purposes of making you know a later action figure right so not everything has to be digital I mean there's still a lot of hand artistry that goes into a lot of these things you can see kind of along the top of this all of the recognizable you know Harry Potter figures and what else we got here we got a Harry Potter we got Star Wars I think I have a I think I have a zoom in of that so like you know here are a bunch of of these figures that all start from Clara so what will happen basically is that you know these will be very carefully hand sculpted and then there'll be 3d scanned and used for a printing later on right there's also a lot of digital work to clean up these things for different purposes and so it may be that you don't necessarily need all the detail that you get from a 3d scan of an object when you're making a toy when you're baking a video game carriers don't like that so you want to have enough detail to kind of sell the idea but you don't want to have maybe every 3d vertex that you got from the scatter and so there's a lot of digital cleanup work later by experts to figure out how to make you know the scan that's at the right kind of resolution or the right quality for the job at hand and so I don't have the picture right here but you know this guy when I visit was working on a I think it was going to be like a I would call it Choi I mean I could a figure for venom after he's been like totally bulked up right and so this artist was looking at these pictures of you know bodybuilding magazines he's crazy steroid booked up guys to figure out how he should edit the musculature of this object to you know make it look realistic right so it's not just like scan and print and go there's a lot of people behind the scenes for making it over so here this is a head of Dennis Hopper from scan of some movie or other that I am holding on to and so I got to say that this was like possibly one of the coolest places I've ever been so this is like a totally unassuming like almost sketchy looking a strip mall like storefront in in California in the middle of this kind of industrial district so I kind of get to this dress in like ah manages to the staked you know am I going to get murdered here in this alley and then you know I go in there and this is like this incredible it's like totally geeking out of all the stuff that you see there so like these are all sorts of action figures that have been produced from different things and so you know all the franchises that you know so you probably have a friend who has like a little thing on their bookshelf from the matrix from Star Wars and this is where all those things are born right and not only that you know not only are these things used for you know like toys and kind of tabletop size stuff they're also used for like comic-con you know massive ten-foot tall you know displays so like all that stuff is also born in the studio and so you go in and you see this is a you know Lord Voldemort that is like 10 or 12 feet high sitting in their display room right this we got is a life-size mech from the sucker punch movie you got your you know 9 foot tall Navi from Avatar in the lobby this is the Sauron from Lord of the Rings again just sitting next to you but it's like huge is life you know so it was like totally awesome I've got some Boba Fett thing extra trash can and be in Jabba the Hutt right so these are the kinds of things that you see at these big trade shows and so on and again they all come from different kinds of 3d scanning and so again my thanks to Steve Chapman at gentle giant Studios who was the one who let me in the door and talk to me about 3d scanning and so you can see in the back of the chapter there's an interview with him about specifically things like lidar instruction of light scanning for movies and how it's changed over the years and what they use and what they don't use so yeah super cool trip ok so where are we going from here we have two more lectures and then we're going to so those lectures kind of get a little bit more back into the world of computer vision so I mean what I've talked about you know 3d scanning definitely falls under the umbrella of what people talk about in computer vision but let's talk again next week about what can you do when you don't have an active so these are all what I would call active techniques in which you're interfering with the scene by pushing a light into it right so you're definitely not just taking the scene as it comes you're trying to probe it with different kinds of lasers in light but you can do a lot with just taking lots of pictures of the scene right so we talked about in the dense corresponds chapter or it was maybe this the batch moving chapter you know if I want to get a 3d model of Statue of Liberty I can do something like structure for motion where I take lots of pictures of it figure out where all the cameras are I triangle it through all the corresponding points and I get a 3d model right so that is kind of a technology called multi-view stereo and that's we're going to talk about on Monday and then on Thursday we'll talk about some of the algorithms that you need to work with these kinds of three-dimensional data so one big issue is registration you know how do I take two three-dimensional point cloud data sets and fuse them together to the same frame of reference because as you can appreciate from what we saw today there's lots of need to do that because just what we were seeing with the arktech scanner you're acquiring a bunch of partial scans how do you register those together into the complete 3d mesh obviously the job is easier when you have some kolo kated color on the mesh like we do in this case we'll talk about how that process works and then that will be the end of Fletcher's okay so many questions but I think all right then I will I guess I didn't actually write hardly anything this week so I didn't need my camera

Channel: Rich Radke

Views: 36,126

Rating: 4.9200001 out of 5

Keywords: visual effects, computer vision, rich radke, radke, cvfx, structured light, laser stripe, stripe coding, gray codes, fringe projection, 3d scanning, de bruijn, gentle giant, artec

Id: ZC7CIlAVkIA

Length: 61min 50sec (3710 seconds)

Published: Thu Apr 24 2014