Getting Started with Photogrammetry Using Your Cell Phone

sometimes the best way to create a new 3d asset is to capture it in the real world but what if you don't have any fancy scanning equipment well as it turns out you can get great digital capture using just your cell phone hi i'm patrick letourneau 3d artist photogrammetrist and secret crime fighter you've probably heard the term photogrammetry before but maybe you thought it was a bit too advanced or complicated to try yourself well i'm here to show you the technique for capturing incredible 3d scans of the world around you using tools you probably already have at your fingertips photogrammetry is the science of making measurements from photographs using multiple input images software is able to infer super accurate three-dimensional models that you can put to use better yet you don't need expensive equipment or complicated software to get started just your cell phone and some supplies from around the house in this tutorial you'll learn how to set up objects for capture and ingest them into software how to clean up and simplify your model texturing and baking normal maps how to export the model into cinema 4d and redshift and the difference in quality between a cell phone scan and a commercial scanning setup before we begin don't forget to grab the project files in the description below so you can follow along now let's get started [Music] so here's my setup as you can see it's just a shoe on a tripod i've got a toilet paper roll there to elevate the model this lets me get underneath it to shoot the soul so you want to be shooting in a camera app that allows you to control exposure and iso and things like that you don't want to just use your straight up camera app because the exposure will usually change between images and you can't set focus separate from exposure in a lot of the default camera apps so here i'm using pro app this lets me get tiff images you want uncompressed images if possible um as the compression and jpeg will lower your detail a little bit but that can be a more advanced step once you've done your first practice runs it's probably fine to use jpeg so a dslr makes this a lot easier obviously i don't need to really explain that you can see my movements here i'm attempting to be systematic and kind of create a dome of pictures around this thing um you know you'll do a ring above the thing and then you do a ring at the same level as this as your subject and then maybe you can go do some orbits of special areas that haven't been covered previously here you can see i'm shooting underneath the sole probably not going to focus on the bottom of the shoe too much for this tutorial but it's good to have there as additional imagery my main advice would be always overshoot never undershoot it's a lot easier to delete extra images than to make images you never took in the first place in addition you want to shoot on an overcast day and that is critical for outdoor scans like this if you've got the sun casting shadows across something those shadows will get baked into your model and then it becomes really difficult to re-light it yourself in your cg application so remember to shoot in the flattest most neutral overcast light that you can the next step up from this of course would be a studio where you have a lot of light control but for today's tutorial we're just going to talk about this sort of entry level shooting with a cloudy overcast day the application we're going to be using today is reality capture reality capture is a really great multi-gpu accelerated cuda application probably one of the fastest 3d scanning apps you're going to find and they have a very unique licensing model called paper input where you can download this application for free and use it scan all you want and you only pay upon export the charge is based on the input megapixels of the images that you're scanning with so a bunch of really high-res images is going to give you a more expensive scan than a bunch of low-res images so if you go ahead and download reality capture create an account we'll just hop over to it here and i'm going to go ahead and drag in all of my processed images that we captured earlier on the iphone and we're going to see that they're all in here in reality capture so i really could just hit start and and get a scan out from that and it would go through all the steps but i'm going to do this step by step so the first step is to align your images aligning your images has reality capture go through each image and look for little landmarks little areas of detail and try to match them up between images to try to solve the 3d positions of your camera you can see here that it's already starting to solve the positions once it has the camera position solved it will go through and generate a black to white value for each pixel trying to estimate the absolute depth the distance to camera of each pixel once that's done it can merge all of these depth maps together and you get yourself a 3d model so just under a minute later we can see that we have ourselves a shoe here and if i'm to pull out a little bit and select all the cameras that i have in my project you'll see little cones representing the position of the phone when each photo was taken which i think is always really neat to look at and you can see the strategy i've adopted here is to shoot a lot of images in a center row and then move down a bit shoot some images maybe a fewer stops when i'm shooting from below because we're less worried about that and moving down again shooting another ring from above and then some orphan cameras up here that i shot from the top to try to catch the inside of the shoe now that we've checked that all of our alignment seems to be working and we can come down on the left here and expand this component and expand the camera poses and just ensure that all of our images have been solved properly and we can see here that 97 out of 98 cameras had their positions found which is pretty good in my books so the next step is to have the software calculate the high resolution model and this can take some time based on how powerful your system is so we can just let it do its thing and we'll be back when it's done [Music] and 22 minutes later we have ourselves a pretty decent looking scan if i just orbit around this a little bit with right click we can see there's a bit of noise in some of the smoother areas but that's to be expected with a cell phone based scan if you have access to a dslr i'd highly recommend it you'll get much cleaner scans and the capture process will be quite a bit easier but seeing inside looks like we've got the laces captured we've got some of the inside walls which is great as i wasn't super duper focusing on those with my capture session more on the outside here the issue with this is twofold one it's 15 million triangles which is just not a fun number of polygons to be working with in any application and two we need to do a little bit of cleanup so we're going to go up to the reconstruction tab here and in the tools area and the selection area we're going to start by grabbing this lasso and i'm just going to line up here and click and drag to grab our stand and then we're going to hold ctrl to add to that selection and just kind of orbit around looks like we've got to capture a little more here so add to that selection okay so we've got our stand selected here and i'm just going to go up to the tools panel and we're going to click on filter selection and that's just going to do exactly what it sounds like and it's going to delete whatever we have selected once that deletion is done we can see that we have a shoe without a toilet paper roll underneath it anymore which is great the next step is going to be to close up the holes in that model and we're just going to hit the close holes tool for that and anytime you hit a tool in reality capture that has some sort of options the dialog is going to pop up in the bottom left here this is sort of like your cinema 4d attributes manager and so we're just going to hit close holes and just like that we have a mesh without holes on the bottom so no more hole where that toilet paper roll used to be the next step is going to be to clean up our models a little bit here reality capture keeps a copy of every version of the mesh every time you do something to it so it's kind of like a history an undue history in zbrush so i'm just going to x out model 1 and model 2 because we know what we're doing we haven't made any mistakes here and we're just going to work with this as our high poly model next i'm going to select model 3 which is the one we're currently working on and i'm just going to rename it high poly [Music] so the process of reducing the complexity of this model to something that's a little more workable is done right here inside reality capture if you were to rig this thing and you needed perfect quadrangular edge flow and topology maybe you'd bring it into something like zbrush and use z-remesher on it or you could use instant meshes which is a great free tool now integrated in cinema 4d in fact for performing quadrangular retopology but since we're not going to be doing much with this other than rendering it i'm going to say that just triangles will be fine so we're going to go up here into the tools palette once again and we're going to select the simplify tool simplify tool is going to pop up here in the bottom left and you can see that we can set a absolute or relative type of reduction we're going to stick with absolute for now and we're going to say 250 000 triangles is our target with that set we can just come down here to the bottom and we can hit simplify with our simplification done you can see that our model is now quite a bit less detailed than it was before the spirit of it remains there are no holes in it no huge issues it is however feeling a little bit noisy and especially when it comes to baking down normal maps and transferring the quality of the high-res model onto this low-res model we're going to want something that doesn't have all of these sharp 90 degree edges on it those just never turn out great so what we're going to do is we're going to go up to the smoothing tool here and once again it's going to come down to the bottom left at the top of our panel and we're going to up the smoothing iterations to 5 just because this is a pretty noisy model we'll leave smoothing weight where it is smoothing type to noise removal all good and now we'll hit smooth as you can see you end up with something a little more melted looking but this will be a much better target for reprojection of our textures so also be a much better target for baking out or normal maps [Music] for baking out our normal maps we're going to first want to delete the excess models that we have here which is just one in this case and delete this noisy model and then come over to model 2 and we're just going to rename this low poly the next step is going to be to actually texture our model now that we have this low poly shoe it's much easier for reality capture to come in and uv unwrap this for texturing so uv unwrapping if you're not familiar with it it's a bit like taking an orange and flattening it out flattening out the orange peel or reprojecting the map of a globe so 250 000 triangles is always going to be easier for reality capture or any application to work with than 15 million triangles so with our low poly model selected we're just going to come up here and we're going to hit texture we're not going to worry about setting any options up for this the defaults are going to be fine reality capture is quite good at making sure that the full quality of your photos is represented in your texture with the texturing process complete after about a minute and 20 seconds we can see that we came out with a pretty nicely detailed texture i'm very happy with this you can see the benefit of shooting on a cloudy day here in that we have no baked in shadows or very few baked in shadows other than things like this um and for the most part this is mostly ambient occlusion that can be processed out you can run an ambient occlusion node in your renderer of choice and just invert it and apply that to the diffuse color of this model and you'll sort of clean up some of the shadowed areas here and be more prepared for something that can be lit from scratch obviously a cloudy day is not a hundred percent perfect if you were striving for perfection you could use something like a light box or ideally a ring flash around your camera's lens to make sure that you had no shadows and even consistent lighting across each shot [Music] now that we have the textures done the next step is going to be to reproject the normal map from our high poly model so i'm going to deselect the smoothing tool here and just come up to the texture reprojection tool and over here in the bottom left you can see that since we only have two models it's already pre-populated our source model as high poly and our result model for the low poly so this is going to bake into a normal map the geometrical details of the high poly model so you always want your high poly model to be the source and you always want your result to be onto your low poly model we're going to disable displacement for this just as displacement is a bit overkill for a model this simple and will add to the time it takes to calculate so i'm just going to go ahead and hit reproject here in the attributes manager with the normal map baked we can see that we get this handy little diagnostic view that just shows us the results of the normals that have been projected onto our model obviously this is not the final product this is just a very handy way to be able to diagnose things like these stitches coming across and frankly some of the noise but again that won't be an issue with higher quality capture methods but it's quite excellent for a cell phone and i love the future [Music] so we're going to go right to the export now we're going to go over to the workflow tab and we're going to hit export model i'm just going to save it as shoe low vo1 hit save and i've already acquired the input licenses for this model a little screen will pop up with a password prompt that says hey this is going to be a two dollar model to export please input and confirm the purchase so i went ahead and did that in the tv magic world and we're just going to check to make sure that all of our outputs are correct here so we've got a tangent space normal map that will be created from this world space normal map we have our normal layer being exported as a tiff 32 bit all very good our color layer once again 32 bit we probably don't need tiff honestly jpeg is more than good enough for diffuse textures the normal map you're going to want in a less compressed method more typically and displacement you just can't compress so with all our options set up here we're going to keep vertex colors off as well and vertex normals off we're just going to go ahead and hit ok and reality capture is going to export that textured and colored mesh for us and the next step is to pull this into our 3d render engine of choice [Music] moving on now we've got our shoe here inside of cinema 4d i've just dragged and dropped the exported obj into the viewport you can see this is pretty low res here feeling really nice and nimble and quick put down a little plane here so we get some reflections uh from the light and put down a redshift dome light with my favorite maximum raw's hdri next step is just going to be to create a redshift material and we'll drag that right onto our shoe with the material selected hit edit shader graph we're also going to open up the red shift render view here and we'll hit play so as you can see we've got ourselves a shiny blobby low-res shoe here um go ahead and make sure you have photographic exposure turned on uh if you're not working the aces color space photographic exposure is great to keep things from getting blown out so yeah we push in here we can see things are looking pretty low res pretty melted so let's go ahead and grab the normal map and the diffuse map that we've exported drag those into the red shift shader graph and we're going to want to put down a redshift bump map node here wire in our normal map make sure that you switch the input type from height field to tangent space normal otherwise you'll get some weird results and we're going to wire it into the overall bump input and you'll see magically boom we've got ourselves a high res nice looking shoe one thing to take note of is gamma override oftentimes with these normal maps especially coming out of uh apps like substance painter or reality capture you're gonna want to turn on your gamma override here uh without it you get this sort of weirdly darkened shoe turn it on you can see things look normal if we toggle our bump map you can see the lighting doesn't change drastically so that's how you know it's working correctly if you've got gamma overwrite off and you toggle your bump map and things get kind of dark or weird that's how you know you gotta have your game over right on moving on from here we plug in our diffuse map into diffuse color and there we go we've got ourselves a beautifully 3d scanned shoe i'm still amazed that it came out this good from a cell phone um but there you have it take our light and spin it around and see that we've got ourselves a shoe so first thing we're gonna notice here is that things are quite shiny maybe a little bit shinier than they would be in real life so we're just going to make a quick roughness map here i'm going to pop down a redshift area light hit shift c there to bring up the cinema 4d commander and we'll just spin it around scale it down so that it's not overwhelmingly bright and we'll just pop it right behind the shoe as something that we can use to judge the overall specularity of this thing i'm gonna turn off my dome actually you know what let's keep the dome light on and push in here and so a great way to cheat a roughness map or specular map is to first of all we're going to disconnect our diffuse color we're going to come into our object here and we're going to manually set its color down all the way down to black so that we can see just the reflection so next thing we're going to do is put down a redshift ramp node we're going to wire diffuse through the ramp and we're just going to preview this ramp node on the surface you can do that by going to tools and connect node to output i have mine hotkeyed to v which i strongly suggest you do because it's a huge quality of life improvement so looking at this ramped texture we can see here um that we've got black to white um so with a roughness map you want the rough areas to be white and you want the shiny areas to be black so if i just kind of look at this and kind of judge it here so i definitely want the shoelace to be highly rough i probably want this reflective strip to be a low roughness so i'm just going to go ahead and hit invert here on the ramp node and then i'm going to play with the gamma a little bit maybe clip the black a little and looking at this this looks like it's probably going to be pretty good for a roughness map so we'll just go ahead and repreview our material here hit v and we'll take this ramp node and plug it into the reflection roughness input and so now you can start to see that we're getting some shine here and some roughness here start to feel a little bit better um another thing you'll want to do is so we're just going to put down the node for reflection color um so this is if roughness is how rough your reflection is reflection color is like a speck map um that you probably have played with in the past any time you downloaded the 3d model anything like that so we'll plug our texture into the ramp node and we'll preview this again and we'll just think about what we want so we definitely want the laces to be lower brightness we definitely want this strip to be higher brightness so let's just go ahead and turn off our invert and bring this down like this so one fun thing that we can do in the redshift ramp node is introduce noise and this is sort of the fun little secret weapon that you can use to break up these ramped textures because if you're just ramping this one diffuse texture both ramps are going to have very similar feels but if we push in here and you know what let's introduce some noise on the roughness one as well if you introduce some noise it just sort of breaks things up in a way that doesn't really look terrible at render time and that also helps sort of keep features from being identical in the roughness in the specular so we plug both of these in here so let's take our color ramp into reflection color so now you'll see that it's not a uniform brightness across the entire surface of the shoe you know this is looking let's just take a screenshot here this is looking much better than it was when we just had this shiny shiny object here so i'll just undo those changes and then we'll plug our diffuse right back into diffuse color and so now we've got a much more realistic looking shoe uh you know this was kind of a ratty old shoe so it was not something that was shiny and new um so this is feeling much closer to the junkyard shoe that i scanned so this has been your introduction to 3d scanning in reality capture using your cell phone i'm going to next go over the differences between the cell phone capture and a capture on a commercial scanning rig that i have and we'll just kind of go over some of the key differences between those so you can get an idea of sort of what the ceiling is well i don't want to say ceiling that might be a bit braggadocious of me but just to see what's possible with um maybe a higher end rig so we'll be moving on to that next [Music] so now we can have a look here at two scans one from our cell phone image set and one from my desktop turntable setup that use really high res camera and a ring light that really blasts out quite a bit of light and gives me completely flat photos with no lighting information baked in and we'll get to that in a minute so just off the top here looking at both of them they're pretty close i think looking at most of the details here you can't really tell one from the other until you start to i mean if you look at the laces here you'll see the laces are much more melted together in our cell phone data set and our turntable setup has much better separation between laces but that's just the beginning here if we go down to some more extreme lighting you can see especially in the lace area one works and one is starting to fall apart here overall still super super pleased with how the cell phone scan turned out but you know in certain areas it's just not holding up as well as the turntable setup so what make it might make it more obvious is if we switch to a clay shading mode here and at the bottom we have our cell phone scan and at the top we have the scan with the uh turntable set up so i'm just going to bring this down here real close and you know just looking at this you can see right away the word fluid ride is very clear in our pro setup and in the cell phone uh can't read it another place to look would be the stitches the fine stitching detail here is much easier to see in our high hand scanned the ridges in the reflective area came in through the the dslr scan the cell phone they did not you can see different textures in this area versus that area whereas the noisy scan from a cell phone you just you can't really see what's going on there so yeah it's it's it comes down to the finer geometry this is stuff that's really going to show up when you have things under extreme lighting scenarios for most really really soft lighting situations you can usually get away with something that's not entirely clean like this but it really helps to have this in production makes a big difference and so the last thing that i'm going to show you here and we're going to switch the orders here this is our desktop turntable scan here at the top and this is our cell phone scan at the bottom so this is us just looking at the diffuse map so this is the pure texture with no lighting applied to it and you can see with the turntable setup since we're providing light for the scene from directly around the lens with this ring light um you can see that the textures are evenly lit throughout so the white on the bottom of this shoe looks the same as the white on the side of the shoe whereas when you shoot outdoors or in a situation with harsher lighting we have a light color on the side here and then a much darker color on the bottom and you can see just looking at them this one just looks like a flat you know if you cross your eyes it's a totally flat image this one has lighting information basically baked into it so even little things like on fluid right here you can see a little bit of shadowing at the top that only exists when you have lighting and when you're providing a diffuse map the goal or an albedo map i should say the goal is to have no baked and lighting no reflections anywhere and you can definitely see the difference here in terms of flatness especially in the contour of the shoe you can see that we start to get darkness down here whereas up here it's perfectly even as you would want because you're taking this information and then you're applying lighting to it and if you have one set of lighting already baked into it that's going to affect the output you know if we were going to light this quite extreme from below or even quite gently from below uh you know the shadow here that we have baked in might cancel out some of that light coming from below whereas with this one it would show up very very clearly and even beyond the shading you can see uh you know the top here is kind of a blue uh hue of the sky and the bottom half has the green of the grass right of all that light reflecting back um whereas the the turntable setup you can see we get we get to really pick up some crazy details and all these crevices all the dust and stuff that um doesn't really normally stick out that much really sticks out versus on this just looks like a little like you're looking at a photograph right so yeah this has just been a quick little look at what's possible with a higher end setup but uh you know we we achieved something pretty great with this cell phone i think i think it came out really really nice so those are the basics of photogrammetry it's pretty impressive what we were able to achieve with just a cell phone a tripod and a toilet paper roll but the difference in quality when moving to a professional setup is quite dramatic if you want to learn how to get the most out of your 3d assets check out cinema 4d ascent over the course of 12 weeks you'll go from beginner to intermediate level 3d artist that's fluent in cinema 4d and familiar with other 3d tools don't forget to subscribe and click the bell icon so that you'll be notified when we drop our next tutorial we'll see you next time [Music] you

Channel: School of Motion

Views: 844,323

Rating: 4.9465127 out of 5

Keywords: Motion Design, Motion Graphics, After Effects, Tutorial, Tips, Tricks, Technique, Learn, Basics, Design, MoGraph

Id: ZIW4XU6Wm8Q

Length: 28min 55sec (1735 seconds)

Published: Wed Feb 17 2021

Reddit Comments

[u/eaddict]

Nice but should try to use freeware/shareware as much as possible.

[u/eaddict]

A lot of assumptions of terms as well. Guess if I was at that school I'd understand...

[u/Position-Hungry]

nice tutorial! way to go OP's bro!