Photogrammetry is DEAD!

what if I told you that you could take a few photographs of the room you're in right now feed them into an AI algorithm and it could give you a live interactive video where you could move the camera around your room and you could see what it looks like from every Viewpoint not only like a photo scan but it would also have live interactive lighting effects like Reflections and opacity well you don't need to imagine that because that's already a thing it's called Nerf or neural Radiance fails in a lot of ways I think it's going to replace traditional photo scanning or photogrammetry now if you watch the channel for a while you probably know I'm a pretty big fan of photo scams I use them a lot I make about three to six minutes of original animation every couple of weeks that's about the equivalent of a feature length movie every year that I make by myself so anything I can do to speed up their workflow is worth it to be so I love the fact that I can just go on the sketchfab find a car scan or a building or whatever drop it in the background and it looks good but what I found over the years is that photo scanning is very very limited whereas this new system Nerf fixes a lot of those problems and actually adds new possibilities that I never even thought possible with photo scanning so in order to get on the same level here I'm just going to quickly talk about how traditional photo scanning actually works it's pretty simple so basically you just take a input series of photographs or a video in this case just imagine that this is a video of a monkey that I've recorded and we want to turn into a 3D model so we take those frames and we feed them into the photogrammetry software and we say okay just track these points just like camera tracking find all the points and then find where those points are on all the other pictures look for the similar points photogrammetry software goes figures out the points that's called a sparse Point Cloud but in order for that to actually be useful we need to say okay well based on how those points are moving in relation to each other can you figure out where they are in 3D space and the photography software is like yeah trigonometry or whatever and they can figure that out I don't understand the math but it can do it once you have that sparse Point Cloud which is now three-dimensional we can say okay can you fill in all the blanks add loads more points based on what you know and of course yeah this is your dense Point cloud so now we have this cloud of points in 3D space which roughly looks like our original geometry then if we want to turn that into a mesh that's actually really simple we just need to connect all of those points together with edges turn them into triangles and slap a projection of our original photographs on for a texture that gives us a photo scanner and it looks pretty good usually but it does have some huge drawbacks because the sparse Pine Cloud only has a certain amount of deer to work with they can't really do high frequency details like the leaves on a tree people's fingers hair you know very very small details cables wires things like that it just does a terrible job with them it kind of just makes this one Blobby mesh from a distance looks alright but usually doesn't look great it doesn't understand opacity it doesn't understand lighting effects like specularity and Reflections all that stuff just gets burned into the texture so if you move around a 3D scan of like a very reflective metal object it doesn't look like a smell at all it doesn't look like the reflection move because they don't but with Nerf we can solve all those problems the way that Nerf basically works is you start off the same way you take the the input video or the photographs but this time you give to an AI system the AI system uses interpolation which basically means it draws all of the frames in between the frames you've took so if you've took a picture every 10 degrees it'll fill in every single degree from every different direction you end up with this kind of fake photos figure now the AI knows what this scene should look like from every single Direction then once it does that it uses something called volume rendering in order to actually make this an output file right something we can view volume rendering is not a new thing it's been around since I think the 70s but the basic idea is that you have a volume right like a bounding box and you have the camera in various points around this box wherever you're moving the camera through you fire out a rare right this is called rare marching you fire out one rear and you can sample along that rear every whatever the distance unit is you take a sample inside that bounding box what we've already done is we've projected all of the the information that we have about the scene so every point in the 3D space has a coordinate every Point has a viewing angle like a vector to the camera it has RGB value and it has an alpha right opacity density so all we need to do then is we fire through array and we take a look at all of those values collect all those values then we go back through the scene and we rank them so we say okay well this pixel right at the back of the scene that's the first one then this next pixel is totally opaque right it's a brick wall so we're not going to say this Vector will make an ignore it and then the next one well that's a piece of glass so that's semi-transparent and I'll stack all these things on top of each other obviously you'll pick things will just wipe out whatever below them but now we have transparency and because we have the viewing angle information we can now have reflectivity as well so you can fold or scan or you can Nerf sorry um like a mirror you can take a photograph of a drink or something like that or a glass object an ornament and you can turn that into like something that can be viewed from any angle and it's see-through you get all these nice high frequency details because you don't have the limitations of a sparse Point cloud it's amazing guys in fact you can even do moving objects I've seen Nerfs of like where it also has a time element to on top of all these other factors it also like records how these pixels change over time so you can record like a moving person with the camera from every view then you can just like rotate that person around or rotate the camera around them you can do all sorts of mad like it's legitimately insane I'm watching the videos of this stuff and it's like compared to traditional photos coming it leaves them in the dirt now there's probably some things that it'll never replace about traditional photo scanning maybe AI in general probably will replace those things but you can just type in give me a tractor model or whatever in a can which we're not far away from but there's so many things that you can use this for and it's it comes with its own density map as well which is like ridiculously clean like for using compositing or for you know changing the background or something like that like having a nice density map is really important and it just comes with that out of the bag because it generates one of those as it works so anyway guys I just wanted to fill you in that this is a thing it's coming it's going to be awesome will blender have something like this in the future well maybe because actually they talked about a blender conference um there's going to be all this new AI initiative with blender to kind of keep up with the times so maybe something like this will come along I'll leave some links for you to read in the description um I know this is another just like talking head video and the last video I did actually say what do you think of these videos do you want to see more of them or not a couple of people actually came up to me at the blender conference which was awesome by the way and they said they like these videos so I'm probably going to keep making them uh by the way anyone if you did come up to me at the blend conference everybody who did was ridiculously cool it was a very weird situation for me to be in like being semi-famous for a weekend or whatever like having people come up to me who I don't know and just introduce himself that was very strange like I expected you know maybe five or ten people to introduce themselves and it was like I would say 60 60 different people came up which was bizarre but everybody was very cool people were pretty understanding that this was strange for me as well everyone was really polite I expected at least a few it didn't happen which is fantastic anyway I'm gonna keep making videos like this so people enjoy them but let me know in the comments what you think anyway thanks for watching this video guys I'll see you in a couple of days

Channel: DECODED

Views: 96,372

Keywords: tutorial, DECODED, Blender, blender 3d, software, computing, classes, blender for beginners, blender online, free 3d modeling software, open source software, 3d software, vfx, vfx artist, NeRF, neural radiance fields, ai technology, ai image synthesis, photogrammetry, stable diffusion

Id: -UEHhjuEXco

Length: 8min 42sec (522 seconds)

Published: Tue Nov 08 2022