The End of Photogrammetry? (Lidar and Blender)

so recently i've been having a lot of fun playing with lidar scans and combining them with 3d artwork now for those of you that don't know lidar is basically a kind of technology that lets you get distance data as well as other things the actual word lidar is a portmanteau of light and radar but some people have other definitions for as well such as light detection and ranging or laser imaging detection and ranging it really doesn't matter it's all of these things in one now when talking about lidar for 3d work what we're basically saying is using lidar data to construct 3d meshes based on the distance data that's returned to the piece of technology and you can use specialized apps to take the data and turn them into 3d models which you can then export and use them in any software you like so recently i've been playing with the iphone and with an app called polycam there are other apps available as well but i haven't tried them yet and the results have been quite impressive now i've always been interested in this idea of taking stuff from real life and using it in your own work because it adds this whole other dimension of realism and it just generally adds details that you would never think about adding yourself because they're either too mundane or would take too much time to put together but one of the reasons i've shut away from regular photogrammetry is because for me it's a bit too much of a time investment there's also quite a lot of cleanup involved when doing photogrammetry scans and lidar is no different but having the distance data there really makes an improvement to accuracy so i'll show some footage on the screen now basically you just have the app open you hold it up and you can see a visualization of it trying to construct geometry around what the camera is looking at and you can optionally choose to show or hide the color data as well and this just proves that it is actually using distance data basically it uses the lidar data to construct the mesh and then the photo data to construct the textures that get wrapped around the mesh so just by walking around at different angles and pointing your camera at different objects rooms environments it will automatically construct the model for you and in the case of the polycarb app if you spend a little bit of money you can export an unlimited number of scans for example you could export to an obj email that yourself then open that up in blender and because the material file is also associated with it it will auto assign all the textures so you can just open it up render in eevee and boom your scan is there now if you have a semi decent understanding of how to manipulate 3d objects and textures you can then clean this up use the environment for your own artwork or even steal props from it now the whole idea of stealing props and items from the real world and using them in your artwork is not new to the community in fact i highly recommend you watch the world building in blender speech by ian hubert on the blender channel on youtube because this goes over a similar process that's only using 2d imagery where if you take images you can then essentially project them onto geometry and then you can extrude areas add whatever geometry you like duplicate some areas use extra images for more detail and then you can basically construct props using the images not only as references but also as sources of texture data and then of course these props can be saved and added to a library and reused again in the future now with the lidar scans this is interesting because you don't need multiple images all you need to do is press record have a look around the object and then you can take it away now there are of course some limitations namely the mesh resolution because in the case of polycam it seems that the mesh resolution is kind of tied to the world space when you're processing the scans you do have some options for the voxel resolution where you can reduce it as far as possible but even then it's still not viable for small props so this is really more useful for doing actual environments rather than small items also there are issues where if you have an object even if it is quite a large one but it's perforated and it has holes in it the algorithm has trouble interpreting where the geometry should be constructed around that object so sometimes you might end up with strange texture data where objects should be see here where i scan this garden the bench is kind of splayed out onto the floor it doesn't exist but a shadow of its former self has been merged into the texture data but putting those inconsistencies aside there's so much potential here for the real-time scanning of entire environments environments that you can then remix and re-use in whatever projects you like so it's been really fun playing around now i've managed to create this ufo artwork i'm calling bad parking where i've basically taken my garden and crashed a ufo into it now you can tell from looking at this that it's a combination of the lidar scan data and regular 3d models and like i said at the beginning with the scan data you get all these kind of mundane details that you would not have added otherwise like take the back fence for example i'm never going to be bothered to make that manually sure you could texture paint that if you had enough time but for me personally my patience is very thin so i like just being able to slap things together and make something pretty being able to have all these realistic mundane details helps you to really compose a scene in a very short amount of time now because data is data and it's all accessible in blender it doesn't mean that the style of your artwork needs to be strictly tied to the scan you've created for example because the texture data is attached to the materials and the shader node system in blender allows for all kinds of procedural effects i can combine my ambient grunge node with the texture data and then procedurally add dirt to the scan so you can start layering up all kinds of procedural effects on the data that you've imported now of course some cleanup is required when you import the scans so for the geometry you might find random bits of mesh sticking up out of nowhere where it hasn't resolved properly if it's subtle you could fix this just by using the scope mode and smoothing it out if there's something extreme that's gone wrong then you need to go into edit mode and manipulate the vertices manually but you'll need to be careful when making new geometry because the geometry that's already been created has generated uv data attached to it so if you make new faces they will not be appropriately mapped to the uv map but of course what you can do is generate geometry from the pre-existing geometry for things like loop cuts and using the knife tool so you had a round surface that didn't look quite right you could use the knife tool to draw a line around it and then scale that down so you're basically manipulating the shape of the scan now of course what's very useful here is the proportional editing so if you're in the edit mode and you press o on the keyboard this will enable proportional editing so if you start dragging vertices around you can scroll on your mouse forwards and backwards to manipulate the threshold so as you're moving the points there's going to be a falloff where it affects other nearby vertices as well now that's not just for moving that's also for rotating and scaling so it's incredibly useful if there's any areas where the distance isn't quite right or things have been blown out of proportion you can kind of smoothly manually move them back into place so the other main issues come from the texture data sometimes as i said things will exist in the texture data that don't exist in the 3d geometry or alternatively because you scanned it in a slightly bad way and the textured data is only available from a skewed perspective from where your camera was looking things may be a bit jagged and strange and really off proportion this is where the texture painting mode in blender is really useful and especially the clone tool because with the clone tool you can take pixel data from somewhere else on the textured object and then paint that where things are inconsistent so with some careful manual effort you'll be able to resolve these inconsistencies and make it look a bit more realistic of course things probably won't look perfect up close but if your camera is in the right position then you should be able to end up with a possible result now another little technique i did in this specific artwork was take the tiles on the floor use the knife tool to draw squares around where the tiles were then assigned these to a vertex group and then once they're all cut out i then extruded them upwards to give the ground some more depth now this is useful because the mesh resolution from the lidar scans isn't particularly high so by going in yourself manually and adding these extra extruding details it really helps to sell a sense of depth then if you have this data and you fixed it up then what you do from here is completely up to you there's so much you can do with this and there's so much potential for making 3d artwork even for animations i mentioned ian huber earlier because he's known for doing lots of this kind of stuff but you should definitely check out his work because he's quite popular in this community and he's definitely more from the side of visual effects and combining real-world data with 3d artistry now of course the elephant in the room is the trying out this lidar scanning technology and these apps with a mobile phone has quite significant monetary barrier because as i said i've been trying this on the iphone 12 pro there are of course other phones and devices with lidar scanners but not every app is available for every device so if you do want to try this technology out then you should do some research beforehand otherwise as i said there's other very viable options like regular photogrammetry so taking images of a 3d object at multiple angles and then reconstructing the geometry afterwards there's tons of software to help with that and alternatively there's the camera projection or uv projection method outlined in the talk i mentioned where you basically take simple images and construct objects from those but overall it's very fun to play with and it can give you an entirely new perspective on generating 3d artwork now this final blend file is available on patreon under the 10 tier and i also want to say a big thank you to everyone that's been purchasing the modular metals pack so far it's the biggest pack for blender i've made up until this point which is naturally why it's the most expensive one okay so hopefully you found this video interesting let me know if you've played around with lidar scanning photogrammetry or uv camera projection from images so thanks for watching stay safe and i will see you next time [Music]

Channel: Curtis Holt

Views: 1,713,959

Keywords: blender, lidar, polycam, iphone, scanning, photogrammetry, curtis, holt, 2.9, cleanup, scan, 12 pro

Id: exeagQ4g3-I

Length: 8min 37sec (517 seconds)

Published: Thu Dec 31 2020