3D Gaussian Splatting and NeRFs for Burning Man Art Preservation

so I got to go to Burning Man this year yes that Burning Man The Burning Man also known as muddy man but before the rain fell I was there with an amazing group of people whose mission it was to capture the art on PLA and preserve it for the future in virtual reality we even got these cool official media crew badges that entitled us to well absolutely nothing cuz that's Burning Man humor I learned a lot in this process especially about gaj and Splats and I thought it would be cool to do a video showing the process I went through to convert raw footage on pla to models for use in VR and in the future the art piece I chose in particular is called farmer of the riger and it's based out of La I'll include a link to this art piece below I chose it for a number of reasons the first of which is its size this is a very large art piece it stands about 18t tall tall and about 16 ft wide so it was a challenge to capture the other reason I chose it is honestly because we kind of screwed up that day when we showed up to this art piece we had a 360 camera that we used for doing just regular environmental capture but we forgot the DSLR so we don't have a bunch of loose photos that we normally would have for doing photogrametry what we did have is me walking around this art piece with an 8K camera and actually the footage that you're seeing here is 8k resolution footage but it was just shot on a Galaxy s22 Ultra and in portrait mode so I put it in 8k and I did two loops around the r piece the first Loop was about I height and the second Loop was about knee height I was of course Very diligent to make sure I kept everything centered as I walked around the art piece in the end you'll notice that we missed some very important data with this video most notably we didn't get high enough even at eye level there are parts of this model that are simply not going to be seen of course nobody is really tall enough to see the top of this model if we present it at real scale but even some parts of the model weren't scanned correctly like the tops of the wrists or the hands and the knuckles on the ground so the first piece of software that I wanted to try for reproducing this art piece was actually Luma Luma has done some amazing work this year on making neural Radiance Fields very easy to use and they also have the ability to export 3D models from their Nerfs so it's super easy to create an account and sign in and then uh once you've got your account created you just go into my captures and go into create luckily Luma takes video files directly so I was able to Simply upload that raw 8K footage to Luma and get a reconstruction super easy when it's done well this is what we got now this looks very impressive right off the bat and uh and it is I mean for the simp Simplicity of just uploading a video and getting this it's incredible but there are a number of things that I think are just maybe slightly misleading about this video this video is a preview of their own neural renderer that they have in the cloud and uh so the first thing that I am going to do is try try to figure out what kind of model we can get out of this for use in VR and if you click the cube button down at the bottom here and let me hide the background or actually let me get close to the model um now what we're looking at is we're looking at a web optimized version of their renderer this is not a Nerf renderer this is a 3D model renderer and you can already see some of the challenges that we start to have the moment that we try to get um from a Nerf a neur Radiance filled into a traditional 3D model there are a lot of struggles with details like chains um and you know of course no no fault of their own but we didn't provide them with all of the detail we're missing parts of the model back here but you know what this is a a web View and not a true 3D mesh so my next step was to see what I could get out of this uh if I download it as a gltf so downloading that as a gltf is a simp simple is clicking the download button and grabbing that and then um I like to use this gltf sample viewer that's available online and free and you can just drag your gltf model into it what's kind of nice about this environment is you can add different environmental images and see what your model is going to look like in different uh environments different lighting conditions so I like to switch over to this wide street because it as a very bright light in the background in fact if I turn off the blur you can kind of see where the light source is and you can even change the direction of the light source we go from negative Z to positive Z you can see what we get out of this model so yeah I mean this isn't the most natural light source for this model but you can really start to see some of the challenges um that come from kind of doing a Nerf based volumetric to 3D model approach this texturing obviously wasn't on the original art piece uh but this is a pretty darn good reconstruction biggest problems that we have are in these spaces back here that we just didn't capture the data again we're missing the a lot of detail about the hands we've got a hole Holes down here in the um uh this is where they had uh natural gas I believe or propane rather for for the burn barrels and there's just a lot of problems with the the chains here again um I don't want to fault Luma for what they've produced here with the kind of limited data that we gave them but any of these details any fine grain details like chains and uh you know thin metal pieces are always going to be very very difficult to reproduce uh in any kind of photogrametry pipeline so the next thing I wanted to try is rather than starting from a Nerf and going to a 3D model what if we were to use a more traditional photogrametry Pipeline and for that we have used reality capture so reality capture allows you to take in a bunch of photos uh and reality capture does have the ability for us to import a video directly but I found when I was importing the 8K video that reality capture really struggled uh with the import it was taking a really long time for it to get individual frames um extract those frames so I decided to help it along and I used a command line specifically I used the tool FFM Peg and this post on stack exchange was helpful um basically what you do is you just call FFM Peg - I for the video file that you're giving and then how often you want it to take a frame from the video and output it so in this case- R followed by a one would mean to export a PNG file every 1 second of video R 0.25 would mean to export uh frame every 4 seconds of video 0.1 every 10 seconds and so on and so forth so I specifically did R-1 I'm sorry- R1 so that I could get a frame for every 1 second of video and then I imported that into reality capture and you can see I still got a lot of uh cameras here plenty of frames pretty good coverage but again the problem is that there wasn't a lot of difference in the height we had a up high down low there's a little more difference in height than you can see here because remember this model is 18 ft tall but still we didn't get high enough to get the coverage that we needed the model that was produced by reality capture is pretty good um if I look at the back it did its best to fill in those holes and it did a pretty good job of filling in those holes um on the the propane tank as well as on the top of the hands but clearly it's still very much struggled with the chains those are completely missing the top of the jaw is missing here and really the biggest problem for us was the the top of the hands wasn't able to reconstruct this there were also a number of holes that I was surprised to see show up uh for example there's a hole here above the nose and um in the eye so we could certainly fix these uh it's not not terrible also another one up there it's not terrible to open this up uh in a tool like mesh mixer and delete those holes and have mesh mixer kind of average the texture across them but it been a lot of work to do this and again if we had been there with the right uh the right equipment and cameras taken a lot more photos different elevations um and and really done a good job of getting Source footage then this tool would have been able to do a better export but that's kind of not really the point of what I'm wanting to show today I wanted to show how far Advanced gajan Splats are with the same or even less input data so this is kind of right now the end of what we can do as far as virtual reality is concerned because once we go into native neural Radiance field rendering uh or native gajan Splat rendering there's not really a good way of doing it in virtual reality today Nerfs are way too slow and um and gajan Splats just don't have the the native VR renderers yet but I'll talk more about that in a minute still I know this is the the path that they're heading so I wanted to Circle back to Luma and give them another chance this time uh if I just kind of assume that we don't have to end up with a traditional 3D model I wanted to see how far we could push it by viewing the radiance field directly in an engine like unre so Luma supports Unreal Engine they have a plugin for it that allows you to see your light field directly and so this is where you download the light field I'm not going to walk you through getting the plugin set up for un reil engine but that's documented here and it's it's very easy to do once you have it set up you go into Unreal Engine and basically you just drag and drop your light field into your project and um and then you're given a number of options you're given the option for baked lighting uh or dynamic lighting and whether you want the entire environment or just the subject really quick here what is super impressive about Dynamic lighting is that this plug-in allows your scene to be relit so we could easily take this model and make it appear uh to be nighttime instead of daytime now there are some other great videos online that show that so I'm not going to walk through that here but it's incredibly powerful what I wanted my goal was to recreate the experience of being on Playa at this art piece as much as possible so I'm going to move forward with the baked lighting uh but just know that there's some really impressive capabilities here for being able to relight the scene as if it was captured at a different time of day different time of year different lighting conditions so now here we are in Unreal Engine and let me close this drawer and you can start to see really the power of having the light field uh the the Nerf itself in its native capability because we're starting to see a lot more details that we're missing before now we can see the chain that was missing here we can start to see um that the bottom of the jaw is more filled in there are uh no holes in this we've got some detail on the hands back here the chains but I do want to point out that this frame rate is not high I have the render quality set to high because I want to give the Nerf the best chance possible to look good but even when I set the render quality to low though the frame rate jumps up frame rate still isn't really good frame rates for virtual reality and look at what happens to the Quality we have uh lost all of the details with the chains and really we just have so much noise uh in the picture here it really becomes pretty much unusable so now I want to switch of course to the topic that everyone's here for which is gajan Splats these are a very different way of rendering uh we still start with Point clouds but rather than rendering those Point clouds as kind of volumetric f we render each of those points as a gradient on the screen and it's stacking those gradients on top of each other that give you the actual blending that is necessary for realistic looking light Reflections translucency um and all those effects that really make a scene appear photorealistic now it's important to realize that gaj and Splats are very much in the res search phase right now and so you're not going to go to the GitHub repo and find uh setup.exe that's going to put this on your machine ready to go you are going to need to go through the process of downloading code and building it uh before you have something you can actually run now I'm not going to walk you through all the steps necessary to get these tools on your machine but there's a YouTuber who goes by the name the Nerf Guru and he's put together a step step-by-step video that will help you get it running on your machine I'll go and provide a link to that video below now training a gaj and splat is like training a Nerf which means that you're going to start with images so if you're starting with a video like I was you'll need to use the FFM Peg trick earlier to export out those frames and when I did that of course I ended up with 8K portrait frames with 4 th000 something resolution by 7,000 um but when it was time to bring these frames into the training model for gajan Splats it really didn't want anything over 1.6k images so I started with the original frames but then I ended up having to down sample them and they uh I ended up down sampling them to images that were pretty much very close to the 1.6 case size and those are the images that I ended up training on now once your models trained and you're ready to view them you're going to drop down to the command prompt uh and run this executable and this first parameter here is the resolution that you want the renderer to run at so you can have the renderer run at a lower resolution than what your graphics card is capable of and that way you can get a little bit more performance out of it the second parameter here the iteration parameter is basically you can think of this as the quality of the model so when it comes time to train the data you specify how many iterations you want it to train and I knew that my graphics card which is a 3080 didn't have the total amount of memory that was recommended for training they recommend something like 25 gigs of vram but my 1080 only has 10 gigs of vram so they recommended that you only train for 7,000 iterations but there is a neat trick where you can force it to train on the CPU rather than GPU and then even have it Tim stamp or I guess do checkpoints as it's training so in this case I had it do a checkpoint at 7,000 iterations 10,000 15,000 etc etc so that if it ran out of vram and crashed I would still have a previous quality checkpoint that I could use to go ahead and visualize so that's what the 30,000 parameter is here which iteration to load and 30,000 is kind of the maximum that they recommend you train to um since I was training on the CPU it it made it all the way through and I'm able to load that model and the first time I loaded it it I was shocked I honestly thought something was wrong because when it first opened up I thought I was just looking at one of the frames for my video but then I grabbed the keyboard and I started to move around now I should mention here that due to the screen capture software running this is limited to 60 FPS but without it um this is easily over1 20 FPS and I can't even begin to tell you the night and day difference I mean you can see it for yourself but just look at the detail captured on the hands from areas that were uded or not even reconstructed properly with the other tools and there's noise there but it's very clear what we're looking at the um propane tank and the valve there there the fact that there's no holes on the top of the propane tank and of course the the top of the wrists again we didn't have the best angle possible when we had the original capture footage but it it did a good job filling it in the detail on these chains is just incredible unbelievable detail compared to before and the metal work too the metal pipes the top of his jaw is completely filled in there are no gaps and even if we come down here on the ground seeing the metal work down there it's just clear as day I was even happy to see how well it reconstructed his nostrils and this was the first time that I noticed what I think are lights up there I actually never saw this piece at night unfortunately but I'm assuming those are colored lights the detail that it captured in the eyes the kind of stained glass work is just incredible and then there's the details about the environment that it captured the barrels not only did it captured the symbols on the sides of the barrels but also these little LED lights that were out on the ground lighting up the barrels at night there's a barrel over here that only showed in a few frames but if we come over here and take a look at it we can see the artwork that was on that the side of that Barrel very well and not only that it kind of captured uh us and our bikes off on the side all from those two Loops of video and don't forget that this is down sampled right like this is isn't the 8K frames anymore this is now effectively 2K frames so look at the amazing reconstruction that it did from even lower resolution video now of course I don't know what Luma does on the inside when you give it an 8K video they may down sample it to 2K as well um but just the Reconstruction work here was really really unbelievable to me so where does this leave us well don't forget that gajan Splats are a brand new research technology they don't have any runtime support today the only true runtime support from any company is the support from Luma AI integrated in unreal and in fact during the production of this video Luma added support for web-based gaj and Splats so this space is evolving incredibly rapidly and of course something conspicuously missing from this whole presentation is Unity um a lot of VR applications today are built in unity and we don't have support for Luma or gajan Splats at least in any open- Source format there are some open source projects or at least projects listed on GitHub where they've promised to provide Source um but we're still a little ways away so I think ultimately this technology gajan Splats are going to provide us with the opportunity to have the most realistic Recreation of a scene possible but we're still probably a couple months away at least from being able to experience it anyway I think that's everything that I wanted to share I am super excited about this technology and if you're excited about it too please reach out to me you can find me on LinkedIn or at my website which I will also Link in the description below thanks for watching

Channel: Jared Bienz

Views: 8,904

Keywords: 3D Gaussian Splatting, Gaussian Splatting, NeRFs, NeRF, Neural Radiance Fields, Radiance Fields, Light Fields, Luma, Luma AI, Photogrammetry, VR, Virtual Reality, Unreal, Unreal Engine, Unity 3D, Unity, Reality Capture, Environment Capture, Rendering, Art, Art Preservation, Burning Man

Id: VXHWXy_pFYM

Length: 24min 6sec (1446 seconds)

Published: Mon Oct 09 2023