10 Things You Can Do With 3D Gaussian Splatting & Reality Capture

so midst the hype around AI image and video generation is another key vertical getting a huge boost the Art and Science of reality capture now reality capture isn't new per se but the level of democratization that we're experiencing has been staggering and of course the cost of computing power has dropped dramatically making photorealistic 3D model creation accessible on consumer Hardware the stuff that you needed massive data centers for and teams of computer vision experts you can now do on a decently powered desktop or heck even in the cloud and damn is it moving fast remember Nerfs only a couple years old and Along Comes gajian splatting which is faster editable and readily integratable we're talking 100 FPS rendering performance on a modern GPU I've already done an 8-minute Deep dive on gaui and splatting so if you haven't checked that out check it out over here but in this video I want to talk about all the cool stuff that you can do with techniques like gaui and splatting and reality capture in general so get ready for some visual Umami it is remarkable to witness the pace of progress in this Timeless quest to connect the physical and digital worlds number one memory capture this is to me perhaps the killer consumer use case for 3D and 4D capture check this example out my parents retired earlier this year and I've immortalize their home forever more in this gajian Splat photos scanning is perhaps the most futureproof medium we have access to today so go out there capture the spaces places people and objects that you truly care about it is perhaps then unsurprising that apple is leaning into this very direction for their own spatial media initiatives 3D photos and videos that are intended to capture moments that matter to you the most of course so you can view them on a Vision Pro headset but it's not just static scenes they're also focusing on 3D video that you can capture with your freaking iPhone and let me tell you stereo magnification off a narrow Baseline is very very possible the next one is reality bending effects given how close gajian splatting is to a 3D Point cloud representation another benefit over Nerfs by the way creators like Reuben are adapting their point Cloud shaders to work beautifully on 3D capture creating reality bending magic sort of like this I think it's only a matter of time until we see these effects in some sort of music video so since we have this easier to edit representation with gajian splatting versus neural Radiance feels the level of creativity and manipulation you can pull off is amazing check out this other demo by Ruben that's giving me Oppenheimer Vibes meets of course the Thanos snap but you don't have to just blow up the world you can also make some very clean animated reveals using gaui and splatting no surprise that everyone's obsessed with lum's loading animation I mean just check these examples out in fact I used it in a recent sponsored engagement with Lenovo as well and they loved it too you can also reskin 3D captures with generative AI in other words taking the output from These reality capture tools and running them through something like kyber or Runway gen 1 I just got the new DJI mavic mini and I just had to put this to the test making a 3D drone combined with a healthy dose of generative AI the results are amazing you can create some very captivating Loops well suited for social media and it's not just static captures of the world you can also create Dynamic 3D scenes there's a bunch of approaches being explored in Academia here and they're a huge step up from the rather GTA 3 looking videogrammetry pipelines of yester year that you may remember from the likes of Microsoft now the potential for dynamic capture is absolutely amazing because you're not just capturing a stat place and reframing it infinitely but you're doing a legitimate performance capture of dynamic entities in the world and then you can do Alters of crazy stuff in post- production looking at some of the tracking markers over here just imagine the potential for visual effects you could easily attach particle emitters objects Etc to Dynamic captures that you do but it's not just quality you can also get really really good performance I mean just look at these results the ease with which you can distribute this stuff is way higher I mean you're getting 80 FPS performance on a consumer GPU right now if you throw in other compression techniques I wouldn't be surprised if you're streaming Dynamic 4D gajian right into your browser very very soon now a caveat with these approaches is that you do need a multicamera array to capture this stuff so you can't just take a bunch of iPhones you need to have cameras that are synchronized that you know the intrinsics and extrinsics for but as we discussed earlier I suspect this will be a very interesting line of research for academics but also apple as they make 3D video capture more accessible with say an iPhone relability is another Direction that's being explored I love this example by infinite realities you can see how this is the same person in a variety of different lighting conditions and we're looking at the gaui and splat the way they pulled this off is they captured Henry in a light stage where you can use an HDR image to actually light the subject in a certain way but there's also a bunch of interesting exploration happening to make this more accessible as well where you can take your static gausian Splat Delight it and then relight it using neural rendering techniques I've been playing around with a tool created by a South Grand company called bble and they're doing an amazing job in this space just take a look at this I can take a capture of myself and swap out a bunch of hdri environments and notice how it's accurately relighting me and if you go beyond just using the albo maps and the normal Maps they give you and use their neural rendering engine they can even simulate light transport effects through your skin stuff like subsurface scattering which will make it a lot closer to that Henry video that we just saw now there's a lot of other cool stuff that you can do by virtue of bringing this into a game engine you could take the photo realistic 3D tiles from Google use that as sort of your scaffolding to Anchor these scans and get the surrounding context from Google but have the high detail ground level Precision that you've captured with your own scans you can make some really really cool stuff with this you've got plugins for Unity and unreal as well so you can bring these into a game engine and do all the other cool stuff that you're used to doing in these type of environments a great example here is this one by bad decision Studios where they did this capture off a boat in Dubai of this iconic landmark and then took it all the way with Unreal Engine and I call this use case kid bashing reality where you can start building this library of the spaces places and objects that you care about bring them into a 3D environment like unreal or Unity compose them together and pull off amazing results like this that would have taken a lot more effort if you did this all manually you can almost think of it like 3D screenshots for the real world right so start building your own stock library of these 3D assets so you can use them in your future Creations speaking of distribution not being a problem here's a really cool demo by lawn Labs running volumetric video captures on a freaking Apple device so you've got Unity you've got unreal you've got browser and you've got metal implementations so again distribution isn't going to be a problem and many of these folks haven't really even gone down the rabbit hole of asset optimization streaming optimization and so forth so plenty of Headroom there for distribution down the line there's also really cool state-of-the-art stuff that's been happening in the past here a Google did a really great jot with deep view video just a couple years ago using this sort of layered mesh representation that they derive from the previous state-of-the-art before Nerfs called multiplane images or MPI this data set is out available in the public so look out for them when you see newer research papers you'll find them reprocessing these older results again this was captured with a synchronized array of action cameras but I'm excited to see what happens when capture becomes a lot more democratized now kit bashing is really cool but what if you want to do more fine grain edits of your Splat I think there's a ton of potential here and new tools on the horizon that make this possible for example spline recently added gaj and splatting support so you can import your py files crop them and compose them with a bunch of web and 3D assets like you could take your landing pages to the next level with this type of stuff and obvious implications for e-commerce as well so play canvas has this super Splat tool where you can say take a subject capture isolate it and then put a different environment map whatever HDR that you prefer really really cool implications for e-commerce I mean obviously sneaker retail comes to mind because that's just such a popular use case when showcasing object scans but also think about the implications for fashion imagine going to Banana Republic or Zara and seeing captures like this to see what something would actually fit like and then in the future maybe you even upload your own skin now prograde virtual Productions have already been using reality capture just look at Mandalorian but gazi and splatting and Nerfs are going to be that next level of detail to have these really rich background plates that you can add your subject matter to and of course relight everything accurately and Beyond the Mandalorian virtual Productions of the world gaming is already been using reality capture just look at games like Battlefront or Call of Duty these folks figured out that it's easier to take that DSLR go out and capture that nondescript La Backlot and capture the complexity of reality even if you have to optimize it after the fact kit bashed together in a broader 3D World it really really brings everything to life and makes it look far far more closer to the real world way easier than trying to do that all from scratch now Heritage conservation is another amazing use case poly cam is actually working with UNESCO to capture various landmarks and monuments that are being impacted by the Ukraine war currently taking place I had a chance to sit down with Singularity University's publication to go into greater detail on why we're seeing this democratization and also the amazing use cases that it unlocks if that's of interest to you check it out in the description below so to bring this all back all of this is moving towards this future of spatial Computing and spatial media now while terms like metaverse are absolutely overused and might draw a bunch of eye rolles the underlying spirit is the recognition that 3D environments 3D captures of the real world truly matter this is one way we can start connecting bits and atoms and create this connective tissue or substrate between the physical and digital worlds that we operate in every single day and techniques like gaui and splatting are going to play a key role in that connective tissue the real large scale ambition here is the hope for a realtime 3D map of the world now we've already had stuff like Google Earth immersive view Apple maps that give you the static 3D rendition now we finally have the Technologies to keep this stuff up to dat in other words there's the building the model of the world and then there's the maintaining that model of the world and with all of these methods that we're talking about I think we may finally have technology to solve that maintaining the model part through crowdsourcing that visceral feeling of being in that space or place is just so powerful but even if we leave out all the VR applications the content creation and commerce potential is off the charts so I hope this video gives you a good idea of the kind of stuff that you can do with reality capture now reality capture is a massive space to deconstruct but I hope this gives you an idea of the types of stuff that you can do today with reality capture techniques like gaui and splatting in the next video I want to go far deeper into human capture what are all the big Tech players doing what is the state-of-the-art in research and Beyond just environments how do we bring ourselves into the virtual realm whether it's for telepresence or media and entertainment so stay tuned for that video all right so that is it for this video If you enjoyed this and want to get this type of content sent neatly to your inbox consider subscribing to the Creative Tech digest it serves as a companion to this YouTube channel where I give you all the links that you need to go into this stuff in a lot more detail that's it for today and I will see y'all in the next one

Channel: Creative Tech Digest

Views: 26,600

Keywords: gaussian splatting, 3d graphics, 3d rendering, real-time rendering, neural graphics, novel view synthesis, computer vision, photogrammetry, nerf, vr, augmented reality, virtual reality, game development, unity, unreal engine, 3d modeling, 3d capture, 3d scanning, deep learning, machine learning, ai, computer graphics, future of 3d, ai for 3d, vfx, geospatial, apple 3d photos, apple 3d videos, apple vision pro, editing gaussian splats, radiance fields, visual effects, ar, ml

Id: RG_1OybkeRU

Length: 11min 34sec (694 seconds)

Published: Sat Nov 25 2023