3D Gaussian Splatting in UNREAL ENGINE 5 is INSANE!

do you guys remember when we use 3D gossan splatting to turn videos into 3D scenes like this what dude well the default use case at the moment is just a 3D scene in a viewer and you can't really do much with that we want to use these g- spass and the only way we know how is to bring it into Unity poor kidding Unreal Engine 5 because once we have it in unreal we can use it for cinematics for VR virtual production and even unreal engine fortnite editor but there was a problem how do we import this into Unreal Engine and that's when we started searching the internet until we found this one guy whose name is Kenji who thought about the exact same thing and decided to create a plug-in in the Unreal Engine Marketplace that converts these gajian Splats into Niagara particles that we can view in real time so let's go and freaking do it so here's the plan we trained three scenes for today's Unreal Engine test the pyramids in Egypt the steam clock in Gast town and downtown Vancouver we're going to judge how they look in ui5 run some performance stats to see how they run in the engine and we're also going to play with some of the plugins tools such as the cropping tool we'll even set up some cameras and see what shots we can take let's Jump Right In if you have no idea of what 3D goian splatting is we have made a detail video make sure you check it out we are right here behind the PC and testing out our first experiment we got to give a big shout out to our friend maget who sent us the Drone footage of the pyramids so these are the image sequences that we submitted for the g-s spats and just so you get an idea right these came from a 1080p footage so they're not anything fancy and the Drone flight path is actually a half orbit by the way just look at how beautiful this looks it's crazy I really want to visit Egypt I know I think we will go we have to after the Egyptian documentary so it's a half orbit and then it goes up in altitude gets closer to the pyramid and then go goes back into the same direction and that is the footage ladies and gentlemen we present to you the pyramids of Egypt in a new L 5 baby let's go this is so good dude I cannot imagine how fast we could produce our documentary if it had this in our in the beginning which you recreated the whole pyramids I swear to God two seconds of video far can you just please understand how crazy this is we saw the input just now with you guys you guys saw the images that we gave we have the exact same thing in Unreal Engine 5 dude this is freaking crazy look at the other pyramid you have the other pyramid at the back and I think you have the other one on this side too we do and you know what's crazy can you see the human can you see the humans here so tiny can you see the humans and the bus the one thing we couldn't get with Nerfs was the tiniest detail like humans or even the smallest cars but it does come with a disadvantage when you look at the gausian Splat if you look all the way into the distance you notice that everything sort of Fades out there is no detail in the background which is something we usually got with nerves have you added any post-process volume to this or not no but I can do it right now I've already prepared it so postprocess volume is for us to enhance the visual aspects of our scene so I just set up a widescreen camera right here and we're going to turn on our process right there so it's a very tiny difference because I didn't want to add too much it shows that we can change the visual aspects of our Splats and we can improve them once we get them into on real engine right just yeah look at yeah look at this look at how even though the background is faded and that could be JY now it's it looks hard yeah you can use it dude there's no way we making a cinematic or what should we make a cinematic right now we should do it right now do it 3 2 [Music] 1 [Music] before we go into our next experiment the only thing I would say is sometimes you have these floaters in the air do you see those grids oh those grids are essentially the ellipsoids they're the ellipsoids that are floating in the air so luckily after they converted to Niagara particles right now thanks to kentry and how they process you don't really notice them in some angles but if I were to go into the blueprint right here and show as mesh see these are all the floater in the air do you see them dude this looks like some Assassin's Creed yeah dude when I was reading the paper it was saying in every single render there are 1 1 to 5 million gossans right imagine if Kenji actually adds in a paintbrush tool that you can use to remove the gos ends you don't need you can clean up all these areas all the mess so the crop tool is there and we're going to show you that in a second but imagine if you just want to go ahead and hand paint these specific ellipsoids right here because they're blocking your view for virtual production for cinematics cuz that's what's happening to us right now shall we go to the next one let's go to the next one I want to show you guys how we import them it's really as simple as just clicking this extra button that gets installed after you bring in the plugin and then selecting the point Cloud that is generated once you create the goian Splat and for the next one fire which one do you want to try let's try the stee clock can show the raw footage ra ra guys mind you that this was done 530 a.m. in downtown R that's why no one is there usually there's a long queue beside this steam clock all right I wonder how this is going to look because there's steam literally coming out of the clock you can see oh oh this was me this was me yeah this was me if you look at me I'm doing Nerfs in the middle of this stre wow you wait did you even keep this left and right thing you kept all of it yeah I kept everything I didn't cut anything oh wow it's three Pi orbits height three Pi orbits and it goes around okay cool now we know what it looks like and it was very dark so there was not a lot of light it was very early in the morning but now we get to see what it look like in 3 2 1 turn the lights on what look at that wait let me rotate it yo okay let me turn on the post post process turn it off this is exactly like Harry Potter can he go back can he go back wow can he go back this is where Harry Potter he doesn't fly he tell ports into Dagon alley and it looks exactly like this going going going yes wow dude it even captured the steam yeah do you see the steam it's like this fuzzy looking you got the signboard perfectly you can read it Water Street water stre when you get closer you can see all the Splats getting pulled to represent Water Street but from far it looks perfect can you try go behind this lamp I don't think it will work because we didn't capture that angle yeah it doesn't yeah you see the lamp because you got oh the lamp L it works from here yeah yeah that's true dude the pavement looks perfect look at these look at the chains look at the chains can we try something now can you crop the end of this street for me like so we don't need it right imagine I don't need it all right we're going to crop it using the blueprint right here wow this actually looks even cooler wow I'm going to bring the scale up so I capture more yes to see this side but imagine if you could just goad and paint it off even in the middle of the scene even right here you have some ellipsoids which you don't need luckily they don't show right now as you're looking at it but you know they could become a problem but just look at this okay I'm going to set up another camera really quickly and we'll be right back look at this wow all you have to do is just find out one or two Dynamic elements that it can add to your scene just make it so much more realistic and just like that matter now yes 3 2 1 61 the W reminds you Sunday December 1st oh no Sunday is another month already so for the final and last experiment we are going to try one of our biggest scans P show us our input data first okay so this one was a drone shot we did in downtown Vancouver and you can see it's beside these trains and it goes to a half orbit or I don't even want know what to call this but it's a giant orbit and it was extremely slow as you can see it was shut at 4K you know really high quality and it goes around this Microsoft building actually if you can think about it that way and now we're going to show you what it looks like in full in on Lon 5 you ready for this let's go boom wow we got a whole town in there man dude just look at the Quality this is I didn't even want to move the camera just look at the Quality look at the buildings do you sp I see the Microsoft building I can spot the you want to get closer yeah let's go to Microsoft building boom okay so if you get really close you of course you can see the Splats but look at the fact that I can even distinguish that there was a table and chair here there's one there's one table in the corner here you can kind of spy on people with Splats and look at this building look at this trees Building look at trees completely behind the building wow the street we're just going really far off into the distance right here but if you really want to pinpoint you can even these buildings back here look at this this is madness this I don't even know how far I can go how far you are now from just super far let's go back look at the other side look at the other side let me see how far you are from the train wow Train's right here so let's talk about the FPS sure actually that's what you want to do let's talk about the performance yeah let's see how fast this is running in engine right now and we're running at 80 FPS that's standing still and if you move around it's still around 80 FPS we're running this on a 409t but mind you I have multiple tabs open on Google Plus recording this in OBS at 4K resolution so you can imagine this would be around 100 plus FPS right now as we speak and found I have the post process on so if I take it off you you can see the difference right it gives a little bit more vibrancy to the colors of the building so that's really good and I would say the only downside to this is the sky again so the same problem we had in Egypt the same problem we had with Gast town we have the same thing here cuz the sky is sort of covered it's full with all these yeah so if you come all the way up here you'll notice that let's say we want to switch this to alra Dynamic sky right because we want to replace the sky it doesn't really work number one because the alter dnamic sky has the atmosphere effect so you do get all these little holes that are now filled with the effect plus the sky doesn't look like a sky because you can see all the ellipsoids being stretched out and that was a problem we didn't have Nerfs right with Nerfs we could easily replace the sky and have it work so that's one of the downsides here but I hope this can get fixed because if you can just remove the ellipsoids you'll have no problem I don't know how to deal with the floor though that's another problem we have to deal with that's it for another you want take a cinematic out of this out of this yes as well 3 2 1 today Mr Huxley says that his fictional world of horror is prob just around the corner for all of us all right there you have it guys that was our first ever experience and impression of bringing in gats into Ang Engine 5 not the last one right not the last one for sure and guess what you guys now know what it looks like and what the possibilities are but farad we're going to do more tests so we are going to try this with virtual production we're going to try this to create more cinematics and are we going to try the D Legend fortnite editor fortnite editor yeah sure why not we can give that a try as well because Kenji actually brought in a documentation on his website about taking this into ufn we're going to test that out too one more thing we're going to test out is Nerfs versus gats because clearly there's advantages and disadvantages to both so if you guys are having fun with us so farun yeah having fun with us experimenting you should subscribe you should consider sub subscribing to to this channel to continue having fun having fun with that's

Channel: Bad Decisions Studio

Views: 78,693

Keywords: radiance field, gaussian splatting, neural radiance fields, Nerf, AI, 3d Scan, 3d

Id: KaAxGr_xck4

Length: 12min 29sec (749 seconds)

Published: Fri Sep 29 2023