How to View 3D Gaussian Splatting Scenes in Unity

hi I'm Jonathan Stevens and in this video I'm going to show you how to take 3D gajon splatting scenes and put them in unity that's right a game engine and from there you can do all sorts of things and I'm just going to caveat here I am not a Unity expert and I know a lot of people like other game engines but this is an interesting way to start and there's a great GitHub project which I'll show you guys Linked In the description on how to get going quick it only takes you a few minutes to get up and running and if you don't know how to make a 3D gaj and splatting scene make sure you watch my beginning tutorial I'll also link that in the description notes anyways let's get into the video and know at one point I kind of get tripped up in this and I will do a quick jump cut to say whoops I made a mistake and hopefully you can learn from the same mistakes I made see you guys in the tutorial okay the first thing you need to do when you're getting a 3D G and splatting scene into Unity is start with this GitHub repository that I'll Linked In the description notes it's called Unity gajin splat and it's created by this um this Creator RS uh I cannot say his last name very well so I don't want to butcher it but I'll make sure it linked it and I also suggest following this guy online he's got some incredible information on 3D Goan splatting implementations and he calls it toy gajan splatting playground in unity uh I assume you know what you're doing with you get into Unity but I will kind of walk you through how to get in there really quickly and he's got great documentation here so let's get to it so the first thing I need to do is bring up my command prompt I'm just going to type in CMD launch command prompt and I'm going to make this little big and I'm going to type get clone and then go to this code box here and copy this https string and hit enter and it's going to clone that repository uh into my user folder you you can always see where it cloned it to by default here it was my C users j n a folder and that's where it'll be so if I was to bring up my my folder structure here refresh it you'll see Unity gajin splatting right there so um that's good it downloaded everything I need and then you will also of course need Unity itself um it calls out in the project that you need 2022 do3 so if I go to mine I have 20223 do9 F1 LTS so that's good um I just did the base install and so if I go to that gajin splatting folder you'll see there's an assets folder and within the assets folder there's this GST test scene. Unity we're just going to double click to open that and you'll see um here we go continue access um I had an error that it just click through that said that there was a mismatch in versions his was for 7 F1 this is for 9 F1 just know that I had no problems with the latest of the 20223 so again this this takes just a second here um okay so now we loaded up um the the first thing we need to do is we need to go up to this top here where it says tools gaj and Splats and we need to create a gaj and splat asset so what essentially we're going to do is we're going to take a gajan splatting scene that you've already created and we're going to convert it to a format that Unity can read and so I assume you've already created a scene if you have not go back watch my beginner tutorial and make your first your first scene or you can find a sample somewhere I think he linked one of the samples within um the GitHub repository going click this input folder and um I'm going to go to this is my folder where I have my uh God splatting models and that it should be in your output folder I'm assuming and for this one I'm just going to pick pick this um this one of me now you want to make sure that you pick the model like the the container folder not this sub folder with Point clouds so just hit select you know it's right so it'll say file size for meet was one and a half million or one and a half gigs with 6.4 million point uh Splats in there and then your output folder um I don't change that but that's that's already a folder within the um 3D the the gajin splatting project that you downloaded called gaj and splatting right there with you're scene so um then there's quality so you'll notice at medium your file is going to be 287 megabytes 5.3 uh X smaller which is great but what does that mean how does that mean for Quality if I go back to the the GitHub repository and you go look at um rs's um profile you can find his blog and on his blog he has this great article called making gajan Splat smaller and I'll make sure I link this in the description as well but if you go down he shows you what those different presets look like what I think is very telling is that from very high to medium it looks pretty close and you can blow these up but like on this one the reflections look a little better when you're at medium but things start to fall apart when you get to low and very low um and so it's up to you for me I'm just going to go back to my Unity project and I'm going to go to very high I like to deal with the best and it's going to make theet even bigger which is kind of silly but that's fine um so hit create asset and this part should only take you seconds depending on the speed of your your your computer this might take a little longer and once this loaded we just got to do a couple more clicks to get it actually into your scene okay so now now it's loaded I'm just going to close this this here hit this X and then you want to click on the gajin Splats in your scene hierarchy and then here I'll say script um we want to take this asset here and we're going to put that in this uh not script in this asset here where it says missing so just going to grab this drag it right in there and Bam so now it's it's in the scene um and from here we can start to navigate around okay so now it's in my scene um I can navigate around it it a couple notes he said that your scene might not come in rotated quite right um but once you get in there so if I go here you see there is already a a rotation Matrix a rotation in there you might need to rotate that even further to get that that horizontal um but that's in there um and so that's pretty cool you have a few controls here you can change your Splat scale just like that other viewer you can make them even bigger than than normal um so let me just reset this back to one um your your spherical harmonic order so like how many colors each Splat has this I think might change the performance of the viewer but I noticed things like in the back here um looking at let's like look at this building you'll see how it makes a big difference in the details so I I like to keep that up high if you can um like so and then yeah there's there's just some other options you can play around with render mode this is all debugging but I think it's really cool you can see all of the Splats instead of as a point Cloud but as these like boxes and get an idea of like what Splats make up the scene so you have some big Splats you have some small Splats um but just make sure you're back into regular Splats when you're navigating the scene um and I am not a Unity expert so I may be doing a little a little bit of botching trying to go around here but I just want to show you um one thing that's cool so I I did notice that when you when you play with this there's no Collision layers and so you're going to have to like figure out how to set up some sort of collision if you want to navigate this as like a firstperson character um but I was also curious does this all interact with um postprocessing volumes so um I'm just going to do that really quickly so that's something that's fun if you are new to Unity and you just kind of want to play with this um you can you can add post processing so if I go to my my window my package manager and um I just want to find my postprocessing install that um it's going to install to the project should just take a second here okay so now it's on my project I'm going to go uh move my face here I'm going to add a component I'm going to add this post-processing layer first um and go down here and make sure the target layer everything why not um and then we're going to add a volume so now um with the postprocessing volume I can um make sure it's Global I can then start to add these profiles and then add an effect um I'm assuming you know what you're doing here I actually have only played with this somewhat but I'm going to put the color grading which I think is interesting cuz I got this nice color grade on here um one thing is I got to uh this will only really affect the game scene and for some reason it keeps coming in upside down so let's see here at this point I need to pause because when I added postprocessing volumes everything got all wonky with my camera I have no idea why so I had to go add a new camera operator in there and make sure that is set as the parent camera or else every time I made any changes I couldn't adjust the camera it was it was all wonky again I'm not a Unity expert but let's jump back into it and I'm going to show you what I did in case you run into the problem I had okay so I went to add um postprocessing to this scene so if I go to G and Splats you can see I have this layer and this volume and I added this camera by default and uh if I go to my main camera here and I and I go to um if this was like the one I was looking at every time I launched this everything was upside down so I had to go add another camera and I just nested in the main camera and that's my parent camera so if I click on it you can see I can move things up and down so if I let's say I look high it's looking up if I grab it again and look further down and look down that's what I was that's what I was expecting to have um so one thing you can do having like two screens really helps but I can split screen these um and so now you can see is I can I can move one the other you can see that okay and so once I'm happy with this camera um whatever I added earlier I got to make sure that I go read the post-processing so I added postprocessing layer not to this gajan Splats but to the sub camera so this postprocessing layer here turn that on make sure you you know you set it to what the target layer everything is um my post-processing volume make sure it's it's Global and then from here so now you can see my right screen with it on I can like change my contrast um I can make it really deep so make sure it's all turn on here okay so here we go I can I can change the contrast when I turn it on you'll see on the right screen things can like kind of dark and Moody um I can turn on you know like I can Crush my greens which is again like another Moody thing you could do um you all all kinds of things you can play with your your channel mixers um everything um what other effects we can add if you go in here there's all kinds of things that you can do um depth of field I mean that one could be very interesting as well um like play with your aperture focal distance um all all those different all those different effects that you can play with uh and get some really cool effects so um just wanted to show that once once you know what you're doing I'm assuming you know what you're doing here in unity you can take this camera you can make animations and you can get you can get some output that is going to be absolutely stunning and it's going to be super high quality so I hope this was helpful again I'm not Unity expert I'm going to dive into this but again I'm I I I'm just scratching the surface so what you can do once you get in here you can go VR you can do all kinds of things um I just want to remind you that this uh 3D godge and splatting project needs to be licensed to use it commercially so if you're just playing around for your own learning this is what this is great for um and if you actually want to implement that training process the revew all that that is not the IP that is not um that is not free IP you got to make sure you license that so if you found this video useful please subscribe to my channel and I'll be making more videos I'll have this in VR and Unreal Engine all sorts of other things that do with Nerfs and photos um and I find this helpful if you're running the same problems I do with unity it's probably just because you're you don't know how to use it like I do but um anyways I'll see you guys in the next video

Channel: The NeRF Guru

Views: 13,235

Keywords: #gaussiansplatting #nerf #nerfs

Id: 5_GaPYBHqOo

Length: 13min 40sec (820 seconds)

Published: Tue Sep 19 2023