#UE5 Series: Introduction to Gaussian Splatting in UNREAL Engine

hello everyone RZA here and welcome to my channel in this video we're delving into the fascinating world of gajian splatting we'll explore what this technique entails its benefits limitations and even delve into integrating it into Unreal Engine with some handy controls for tweaking the final outcome without further Ado let's get started [Music] [Music] let's start by understanding Gan splatting unlike traditional photogrametry methods Gan splatting is a volume rendering technique that directly renders volume data without the need for converting it into surface or line Primitives that's a big difference there are two main types of Gan splatting techniques the first one is 3D Gan splatting and this method probably the one you've seen showcased on platforms and social media utilizes the radiance fi approach to connect the parse Point clouds generated during camera calibration it offers realtime control control over the size position rotation and translation of points resulting in higher quality photo realistic outcomes however it comes with some limitations it uses high vram usage so definitely check your computer spec before you dive into gajian splatting also it occupies large disk space some of these scenes are up to one gigabyte for a landscape scene it is at the time of the recording incompatible with existing rendering pipelines for example the one that I'm about to Showcase in on real engine does not support Lumen and of course real time results are static but for time being who knows what's waiting for us around the corner the second method which to me is a bit scary is 4D go Gan splatting and that builds upon the foundation of 3D gajian splatting and this technique introduces the time component adding another dimension to your rendering process I got to love that both methods are evolving rapidly hinting at the dynamic future of computer Graphics now we know what Gan splatting is let's actually dive into unreal engine and explore ways we can bring gin splatting into Unreal Engine and some handy controls that are available to us to view the [Music] result now before I start with Unreal Engine I just want to show you what plugins are available and how to install them you have two ways of doing this the first one is free and the second one is paid on Marketplace this is the website by radians fields.com xers is the name of the plugin as you can see it's free Unreal Engine plugin only compatible with 5.3 at the time of the recording so you basically follow the breadcrumbs this is the GitHub page which allows you to download the zip file and some instructions are in the same page available to you to be able to follow and install but just to show you uh it's it's very simple once you unzip the file you bring in your program files find epic games in the program files Unreal Engine 5.3 engine plug-in and all you need to do is to select your zip file and drag and drop it into this folder I've already done that as you can see XV for X verse 3D gajian splatting now with that in mind let's go ahead and open onreal engine 5.3 here I am inside on real engine 5.3 the first step we would like to make just like any other plugins than we bring into Unreal Engine for the very first time is to activate it so let's go to edit and plug in and in here I'm going to type in XV for xers and it brings XV 3D gajian splatting I'm going to enable and restart now we have everything loaded you may have noticed that I have a letter X and if I go into Windows I also have XV 3D GS and both buttons are doing the same thing and they allow you to bring in a gan splatting scene into Unreal Engine what I'm going to do to make sure that things are visible I'm just going to remove some of these guys sphere reflection we don't need Sky sphere we probably don't need and Atmospheric height and fog we don't need now I'm going to click on this icon there is only one button import plyy file and plyy is the extension for any gajian splatting scene before I import the scene I'm about to open is created by Alex devvo he graciously granted me permission to utilize his scene for this specific demo so be sure to check out his work I will include the link to his page and the model used for this particular demo in the description below so I'm going to go open and select the scene we have the information inside Unreal Engine I can go ahead and close the plugin window we've got the blueprint and some supporting files position rotation scale and color information to allow this blueprint to work it's as easy as dragging and dropping into the scene you can see the boundary of the gajian splatting any type of scene that you bring comes with excessive amount of information things that you don't need I'm going to go and set this one to 90° my rotation pressing e to switch to my rotation and I need to rotate it in Z and rotate it in y AIS now if I zoom in you can see that the scene is coming through now before I dive right in and show you how to take what you have to the next level let me show show you the second method of doing exactly the same thing the second method is to use 3D gajan plugin created by Ikea Research Institute again the way that you work with this is exactly the same thing let me show you how once you download the plug-in simply drag and drop it into your plugin folder I guess you know what to do let's fire up andreal engine 5.3 again I'm going to go into edit plugins and this time I look for 3D gajian I'm going to enable that and let's restart and now that's done instead of having X I have an icon here that I can click on or window import 3D gajian they don't offer any particular options they're just simply allowing you to open the file and the rest happens inside onreal engine and it's almost identical if I were to just point out a minor difference is with the paid plug-in I get slightly better consistency so during working with the XE I had moments where Unreal Engine crashed it never happened with the paid plug-in that's the only difference now with that out of the way let's go to the next chapter pick one of these plugins and see what we can do with the pl gajian splatting scene files now before we talk about how incredible gajan splatting is and how it looks inside onreal engine let's talk about limitations the first one is if I go into the content drawer and look at the blueprint class you're unable to access the internal structure of that even if you go ahead and double click on it it's only going to clone the properties that you already see inside the details panel so it's not going to give you anything extra all you have through Niagra system is the user parameters and we did talk about user parameters in our previous videos where you can extract a property from an internal structure and showcase it in the main user interface and anything you see in user parameters is all we have to tweak the content so that's the first one as for real limitations would be the plug-in renders 3D Gan splatting data using Niagra particles so because there is a limit to the number of particles in the Agra system sometimes any plugins that you use import the data into multiple blocks and those blocks sometimes create seams between the blocks so that's why sometimes you look at the data that you bring in and you see something like cards if I go into wireframe you probably see how they look so sometimes you see these seams and each card contains the data and the system is blending them together so that's one limitation is um well you see seams and artifacts the other one is lighting support is experimental while there is a way to somehow relight the scene it's not going to work quiet as you expect anything with translucency and Subs surface scattering are going to pretty much fail because the light cannot go through and Scatter these are points they're not Primitives remember that's the main difference between this and a geometry and of course Lumen is not supported Shadow is not supported Ray traced translucency is not supported now you may say all right can you res it say what we can do in the engine well there are things that you can actually do to present your work and showcase it the first one uh which in my opinion is one of the most important ones would be Point Cloud image size now Point Cloud image size shows how big those points are and based on the size they are going to blend with each other and as a result of that you get a better result I am going to centralize this point Cloud inside my bounding box and this bounding box is very important why because we can crop excessive content using this bounding box as a matter of fact I'm going to scale up my data five times maybe 10 times yeah 10 should be good so I can see things better sometimes you bring let's say a city into your scene and it's very small and because of that you will struggle orbiting around the scene so make sure that it's big enough so you can actually get into the point clouds and tweak with regard to this um Point Cloud size if you go too low it create bucket holes because those points are not close enough to cover the surface and if you go too high it somehow softens the result so if I go let's say 800 that's way too much not to mention the blending of the points are going to create also artifact if you go crazy high you cannot see anything because there's a limit to what Niagara can show based on your computer spec so if I go let's say 2,000 chances are I'm not going to get anything I think for this particular scene uh 600 would be a sweet spot and if I zoom in you can really see how in in credible the main object is and it's just mind-blowing to think about the fact that it's not a geometry so you may say that's all good how can we get rid of this excessive amount of information around the main model well you can actually crop so if I search for crop it says in the control type what you would like to crop and in the cropbox scale it says how much based on the bounding box you would like to remove so if you change this without changing the control type obviously you're not going to see anything so I'm just going to switch from Kill nothing to kill outside this bounding box and immediately you see that I'm getting a really good result I can bring my point cloud ever so slightly forward and I can start removing some of these let's say 400 300 I'm really pushing it 200 should do the trick if you get some artifacts at the front you can actually move this back and forth you can see I'm removing some of the data here so I'm getting too close and few inches back so I don't get all of those um sort of hazy looking cards in front of the model you can do the same thing for the sides what is the default 400 let's try 350 maybe 300 and you get the gist you basically take it from there it's just about trial and error to see which one works for you and which one doesn't for the height you pretty much do the same thing the default is 285 let's try 200 and that should do it something along those lines and you can of course go to Game Mode you can change your field of view to get less Distortion and take it from there you can see no time at all you get something presentable and the result looks spectacular it doesn't end there you can go into visual effects and bring postprocess volume into the mix you can go I'm going to clear my filter and play play around with exposure so if I go infinite to go infinite bounds so post process volume effects will have an impact on the entire scene now I can go to exposure enable exposure let's um bring it down to8 maybe 65 to just intensify the black areas get a slightly better result and of course you can have access to uh not only that but also your color grading you can see I have color grading where you can play around with temper temperature lower the temperature let's say 4500 and of course from this point onward you can play around with saturation I'm going to double click on that you can increase or decrease the saturation to just fine-tune that sweet spot and play around with it and you can add chromatic aberration you can bring in vignette you can intensify your shadows so on and so forth you can also relight the scene so if I go in here and type in relight in my point cloud data you can see with righting is one of the user parameters that I can enable and immediately I get a far better result I can now go to exposure and maybe crank up the exposure if it's so slightly maybe 13 will work just fine and you can see how gradually but surely you are actually getting closer and closer to the ideal look you can go ahead and relight your scene you can go to light bring in a point light or even better let me bring in a spotlight I'm getting out of the game mode just to make sure that I'm actually covering the character in there and let's go ahead and intensify that to a crazy number let's say 3,000 I'm going to go with blue and you can see I'm somehow I'm kind of getting the result that I want it's impacting the image having said that as I said with one of the limitations it's not really changing the direction of the Shadows so that can be a deal breaker and I understand why but at least for now it's really not supporting the way that you change Shadows within the application so some of these small tweaks you can do to ever so slightly get the result that you want but that would be pretty much the extent of what you can do that should do it for uh an introduction to gajian splatting what this technique is what you can do with it in onreal engine at the present time and all I can say is the future of computer Graphics is a big question mark to me with the pace and the rapid growth I'm witnessing the future is very very exciting the big question we should ask ourselves is is gajian splatting gets fully implemented in a production environment and if that's the case then we need to rethink the way that we look at the traditional Studio Pipeline and the way that we produce work so it's a huge change that is coming our way and this is to me just a starting point well there's one way to find out and that is to wait probably not for too long to see how this technology is going to evolve I hope you found this video useful as usual follow me on X Reddit and Instagram have a great rest of your day and see you guys later

Channel: SARKAMARI

Views: 5,555

Keywords: gaussian splatting, gaussian integral, ue5 gaussian splatting, gaussian splatting ue5 plugin, ue5 gaussian blur

Id: pkiBGdWtp4g

Length: 20min 40sec (1240 seconds)

Published: Wed May 01 2024