Let's Clean the 3D Gaussian Splatting models and remove floating artifacts!

even though these new 3D gaussian splatting models are impressive we certainly have noticed this floating debris that annoyingly accumulates in the front of the actual model no matter how carefully you try to photograph your subject with the camera these floating artifacts can still appear and obscure the actual model making it difficult to see the details then therefore we need to find ways to remove these unnecessary [Music] points hello boys and girls it's all here again this time we take a look how we can clean our 3D cian splatting models since we are on Words of a very new thing there aren't that many ways or tricks to to deal with gaussian Point clouds at the moment currently the best and most versatile tool can be found in the plugin that is designed for the unity game engine this tool is a project of a talented Lithuanian coder Aras pranovich use who has also developed an efficient compression method to significantly reduce the size of the py files together with the German developer hybrid herbs they have now also created cutout tools with which you can conveniently delimit the point Cloud this tool can be found on GitHub I'll put the link in the description so now let's jump into how it works and how you can modify your gaussian models to make this work of course you need to have the unity game engine downloaded and installed on your computer we can download this gaussian splatting plugin for example as a chip file from the GitHub site and extract it to your own directories wherever you want the project with the necessary tools is located in the asset directory from there we can launch the unity file and now that we are inside the unity game engine we of course will need the gaussian spling point cloud files that we are going to edit I assume that you already know polycam and Luma AI Services where you can generate gaussian splatting models from your own images these Services have a feature where you can export this special Splat plyy file which we will now open in unity gausian splatting Creator settings can be found under the tools menu in here we just simply choose our py file and then we we have the option to change the quality how heavily the file will be compressed I choose the very high quality and then just press the create asset after the importing punch of these different kind of files will appear into the gaussian asset directory next we will need to activate this gaussian splatch object from the hierarch list and then to the right side into this data asset field we can drag the file that we just imported the gaussian splatting model is now visible in the viewboard but it probably will be oriented in the wrong direction so we can now try to reposition it to better alignment by using the move and rotating tools my model is a scan of an interior and I can try to align these rooms in a straight line using the background grid and the orthographic view when the alignment is more or less in the right place we can start the actual editing of the points it happens when we activate this selection tool from this ptn it means that now we can drag a selection rectangle and select group of Splats from the model every point that is inside the selection will be painted as a purple color and this will very effectivly show how the Splats are placed inside the point Cloud but as we can see there is a problem when we select points with this method and in this perspective point of view rectangle selection will take the points in the foreground but it will Mark also all the points in the background through the whole model so so it is too powerful tool while we only want to select and delete the points that are floating in the middle of the space and since we don't want to make hole in the wall one solution would of course be the turn to the side view and try to remove from the selection all the points that touches the wall or the floor but we quickly notice that this selection tool completely lacks a substraction feature we can only add more points to the selection by pressing the shift or control buttons but we cannot take points off from the selection with the same logic as we could do in the Photoshop or another 3D program like in blender of course it may also be that Unity has some other ways to reduce things from the selection but since I'm not very familiar with this game engine I didn't manage to find such a feature but since that that seems to be the case we have to look for other creative solution to figure it out fortunately in this latest version we now have new feature called cutouts cutout can be added from this button and it will by default add an ellipsoid subobject in the Splat model it means that everything that is inside this round object will be displayed and all the other parts that are outside will be cut out so we can use these subobjects as a cropping tool and get rid of unnecessary point this way too shape of this cutout object can also be changed to PX cutter type which is especially much suitable for my room model here there is also option to make an inverted selection where the points are then displayed in opposite way using this cutout function function we can now temporarily cut this room and scale this box sub object so that we remove all the points that are on the floor and the ceiling so now when we move to the top view we see that only the walls are visible to us and from this angle we can draw a selection rectangle and select all the points that are inside the walls in the empty space we can safely remove all these points by pressing the delete key on the keyboard just see how many chunk points we can find in this empty space I have to be careful not to accidentally remove the lamp hanging in the middle of the roof by selecting areas against an empty background we can now safely remove points without accidentally removing pieces from the floor or ceiling because we know that the points that belong to the ceiling or floor are safe behind the clipping area we can now easily fly inside the walls shirts and remove floating points against an empty background deleting process can be quite satisfying when you find hidden artifacts and can clean them away in this way you also start to understand how these gaussian splatting points are distributed and how their 3D structure is made this cleaning method will also reveal that gaussian models are not that accurate for example these wall surfaces are not that straight when you take closer look and the layer of the Splats can be quite thick and you cannot delete all the the points near the surfaces that seems to be flat to you it can lose some important quality of the model if you are too eager to remove all the points and want to make straight lines so you need to be careful you need to be careful also because there is no undo feature when you are inside this selection mode if you try to press the undo it will only remember the major movement sections that you made earlier for the cutout object that's why you need to be extra careful when you are cleaning these plats out when you are finished you can now scale this box cropping object back so that it will reveal the ceiling and the floor and now we have successfully cleaned the room from the floating artifacts probably the best and most important feature in this plugin is that you can export the edited model back to the plyy format and save all the modifications that you have made in this way the point Cloud will not only remain as an asset of the unity game engine but it can also be used in other programs such as Unreal Engine okay I hope that this short tutorial showed you the options how you can remove and clean floating artifacts from your gaussian splatting models even if you wouldn't be a big fan of the unity game engine I recommend you to try it because it is the only tool at the moment that has such a versatile options to clean points out of gaussian models so if you like this video hit the like button and subscribe to my channel I will continue to research these 3D models and die deeper into the Splats until the next time thanks for [Music] watching

Channel: Olli Huttunen

Views: 6,385

Keywords: Nerf, neural radiance fields, Radiance field, Luma Ai, 3D model, 3d Rendering, 360 scanning, 3d scanned, created with ai, Ai modelling, Training NeRF, Cleaning splats, Cleaning 3dgs, removing artifacts, deleting points, cleaning gaussian splatting models, cleaning PLY files, Cleaning point cloud, 3D Point cloud, unity tutorial, Gaussian splatting, editing point cloud, remove floating artifacts, floating points, floating artifacts

Id: YtjvfRAEJE0

Length: 11min 18sec (678 seconds)

Published: Sat Oct 28 2023