What is 3D Gaussian Splatting?

this is a video of the pyramids in Egypt and now it is 3D this is a video of Gast town in Vancouver and we also made it in 3D this is Castle de Verdera in Catalonia and guess what it's now in 3D but how on Earth did we create each of these 3D scenes in less than 1 hour there is absolutely no way we could have modeled these environment to this level of photo realism let me introduce you my friends to 3D goian splatting or short g-spots which is the latest method to create an entire complex 3D scene using only a video it falls under the category of Radiance Fields much like Nerfs but it's actually quite different same same but different same same but different here's the thing gplat are faster to train and you can achieve unprecedented level of detail need Crazy Fast realtime rendering so let's go and find out how it all works first we begin with a setup of input photos that are taken at different angles from a subject now these input photos can also be extracted from a video as an image sequence next using a mathematical method called structure from motion the differences between these images are calculated to form the 3D Point Cloud think of it as many dots in 3D space with each dot representing a point from your photos now instead of trying to create a detailed model with polygons each point is turned into what we call a gausian gossan on a graph takes a shape like this so if you were to imagine it in 2D space it would be like a Photoshop paintbrush that has a dense Center which Fades out towards the edges now imagine in 3D space these gossans take the shape of elipsoid what's amazing about this is that these gausian can take various parameters such as transparency values different sizes and even color at this point the gausian enter an iterative optimization process which is key to its quality and performance the these gossans get fine-tuned constantly to match the original photos that we took by adjusting their 3D positions color opacity and how their details vary from different directions the training also uses an Adaptive density control which means when a gosan is too transparent it gets removed or when a gosan is too big for a detailed part of the scene it splits into two to view these gossans we use a technique called rasterization which projects them onto a 2d surface we consider their distance from the viewer for depth and then for each pixel we'll calculate how much each gossan contributes to that pix and with all these ingredients you tend to end up with a quality that is better than the other Radiance field approaches and trading times that are faster there you have it guys this was the simplified version of our own understanding from the paper that was published for 3D gosin planning yeah the paper's pretty complex and if you're a science geek and you want to learn the entire breakdown the link of the paper is going to be in the description below now as for us after we're done with this explainer video we're moving on to more experiments with gosi splatting plus we're going to be testing it with virtual production so if you are interested at all in any of this make sure you subscribe wait wait actually haven't subscribed yet haven't you you mean you watched all of this and you still haven't subscribed what is going on

Channel: Bad Decisions Studio

Views: 45,871

Keywords: radiance field, gaussian splatting, neural radiance fields, Nerf, AI, 3d Scan, 3d

Id: Tnij_xHEnXc

Length: 3min 23sec (203 seconds)

Published: Thu Sep 28 2023