Dreamgaussian (3d gaussian splatting + dream fusion)

the main thing that's really exciting about this is it's like one image 3D reconstruction which is really like that's an insane topic and it does it so quickly and it also does it pretty well so today we're going to be looking at this repo called dream gausian and so it's basically combining gausian splattering and the ideas from dream Fusion so just to break down those two ideas goian splatter is sort of Nerf adjacent in that it actually isn't a Nerf because it isn't a field um it is a bunch of goian splatters instead so kind of the underlying structure is quite different but it uses very similar kind of differential rendering because you're using kind of these larger gausian objects it ends up being incredibly efficient um and very quick and there's a lot of really positive sides to it and the other thing that it does is that if gausian become small enough uh they disappear and this really solves a big problem with generative 3D because often times uh errors and kind of uncertainty in this generative 3D ends up kind of imple like creating this a little bit of smoke around the object where it's like unsure if there's a pixel there or not but these gausian splatters often times they just kind of clamp down on that and then the gausian disappear and there's no more smoke and so it creates a really good environment for generative 3D um and then the other idea is the kind of dream fusion and that's an idea that's been around for quite a long time but earlier it was like you needed a really high-end equipment to do so it was like you you needed like an a00 and like a lot of these experimentations were just kind of exorbitantly offensive and it was really hard to actually get involved with uh but this has a lot lower ceiling than the dream Fusion so I can run this all on an 8 gig card it allows anyone that can kind of do stable diffusion to do this and so that's something that I'm really excited about is um and then also the speed of it is much quicker cuz some of the things from dream Fusion some of those papers would take like a half an hour whereas a lot of these converge in about 2 minutes so it just has a lot going on for it that's really exciting so the goian splatter can actually be converted into a mesh using using the marching Cube algorithm hopefully at some point someone will come up with a better algorithm for this because I think a lot of the problems with kind of this Nerf Tech or like differential rendering is that when it gets converted over people almost always use marching cubes and marching cubes it kind of voxyz things and it makes them a lot lumpier than they would otherwise be um so I think that like that's an area that I really hope to see some improvement in in the near term cuz a lot of these things look really amazing in differential form but but in mesh form they're like a little bit wanting so the textures are a little bit cloudy and you might want to kind of refine them a little bit more so one thing that one trick that is always pretty helpful to do use is projecting from View and so you can select the front faces um and then send that to the UV map and then just kind of rescale it appropriately and that way you can have some nicer textures um and since all the textures were generated from the objects it like they should look really like everything should just line up perfectly um it is you will end up with seams and stuff so there are some disadvantages so the guian splatters that it creates are really high quality in kind of 2.5d but then as soon as you start moving out of that a lot of times it loses some of the detail I think some of this might just be like my skill issues um because some people have actually been able to have really high quality images all around so sometimes if you have a really obvious back of the object then it'll kind of generate that on its own but um it does end up having some difficulties with the kind of back part of some models one thing that I found is a way around this is you actually can use multiple images there's this really nice answer on GitHub of someone asking about multiv View and providing like where you would modify the code and the code is so nicely written out that you actually can pretty easily just add in a second camera and put that on behind the object and then have kind of both images and that's something that's yeah nice like it's it's just nice how clean this code is and how easy it is to modify so for the most part using this library is kind of a three-step process so you do a pre-processing which basically cuts out the background centers the image um then there is a second step where you can build out the Nerf and then build a basic mesh and there's a refining mesh step so if you don't get a good enough mesh the first time around you can refine it down so you are able to include stable diffusion in the loss function and kind of uh add extra details to it I think there's a lot of fine tuning that you might need to do in order to get this feature to look nice cuz I was finding that it was just kind of creating some really awkward looking things uh in particular uh it was using older versions of stable diffusion which are really susceptible to this problem known as the Janice problem and that's basically the idea that stable diffusion and a lot of early generative models really like having front-facing faces if you have some s sort of 3D generative person a lot of times they'll have faces on the back of their head so um yeah that comes the Jan is problem comes from this Jenna statue which is has faces on the back of its face and that's like why came to that but it's it's a really big problem in j generative AI um but it's starting to be solved or Sable diffusion xdsl sdxl ends up being very good at drawing the or drawing the back of faces so some of the other kind of repos that have uh similar qualities is the shapey and 1 two 34 five both of those end up having like they both work but like I've had issues with them I think this actually is a step up in quality like they they show some of the images here and this is kind of like how my experience with it like shapy was really underwhelming and then 1 2 3 4 5 like it sometimes does a really good job but like other times and it it does a really good job with generic objects but once you start getting into kind of like weird or quirky things I feel like it just falls apart whereas this new model like it it can do stranger things and yeah so I just want to do a quick comparison with depth maps and this new dream gausian method so um depth maps in particular have a lot of things that are really nice about them uh they work really well with the actual texture so you have kind of a very texture to depth map comparison that's really nice they also do a really good job with high details a lot of times these other 3D generative models end up kind of flattening out a lot of the smaller end details and so that's that's a difference whereas dream goian does a lot of really nice things too so it has all the objects that it creates tend to be uh have an inside and outside so they tend to be self-contained which if you're doing something like 3D printing might end up being incredibly important so once again you can find this project at this GI hub uh it's really exciting um it's an amazing project just like I'm very impressed and I think more exciting things are always going to come up in the future so that's all I have for today thanks and have a great day

Channel: Grae n

Views: 15,924

Keywords: gaussian splats, dream gaussian, ai art, generative ai, stable diffusion, 3d, blender, img-to-3d

Id: 1xv3NBIYT44

Length: 6min 42sec (402 seconds)

Published: Thu Oct 26 2023