Use 3D Gaussian Splatting In Game Development, What The Internet Doesn't Tell You | GDC Talk

thank you everybody for coming in uh it's really crowded so well I'm flatter um yeah so my name is Jack um I'm the CEO and co-founder of a tech company called cury Innovations and basically we develop 3D scanner apps that you can you know download on your phone like you just saw the video you can turn uh photos around an objects and then we use our magic power to turn that photos into a 3D model uh that's ready for um a real engine blender like you said uh like you seen uh stuff like that yeah so but today it's not really about my product it's about uh 3D Gan splatting um but we have a really packed agenda for 30 minutes so I'll I'll try I'll try to make it in time uh but I was informed before I got it started uh because this session was is actually being recorded so uh if you guys please can you like turn off well not turn off but like site on your phone uh so like you know um your your your sexy tone won't be appeared on the video on YouTube all right yeah so um we have a really packed agenda uh but I'm sure we can make it so basically we're going to go from a little introduction about what we do um I'll I I'll go through really fast because it's really not about K engine uh this this time but basically KY engine um it's a it's a 2d scanner app you can download on your phone um both Android and and iOS and you can just turn uh take videos or photos around an object and and you get a 3D model but what really cool about Cy engine is like if you look at um how we actual game developers develop um you know 3D models uh we actually need professionally prepared 3D models with a lot of things right imagine you're you want to Tod model yourself you have to go through all these complicated and tedious steps uh but normally like normally 3D scanner software doesn't matter how how like you know how expensive it is it sometimes always give you 3D model but you know it's not perfectly uh UV and wrap it's not perfectly you know um um you know a a PB R big stuff like that so what we are trying to do is ciry we really wanted to help game developers like you guys to solve the 3D model problems to give you a really professionally prepared 3D model and how do we do that basically uh we can turn triangular mesh to quad mesh so you can you know work on your product to the models um better and also we have a magic power to turn on unra UV to a WRA UV uh with PBR materials uh which is pretty cool um so basically this is a just a Showcase from one of users on Instagram and look at how he turned that model toy car into a quad mesh uh fully unwrapped uh you know uh with PBR materials that's actually ready uh for Unreal Engine it's it's really cool so um yeah but uh another really really cool thing I personally really like what's really set ciry engine apart from other software is like we can first time Tod scan transparent and and shiny objects which is you know unbelievable before for pH gretry uh thanks to a new technology called neural surface reconstruction or NSR for short um in ciry engine it's um uh it's featureless up scanning and yeah but basically until now everything I present is pretty cool right uh but the problem is sometimes people have or will have really bad experience with cir engine uh like like this guy um yeah so basically uh the reason why people are divided into uh a group of user who has really good experience with cury engine uh versus another group people who don't really have good experience cury engine is because uh doesn't matter if we use pH Gramm before or the new NSR you saw just now um they do require um you know fairly steep learning curve to learn how to properly photo scan an object right uh you know doesn't matter if K and's been around for like two years but like we been around for two years um but sometimes you do see bad reviews about you know people trying to photo scan something but it didn't work because to get a successful 3D scan sometimes you have to know what photogrametry is right you know what you have to know what photo scanning is because not everything can be scanned you know that right basically you have to pick up the right objects that has you know featureful or Rich texture on the top which can be recognized by the computer vision and so you can in turn you know get a point cloud in turn get the 3D model but but in contrast if the object itself doesn't have enough feature points like uh this little uh I think it's WC toy figuring which is like pretty smooth you know plastic on the top was fairly simple color uh it's really not the object for photogrametry so that's why uh if you don't know the technology behind it sometimes you'll get some really really creepy results um actually that's why I think this talk is so important because we have new technology to trying to save the world and that is 3D Gan splatting uh so before I jump into this topic um Racor kind if be know 3D G planning like heard of it yo that's so cool uh so cool okay we have like okay first you not be able to see the back we have like about 90% of people who actually heard of c g spotting this so cool um but still like I just want to give a introduction about explain explanation about what really 3D Gan spotting is because uh you know surprisingly uh a lot of people I talk with although they are real experienc in game development and they not necessarily know the exact difference between a mesh or or a or a 3D model versus 3D Gan plotting so I want to use this opportunity to explain try to explain really well what 3D gouon spotting is um because yeah sometime it's it's the term is just so weird all right um so let's go it uh so basically um 3D Gan plotting is um is it's basically you have two ways to represent a 3D model before you have 3D mesh everyone know that that right but with the 3D G plotting it possess a another representation form of 3D model So when you say 3D model before it's generally just mesh but now when you say 3D model it could be mesh but it could be 3D gou and spling so it's completely different from mesh so what I mean by different is if you look at the video basically if you mesh if you go all the way in if you zoom all the way up basically you will see the closed right uh surfaces between points and this is mesh mesh basically means the surfaces with the little grits and basically the Grits are the connecting points between points so basically if you if you zoom all the way into a mesh you will see a perfectly closed surface with texture in that's mesh but if you try to do the same thing on the 3D Gan plotting you will see something that's really interesting happening so from the distance it look all almost the same but if you zoom in keep zooming in you will know is actually it's not the surface that represents the 3D model but rather it's like those kind of weird spikes is those spikes I'm talking about basically have like millions of those spikes that like overlapping on each other and the spikes are actually on a different color so when they overlap each other visually from the distance you will actually see the actual object that will represented not by the mesh but by the 3D gaes those are by those are called 3D G by the way and I'll talk more about it yeah so um kind of makes sense right and let's all right let's keep going let's let's go further on this um yeah let's go further to explain so basically when we generally say 3D model before right it basically means a combination of two things right mesh and textures and what I what I mean by mesh is uh the pictures on your right is basically a gray 3D model uh you know you have like those little points those are called Point clouds like little points and then between the points you have the you have the uh the grids that connect the points and this is actually the mesh and the texture essentially just the colors and materials you know WRA on the top of 3D model so when you unwrap it you have the texture and a mesh to represent a colored 3D model but 3D gouon spotting is different story uh like you like you saw before basically if you zoom all the way in the 3D GS plotting you will see the overlap spikes like what I showed on the left side of the screen so basically you have these colorful spikes overlapping on each other and when you zoom all the way you know when Zoom all the way out and and see like millions of those color spikes you know collapsing each other um you will see a visually represented 3D model by a completely different methods and that is basically 3D spotting and yeah I'm not sure how well explained but I have another video to show you what a spike look like so look at just you know uh pay attention to the uh to to the uh to the to the that um Gray Line that I Mark in there because basically this is called a 3D GAA right but basically As you move your camera around it you will see the 3D Gan actually got blurred out and basically this is how you represent that points of the of the object by a one color of 3D Gan so yeah like I said before when you opap millions of those basically from any angle you will see a color representation of the 3D model can that make sense I I hope I I explain well um yeah but if you if you Google it will you will see some really really heavy words like you know implicit representation a hybrid of explicit and implicit I'm not going to go over that it's like next level stuff but hopefully uh the the the the the the explanation I did was uh was clear enough for you guys to understand have a clearly understand what it is so why do we need 3D gion spotting like we have mesh right and it kind of served industry for like 30 50 years something like that so why do we need TWD gouon plotting um so I have two points to claim number one is um 3D gouon plotting it will never replace traditional 3D modeling all right because you know uh in KY engine you know we talk to investor all the time and sometime investor like oh guys you guys are replacing 3D modelers I was like we'll never do that because you know 3D modeling is one thing but 3D scanning either with measure or 3D G pling is another thing you can never replace 3D modeling so it will never replace 3D modeling but it will be a a huge Plus for traditional photogrametry what I mean by that is like look at this so what I'm going to show you the video on the right this is a traditional capture of a photogrametry basically you turn the mesh off and you turn you turn the texture off and turn the texture on you will see this so basically photogram is really bad at representing something right for example this really thin glass and dirts on the ground you know it's really really hard to be represented by mesh because it's so thin and you know it just sometime it's really hard to get captured from photogrammetry techniques but if you look at look at the same scene but from T Gan plotting you will see because you know it's not represented by mesh it's actually much much clearer to be represented so basically uh like like like the point I made before um 3D gon spotting isn't set to replace anything but it's a huge improvements on photogrammetry and that's actually the goal I'm trying to achieve and another really cool example sorry when I click my mouse the video just just okay try this so you guys have you guys tried photogrametry before I think yeah I see a lot of n perfect so you know photogrammetry it's it's impossible to capture a water bottle or glass bottle like this right but with 3D Gan plotting you can actually capture well I said n NSR like our NSR can capture it as well but like 3D Gan just better right 3 the gon spotting it can actually capture the transparent water bottle that was never done before right and actually I've seen a really cool uh project that was done by one of our users on on Instagram was so cool that people can actually do something with the 3D G plotting today yeah so take a look at this this this is like next level stuff so what you see here is the whole scene is 3D G plotting so that's why you see this little weird balls like in the distance these are actually the 3D gions that got rendered by the actual uh game engine where I think he used blender in this case but this is actually really really cool but um it's it's a problem so basically uh imagine how how you create games guys because uh what I just showed you earlier everything about 3D gion spotting I showed you earlier actually these are the 3D gion spotting scenes right when you capture for examp examp an object the 3D G plotting is not only going to capture that object in Center for you but also all the background scenes like around it but when you play when you create games guys you don't need a whole scene you need the assets the individual assets so you can like combine those assets manually to create your own world right but the problem is um previously the 3D gouon spotting isn't really capable of that uh but that's actually a really cool thing that we have here is um I think a couple months ago we introduce the 3D gons plotting um cropping and editing in the K engine app so in the first time ever you will be able to crop and edit the whole 3D gou spotting scene on your phone and to get a standalone 3D gou spotting asset but like I wouldn't say it's asset because like you know although it's a standalone 3D model but it's not you know it doesn't really have you know uh PBR Maps stuff like that so it's not like 100% uh you know assets yet but like that's definitely where we're trying to go yeah but you can see like we provide a vivid um editing tools for you in the app so you can actually edit the 3D gaan to get a stand 3D Gan splatting model this is actually uh this actually one of my favorite uh updates uh a couple months ago that's actually pretty cool all right so and with the 3D Gan plotting cropping scam uh you can do something like really cool like you know uh cutting from a whole seen into a standal assets it's it's mind-blowing for me all right um okay now let's jump to the topic like um how do you use 3dgs add-ons uh well how do you use 3dgs in game engines and I just uh disclose that we use addons um like I said uh well basically all the game engines we used before uh and real engine blender G blender is not real a game engine but like 3D software and unity stuff like that uh these softwares are developed based on mesh right it was it's been developed like tens of years ago um but when TD gan's planning CES on you know uh because it's not necessarily represented by mesh so it's not really natively supported by those game engines so how do we solve that so we have to introduce 3D Gan spotting addons in the existing game engine so actually the the right now game engines can support 3D gou and spotting um but how far are we though like like as you can see we're not really far so basically uh so if we are going to rate the like compatibility um you know uh of uh each game engine major game engine with the 3D C splatting uh if you if one out is highlighted that means it's barely importable but you can't really do much with it but if it's five like five star like or or five dots then that means you can use it like like just like you would do as a tral 3D model but even though right now the highest capability we have is UN real engine five but you only have like two dots so you can barely work on it but I mean that's the exciting thing of the tech right like you don't get the the perfect Tech from day one you kind of build it on it like gradually and hopefully one day maybe next session or five sessions after this year we'll get a fully functional 3D gion spotting where we can even see some 3D gion spotting uh games uh eventually but like we have to we have to walk on it like step by step right but right now it's not really cool but I just want to show you um what you can do right now um so actually um so here's a uh here's a link that you guys can scan or save on your phone it's actually a free uh un real Engine 5 3D G spotting add-on and it's pretty cool and you could just download on your uh on GitHub and you can just install it on your enre engine 5 and you can start working on it so what does add-on can do it's actually I Tred to make it really straightforward basically once you install the um the and real engine addon you will get this oh I'm playing it no hold on sorry wait what oh sorry no it's not a video I thought it was my bad so basically if you can see in this scene um everything except that car uh is actually a a re engine scene like made by mesh right except the car the car is actually the 3D gouon spotting asset that we uh we just talked about before right and when you install that 3D gou spotting add-ons you can actually just import that um the 3D GS as it would do as a normal 3D model and just start to position it right and when you position it you will get something like this in the end but more importantly I want to show you a glimpse of the future that we for to mute myself but don't worry yeah so what you see here is basically you know you have this mannequin walking around a UE scene but you know by walking around that car right uh it's pretty cool but um it's actually okay let me show you this slide so um you've seen this right but um yeah the video looks pretty cool um but it does have very limitations actually I try to hide that uh the disadvantages from that video but basically uh if you look at that video again um actually uh for the 3D gon plotting in today's technology although you can import it to engine and and kind of start to do cool things with it but the problem is it doesn't have physics number one like basically if you walk to that car right into that you will just walk right into that like you don't really have physics and right now there's like kind of limited ways for you to add physics well there's some other ways for example you can you can add invisible mesh inside that car so when you walk into that car you're actually bumping with the invisible mesh but not a 3D C spotting but it's not smart enough so if we try to solve this kind of problem uh in a smarter way and that's actually one thing we're working on right now and also um there's no relighting it's actually very very crucial that's why I don't really think that 3D gon splatting is usable at this moment it's mainly because there's no relighting and what about what I mean by relighting is when you C capture your 3D Gan plotting um you know around a scene basically whatever color condition or liing condition uh the moment you capture it it's baked in your 3D G spotting so right now there's almost no way for you to take that lighting condition off and get a fully even out albo map from the uh from the Gan spotting so right now for example if you're taking a 3D gaan in this condition you have a strong light shooting from this way in your 3D gaan model you will have a strong lighting surface on that side but if you're trying to to to shoot a Different Light direction from the other side uh then well there's really no lighting so uh in the 3D gaan you will see kind of a weird kind of uh you know Blended lightings from everywhere like One Direction from the mesh but in the gaan you see the the other D direction of lighting so that's actually a really really a main challenge that we are trying to solve but um hopefully it's not far away um I I hope in the next session next year uh I should be able to show you guys something something that's really cool that you can actually uh do your stuff with it and also LOD is also very important but we don't have that in 3D ction right now so remember the um the the the the G statue that we looked at in the beginning of the session um you know if you zoom all the way into that right basically you start to see that spikes um and in really low resolution because that's something that we also need to work on as well because uh in the actual game you want to have an LOD like doesn't matter how close you go you know the resolution goes up with it but right now uh 3D G spotting um in practically uh it can't do that and also there's no PBR um that's another really um limitation tool I thing I think uh I think is I think it's limiting 3D gou spotting from game development so yeah I mean uh so in conclusion I think we have a long way to go um but uh like I like I said put the last sentence it has a lot of potentials uh what I mean by a lot of potentials is because although 3D Gan you know it's has a lot of limitations but it's not something that is unsolvable but because I'm not sure if you guys know it but 3D Gan plotting the format of it it's by a format called py format and do you know what py format is it's actually a point Cloud format and when we look at Point Cloud then we know a lot about it because Point Cloud we've been working on it for like hundred hundred of years so so 3D gou plotting because it's represented by this point Cloud format although the the training process different but there's a lot thing we can work on the point Cloud file so that's why uh you know 3D gou plotting it was just introduced a little bit you know uh earlier last year was like November of last year we start to see 3D Gan plotting applications like KY engine uh you know go to the market uh it's only been like 4 months five months so like it's really young but but yet we achieved so many things in just four or five months so so especially because it's py format I think we're going to achieve much more uh next year and the year ahead so yeah so I'm really really optimistic about the 3D Gan spotting and yeah so I yeah I count about 20 something minutes uh for this session and but I just want to leave the last five minutes for you guys if again is QA um if you have any questions about 2D gaan spotting or Nerfs or these kind of new technologies that you heard you know all over the internet but you don't know what the hell is going on uh this is your chance to to clear your mind because I've seen so many questions on the internet uh asking what's the difference between nerve and TD Gan like what's like like why is it so cool like stuff like that so so don't be shy guys uh I'm the guy that's the whole point I'm I'm offering the session so if I have any questions uh Now's the Time to to ask around all right cool thank you guys hey um great talk thank you uh I wonder if you've seen any good examples of Animation via skinning waiting rigging that sort of thing or whether there's another animation solution you mean uh 3D G sping animation yeah uh very good question uh right now if if we're talking about the actual real projects then I haven't seen any actual projects that were done by 3D gaan but I've seen many uh research works that's done around it basically why we're not seeing so many animations around it because 3D Gan is not mesh so it cannot be perfectly rigged right now so that's why you can't really do too much with your with the 3D Gan uh but actually we have many research going on not only does but like we have a lot of research going on around the world that's trying to solve the rigging problem on the 3D gaan and I I I can't really discuss too much because we have CRA and cvpr coming in like those are like undisclosed papers but what I'm trying to say is like we are really close to animate at TWD gou and start making it work cool thank you yeah thank you so much hello thank you for the wonderful talk thank you um I had some questions about performance y um were there any like comparisons like mesh versus gussian spots and um since you're drawing so many spikes are there any concerns for overdraw uh good question um actually uh short answer is if you look at compare about the FPS right now um when you render a uh a a 3D gouon spotting the FPS is um indeed is lower than you would actually render on mesh um but it's not actually solvable because um in the 3D G spotting actually uh you have lot of information you not only have this kind of appearance you know not only have that color thing you also this kind of format also include the the kind of reflectiveness kind information in that while you were capturing basically if you're trying to capture this uh this this talking stage for example you can actually physically visually see the lighting moves with my with the camera but with the actual 3D C spotting that information is embedded in the S go spotting but that's actually useless when you're trying to do your own thing right so if you try to uh extract that out of the file format you can have a relatively uh you know a smaller size of the 3D C spotting so that's actually another uh Direction research where uh some schools are actually heading toward so right now it's it's it's it's worse than mesh but but it's doable awesome thank you so much y thank you hello thank you for that um I have a question about the rendering part since uh it is as you said it is not a mesh but our current rendering pipeline is still like R using mesh to render everything so there's like a conversion maybe uh for that so do you think uh there could be an optimization the render rendering pipeline itself or rendering Point Cloud format better yes uh that question is so good because because we just introduced 3D gaan splot into mesh uh in our app like last last week something like that so we do support a mesh inclusive 3 gaan but um but from my own personal standpoint I think it's just intermediate solution this moment because if you try to convert 3D Gan to mesh basically 3D Gan loses all the points like right basically because like what's so cool about 3D Gan is like it's so tolerant to the camera post calculation if you know what I mean right basically before if you're trying to do photogrametry the reason why some photogrametry model is so rough is because when you're trying to align the cameras or photos from the first step it's not necessarily give you a perfect camera post calculation and if you don't have a perfect cam camera post calculation you're going to in turn have a really rough surfaces and 3D gouon is plotting because it doesn't represented by the mesh it's so torent to the camera post calculation but if you're trying to get back get back the mesh on top of it then it's it's again it becomes you know camera post dependent and that will pretty much uh loses all the point because the whole SE point from 2D gouch in is from K engine app side of VI like you've seen a tremendous amount of increase on the successful successful rate on people creating 3D models so we don't want to lose that so although we have mesh is because like you said like mesh is pretty much the only way can render an actually thing so we want people to actually be able to use three G stalling so we have uh mesh inclusive 3dgs right now but if you're talking about five five years in the future like I think we really need to work on the way to be able to run the 3dgs alongside mesh like that's why I'm talking about relighting you know PBR I'm really looking forward to a way that people can you know have a mesh off a you know have a have a table that that is mesh but on the top on the water bottle that is 3dgs but when you put the same like lighting from every engine I I I really like the automate goal is I want the 3dgs to be able to response to the light the same as the mesh so that's to over over top out in the five years all right thank you y thank you guys all right so if there more questions so enjoy your day and your last day on the GDC and hopefully I'll see you next year so yeah oh by the way um feel feel free to join our booth in South Hall S1 1421 because uh we going to show you the world's first 3D Gan spotting rendering on Apple Vision Pro it's like it's amazing it's like you're standing among your dream like you captured before it's like cool so feel free to drop by and check it out yeah see you next year [Music]

Channel: KIRI Engine - 3D Scanner App

Views: 1,812

Keywords: polycam, luma, luma ai, 3D scanner, free 3D scan, 3D scanner app, 3DGS, NeRF

Id: zTwHmxfKvOs

Length: 28min 58sec (1738 seconds)

Published: Tue Jun 18 2024