Interactive Gaussian Splatting Scenes in Luma AI

[Music] the year 2023 has been very interesting time in the field of artificial intelligence I have been particularly interested in development of 3D Graphics Gan splatting technology has been very hot topic especially in the last few months and we have seen lot of bus around it recently Luma laps has also now entered this game and implemented this technology into their Luma AI application as well but they have interestingly decided to rename this feature and call it interactive scenes so let's find out what is so interactive about [Music] them hello boys and girls it's oie here again and this time I decided to go out and try the new features that have been updated in the Luma app normally I would 3D scan something with my smartphone but since my iPhone is a bit older model the cided scanning feature of the Luma app does not support it however it is not an obstacle to using the service through the website 3D models can be created from video files and they can be shot with any camera and it also works with Android devices that way but the mobile application itself is currently only available for iOS devices and in order to explore all its features I decided to use my iPad Pro which has the necessary camera model with all acceptable functions the 3D scanning itself is exactly the same as before it is very easy to do through the cided feature of the app but what happens Under The Hoot when the scanned material is sent to Luma service for processing is interesting all new scans are now processed and displayed using the new gaussian splatting method this function has been successfully built as a continuation of a previous Nerf pipeline as the Nerf model is still generated in the same process however now in the foreground is a very well optimized and highquality gaussian splatting model when you open a fresh model in the mobile application it is animated during the loading and is revealed very elegantly first as a point cloud and then with the gaussian layer this is a small but very nice effect that immediately shows that we have managed to create something unique when we open the same model on the computer's web browser we notice that the reveal animation is little simpler without the point Cloud pH in any case both in the mobile application and on the desktop the 3D gaussian splatting model turns very smoothly and the accuracy with which it is calculated ated is excellent a considerably high amount of iterations have clearly been used for training these models and yet it has been successfully optimized so that the million of doch can be displayed both on a smart device and in the web browser but what can we do with these models then when we look at the download window we still find options to export Port files in common 3D surface model formats but the newest thing in here is the point Cloud format that is named Splat and it is new kind of a Ply file at the moment there are still only a few programs where you can easily take advantages of this new format but fortunately some competent developers have already programmed plugins for Unity and unreal game engine where this file can be opened as it is intended and as I already mentioned the older Nerf models are still included and they are left as an export option where you can still download the radiance field model and use it in Unreal Engine through a plugin developed by Luma itself but I also noticed that the Nerf models are still present in other parts of luma service for example if we use the re-shoot feature where we can easily create different camera movements around our model and when we render this out as a video file it can be seen that the camera animation is still made from the Nerf model I'm sure Luma will fix this soon and we can get real gaussian splatting videos out in other ways than just using the screen capture but at least for now G splatting models can only be admired inside the service except of course that new py format another aspect where Luma cannot yet offer this new feature is 360 footage or fish eye images if you decide to upload material that uses distorted WID screen or equi rectangular material they will still be processed as Nerf models and it is not yet possible to change them but then there is also this cool new feature called an upgrade if you have already used Luma service and made 3D scans with it now you have the opportunity to upgrade your older Nerf models to this new interactive format there are limitations models can only be upgraded one at a time and maximum of 20 request can be made the upgrade process was interesting because I at least noticed one significant Improvement when I converted the Nerf models to gaussian splatting format it is related to thin structures I had this 3D model of a piece of a fence as we know in photo chetry and 3D scanning it is very difficult to make the fine details of the thin mess struct structure visible in the 3D model the Nerf model I implemented earlier left a big hole in the fence when I upgraded the model to gaussian splatting I was able to see the details of the fence goian are therefore capable of very precise details and manage to bring out what is present in the original image this is a good example of that but let's go back to the original question why has Luma named this feature as an interactive scenes is there something particularly interactive when you can rotate a 3D model in a web browser I think that's exactly what it's all about the fact that these Services work online when we are now able to present 3D models with such high quality directly online and anyone can now produce photo realistic models through these services so just think how it can increase the features of for example online stores for some time now Luma has had an excellent tools for building various embed codes with this you can take your 3D model and easily attach it to your own website or other web services real interactivity will therefore arise in the future wherever we come up with these 3D models to use I recommend you to check out what is behind this sare button and see what different options there are for embedding Luma viewing window on the web in any case it is very inspiring that now we have these services is where everyone can create 3D models with the latest technology without an expensive PC or a fast graphics card Luma AI service is free and very easy to use and it's now capable of producing very high quality interactive scenes what do you think about this have you already tried Luma new features yet leave a comment below and if you like this video hit the like button and subscribe to my channel I will continue to scan objects and research these Splats I can't wait to see what happens by the end of the year until the next time goodbye Liv it's talk a big game but it's just a matter of time have you ever really Liv

Channel: Olli Huttunen

Views: 15,294

Keywords: neural radiance fields, nerf, Radiance field, Luma Ai, Ai, 360 camera, 3D model, 3d Rendering, Camera movement, camera fly, drone like footage, Nerf model, 360 scanning, scannig with 360 camera, 3d scanned, created with ai, Ai modelling, Training NeRF, Gaussian splatting, 3D, Interactive scenes on Luma AI, interactive scenes, Luma on splats, 3D web viewer, Embed codes, Attach 3D models to website, 3d environment

Id: mbR-Wj3fG9E

Length: 10min 0sec (600 seconds)

Published: Mon Oct 09 2023