How to Use 360 Video for 3D Gaussian Splatting (and NeRFs!)

hey everyone in this video I'm going to show you how to take images from a 360 camera well not one like this more like one like this and take those images and prepare them for 3D gaussian spotting or really any Nerf project that you want to use so follow along this is kind of a beginner's tutorial there is ways to do it gives you way more control but this one should work for you to get started so here we go okay so there's more than one way you can take the imagery from a 360 camera and in this example I'm I have this 360 video footage and this is for my insta 360 rs1 1 inch whatever and you can see that I kind of have it on a pole above my head this is not a perfect capture I got the sun flare coming in you can see this I had a friend walking with me um I would suggest putting this camera right above your head so you have like the least amount of Shadow the least on whatever you can do it on a a day that's not very um a a lot of a lot of sun coming in this sun flare is definitely not great um but another thing I always think about when doing something like this is that you need this Parallax motion so we don't really see like this wall from multiple places right so that's happening that's great if you're out in like a middle of nowhere and things aren't really moving across the scene that's not gonna help um so you can capture a video or you can capture intervals still so I also like taking like a two second interval still getting the highest resolution possible but then when you go to export this thing so I can go over here this is insta360 software just make sure when you say export 360 video you bump that bit right up you bump everything up the resolution's the top resolution it's gonna be a gigantic file I'm guessing but yeah you want to make sure you get the full quality of the imagery another thing to think about is it let me just bring up my my drawings software here let's just say I'm going down an Alleyway here so like this is a wall like this right and here's another wall and I'm gonna be walking down this thing like so with my camera just like I did in that video um well that's great and I'm going to see Parts as well as I move along uh the best thing you could do is actually walk down to this part and then you would take a quick turn well not quick but you just want to then move over and then move over again so in the end you can get like this box and what's happening is I'm going to capture this scene from not only one walk down but then on my way back and this could be a box if it's a bigger area you just kind of want to move around and what that means then is that as I'm looking at the scene later on I can move anywhere between this as I let me bring my school tool as I'm moving through this this gaussian Splat I can kind of like move around that is the wrong tool to show you what that looks like um here's the pencil um you know I could kind of like move anywhere around inside this area and things should look fairly consistent another strategy is perhaps move down this Alleyway or scene at one elevation and then have a selfie stick and raise it up several feet and that way instead of moving back and forth between that area you can move up and down and everything will stay consistent because if we get too far from your your camera poses Things Fall Apart so just kind of getting back to the imagery you're just going to export this export to a folder somewhere and you should be good to go so the next thing we will need to do is either extract images from this 360 video output and you want to make sure it says like equal rectangular or you want um or you can work with the Stills but I'm going to assume you just have video so let me just jump into command prompt and show you using ffmpeg how to then strip out these like frames from this walk down this Alleyway and use those um for my insta 360 Capture to a Nerf or Goblin spotting okay so now I have this mp4 file which um you can see was me walking down this Alleyway that x i extracted um oh okay so I'm actually using a different one but me just by myself it'll be a little bit more clean um again the lighting's not perfect but this is one I'm going to show you how we're going to extract image stills from this so let me just close this file and I'm assuming you have ffmpeg installed you're gonna need to install it no matter what you do it's pretty easy okay so you're going to want to open up command from this screen and open up a fresh command prompt you just type CMD in the search bar here and I'm going to make an output folders it's called this new folder I'm going to call this um images under bar 360. this is my 360 images and I want to get these to 2D images when I'm done and then I'm going to use ffmpeg so ffmpeg ffmpeg and that's something you should already have installed if not you can look up how to install that um and then you got to go space Dash I and then your input video so if I type in ritters and hit tab it should complete so we're in this folder and then space Dash VF here I'm going to copy and paste it for the heck of it I'll make sure it's in the notes Here VF FPS you can specify how many frames per second let's look at this video it's only 23 seconds so I'm going to want like three or four frames per second so let's make that let's make it three and then don't worry about this like Q scale but then it's gonna do this image 360 folder that I created and then name them so hit enter it's going to run and if it goes image 360 folder you'll see it start to populate with these 360 images and that's me moving down the alleyway so that's perfect so the next thing I want to do is I got to split up these 360 images into individual um Cube map images those are like if I took a cube and mapped the images to it it'll make it up and that'll make sense in a second so I'm actually going to quit this command line and at this point you're going to need to install some special software called mesh room okay so you're going to need to download mesh room and when you do that you're going to go to the older versions because in the newer version they've changed how things work so you won't be able to follow this so I'm going to make sure you go down and you pick the 2021.1.0 link for Windows I'll make sure I link that in the the show description as well as in my instructions I have written out um and so once you've downloaded that you end up with a a zip folder so if I go to my downloads um like so if I go to my so if I go with my downloads and I open this up I'm going to extract it all and this will take a second so let me just jump into when that's done and I'll show you where to put this folder okay it's just finishing up uh I will have in my downloads this mesh room folder I'm just gonna open up there's a subfolder I'm just gonna cut this out and then I'm gonna put it in my uh my just kind of like my Joe on t file my my user uh folder you can put this anywhere put in like C just put somewhere you can always find um I'm gonna open it up and then I want to go to Alice vision and then the bin folder so now okay I got we're the root directory I had it in mushroom Alice Vision bin folder from there I'm just going to type in CMD to launch command prompt from that folder so I don't have to path to it or anything at this point I can I can I can close out this this window I'm gonna put this here like so get rid of this I'm just going to make this big and then um then you just gotta you're gonna run a script that's gonna take these 360 images and turn them into 2D images so let me just uh let me just show you how that works so first thing you're going to go to is you're going to type in Alice Vision um and then under bar utils and then just start tabbing what you're going to look for is one that's called um you're gonna find one that's called split 360 images dot e x e and then from there you're going to put in a couple of parameters and I'm just going to copy and paste this from a cheat sheet I have um but um the first thing you actually do is go Dash I and then that's the image directory so it says 360. dash dash o I've created this images folder so that's one of the 2D images to go throw that in there super easy then space and this is again where I'm going to copy and paste I have two parameters so the first one is I'm going to split it into eight cubes so like eight Cube faces so you're going to end up with like a bunch of different images Cube mapped and then you're gonna have this split resolution so each image will be 1200 by 1200 you can kind of do the math figure if it actually makes sense for the size of input imagery you did but that's a pretty standard number there I'm going to hit enter and it's going to start running um doesn't take very long so it's going to go through these images and it's going to image at a time if I go into this folder you'll see that it's starting to split this up and I'm going to show what this look like so if it's open one up you'll see there's my Square 1200 by 1200 image notice it's not the highest quality probably because I mean I went video to to extracted image um it would look better definitely if I was using Stills as high-res as possible but that's what you end up with this like these kind of Stills and so as I I didn't mean to open up more than one as I go around you can see I'm kind of going to circle and then I eventually progress as I keep going through these I'm getting further down the alleyway tons of Parallax seen in all directions let me just close that so this is still running and when it's done we're pretty much ready to start making our our 3D gaussian Splat scene so wherever this ends up being so I have this in some working folder when you're done you'll have your folder of images and I'm just gonna copy and paste this into my gaussian splatting project so um I have my data folder if you ever watch How I get ready to get started I had a data folder um actually I have a archive here we go so this data folder of this exact alley and you can paste it in here but you want to paste it in as input so this images folder will you know you would you would go and I would paste it in here and then I would rename it to input because that's what you're supposed to do and then from here it gets pretty easy so once this is done I'm just going to let this run when it's done you're gonna you know you're gonna start your your prompt I'll start fresh prompt here and then make sure you change directory to your um your gaussian splatting folder and make sure you kind of activate and G activate gaussian splatting like so once you've got that set up then again you just do the you're gonna do the convert Dot py conversion and then you're gonna do dash s and then just drag that input in there in a run cold map it'll do everything for you and in the end you'll end up with this this um this great output where you'll have your distorted you'll have your images which I I think I renamed here like so and then all the other cool map stuff you need and then again back into your your condo once that's done and uh you're just gonna run then the train dot p y and again that same Source folder um which is the you know the top down folder here so you're just you're just going to run that okay so then when you're ready to so this is it's okay now it's all been trained and I'm not going to go through all that because you should know how that works by now um if you don't go watch my video make sure I have a link in there as well um you're just gonna go then take the the results and I'm just going to show you what they look like in the viewer really quickly so at this point I'm just gonna change directory to the viewers slash in and then the siber viewer slash m the output model let's see what that looks like output Ally is going to copy and paste that path and we'll launch the viewer and we can see what this looks like all right so here it is um so one downside is because again I didn't I didn't extract like my head out of it I'm like in all these so um let me just change this to trackball and make this a little bigger like so um it doesn't look pretty unless you're kind of like near the path which you've taken it Nerfs Can Do Better or Worse at this um but here we go so now I'm kind of moving things look great um it's going to kind of have that spherical look um I don't have I I don't have the top above me so if I go look straight up there will be a hole in the ceiling which may or may not be good um but here let me just kind of move around you can see from certain angles things look great but that's because the capture on this was not the best but here we go we have a 360 scene I'm moving through the alleyway a lot of the cars look awesome um if I was to walk back and forth and get a lot more imagery this would have been great um one thing I do want to point out with the myth that I did um if you go and you look at these images again you notice it split it at like horizontally I did not get above me I did not get below me so that's why it's kind of nice too as you go along make sure like what's the goal what's the intended goal of this scene do you want to look up high do you want to look up low I mean what's your focus I will have more tutorials on how to capture like a bigger viewpoint but this is the easiest way to get started so I hope I didn't lose any you guys on any of this and notice like that you know this isn't the greatest imagery in the darks here probably that's the camera part of that is the day and the light but that's how you do it in a nutshell I've had some fantastic results um and that's how you do it and you can take the same image set and you can train it with practically any one of these you can use this with photogrammetry apps it will work so there you go that's just a simple way now go make it some 3D gaussian Splats or some Nerfs and make sure you share them with me because I will put them all over the planet and put them on the internet I love to see what people are doing with these cameras um and if anyone's interested I do have this really nice camera I do have this six camera version this is 200 degree images and I can show you how to take natively each one of those and make a really nice Nerf as well so I look forward to seeing you guys next video next one will probably be how to put this in unity see you guys next video make sure you subscribe

Channel: The NeRF Guru

Views: 17,446

Keywords: gaussian splatting, nerfs, nerf, radiance field

Id: LQNBTvgljAw

Length: 16min 54sec (1014 seconds)

Published: Sat Sep 09 2023