How to Create Gaussian Splats with Nerfstudio

today we're going to make gajin Splats using Nerf Studio that's right they soft launch gajin splatting what does that mean means they launch the code to their main branch so you can install it on your PC and you can start making gajin splatting and not have the same licensing restrictions of the original project because this is an open-source project you can more easily license it for your own commercial uses so if you are new to Nerf studio and you have not installed it yet please please look at the description below to this video where I linked how to install Nerf Studio this video just showed you how to run it once you have it installed but this should be pretty easy to follow through and if you've used Nerf Studio before this is going to be super easy because it's only a few different lines of code well let's jump into my computer and start making gajin Splats okay so the first thing we're going to do when we create gajin Splats with Nerf studio is to follow this custom guide the unofficial guide that I have made in GitHub again I'll make sure that's Linked In the description to this video but this is a fork project from the main code so do not use the code that I have at the top here but the instructions are great and I was able to reuse some of Nur studios's instructions and this guide will specifically just walk you through how to install and run gajan Splats Nerf Studio not how to run anything else um but I'm also going to skip over the installation as I created a separate video for installing Nerf studio and keep in mind that if you have installed it in the past let's say um couple months ago prior to 2024 you most likely will need to reinstall Nerf Studio to get gajan Splats um so just keep that in mind but if you're inell For the First Time watch that video re installing it that video is also great um so I'm just going to skip over this section because that that is for installing but so now we have it installed we're at the point we want to train our first gajan Splat and so if you're new or you're not new some of this may be pretty easy you might need to skip over but we're going to walk it through as if you've never used Nerf studio and you're creating your first gajan Splat um so again the first thing you always do when you run Nerf studio is you actually have to run it through command prompt so I'm just going to search for type in CMD will bring up command prompt and I'm going to make this a little big and I'll probably just go back and forth between my instructions which maybe I'll put on the left side of my screen like so and my command prompt on the right side and we'll work through it that way so if I can make this there we go a little bigger for you so okay here we go so we're going to first we're going to go to cond and we're going to activate our Nerf studio um session and if again you're new to to GitHub you can click this little button here it'll copy it uh and we're just going to hit crl +v to paste it in and now we're in the the Nerf Studio environment that we created when we installed the software but we also need to be working within our Nerf studio folder so I need to change directory to Nerf studio and that might have been installed somewhere else for you but most likely it's in your root folder in Nerf studio if you followed my directions for this spefic specific training I'm going to start out by just using some of the sample data that they provided and to do that you're just going to take this NS download data Nerf studio just copy that going to paste it in and it'll download the data so if I hit enter you'll see It'll download a zip and it'll unpack it into your Nerf studio folder and it show you how to find that once it's complete so there is downloading uh keep in mind this is kind of large but what it is is a bunch of photos I'll show you what that looks like and when it's done it should look like this on your computer so I'll show you that in a second okay now it's downloaded uh I can now find that data if I if I open up Windows Explorer and I go to my uh Nerf studio folder you will now see that there is a there's a data folder uh I already have some other projects in there but there's one uh called Nerf Studio that's the Nerf Studio data and then poster then within that there's a folder of images like so and I can kind of scrub through them these are suboptimal images as you can see they're kind of low resolution they're not the best but they're they're great for just making sure what what you train the first time works and it's not your imagery uh but that everything's just working correctly so I always say use their test data first and then um after we've trained the test data at the end of this video I will also show you how to prepare a custom data set as well okay so now I have that that downloaded um we're we're pretty much to uh to we're almost where we need to be to start training so again you have this full of images you should have some coal map results some of these um camera Json and transform Json and we're just going to say uh we're just going to run the training so again it's all command prompt and anytime you want to run training within Nerf Studio you just type in NS train and then the type of training and so for this it's the gajan splatting so if I copy and paste that in and say we're going to train the data using gajin splatting and then where is that data so it's uh for you it should be in this same place as um where it installed it'll be this uh you're giving it a flag to say okay the data is in data Nerf studio poster so if you look at this you'll see um we're data Nerf studio poster poster and we're already in this folder working so I'm going to um going to hit enter and it will start training the gajin Splat now do keep in mind if you hit enter and it's not doing anything right away it's it's probably loading up the all the data just be patient so here it is it's loaded up and there's all these parameters that we can change later I'm not including that in this video purposefully I'm just showing you the basics this will train a 30,000 um 30,000 steps but uh later on we can we can have an advanced video on how to change some of these parameters okay so now it's going um it's going to start training you'll start seeing this data Cascade across your screen that's telling you the percentage that it's done in estimated time so I have an RTX uh 3090 TI with this scene it's only going to take I found it took about N9 10 minutes in total um so not very long uh it it could go slower or faster depending on what youu configuration you have and so if I go down here um we want to launch the viewer um and then we can see this training in real time at the bottom of this of the screen it'll say viewer beta and and it'll say for me it says 0.0.0.0 then the 70007 uh there is a some sort of error in this system that some people are running into where it's not actually showing you the correct address to go find this um so I'm actually going to show you how to find the right one you you basically need to put in your IP address for those zeros um so I'm going to go look those up on my computer I included instructions here how to uh find them but it's pretty easy just go to your network settings you're going to go to your Wi-Fi and then go to details and I'm not going to show you the details of mine but I will go find them really quick okay so now I've looked at mine up um I'm just going to start a new tab in Chrome I'm going to make this full screen and I'm just going to type it in and it's going to launch the viewer and we can see this data um being trained in real time which is fun so if you just let it sit you'll notice it'll continuously update um I do want to point out though if you watch it training you'll notice that your time will you know for me it doubled or almost doubled so it does take a lot longer for it to render and train at the same time uh you can change how fast I can say fast training speed and what it does is it gives you a lower res preview or you can slow down the training speed and see it a little bit better um I per I actually just like to close this all together and just let it run and come back later um I also want to point out on this this as well that uh you can pause the training at any time uh you can hide the training cameras there's all sorts of things you can do but um just let this run so I'm going to close the viewer because I already have this trained on my computer just to speed things up I'm actually going to close this whole command prompt window if you hit control C that will also stop it but for the purpose of this training I'm just going to bring up command prompt again a fresh one sometimes makes life a little easier and anytime you bring up a command prompt I just want to tell you again you need to type in cond activate Nerf studio and change directories to Nerf Studio like so so you're back to square one make this a little bigger for you um and if I already have finished running or let's say actually I P I stopped the training want to go back to a checkpoint and keep training you have two different options for me I'm just going to launch the viewer on one that I've already trained so I'm just going to basically copy part of this the NS viewer load config which I'm saying Hey I want to launch the viewer and then I need a configuration file from the training and anytime you start training it's going to put those results in an output folder so if I go to Nerf Studio you'll have an outputs folder and then there's that poster project I'm going to go to the gajin splatting and I've ran it twice so um I'm going to run the first one here and there's my config file I'm just going to copy right click and hit um copy his path paste that in and then one more flag I need to I need to add off the instructions is this you want to make sure you're launching the view the beta viewer which is the visualization we want to use the viewer beta or else it won't launch correctly it will you cannot actually view uh results in the original viewer okay now now it's loaded again you'll see the wrong for me I keep seeing the wrong numbers that's fine I'm just going to bring up the correct one load it so now this is a fully trained scene here he goes Pops in uh the first thing I want to notice it doesn't always look great that's because the max resolution is set at 512 resolution we want to push that up to uh 2048 and now things look really good uh they should move really fast on your on your browser and again this is not the best example this was just a great training set that you didn't have to worry about does is it going to work is it not um you can see your train training camera positions you can click on any of those and jump to them like so um and there's all sorts of things you can do along with that um just going to walk you through the viewer really quick as well because this might be new to you um you can also look at everything in a depth mod model or uh when you're actually in that depth model you can change what type of colors you want to look at um to me these more fun than anything uh go back to RGB and then you can do a split screen so we can see half and half these are just ways to like evaluate your results they're not necessarily going to be useful for everyone uh and you'll notice if I move my picture here you can also like change the split um you can invert all kinds of things so just play around with that if you want and then you can also crop your view so I can crop down to let's say just that chair and again you can you can play with these values you can move this this cropping um tool here if you just want to say render out a crop part of the scene um which is nice but if you uncheck it goes right back to the full scene um so if I go back to my directions I talk about exporting results there's two things you can export from this viewer we can export a video through the render tab we can also export the gajan Splats itself which actually isn't done through the viewer I'm going to show you how to do that cuz right now it's not supported in this viewer so let me just quickly show you how to make a video this is not going to be the best but it's just going to give you an idea so the first thing you do is you need to add some key frames so let's say I'm further out I'm going to go around that that poster like so um just start like way up here add a key frame I'm going to come in lower add another key frame even lower like so maybe I kind of want to almost pause there and then pull out and what it's going to do is it's going to start to add ah it's going to start to add an animation path as I zoom in back in here and you see it kind of this bezier curve and if I want to play around with different things you can play around with a spline you can Loop it um you can move your key frames around uh you can get crazy with it but if you want to then get a preview hit the preview render and then play and so you get an idea of like what that looks like so again that wasn't great that's that's fine and you can change the length between transitions your frame rate how long you know the that that'll all dictate how long the video ends up being um but once you're once you're happy with that video and I'm just kind of showing you like the worst of the worst here um you give it a name so we'll call it poster uh and then hit generate command and it'll give you the command prompt you need to create that so let's get contrl C and then I'm going to add this to notepad because I usually end up losing that when I close this so I make sure I don't lose that um I can actually uh exit that that preview and so then I want to render that uh but before I render that I just really quickly want to point out again there's another tab export um I haven't really played with these but I'm not sure these work but a point cloud is not actually what you think it is it's not the py for a uh gaj and splat that you can then use in Unreal Engine or Unity that I'll show you how to do in just a minute uh you can try these I haven't actually tried them myself but they're they're they're not actual Splats you're exporting then in the export so going back to render again I I got that code so I'm actually going to close this viewer and then I want to um stop the processing here so I hit control+ C which interrupts and also then messes up your copy so I'm just going to go here going to copy this whole entire string paste it in hit enter and now it's going to render that video that we' set up uh into my Nerf studio folder uh and there will be a render folder so if I go to uh um not outputs Nerf Studio you'll see there's a renders folder and we should see a new subfolder show up in just a second um I find on my computer it renders at about 15 to 18 frames per second so if you set up a 5-second video it will take 10 seconds to render not too bad and so now I got this poster which we can watch so not the best video but just showing you how it works um super simple get creative I would love to see what you guys create make sure you link anything in this YouTube video uh comments because I'd love to see it and repost it for you guys um so at this point there's not much left we can do except for a lot of you guys will want to be able to export that py so generate a gajin Splat file okay so now I'm going to show you how to generate these gajen Splats um what all you have to do is this NS export gajan Splat load config I'm going to copy that um I did notice I made a mistake in my GitHub which I'll I'll update before you guys actually use this I left out the you so it says gajin Splat and then I got to find that config file so again that was in my outputs poster gaj and splatting and then um my latest one I'm just going to rightclick and hit uh copy as path just like we did earlier in the tutorial and then I also need a D- output ddir again which is right from the the instructions here and after you do that you just have to say where you want to Output to so we're already in the Nerf studio folder so it's going to say it's going to be in data for slash Nerf studio for slash data or you could pick a different folder and copy its path and put it in there um but that's just oops not data poster that's just going to put it in the the folder that I processed everything in and hit enter and it will start processing a gajan Splat file that we can then open in Unreal Engine or Unity or some other third party this only takes um a few seconds okay so now it's done okay so if I go to Nerf studio if I go to data um Nerf studio poster uh you can see there's that point cloud in there it's in that data folder I could have put in the outputs folder as well either one would have worked and so there's a 14 megabyte file it's not very big scene I could open that in Unreal Engine or unity and I have some other videos or you guys can figure that out pretty easily um but there you go so pretty easy if you go back to my instruction set here there's not too much left um I do say you can you can learn more about exporting options here by just if you do d-el on any of these it'll give you basically different options um and then as well as modifying your training configuration so again if you want to train to a different amount of steps or anything like that if you type in this NS Trin d-h let me just put that in so you can see what that looks like it'll give you a bunch of these called hyperparameters that you can then go change your processing options which is really helpful if you want to fiddle around with things but first I suggest just running a simple model first and seeing what works and what doesn't work okay so now that that's loaded that up you'll see there's a ton of different things that can be uh chose for optimization um it can be overwhelming but if you kind of look through here you might be able to pick out a few that um that might be helpful for you as well um so as you look through here um check that out most of these things are probably something that you actually won't need or use um but you can here we go here's more want look for the arguments um where you can go and you know steps per Val image things like that Max number iterations those are the sort of things you'll probably want to change and play around with okay so now now we've made a gajan Splat uh you may if you're brand new to to Nerf Studio you might ask how do I use my own imagery um I'm going to walk you through that if you've ran Nerf Studio before my guess is that you already know how to use your own data so you can kind of skip over this part of the video um but for you who want to use your own imagery we're just going to walk through this really quickly we're going to use the video method if you click on any one of these it'll basically tell you how to run it and you know like which ones are fastest when they say these are slow it's because they have to run coal map which takes a while and or can take a while so I'll probably do some shortcuts but there's these other you know apps and things you can use to kind of skip some of that so if I click on the video it's going to take us to the Nerf Studio documentation page and um the first thing it'll tell you you need to do is you need coal map and ffmpeg and those are really easy to install so for Co map um I'm just going to open that link in a new tab we're going to go to the download page which I think is down here like so you're just going to download this pre-release version it's the one I recommend um and if I go down here um you know what 3.9 was released 14 hours ago as of doing this so I'm going to skip over that CU it's just say source code we want to actually use the pre-built binaries so we're going to probably use 3.8 instead yes so we want to go down here to 3.8 look at the assets and get this uh colap 3.8 windows ca.zip it's going to download it'll take a minute here being a large file and as you can see I've already done it before here recently but if I just take this ZIP file I want to basically extract it to my PC um extract all um I'll sure I'll put this in my downloads and so if I go open that up you'll see there's this folder that that you've extracted and I just want to cut and paste this somewhere that you will know where it is so um I always like to put in my root folder which for me is j n a t it's just kind of like your root root user folder like so I'm going to paste it in here it looks like I already pasted in there so I'm not going to I'm not actually going to replace anything um but now we need to add it to your computer path so it knows how to find this so you just go to search here and you're going to click edit system en environment variables like so go to this environment variables button and then you'll have a system path down here you want to go and you want to edit that and then you're going to go add a new line telling it where um coal map is so if I go find that coal map folder right here um I'm just going to open it and I'm going to copy this Top Line out go back to my en environment variables editor um click new paste that in hit okay for me I'm not actually going to add this because I already have but basically t on the computer here it is here's um here's the folder with all the code go ahead and and and use that so um pretty easy don't build it from Source it'll take a lot more effort um so going to close that up and then the second one you need to go and you need to download FFM peg this works just the same you want to go and just make sure you download the the Windows executable file um either one of these Works um I went and downloaded the essentials you can get the full either one works um and so for me again that's I think I just did that PRI of this yes this FFM Peg you'll end up with this folder as well called FFM Peg and you want to take that folder and add it to your path as well so be once you've extracted you copy in here and you want to uh actually you want to go to the bin folder for FFM Peg which um you'll copy and past paste I'll make sure I add that to my GitHub repo as a gotcha but if you go and you copy this out and then go to edit your system variables like so go down your your path again and you're just going to hit new and again copy and paste that again I'm not going to add that cuz I already have and now your system recognizes both those software you don't have to actually like run through a traditional installer or build any source code um okay so now once you've done that um now you need a video uh or photos for the method I'm about to show you a video uh the video I'm going to use if I just open up my PC and go to my Nerf studio folder um I have a data folder and um this selfie folder so I I'm actually going to delete everything in here cuz I've already pre-processed this once but we have here is a video of me that my wife took she just walked basically a loop around me very slow very steady um that sunflare was not great but it ended up a cool effect in the end and that's all you want to do just kind of like walk around a subject and you can do multiple up and you can go up and down at different uh elevations which is great and just make sure you're kind of far back so I want to get I want to get the building and everything else I want to get the trees and that's it so we're going to turn that video into the data you can use for training and so I have it in here in Nerf Studio data selfie and you're just going to run this NS process data like so so if I go back to my command prompt just going to make sure make sure you're in your nurse Studio environment still in the Nerf studio folder and you're going to do NS process data and then what type of data am I processing so images or video I'm going to pick video so just again put in video then I need the data and data path so I'm G copy that data and then the data path was for me it was Data see here go here is this uh data selfie so for SL selfie and then um you actually need this video name as well so I'm just going to copy and paste it could have just copied this file path as well in explorer that probably would have been faster and then um I need an output directory as well and I always like this to be kind of the same folder so data selfie like so hit enter and now it's going to start the processing so it'll say your your video has my video had 1,851 frames it's going to extract 309 images when you use your own images you want between 150 to at least 300 images um and then it'll start processing the data and this could take a while so if it's not a very long video and you do nice Loops good clean images this might take 5 10 minutes this might take an hour might take even longer depending on how many images and how many pixels um so this is just going to run so now I see it's running coal map just going to let it I'm not going to let it go in this video because it's just going to do itself so if I just going to um kill that process because I don't care to watch that whole thing but you'll see now it created all these folders it made smaller versions of the images um well extracted images and then made a half resolution quarter in eth resolution and then Co map when it's done running we'll have these different files that um you'll need for making gajan Splats um so I'm going to actually go to a different one called Weatherford which is the same video but it's it's done processing so once it's done you should see all this data in here as well and so now that's ready so now if I want to go once it's done I want to train this data all I have to do then is go to um NS train gosan Splats splatting like I have in instructions and then data and then it would be this folder so I can just copy the path to that folder and hit enter and it'll start training on that prepared data so pretty easy I hope I didn't go over to that too fast but it's all in here pretty good pretty pretty well described so you got the process data then train but they show you how to train a Nerf not a Godin Splat and it's pretty easy um and they do have some help here on installing Co map um from Source I I wouldn't I would not do that I would use the the the pre-built binaries and just add it to your path so I I hope this all makes sense um but it's that simple to make gajin Splats very easy and I um should be pretty quick once you kind of get everything set up and ready to go well there you have it that's how you train your first gajin Splats using Nerf Studio it's super simple not that much different than doing the Nerf facto method or any other sort of Nerf using Nerf studio so if this is not a new software to you this should have been pretty easy for you to follow along um if you do have questions about how this works please ask questions in the video uh um the video comments also I have that GitHub repository you can ask questions in there um but I always say if it's something that's truly going on weird go ask on the Discord or go ask um on the original gajan splatting um project because there's people who really know how to troubleshoot things in there as well so again if you if you like this video you make something interesting please please please post it on social and then link that social post on the comments I'll be happy to look at it repost it give you feedback this is great and I hope you guys dive into the world of gajin splatting and I will be producing another video on how to run Nerf studios's just normal Nerfs their Nerf facto method as well uh I'll probably put that video up here eventually if it's not up it's because I haven't produced it yet anyways thanks and subscribe and I'll see you guys in the next video

Channel: The NeRF Guru

Views: 8,543

Keywords:

Id: LhAa1B9CFeY

Length: 30min 51sec (1851 seconds)

Published: Tue Jan 09 2024