Depth Map Stable Diffusion Tutorial: INSANE 3D Models & Animation

[Music] foreign concept is really quite simple we want to be able to generate an image in mid-journey let's say for the sake of this a landscape we want to then take that landscape image that mid-journey generated and we want to plug it into stable diffusion more specifically stable diffusion with a depth map extension installed that's going to generate something like this now this may look quite rudimentary or simple but if we take this depth map along with our original image and plug it into blender we can get something like this which takes that and with a little bit of tweaking we'll turn it into a three-dimensional plane you can do this with a plethora of things and then actually animate it if we hit play you'll see this black box which is the camera icon we have rigged to flow through our scene if we click into our camera here we can actually see what it's doing this little box here will be our final animation now we're going to walk step by step through all of this if you've never used blender you've never used stable diffusion depth map it's no problem at all this is going to be for absolute beginners and by the end of this you will have an animation of a 3D plane for the sake of this tutorial and the fact that we're going to go through this together step by step I've generated a new landscape here's my original very simple prompt NV 5.1 on mid-journey this was my favorite and I would love to have the camera actually flow through this Valley over top of all of this and in between the hills so I think we're going to go with this as well here we are on automatic 11 11. this is the local web UI for running stable diffusion if you don't even have a 1111 installs then go ahead and click on this card at the top of the video you're going to get to this point in about a minute and a half or two minutes and then you can come right back what we need to do here is actually get this right here which is the depth extension in order to do this go ahead and go down to what is labeled as link one in the description down below then go ahead and navigate to the extensions Tab and click on install from URL you're going to paste that URL right up here and I'll go ahead and do that now go ahead you're going to paste it right here and you're going to leave these other two completely blank now go ahead and click install only click it once don't worry it is performing a function even though annoyingly so there's no order completion check it doesn't let you know that it registered a click if it worked go ahead and go to the installed tab this will be kind of grayed out and you'll have a little loading icon here and a percent right up here on the right once that's done go ahead and click check for updates and then apply and restart UI you may have to restart your entire a11 terminal but nonetheless you'll get back here and you'll now have the depth extension go ahead and grab your mid-journey generation you can do that just by opening it right clicking and saving we're going to go ahead and then drop this in right here we're going to drop this into the single image in depth now the model is up to you I prefer either to use res 101 or Midas Midas and res 101 are both just enough for the job we'll go ahead and leave that as the defaults one thing cool we can do is show the heat map if we'd like to and there's some other options down here but don't worry about these for our needs we're just going to leave these blank make sure you show the depth map and if you want you can save the depth map as well we're going to leave these blank and we're going to have boost on as this is just going to give us a better resolution now depending on your GPU you can also choose to process this with your CPU power if you have something say less than a 2080 that will that will definitely be relying more on the CPU when you go lower end so all we have to do is load that in there and then we'll go ahead and just click generate you'll see a little loading bar right here which is very deceptive because this will not be done loading when the bar is done it'll be done loading well when it's done loading so we'll be right back this will probably take right around two to three minutes all right just like that we have our depth map you can also see that we have a heat map I just wanted to show that that was a possibility and so you could see the difference but let's go ahead and click on open the steps map and click save image as I'm going to go ahead and save this as depth map 3. you can actually see a little character in my file up here as I'm also working on 3D character modeling so if that's going to be something you're interested in seeing like this video And subscribe there was my generic pitch but you're gonna get something for it now let's go ahead and open up blender that will be link 2 in the description down below this is completely free to use when you open it your window is going to look a little bit different with the first time but go ahead and it's just going to prompt you to create a new file and we're just going to click on General and it'll bring you to this page right here first thing we want to do is get this cube out of the way we're just going to left click it and hit delete now we're going to hold shift and press a it's going to bring up this little menu right here if you're completely unfamiliar with or have never used blender don't worry we're going to walk you through every step we're going to hover over mesh and click on plane this is going to create this flat little plane surface if you're wondering how I'm moving this around I'm simply clicking down the center scroll wheel on my mouse if we want to move up and down we have this little hand where we can move up and down left to right along our plane as well those are going to be the two main functions that I'm using other than Control Plus to zoom in and control minus to zoom out that we have our plane we want to do something called subdividing so let's go ahead and click on it and then on the top you'll see that this has a modeling section go ahead and click on modeling something you'll notice that you may have missed if not mentioned right here this just changed from object mode to edit mode we're going to talk about this more in a moment but just make note that we are now in edit mode we're going to take our surface and right click on the plane and then left click on subdivide now you may miss it but this is going to open a little menu down in the left called subdivide there's a lot in here but all we want to do is change the number of cuts to 100. you'll notice that your surface becomes a little bit orange and if you click off it becomes black that's because we just subdivided this entire plane into 100 by 100 individual cubes think of these cubes or the points connecting them as really the VAC factors of where our 3D is going to pop out or how it's going to define the points and edges so we can make it 3D from here we want to get out of edit mode now there's some shortcuts but let's go ahead and just go to the top left and click from edit mode to object mode you'll notice that our Cuts go away they're still there they're just not crowding our screen to say from here we want to go ahead and add a modifier this will be in the little wrench icon right here that you see I'm hovering around it should be here by default but if not we're in the wrench go ahead and click on the add modifier there's a lot to take in here but all you need to know for this is we need a displacer this displacer modifier is going to tell our cuts that we want to displace This Plane based on a certain texture so or not texture but displacement image which is going to be our depth map go ahead and click on new now it didn't seem to do anything but you'll notice now it has brought up texture we're going to go down to this little Checker or chess board here which will be our texture properties on the right side of the screen go ahead and click on that and it's going to bring up this black image from here we're going to actually open our depth map which I believe was named depth map 3. so I'm going to go ahead and go to downloads and we will look for depth map 3 dot PNG and we will open image now you can see we have a little bit of a valley going on now let's get this less rigid looking let's go ahead and click on it you'll notice on the top right here we have camera light and plane you can either click on the object or click on the container up here in this case we're going to left click on the plane and then right click and we're going to shade smooth you'll notice that this just made our plane a little bit more smooth a little bit more easy on the eye from here we actually want our image or all of the coloring in our image this right here to be portrayed onto right here now what we're going to do is go up to the shading tab go ahead and click on that and it should bring you to a window like this from here you're going to click the new button and it'll give you these two containers we're going to click off of those hold shift and press a go down to texture and then up to image texture go ahead and drop this box right to the left of the original Green one and then go ahead and click on open now when we open it we want to actually grab our original saved image so for me that is going to be in downloads and then that is the overhead view um so we will go right to here overhead view and open image now where it says color we're going to connect the yellow dot to the yellow Dot like so and we will then have our texture so let's go ahead and zoom in and we can now see our image starting to come a little bit more to life from here let's go ahead and go back to layout and on the top right we actually want to see our image with all of the coloring so let's go ahead and control plus we will zoom in a little bit and on the top right we're going to click the furthest right bubble that will be Viewpoint shading which will let us view our image or our 3D render rather with it completely shaded now as initially stated we want this to be like going through a valley right now we do have a bit of 3D texture in there but we want to extend this out towards the viewer so on the right side we're going to click on that little wrench again and this will actually bring us to the settings for the plane itself remember this is texture and image generated onto that blank plane so whenever we want to manipulate this mountain range we want to manipulate the plane itself which is going to be in this little wrench icon what we're going to focus on is the strength and I'm going to take this one up to about a two and you can see what happens as I do that we start to elongate now from this view it looks pretty messed up but from here going through it and once we set our camera we will be able to see it a bit better and a bit more the way we want from here we actually want to set up our camera so it goes through our scene go ahead and click on the video camera icon or the toggle camera view and it'll take you to a view that probably initially will look like this right here now we're just using our Center Mouse wheel to position this and Control Plus to zoom in one setting that you do want to make sure is on is your camera to view lock on the right side you'll have a little view tool and item menu if you don't there may be an arrow there which will pop this window out to the left for you inside of the view tab you just want to make sure that this box right here is checked where it says camera to view this will lock your camera to your view so wherever you go is going to be what your camera is depicting in your animation go ahead and get it positioned where you'd like as we want it to go down through the valley we're going to go ahead and position it right here and send it down through that way so let's go ahead and get it right there now one thing that you do want to make sure is on the right side you have camera selected versus plane and we're going to set our first frame so press I and then click on location rotation and scale this will have dropped a small diamond in the very bottom of your screen on the frame by frame playback now let's go ahead and set our second frame all right now you should actually be able to see this Frame bar down here so I would like to change it at the halfway point as you can see we have 250 frames we can extend that or start it from somewhere different with these right here we can have as many frames as we want just no rendering is going to take a lot longer so let's go to the 100 point and then let's go ahead and send our camera a little ways down the frame let's go ahead and go here and then go up like this and we will press I location rotation and scale now let's go ahead and go to 190 frames we will Control Plus even more open up say this way and down into the Mountain Still and let's move it over just a tad bit and then we will again press I location rotation and scale and let's go to our final endpoint at the very very back of our Valley and set it to be 250th all right we'll go back in right now and we will go ahead and for the final time press I and then location rotation and scale this has set all of our frames so now if we play it we can see that it comes in and swoops through our Valley very very nicely now we can go ahead and get our initial render or the video that we're going to download for whatever our use is on the right side above the wrench well above the wrench we have render properties on this all you're going to change is Bloom go ahead and take that and motion blur leave those two ticked and go one down to Output properties the only thing you need to worry about here is this file format as you can see right now it is going in PNG we're going to click that and put it in ffmpeg video just the best output for our use or for a video use case rather now on the top left and I will go ahead and reposition my display capture here so you can better see that on the top left I will have a render tab or well you will have a render tab go ahead and click on that drop down and we're going to click on render animation this this is going to start the rendering process don't worry if the screen looks blank you're going to see some flashing here it is rendering each frame into our animation and if we go back to it we can actually see that happening in real time let's go ahead and let this render and then we'll talk about how to actually download it all right now that our render is done we can go ahead and check it out see if it's what we like and see if we need to position the camera any differently go ahead to the top left and click on the render drop down again and then click on view animation this will pop up our new animation of course we'll have to position the camera to take out the gray sides there but we have our fully rendered animation go ahead and back out of this and if you want to download it you can do so from here as well or rather it will already be rendered and downloaded we can just re-render any changes we make from there you can find this by opening your files now I use osc for it but it will make this file called TMP if you open that file we can find our render right here for now what we can create is limited to what people have tried that's why on this channel I'm going to be diving extremely deep into this as I think we can create some really cool things so for starters leave me a comment let me know if this is helpful to you and if you'd like to see more videos like it and if you would don't worry they're coming including character creation in 3D and animation once we get access to things like Runway Gen 2 which is coming very very soon if you want to learn more about AI tools check out one of the two or both of the tutorials on your screen now don't forget to like the video for the algorithm And subscribe

Channel: Hustle Millennial

Views: 20,186

Keywords: stable diffusion, Depth map tutorial, depth map blender, depth map stablediffusion, stable diffusion depth map, stable diffusion depth map blender, blender stable diffusion, stable diffusion blender tutorial, depthmap tutorial, stable diffusion depthmap, stable diffusion depth map tutorial, stablediffusion depthmap, 3d image transition, trending 3d image transition, 2d image to 3d model, image to 3d model, 3d image video, image to 3d model ai, 2d image to 3d model blender

Id: yfWHVigsOk0

Length: 17min 59sec (1079 seconds)

Published: Sat May 13 2023