Use AI to create amazing 3D Animations! [Blender + Stable Diffusion]

I'll show you how to create a super simple 3D scen in blender and turn it into this and this and this or even this in a matter of minutes this is one of my absolute favorite AI workflows because it is super easy and so much fun and you can use it for brainstorming developing consistent environments for your short film projects or video games or just to finally start getting into blender one of the first steps for creating a 3D rendering is to create layout a very rough model using primitive shapes to kind of map out the basic geometry of the scene to get a better understanding of the final shot it's basically a sketch but in 3D and with this workflow we basically want to go from layout to the final rendering speed running that whole process so let's say we need an establishing shot for a short film depicting a futuristic city surrounded by a giant swamp so let's go into blender and delete everything and I create a new plane as my ground plane I want to have this futuristic city sitting on like a little island in the lake so let's subdivide This Plane a few times and I'm just grabbing this middle part here and dragging it up to create a little island so once I have like a very rough very very rough version of my geometry I'm creating a new camera and putting it roughly where I want it to be I'm also setting my camera resolution and I'm using an aspect ratio of 2:1 but you can use any other that you like you can also come down to camera viewport displays and I like to increase the ppat two a little bit and as I said this technique is very good for learning framing like composition and stuff so if you're just starting out I would recommend to come to the camera settings and turn on the composition guides like you can turn on the thirds or the diagonals or the center and that just can help you to create a satisfying image so now I'm just trying to make the geometry of the scene a bit more interesting by breaking up that horizon line a little bit by creating these these mountains in the background I also want to create a middle ground where the city's going going to be a background and a foreground right now we don't have a foreground so let's create like a little foreground Mountain here keep the resolution of the geometry really low so you can play around with stuff very freely using simple cubes I just put in my skyscrapers or like my buildings and let's first try to keep this pretty symmetrical and now I want to have the city surrounded by giant trees so I'm just adding in cylinders so my city is very symmetrical so I want to break that up a little little bit with these trees I also want to try to create some vegetation so some some canopy and for that I'm going to use metab balls these are just these little bolts that you can put anywhere in the scene and if you duplicate them they will create this merging effect and they are amazing for blocking out like organic shapes and it's just like painting you want to start really big and then start working in these finer details now these this will look a bit weird but we'll see if this will work now the geometry is still very rough so what I'm going to do is I'm subdividing this plane and then I'm using the scub tools to add in a bit more detail so yeah finally I'm just creating a new plane as my water level for the swamp and then I'm just duplicating that and put it right here as my sky so this is my final geometry my final layout of the scene now we need a way to transfer this 3D information to the AI and for that we are going to use a depth map or a zdepth pass a depth map is basically just a black and white image that tells us where the pixels are in space so bright pixels are closer to the camera and darker pixels are further away and to create one in blender we just need to go to view layer properties and turn on Z now it's probably a good time to save your scene by the way then go to compositing and click use notes now go to render and render image and we can see we can pretty much see nothing because it's just basic geometry and there are not even lights in the scene but we don't need this go to the compositing window and you can also add a viewer so we can actually see what is happening so when we activated the Z option here in the view layer properties this depth window appears and we want that so let's connect that to to the viewer there is some information in here we can see some areas but overall it's just a pretty wide image and the reason for that is that all these values in this image are all over the place but we want them between zero and one and to do that we just need to add a normalize node now we can actually see the depth information of our image but it's the wrong way so now the black pixels are close to the camera and the whites are further away so let's switch that by using an invert node let's connect this invert node to the composite um output so when we render the image we actually get this image let's save that image you can just do that up here image save as and put it somewhere where you will find it make it 16bit color dep and put the compression to zero and save it for the next step I'm going to use the automatic 1111 stable diffusion webui but you can use stable diffusion in any client as long as it supports control net just go to the text to image Tab and select a model that fits the style that you want to generate so I'm using cyber realistic in this example let's set the sampling steps let's put them a bit higher to 25 and the width and the height should match the one of your blender camera so 1,24 to 512 in my case I like to put the seed to a fixed number so that prompt building becomes a bit easier and then we can come down here to activate control net enable choose pre-processor none and the model should be the depth model and import your depth map to get a bit more interesting results you can also lower the control weight just a little bit so I will put it to9 nine and now we can start filling out the prompt I like to describe everything that I've modeled and that I want to see in the final rendering so let's try a futuristic Metropolis on a lake Island surrounded by giant forest huge trees tree trunks swamp Moss evening sky and warm lighting okay this looks pretty cool it looks like a painting though so let's put in photography uh Arc VI and maybe octane rendering now this already looks pretty cool but it's a bit Gray and not the warm evening lighting that I had in mind so let's crank up the CFG scale a little bit so that the image conforms more to the prompt let's try 15 okay that's that might be a bit too much let's lower it to 12 and also put like high contrast in the negative prompt because this looks a bit weird with this contrast yeah I think I think that's it that looks pretty cool so all the images that we generate are already saved so it doesn't hurt to try around a bit more so yeah this only took a couple of minutes this is my final prompt uh these are my final settings but now let's bring that image back into blender into our 3D scene so let's import the textures into blender for that we're going to use a technique called camera projection basically we want to use the camera as a projector and project our texture back onto the scene you just need to select all the objects in your scene go to edit mode and make sure that toggle x-ray is activated so we can see through all the surfaces and then you want to select all the geometry in your scene and then you just need to go to the camera view and click UV project from view now you can go to the shading Tab and add a new material so I import my upscaled um AI image so so now you can see there is a material added to one of these objects here but we want to add that to all of these objects so click control L and Link materials now this looks a bit weird but when we go back to our camera view it actually looks kind of correct but only kind of because some of these areas are still a bit stretched for example the sky doesn't make any sense at all and the problem is that the sky didn't have enough geometry and this is really easy to fix just select the sky go to edit mode and subdivided a few more times go back to the camera view and click project from view again and now the sky is correctly projected onto this plane let's work on the Shader a little bit so I like to go to my active camera and now this looks a bit weird because specular is so high so let's reduce the the specular and play around with the roughness now I'm just noticing these colors look a bit weird not like our original picture and that's because we need to go to render properties color management and under view transform select standard so now we projected our AI image back onto our layout scene and let's now start bringing this image to life here are some examples of how we can bring these scenes to life let's start with the animation of the camera you'll notice that if we move it too much we can see a doubling effect as the image is also projected again on these background elements and The Simple Solution is just to take those foreground elements and just move them around a little bit scaling them up and just covering up the doubled images in the back ground you can usually get away with it but in this shot for example I wanted even more flexibility with how I can move the camera so I just opened the AI image in Photoshop and used generator fill to remove the foreground elements I then selected the sky plane duplicated the material and switched out the image with the one of the sky I also wanted to add a little UFO to this Martian City so I used the same depth map workflow to create one I modeled a simple UFO exported the zde pass put it into stable diffusion with a prompt for a UFO in nice weather I projected the image back onto the model copied it over to my original scene and animated it but back to our futuristic city in the swamp the next thing that I wanted to animate is the water so I selected the water plane and created a new Shader with a glossy bsdf I then created a noise texture connected the color to the height reduced the strength of the bump map a little bit and then using the node Wrangler plugin I created these mapping noes using control I then animated the location over the duration of my shot and it created this animated water effect a cool thing with this technique is that we can also easily add elements to the scene because we have all the geometry and we have all the colors and so these objects tend to integrate very well into the scene so let's just put a man that I AI generated for another project right over here in the foreground and here's our final rendering or is it let's say we changed our minds and we want this city not in a swamp but in a desert no problem we just go back to stable diffusion change the prompt generate an image and change it out in our blender shaders and the new scene is ready took like 2 minutes iterating and trying out new looks has never been so easy City made out of sweet surrounded by candy cane no problem original scene but in an anime style done or is old school Dungeons and Dragons illustration here it is okay I think you get the idea and see how much fun it is to play around with this technique I hope you had as much fun watching this tutorial as I had making it and I can't wait to see what you create with this technique oh and if you're interested in AI film making I'm offering a cause on my patreon page right now it already started but if you join now you should be able to catch up the goal of this course is to learn traditional film making and cinematography techniques with AI and every participant will create their own AI short film over the course of a few weeks see you there or see you next time

Channel: Mickmumpitz

Views: 44,707

Keywords:

Id: YW3FzfVxAEs

Length: 12min 15sec (735 seconds)

Published: Sun Oct 15 2023