Quick Tutorial - Mixamo to Stable Diffusion

hey guys welcome to this quick tutorial where I will show you how to use a miximo animation for creating a stable diffusion video we will be using blender for creating our starting animation and then feeding this animation into automatic 1111 batch image to image using control net as a guidance tool for those who don't know mixmo it's a free website from Adobe where you can download high quality 3D animations and use them in your own 3D software just create a free account and login into this page so you can find a great variety of 3D models and animations for your own use so here we are on the starting screen on the right hand side you can see the standard 3D character which we are going to use and on the left hand side you can find all the animations available we are looking for a dancing animation so we type in dance into the search bar and then we click on the animation we want to use I think I'm going with the samba dance it's a long animation and it's looped so you can use it over and over again if you need a longer sequence click on the download button choose fbx as the format with skin and 24 frames per second because that's the frame rate we want to use in blender once that's done we are finished with mixer mode and we will switch right over to blender if you don't know blender it's a great 3D modeling software which might look a bit intimidating at first but after all it's not that difficult to learn so here we are at the default scene of blender with the light camera and Cube let's delete the cube and let's import our mixer more animation so go to file import fbx and search for the file we downloaded before here we go so here we have our model with our animation now click on the Run button to see if it works seems okay what we want to do next is opening a new window move your mouse to the upper right part until you see a plus sign and drag the window out next we adjust our render properties we are going to use the EV rendering engines because it's very fast and we select ambient occlusion bloom and Screen space reflections next let's adjust the scene we want to render you can use any size you like but I'm going for 1K Square image so 124 by 124. let's click on the camera button on the right window which will show you how the rendered scene will look like then select the camera in the right upper window and change the camera position and rotation until you're happy try to bring the animation object as close to the camera as possible because it will produce better results later in stable diffusion now hit the camera icon on the right part of the screen and adjust the focal length I think we are good for now let's switch to rendered mode on the right upper part of the screen we still need some background image in order to get better results later in stable diffusion so I'll just go to pixabay and download some random image it's also a free site with lots of great free stuff I'm looking for a citizen I'm just going to use the first one doesn't matter much what background you use as long as it vaguely resembles the video you want to create in stable diffusion so let's download it we don't need high quality let's go back to blender and import the background image go to file import images as planes and select the picture we just downloaded let's adjust the size rotation and location of the image the so it can be a perfect background for our scene and once that's done let's select render render image from the upper menu it's not a perfect scene but I think that's all what we need next let's go to the render menu and all we need to do is selecting the folder where we want to store our video animation let's call it blender render let's select the Armature and see how long the animation will be it's about 476 frames so let's enter that value to the end of the animation and then we are almost done go to the render menu and select render animation going to take one or two minutes to render the whole animation and once it's finished you will find 476 image files in your blender render folder but that's all we need to do in blender and we can switch over to our automatic 1111 web UI I'll leave a link down below how to install it and run it on your PC or Mac let's click on image to image because that's what we are going to need then find your blender render folder with your images and drag anyone into your web UI next we need to enter a prompt to describe what kind of scene we want to render a beautiful young woman with long red hair dancing in the street of New York let's select the batch count of four which means four samples will be created and then hit the generate button let's take a look at the images and choose the one we like most I think that one is good then hit the Green Recycle button to get the seed of this image you will notice that the balls of the image which we just created differs quite a lot from our input image and to change that we will use the control net extension if you don't see it on your screen you need to install it first I will leave a link in the description where to get it and how to install it you will also notice that I'm not using the standard model which you can but the custom model from civit AI called deliberate which in my view gives better results here's the website you can download the model for free and copy it into your models folder I'll leave a link down below where and how to get these models when you find an image that you like click on the information button copy the positive and negative prompts and paste them into the prompt fields of your web UI automatic 1111 this might give you a better look and feel for the final image and animation you can also add some more positive and negative prompts just make some test renders until you think that the scene is fine now let's enable control net you can stay with the standard settings but you need to select the preprocessor enter model I'm going with a depth map as a preprocessor and same for the model because in the city scene a death map could be the right choice just play around with it and you will see what fits your animation best let's make another test render it's not perfect but you can see that the pose exactly matches now now we don't just want to have a single image but an image animation so you need to select the batch Button as an input directory we enter the path to the blender render folder where all our images are located and for the output directory we enter a location where our stable diffusion images should be stored if that folder doesn't exist it will be automatically created by automatic 11 11. you might also want to play a bit more with the settings like sampling steps CFG scale and denoising strength which have great influence to the final renders but for this tutorial I will just leave them as they are hit generate and wait for maybe an hour until the animation render is finished okay we are done you can see that a new folder has been created with 476 images created by stable diffusion you could now import this image sequence into your video editing software to find the output even more but I'm just going to use QuickTime to convert the image sequence into a movie and see how it looks I'm also adding an audio track and we are done for this tutorial it's still a bit flickery so you might want to reduce the denoising strength before rendering but overall it's okay I think anyway hope you enjoyed the tutorial if you have any questions please leave them down in the comments and see you next time

Channel: Render Realm

Views: 15,319

Keywords:

Id: eFjECaeY4Go

Length: 11min 5sec (665 seconds)

Published: Thu Mar 09 2023