AnimateDiff Stable Diffusion Animation In ComfyUI (Tutorial Guide)

do you know we can use text to animation in stable diffusion we can make some short animations just like this let me show you how watch until the end you will see how we fine-tune the animation like this without any flickering today we are going through a new update of animation tool that used to be in automatic 1111 extension now it has support comfy UI it is called animate diff and the processing times even faster than it used to be in automatic 11-11 now there's two way you can install this into comfy UI you can download zip file in GitHub or the more easy way directly download in comfy UI manager let's open comfy UI first you will need comfy UI manager this is like the extension tab in automatic 1111 we have mentioned about this custom node before so check out the previous video if you need to learn more detail now back to comfy UI first you will need to click install custom nodes then in the search bar search animate diff install the one published from art Venture X click the install button it will take some time to download the files and install in your comfy UI once it shows finish the process you will have to restart your comfy UI but before we restart the system head over to hugging face and download the animate diff motion model this is a compulsory module in order to work properly I will recommend you to download the latest version file name mmsd v15v2 it will also take a while to download load this 1.8 gigabytes file take your time once you download and locate this motion model into this comfy UI anime idif folders models subfolder you can pause these videos and go into your comfy UI subfolder path once the file have been located in there go ahead and restart the comfy UI as you can see in comfy UI manager this custom node have been installed it will show that disabled and uninstalled button now we can start loading the animation workflow in comfy UI simply drag and drop the Json file onto the screen the workflow diagram is looks like this now I will be provided the Json file in the video link description below now this workflow are using text prompt to animation only the anime diff are able to run with image to image as well but text to animation are going to generate more movement so I am going to use text prompt only in this video now as you can see we have the load V below that it is the animate diff module loader as we have download the motion model previously this is where it belongs to and the common checkpoint loader with our list of checkpoint models I am going to use rev animated checkpoint to get started now let's go through each nodes in this workflow we have the text prompt node on the top here which is all the image you want the AI to generate like the beautiful girl or beautiful house Etc and below that this text node are using for negative prompt so we will do a lot of typing in this two text box later let me predefine some negative prompt first now on the right hand side we have the sampler for animate diff and below that is that empty latent image which is basically for you to set the image size and the batch size and we have the vae decode Gathering the samples before it outputs to animation okay very simple workflow here nothing fancy and friendly remind one more time before run this workflow remember to download the motion modular model let's try some prompts and generate animation great let me reorganize the layout and set that image size this time I want to try 10 24 times 768 pixels for the sampler setting I would use random seed number because we don't have any preference yet in steps number I'm going to try 25 CFG is going to be seven for sampling method you can try a lot of way and I will try DPM plus plus 2m sde for this time for the noise I will leave it as default and frame number let's go for 30. I know 16 is the best performance but I wanted to do more experiments in our tutorial now check one more time everything are set if no error then we can click run and see our first generation now as you see the green highlight on the sampler node that means it is thought processing you can always go back to the command prompt and check if there's any errors from the execution okay looks smooth to me okay I open the image on the other tab okay this animation is looks like falling leaves but the girl is Missing let's change the dimension looks like 10 24 times 768 are not good [Music] oh [Music] foreign looks way better Japanese cartoon style we can do a guy jump into the water animation that's funny okay I am going to civitai.com and get some inspiration thank you I have set blur image for NSFW images so you are seeing lots of blur on here [Music] um the World War II Japanese Army image we can try out with this prompt [Music] for sampler setting I will set to CFG equals to Fourth Chaco random seed number and sampling steps to 13. rescue prompt to proceed okay pretty cool oh yes I love this image let's try this with sleepy JB [Music] by the way this is how I usually get some prompts ideas from browsing on civitai.com so I saw some comments from you guys mentioned that having trouble on using prompts and see a lot questions asking which checkpoint model do that best there are no checkpoint models can do everything to be honest maybe today someone released a really good realistic style checkpoint and tomorrow someone else make a better one and that is very basic principle for AI when the model train more it will get better and better so don't over relay on one model or just stick to one algorithm I remember there are one jerk comment in on of the older video what he means was like oh so the animation is not create and stable diffusion and I was like um so effing what as the title and beginning of video I mentioned it was using stable diffusion with that tool but those stubborn people now I don't bother to waste my time in reply I will just block the F out of him okay back to the animations here now as you can see not every style of images text prompt are going well with animation and sometimes you may need to generate few times with a prompt to come out with good animation result so here you go feel free to try out this animate diff custom node in comfy UI and comfy UI do perform better in way of loading time oh this is a cool one it feels like the Subaru WRX desert rally racing okay that is it for today's video and I will see you guys on the next one have a great day

Channel: Future Thinker @Benji

Views: 12,879

Keywords: Text-to-Animation, Stable Diffusion, Animation Tool, Comfy UI, AnimateDiff, Image Generation, AI Workflow, Training Process, AI Models, Motion Model, Prompt Generation, Creative Animation, AI Experimentation, Visual Storytelling, Animation Workflow, Custom Nodes, Image Size, CFG Setting, AI Prompts, Model Training, AI Control, AI Artistry, stable diffusion tutorial, stable diffusion, ai art, stable diffusion ai, stable diffusion xl

Id: gBMuJ-_lukY

Length: 10min 33sec (633 seconds)

Published: Mon Sep 18 2023