How to make AnimateDiff Videos in A1111 and ComfyUI - AnimateDiffv2!

AI generated animations using animate diff but without watermarks and with control net 2 the order me spaghetti and change my name to will because that's exactly what I'll be showing you how to make today yes hello and welcome to More nerdy rodent geekery as I dip my little paw back into the world of anime diff because they've released a new model not only does it give even more animation when compared to the previous version but they've also got rid of that annoying Shutterstock logo which used to plague everything with a typical animation taking under 30 seconds to generate on a 1390 you can quickly make a variety of people eat a selection of different foods all in the most normal way possible plus as you can use control Nets to guide your output you have that perfect cinematic control the new motion module was trained with a larger resolution and batch size to improve quality and I've been getting nice outputs at 768 by five seven six you can use either automatic 1111 or comfy UI and I'll be showing both of these here links for all of these things are as usual down in the video description as far as Hardware requirements go the more vram you have the better but basic animations can be done with just eight gig first up is automatic 1111 where you'll need to have the animate diff extension installed you can do this by going over to extensions then available click load from and search for animate diff then click install and then restart you'll also need to download the new animate diff model which is available on their hugging face page the extension expects the file so that one there the new version two in the extensions model directory so you can download it straight there or if you have a different location and you'd like to save models there like I have you could just make a link another option is over in the settings under the animate diff section where you can see path to save animate diff motion models so you could change that to another location if you wanted to as well now while we're looking at settings a couple of things you will want to enable are the pad prompt and batch condition unconditioned check boxes so make sure you've got both of those ticked if you've got any changes that you've made don't forget to click apply all the way up at the top there and that's it you've got everything ready to create your own AI animation like a true film expert this extension Works in either text to image or image to image but personally I think it works best in text to image make sure you've got a stable diffusion one checkpoint loaded otherwise things won't look good start with getting your prompt up and running as usual there I've got a fantastic picture of a teddy bear but do be aware that when you enable the motion module that is going to change how the image looks so just get the general sort of thing you want rather than some Perfect final image all right let's open up that anim diff section there and click enable Alim diff and that's pretty much all you need to do we've got the motion module selected there version two and then let's generate and there he is we've got the skydiving bear now while changing the other settings is possible more than 16 frames doesn't tend to look very good with that being the case 8 FPS is also the best setting and and personally I don't care for fixed numbers of display Loops if anyone can get anything decent from 32 frames then do drop a comment on how that's done at just 512 by 512 this is blazingly fast as you can see that took about 17 seconds to generate that image so you shouldn't have to wait too long before you can experience your own perfectly crafted cinematic Masterpiece and you know there it is that animation is fantastic and the GPU didn't appear to go over 8 gig at all Hollywood watch out however as one tip I have found that generating action scenes tends to give the most movement things such as skydiving eating running or whatever now some images won't have much motion because that's what they're like in reality and that's going to be reflected by the model take this picture of a tree as an examples there photo of a tree let's open that up so we can just see it more there and yeah and it's got some motion but trees don't really have a high amount of activity so that's what you'll get as your output and it's this lack of motion which is the main reason I prefer text to image over image to image let me show you what I mean so we've got our teddy bear there we're going to send him over to the image to image Tab and I've got animate diff enabled as normal and the denoise all the way up to one so if I generate this then as you can see the motion is not dramatic at all it has changed quite a lot obviously from that original image not moving very much and if we go to the text to image version of that yeah there's loads more motion there so that's just why I say I prefer text to image but of course you may want a small amount of motion that all depends on what it is that's your main hiking one thing I haven't yet found a way to do is use control net with animate diff in automatic 1111 but that's certainly something you can do in comfy UI thanks to a whole bunch of pre-made easy to use workflows one simple way to get animate diff working in comfy UI is to use comfy UI manager just load any of those workflows open the manager then click install missing custom nodes and restart all those red nodes will disappear and then you can get on with your animation you also have other installation instructions on the GitHub page there as well should you wish to follow those as with automatic 1111 you'll also need to download that funky new checkpoint and save it in the models directory for that custom node rather than the usual location which you would expect in models control net so it's in the extent engine one for this they provide loads of example workflows for use a trip of free so let's just quickly check these out first of all there you've got text to image then you've got another one there text to image with latent upscale there's a variant on that as well that's got full denoise then there are a couple of control net stabilized latent upscales one there with a soft and the full D noise as well and finally we've got down here another couple with control net using the line art input preprocessor and also the one I'm going to go through in a minute as well the initial inputs using open pose images plus the latent upscaler so basically going from quite simple to reasonably complex there is also a work in progress version of image to image at the bottom there but as with automatic 11 11 it's not really all that good yet let's take a look at the first simplest example it's much the same as in automatic 11 11 be sure to select a stable diffusion 1.5 checkpoint and the anime diff model you want to use as well obviously you're going to need a vae unless there's one baked into the model you can link that over there set your positive and negative prompts to whatever you want ideally remembering some sort of action in there and generate a way once again generation is pretty quick at around 20 seconds but this time I did see vram usage go up to 16 gigs so twice that of automatic 11 11 but it may automatically swap out models if you've got a lower vram card so do give it a go and see but there we have our final image let's have a look at that so we can see the animation there is our skydiving teddy bear looks rather cool doesn't he now rather than go through each of those other workflows I'm just going to jump straight to that most complex one with the open pose control net and latent upscale because this has everything in it anyway I'm using the example here from the GitHub page so we've got a directory comfy UI frames and that's got each of those frames from the open pose control net there it is so there's the directory 16 frames and each of those images 512 by 768 obviously you're going to need a control net model as well so there I am using the open pose one that you can download There's a variety of options there but I went with the smaller fp16 safe tensors one the animate diff loader is down here and once again I am using this new version 2 checkpoint as before set your prompts to whatever you want you got your positive and negative prompt there and generate whatever you like now this will give you two different results you've got the first one there which is the normal resolution and then you've got one which is upscaled as well so let's have a quick look at the normal resolution there's the alien saying hello and then the upscaled one is a little bit different but obviously a higher resolution it does change on the upscale so there you go of course you can use any stable diffusion 1.5 control net model and whatever images you fancy such as depth or Kenny Edge it doesn't have to be the open pose as shown in that example now as a quick summary that Shutterstock logo is gone so animations look much much better and the extra Motion makes it even more fun as well both extensions appear to have issues going Beyond 16 frames so even with that new 32 frame Max the best results come from 16 frames while image to image is possible in both automatic 1111 and comfort UI I don't think the results are quite as impressive for more advanced features comfy UI seems to win out thanks to the easy pre-made control net workflows and if you don't like the GIF output simply use the PNG files and interpolate upscale as required and of course most importantly the next nerdy rodent video will be fun too

Channel: Nerdy Rodent

Views: 24,020

Keywords: animatediff, animatediffv2, AnimateDiff, AnimateDiffv2, AnimateDiff ComfyUI, AnimateDiff A1111, AnimateDiff Automatic1111

Id: w9O-qkgnoi8

Length: 11min 44sec (704 seconds)

Published: Thu Sep 14 2023