How to AI Animate. AnimateDiff in ComfyUI Tutorial.

I made these animations with AI in just a few minutes and I'm going to show you how to do it too you can do something like this I'm going to show you several ways I'm going to show you a text to video workflow I'm going to show you a video to video workflow I'm going to show you a couple of tips and tricks along the way to get you to the best results possible I'm also going to show you the free version and the very cheap version let's get started oh I've never been a fan of facial hair but it's starting to grow on me hey for the very cheap option you're not going to require any hardware at all well a computer or a phone that's about it for the free option you're going to need a GPU with at least 8 to 10 gigs of vram we're going to be using the inner Reflections guide and workflow today and I'm going to start with showing you the workflows after that I'm going to show you how to install the custom nodes if you need that however this first version is going to be the EC and then later on we're going to move on to the free version the links are going to be in the description below you're going to find here attachments and then you're going to download one of these workflows so you have a video to video we have a video to video with multiple controll net that's a little more advanced we have a text to video we're going to start with that one and then we have something that's called prompt scheduling so we're going to look at that in a bit as well but we're going to start with number three here the basics text to video so download that file then we're going to head over to think diffusion we're going to get started up here we're going to load a comy depending on the speed you want you're going to choose the machine you want here the faster ones are obviously going to be more expensive launch the one you prefer and once that's loaded you're going to see this so this is the paid option but it's going to be super easy to work with because you're not going to need to install anything for the free option later we're going to need to do a little bit of installation once you've dropped the workflow into comy UI you're going to see something like this but don't be afraid it's not as hard as it looks because you're not going to need to disconnect the reconnect connect any of these nodes unless you want to you can just load a workflow like this and press Q prompt and basically get started but we're going to look at some of the settings here so you you know what's going on for the text to video first we have some inputs which is the number of frames there are two sets of frames you're going to be setting for this first the number of frames that's how long your animation will be in this case it's 50 here and then at the end we're going to have frame rate so that's how many frames for each second of Animation usually for film and TV it's about 24 or 25 I usually set this at 12 which is half of um a film frame rate eight the default eight works as well in this example at a frame rate of 12 FPS and a value of 50 frames this going to run for a little over 4 seconds now bear in mind anime diff can only make animations that are 36 frames at most but we'll look at that in a bit we can still make longer animations they're just going to be chained together uh you'll see what I mean within height here that's the size of the animation 768 x 768 is the default here I'm going to just set this down to 512 by 512 so you can get some speeds in the generations for the tutorial if you want more speed set this to 512 the checkpoints here are loaded default you can change this to whatever you want I'm not going to change them for now if you're getting red errors here with when you're loading or queuing your prompt just make sure that you have these models if you don't just click here and press some something else bear in mind you cannot use SD XL models so pick a model that is not named XL then we're moving on to the settings for animate diff and this is where it gets a little tricky you don't need to change anything here so if you if you're not interested in fine-tuning or animation you can just skip a few seconds but it it might be good for you to to know this so the context length here that's how long animate diff will make each segment of Animation remember I said that anime diff at least version two can make animations that are 36 frames at the most but we can make them longer and that is by chaining them together so this is how long each of those chain links is going to be and then the context overlap is how many frames each link is going to overlap with each other so if we have a a 16 here so that's going to be one animation and then I'm going to have four here so you have another chain 16 frames which is overlapping four frames and then it keeps going like that so you can make this longer or shorter and you can make the overlap longer or shorter setting here 16 and four it's a good Baseline but it's it's fun to play with if you're an advanced user now the motion module here is going to be set at um V2 as a default there are other to play with you can play with the older one you can play with temporal LIF if you're watching this tutorial at the later date there there might be a newer model that you can load but this is currently one of the latest ones the motion scale here I'm not going to get into technical terms but it's basically think of it like how much you want your animation to move so if you set this to a higher value it's going to be more wild more animations and if you set this to a lower value it's going to be slower next up we're going to move into the prompt here so at default we have Masterpiece best quality close up a girl on a snowy winter day and we have a negative prompt a bad quality worse quality so here is basically what you want in your animation in the green part and the red part is what you don't want for this model that is preloaded the dark Sushi mix here which is sort of an anime model these Masterpiece best quality is something that people use quite frequently and you know it kind of works it's one of those things that just um tagged long same as Greg rowski back back in the day so we're just going to run this default closeup a girl a snow winter day we have the seed here and this is if you want to keep iterating an animation then you want this fixed because if you run a new generation then you will get the same animation same as with images then you can tweak The Prompt or some other settings to see how it changes if you have this to increment uh or randomize then for each new generation the seed will change and you'll get a new generation bear in mind this seed is for the animation generation this is not around if you set this to random you will not get a new random seed for each of the images so it doesn't really affect quality it's more if you want to change settings and iterate on on your animation moving on to the sampler the default value here is set to 25 steps cfg7 and Oiler a this works you can use this personally I prefer to M caras because that is is a Divergent sampler so if you have an image uh generation that goes from 0 to 25 steps it's going to diverge into the same image while Oiler a and other non- Divergent Samplers can diverge into well non not diverge into the same image you can go wildly into different kinds of images depending on your steps an image can be different at step 10 than it is at step 25 while Divergent Samplers are while blurry step 10 they are moving towards the the you know the same kind of final image you don't need to care about the D noising here uh however in the later Parts if you're doing video to video you can lower this value if you want to resemble your video more but here in text of video it doesn't make any difference we already changed the frame right here so this is the frames per second Loop count is how many Loops you want zero will make an infinite Loop and that's basically what we want you can change your format I usually do 264 or webam if you do a gif it's going to be easy to just copy paste into Discord whatever it's going to out to play however you're not going to retain as many colors in a gfin and I'm going to show you that I'm just going to add a combine here anima if combine I'm actually going to do multiple of these I'm just going to drag that into there and there now usually you want to use the video combine the VHS if you aren't getting a preview image on that you can use the animate diff combine but if you have the latest version of this one you should get a preview image I'm just going to load up a bunch of the same ones to see what we're do doing here so to's say we set this to frame rate 12 we make it a gif let's set this to frame rate 12 we make it in 264 here let's set this to frame rate 24 and make this a 264 place them like this because the preview is going to be below now the ping pong here that means that the image will once the animation once it's finished it's going to reverse and it's going to ping pong back and forth in reverse while otherwise it's just going to repeat from the beginning we're going to set this to true so you can see what's going on there now let's quickly generate this and you can see what happens if you zoom out you can see in the green box that's where comy UI is working most of work is going to be in the K sampler here you're going to see a loading screen in a little bit there we go here's the green loading once that's reached the right part here we're going to be finished I'm just going to skip to the end so you don't have to wait once that's finished you're going to see your frames up in the save image here I changed the promp to middle-age woman instead and here is the result so here we have the two left ones are 12 frames per second while the right one is 24 frames per second so you can see that it's much faster and it's kind of I mean it's smoother but it's a kind of janky animation so you're probably going to need to to get less detail in the image there's a lot of snow and lot of details so you can try to lower for example the motion scale or use a model that's um more flat maybe all your animations will be in the comy UI and then the outputs folder let's try something else to show you uh what different of of the motion scale here so it's going to lower this to5 going to delete this and then we're going to change here wizard weaving fire magic blue flaming hair let's see what that gets us and then we're just going to queue this up again so in this example we lowered the motion scale you can clearly see that it's less going on and you can see much clearly also the chaining between the images the character in actually multiple scenes here and you can see sort of the blend between the chains so if you have a lower motion scale you're going to see much clearly the different chain lengths and um the Blends between them so i' i' prefer to keep this at uh at one or but you know play around with it see what you feel happy with now for the next part here we're going to create something that looks like this and this is video to video and we're going to be using this with our own local install comy UI and if you need to install that I'm going to post a link up here now make sure that you install both comy UI and the comy UI manager once you load up your comy it's going to look like this and we're going to drag and drop our basic vid to vid one control Network flow we're dropping that in now and this is expected because you're going to see a lot of these red errors that means you are missing custom nodes and what you're going to do now is you're going to go into the manager here click install missing custom nodes it's going to give you a list of all the noes you're going to be missing you're probably going to have more than I do just click all of them or the little button up here if you have multiple and press install now this might take some time for you but if you open your command terminal you're going to see the progress once that's done you're going to need to exit this and restart it so now we're restarting our comy and we're going to get a new com UI window and now if you take the same workflow and drop it in you should be able to load the workflow without any errors now if you are getting errors you might need to check what nodes are missing and if you can't install them from the missing custom nodes you could try installing them manually from from here or go to the respective GitHub page now this workflow is a little different but not a lot the main part is this here A different input node so now you have something that's called load images and a load video and if you have a video input which we're going to have we're going to use this woman here from pixels so she's just raising her head and rotating slightly and this is a video so we're going to use this and load it into the load video path here now if you have your video as frames or imag you can use this load images path but we don't need that so we're just going to right click and remove that and we're going to take this one and drag the blue here to the blue there that was previously connected to the load so here you can either show choose the path you want but I'm actually going to remove this one as well not because it's bad but just because if you double click here and right video we're going to have we're going to take VHS load video now I have already couple of previous videos loaded here that I've been testing with but yours is not going to have that so you're going to press here choose file or video to upload and then select your file and again drag the little blue here to the blue there so now you can see the video that you are going to use as an input and you're going to a lot of numbers here and this is probably be set at 00001 the frame load cap here that's how many frames that you will use now this is a it's not a long video but it's a fairly long when it comes to frames so we're just just going to use 50 frames here so we're going to use 50 frames of this video and we're actually going to skip some of the first ones so if you don't want the first part of the animation which would be her head here leaning very backwards we can skip a couple of frames so let's skip 30 40 frames here so we're going to skip the first 40 frames then we're going to run for 50 frames and this means you're going to use every frame if you set this to two you're going to skip every frame to reduce to we want to skip a little more than that so we're going to set this to three make sure that your width and height is similar to the input here load a checkpoint that you would like to use I'm going to select the Epic realism and now we're going to have something new here which are the control net nodes and you can leave this at default for now but I'm going to tell you what they are so first we're loading a control net model which is the line art model here now now if you're completely new you're not going to have this so you can go into the manager install models search for line art for example download that by pressing install you could also get some of the other ones like open pose you see the ones I have I have the depth the canny line art open pose if you want to learn more about control net check one of my control net videos we're going to use line art and we're going to keep this default the strength is how much the control net is going to influence your end result and this goes from zero to uh a lot uh one is usually a good like a good Baseline I'm going to use 0. five which was the set value for this workflow the start percent and the end perent is how long controll is going to run and this is from 0 to one so it's going to run for 100% of the animation and up here in the preview we're actually going to see what's going on so if we quickly start this here we're not really finished with the prompting yet but I'm just going to show you what's going on so first we're getting the frames of the video here and then these frames which are the line art input now you can see these are all squished together so let's go back here and make sure that we set this correct so we have the width at 1,000 and the height at 500 so we got it mixed around so let's set the width at 512 for example and the height at 1024 and then we got a red vae here so just load one of your Vees and if you don't have any you can again go into the manager start models search for VA e e and for example get uh the 84 here now let's quickly run this again and see what happens now we should get these in the correct format and we do so these are the frames we're going to use for the video so we skip the first 40 and then we have 50 frames here and for the line art you can see the pre-process has worked with our input frames has given us line art for these frames the control net is going to use for the end results and now the animation is actually being achieved here now we haven't prompted yet so we're going to be trying to do that it's just running here the default prompt which was Masterpiece best quality a girl with blue hair sunshine and Beach and the negative prompt was bad quality worst quality so just going to look at this result and then we're going to change the prompt to to get something that we want and we're also going to look at the settings here in the K sampler and the video combine box so with a default values here we have an animation that has finished it's fairly okay but it's not really perfect it's kind of it's not very smooth and the colors are all you can see all the lines here from you know that's a typical GIF kind of thing so first off we're going to change the frame right here we're going to add this to 12 12 is good 24 is good as well and we're going to change the format here to we use webm we can use 264 for example we're going to use the 264 one for now if you set the ping pong value here the animation will run will run forward to the end and then reverse back back and keep reversing back and forth so it's going to look a little smoother so let's have a quick look at what this changed for us so as you can see now we have a smoother animation and the colors are not as weird as previously you can also see that it's ping ponging or reversing now we can use oil aray here I prefer to use 2m Caris because Oiler a is not a Divergent sampler to Caris is generally giving me a little more consistency in general I prefer Samplers that are Divergent Divergent means that while you're generating an image from the first step to the last step it's going to keep diverging into the same image Oiler a and other ancestral Samplers are not Divergent and they can go all over the place once an image is generated now if you're not happy with the animation you're getting you can change the seed here now the value here is set at 4444 as a default and it's fixed so whatever changes you make is going to be on the same seed and if you keep generating this you're going to get the same result so if you want a different result with the same settings you're going to need to change the seed here or change this to randomize just know that whatever you choose here this will change the seed for a new generation not for a new image so if you set this to randomize it will not give you a new seed for for every new image uh so the the way you create animate diff animations is really not going to change whether you have this fixed or randomized the quality of the result is not going to be different now we're going to change our prompt here to something else so let's remove all of this we can put in Masterpiece people like that let's try woman in red dress La hair bar background and we're going to generate this again now if you feel that these are taking a little too long to generate you could change the frame load cap here set that for 12 for example just to get a sense of what's going on and once you have something that you like you can see the images and the generation you can increase that value generating your your longer animation but if you're doing that make sure that you have fixed down here so that you're not getting a new generation after you checked your first 12 frames we can now see our generation has finished here so here's our woman in a red dress and blonde hair now if you want this to run even smoother you can change the frame rate uh you can even have multiples here so if you clone this and from that one that goes into that drag into this you can have multiple outputs so let's say we want a 24 frame here maybe you want a gif on the second one we're just going to keep keep h264 for now or maybe let's do a webm for example so you can have multiple outputs in multiple formats multiple frame rates whatever I usually have one that's a gif and one that's an webm or age 264 because the GIF is just easily playable and you can paste that on Discord or whatever just to quickly show someone file sizes tend to be a little bigger though so be aware of be aware of that and why we generate our new one here with two frame rates I'm I'm actually going to show you here uh that if you don't want now you can see from our results we're fairly close to our input image now it's obviously a new generation we have a new dress new hair new background Etc but if you want something different here and you feel that you're too close or too far from your input you can play with these values so the strength if you lower this you're going to be more far away from the input if you raise this you're going to be closer to input and the same with the start percentage and the end percentage I would probably prob play with the strength value and the end percentage so let's say you set this to8 for example then control n's going to run for 80% and then table Fusion is keep going to keep working for the the rest of the 20% without the control net which means without this input but as you're just starting out the default values for this workflow are going to work good for you now we have our two outputs running here we have the left one here at 24 frames per second so it's the same amount of frames but they're just moving at a different pace so the left one here is much quicker it's two times quicker than the right one here depending on what you want that's a value you can play with as well 24 frames per second 25 frames per second is a regular number used in in film and and whatever you see on TV nowadays so that's what your eyes are used to now if you would drag and drop one of the prompt scheduling workflows you're going to see something like this now we drag the text to video with prompt scheduling so if you remember from the start of the video we have the number of frames up here the thing that's going to be D different it's going to be this box down here the green one that says batch prompt schedule and what happens here is that you can set a prompt for each frame in this example here we have for zero we have spring day cherry blossoms we have coming in the 25 summer day vegetation prom 50 fall day leaves blowing in the wind 75 winter during snowstorm ear muths so let's say that we put in man long hair here it would try to do man long hair for the full duration but for these frames is going to try to change from a spring Day to a summer day to a fall day to to Winter Etc so let's try this out and first we're going to press Q prompt here and we're going to get some errors we're going to go check out that what that is so we have two red boxes here these are models that I don't have so just change them to some of the models you have once you press this remember from before one of the ones that you have you had another red one here so this indicates an error I'm just going to change from the input left here to the latent image here and we're going to run this again so now it's not giving us any errors anymore and we can see that our animation has started rendering now our generation has finished here now we didn't get a man we got a woman that kind of turned into a man however you can see that during the 100 frames here that we are moving from Winter from swing with the cherry blossoms we're getting some summer here and we're get getting some some fall out in here so that's the power of prompt scheduling and you can set these however you want now let's say for example that you will want to change this let's say you copy this one here and you want to add to let's say you want to set this to frame 100 this to one frame five and you want this to be Beach summer and you want this to be spring Rocky Mountains and then you press Q prompt and you think everything's going to be great and it will not because you're getting an error and you think well Seb I did what you did what's going on you're right but I need to show you one thing to to get this order now if you look at the previous here that every one of these prompts end with a comma and that's true except for the last one and the last one previously here was the 75 this one does not have a comma so you need to make sure that every row has a comma except the last one so now we have a comma at at the 1 2 3 4 and the fifth row but not at the last row so if you press Q prompt now it's going to work and it's going to generate and this is same for all prompt scheduling whether it's in anime diff or the Forum and this has been the same way well since the start of stabil diffusion over a year ago now if you are running this locally you're probably going to want to install ffmpeg and that is to show the preview combine actually the frames to a video or or a gift so if you're have having problems doing that you should probably install ffmpeg and I'm going to have these links below but this is a guide that's fairly straightforward you can either follow this or follow along with with me here and first you're going to install 7even zip just get it from the link there so go into 7even zip download the 64bit one here install that super easy then we're going to head over to this link here we're going to find release builds we're going to take ffmpeg release full on the 7even zip file here download that once that's finished download you just going to go into your downloads folder and you're going to rightclick the 7even zip file if you are on Windows 11 you need to click show more options if you're on Windows 10 you're going to see this 7even sip we're going to do just extract to FFM it's going to give us a folder we're going to take this folder we're going to rename it FFM Peg and then we're going to either contr X or right click and the little scissors here which is cut then go into your root folder which probably is C and just copy paste that in there so we have a folder that looks like that it's not case sensitive so you can do it however you want then you're going to open command prompt as an administrator and you do that by typing CMD and once you see the command prompt you just going to right click and select run as administrator press yes and you're going to see this window and you're going to take this here set x copy paste that in there and it's going to say success specified value was saved and you are finished I hope you learned something today thanks for watching if you want to learn more about generative Ai and AI check out this video here as always have a good one see you

Channel: Sebastian Kamph

Views: 92,864

Keywords:

Id: iucrcWQ4bnE

Length: 27min 46sec (1666 seconds)

Published: Fri Nov 10 2023