AnimateDiff Tutorial: Turn Videos to A.I Animation | IPAdapter x ComfyUI

the quality and consistency of AI animations have improved significantly in the past 2 years in this video I will show you the easiest way to get your tools ready and share with you the settings to transform your videos into anything you can imagine these AI animation methods are only going to get better so make sure you subscribe to the channel to stay ahead with new tools and learn how to use them to get started you will need to install comy UI I will leave a link to the complete guide in the description below there you need to scroll down and find the direct link to download right click on it and choose save link as you can save it anywhere on your computer I'm going to choose desktop then extract the archive open the extracted folder and open confy UI inside navigate to custom nodes and here you will also need to install the comfy UI manager to do that select the folder path type CMD and hit enter to open the command prompt window then paste this command with the link to the comy UI manager you can copy this directly from the description box and hit enter once it's done you will find that the comy UI manager has been installed under custom nodes go back to the main folder and run this file which will launch comy UI on your browser now if you already had confy UI installed before watching this video open the manager and click on update all to make sure you're running the latest version to get started with video animation work head over to this guide on civit AI I want to give a shout out to the Creator in the reflection for putting this together along with several other guides so feel free to check them out open the attachment Tab and download the file that says IP adapter batch and fold this is a Json file that you can drag and drop onto your comi interface to load the base workflow you will most likely get an error that looks like this and that's because this workflow is using some nodes that are not installed on your computer to fix that open the comi manager click on install missing custom nodes here you will have a list of extensions that need to be downloaded go ahead and install all of them one by one and once you're done click on restart to relaunch comi and now you should be good to go we have a few more essential files to download starting with the main AI model that will Define the style of your output and I will link some of my favorite models in the description box in this video I'm going with protovision XL right click on the download button choose save link as go to confy UI folder open models checkpoints and save the file over there the second file you need to download is this sdxl vae module right click on the little download icon choose save link as and this time go to the vae folder and save the model over there next file you need to download is the IP adapter plus model there are a few to choose from so make sure you pick the one that says IP adapter plus sdxl v right click on it save link as and this time go back to the comi folder open custom nodes open the IP adapter node go to modules and save the file you will also need to download this image encoder simply look for this file right click here save link as go to comy UI and open the models folder open clip visions and because the name is to generic I'm going to change it to image encoder and hit save you will also need to download this control net model here you have two versions of the same model to choose from go ahead and download both and I will explain the difference between the two later in this video make sure you save this under models control net and do the same EX exact thing for the other version as well last but not least you will need to download this hot shot motion model this two has two different versions to choose from so make sure you download both and place them under comy UI custom nodes open the animate diff node models and save them over there now let's go back to the comy UI interface and start working on the settings first thing you need to do here is load the video file that you want to transform I have a few clips that I shortlisted and I'm going to try and transform this video right here to do so hold shift and right click on the video click on copy as path and paste it over in this input make sure you delete the quotes then hit okay here you can increase the select every nth frame setting to tell the AI how many frames to process so so for example if you set this to two the AI will only process every other frame this is useful if you want to cut down the processing time you can later use another AI tool such as video AI to interpolate the video and make it smoother I'm going to stick to one here because I want to process every single frame here you can simply choose the dimensions of your output this can be different than your original video but make sure you stick to the same aspect ratio to cut the processing time down I'm going to set the output to 720x 1280 right below you can choose to upscale the processed animation to a higher resolution which will drastically improve the quality and is supposed to run a little faster than running the animation at a high resolution from the beginning so I'm going to upscale mine to 1080p below that use this node to select the AI model that you want to use to stylize if you've downloaded the model but still can't see it on the drop down menu make sure you click on refresh and it should appear on the list I'm going to select the protovision XL model right below you can load the sdxl vae model now let's move down here to load the IP adapter model and the image encoder right below it right next to that we have another IP adapter node which contains some of the most important settings in this process and the correct values will highly depend on your input video I've managed to get really good results by only changing the weight and the noise you can start with the weight set to 0.2 and the noise set to 0.3 changing these two will have a significant effect on your output so I highly recommend that you play around with the values next let's move over to the control net nodes here make sure you load the control net model and to my understanding choosing the fp16 version will give you less Precision but it will result in running a little faster fter since I want the best quality possible I'm going to go with the normal version but if any of you guys has more information on the difference here please let me know in the comments below the control net strength will Define how closely the animation should follow the original structure of your video and I usually set this to one down here in the anime diff node make sure you load the Hot Shot motion model and now let's move on to another very important node in this process and that is the K sampler let's change the seat control to randomize increasing the steps will usually result in better quality outputs let's set it to 30 the CFG value will determine how closely the output should follow your prompt the lower it is the more creative it will be and a value of eight is usually a good start for the sample I like to use DPM P 3M SD GPU let's change the scheduler to caras I found that the start at step setting has a significant impact on the transformation level I usually set it somewhere between two and 15 six is a safe value to start with and the higher this is the less transformation you will have but of course this works in combination with all the other settings that we talked about so keep that in mind down below we have what is probably the most important input in this whole workflow and that is prompting you have two boxes the green one is meant for you to input the positive prompts you can use this to describe the final output that you want to see I tend to describe the subject and I like to be precise with the clothing and environment to help achieve better consistency I have another video on how to find the perfect prompts and use them in different AI tools so feel free to check it out the two input boxes are supposed to serve different purposes but I found that using the same prompt in both boxes is a safe option to go with right below that you can input some negative prompts describe the things or styles that you don't want to see in your video and I will paste the prompts that I'm using here in the description box for you to use moving on to the video combine node here you can set up your export settings I usually match the frame rate to that of my original video which in this case is 25 on the far right of our workflow we have the upscaling nodes here you don't have to change much you can keep the upscaling case sampler settings to default and at the end of that there's another video comp by node here again you can match the same frame rate as your original video you can also customize the way your output videos are named or change the final video format once you get all your inputs and settings done you can go ahead and click on Q prompt to start processing you will see that comy UI is going through the nodes one by one the K sampler node here will be taking the heaviest load of the whole process once that's done you will have a preview in the video combined node this is the 720p version prior to upscaling and it already looks pretty good comi will move on to upscaling the video right after that and you will still be able to see the progress here as well once that's done the final video combined node will display a preview of your upscaled output I'm honestly very happy with the results here it's everything I've imagined the video to look like and chances are if you're running this for the first time you will not get the desired output and this is where experimenting comes in I highly recommend that you go back tweak settings and execute multiple times until you get a really good output to access the generated animations go to comy UI look for the output folder here you will find the final upscaled videos as well as the individual frames and pre- upscaled outputs stored in different folders the great thing about this is that you can take whichever output drag it over and drop it onto the confi interface to load the exact same setting used in that video if you want to access more examples to play with I've included multiple animation exports along with their workflows on my patreon page so if you're subscribed feel free to grab them other than that stay creative and I'll see you guys in the next video [Music] peace

Channel: MDMZ

Views: 86,935

Keywords: ai video, ai video animation, ai video generator, animatediff comfyui, animatediff tutorial, hotshot xl tutorial, ip adapter, stable diffusion ai, stable diffusion tutorial, stylize video with ai, stylize video with stable diffusion, transform video to animation, transform video to animation ai, transform video with ai, video to animation tutorial, video to video ai, warpfusion tutorial

Id: AugFKDGyVuw

Length: 11min 25sec (685 seconds)

Published: Thu Jan 25 2024