Make AMAZING AI Animation with AnimateLCM! // Civitai Vid2Vid Tutorial

[Music] what's up guys Tyler here from cai.com and today we are going to walk you through an animate LCM videot toide workflow for animate diff and comfy UI now this workflow will allow you to take an existing video and do a complete style transfer to it using either just prompting or reference images from the IP adapter and it has control Nets it has a high res fix so you can get good upscaling out of it as well as a face swapper if you want to keep your subject's face or put someone else's face on your subject from an image now this workflow does require at least 10 GB of vram if you have less than that use at your own risk and you might have to find a few workarounds without any further Ado let's hop into a computer I'm going to walk you guys through it and a special shout out to sir Spence and PES flows two of our community members who help me with a few of the finer points of getting this thing hooked up I'm going to put their Instagram handles right here please be sure to check them out on Instagram let's hop into the computer all right guys so we are in the workflow right now a little housekeeping first this tutorial is assuming that you already have an instance of comfy UI installed on your computer this is not going to be a how-to to install comfy UI this is not going to be a howto for the basis for this is not going to be a how-to for the basics of animate diff however I will leave some links in the description down below to some videos and articles from cai.com and from some other really great YouTube creators that will teach you everything you need to know about the basics of animate diff animate LCM and how to install comfy UI so if you are brand new to this check the links go watch and read those first and then come back here but right now let's hop into this animate LCM workflow and let's talk about it and I'm going to walk you guys through how it's built and how to use it and by the end of this you should be able to make some dope video to videos for social media or for whatever so let's get into it now this is what the entire workflow looks like it is not very big at all and I tried to build this out as simply as possible while I was putting it together and a couple of my buddies were helping me with some of those finer details shout out sir Spence and Shout out patches flows go check them out they make great stuff on social media and they're just super talented in general now the workflow starts here as you can see all of these different groups that's what we call these colored boxes here these different groups are colorcoded so that the separation is easy and they are numbered and labeled so you know what each group is and what order you should be looking at everything in so as long as you just work your way down the chain and boom boom boom boom boom you should be good to go and it should be super easy so let's start right here with group one which is your video Source SL your resolution so the first thing we want to do here is load our video so all you do is click choose video to upload and it's going to bring up your browser this is where you pick the video you're going to be working on now the only numbers you need to really worry about here is your frame load cap skip first frames and select every in so let's talk about what each one of those do frame load cap means how many frames of this video do you want animate diff to render if I put 50 in here if I just punch in 50 it's only going to render 50 frames of the video skip first frames is how many frames do you want animate diff to skip before it renders so if I was to put 100 in here and hit Q prompt well it's going to render only 50 frames but it's going to render those 50 frames from frame 101 so it's going to skip the first 100 frames and render the next 50 but we keep that at zero best practice that is great if you're working on a longer video and before you run the whole thing you want to test specific parts of the video so you can have a better idea of how the whole video will come out either for consistency or for your control net settings let me put this light on feel like see boom now we're a little bit more Moody okay now select every in frame means how many frames do you want it to render at a time so if you have one that means that anime diff is going to render every single frame of the video this will give you the most movement accurate and smoothest video output if you put two then anime diff is going to render every second frame that means it's going to skip a frame as that number gets bigger you're going to get more of like a choppy kind of like broken apart animation I always keep mine at one do as you will with that if these numbers are at zero frame load cap that means it's going to render the whole video skip first frames zero that means it's not going to skip any frames it's going to render the video from beginning to end so that is your video upload node right above that we have your width and your height nodes this is for your base resolution I typically render all of my videos in vertical format because so social media everything is vertical and I do that at a resolution of 512 by 896 this is my this is my low resolution 9x6 aspect ratio so make those numbers whatever you want if you're doing a 16x9 wide video then you could just flop these it would be 8.96 in the width 512 by the height and then the upscale image node you don't really have to worry about this you just don't have to touch it so at the bottom of the load of the source video you have your Laura stacker I have this set to have five luras available for you so you would just have to click right here and this should be pulling from your Laura folder that you have linked to your comfy directory so all my lauras are in here you have you can select theur and then you have the model strength and the clip strength I find that if you leave the clip strength at zero and you just adjust the model strength that you will get similar results to what you would get with aura as if you were just using it in like automatic 1111 and generating an image so I like to keep my things as simple as as possible and keep as little variables as possible so I can get to actually creating videos and making art so do what you want with that that is your Laura stacker and that concludes group one so just to recap you have your Source video you have your resolution and your aspect ratio and you have your Laura and a matter of fact let me actually change the title right there and also put [Music] in Laura boom now group two your model SL animate diff loader so this is where your animate diff motion model loader is and this is also where you choose what model or checkpoint you want to use for your render as well as your V okay as well as your V but I'm pretty sure we all use this 840,000 vay maybe use anime one if like it's something super specific to you but almost everyone I've interacted with uses the 840,000 Bay now something important to note because this is an animate LCM workflow this is not a normal animate diff V2 V3 workflow you need to download and have the animate diff LCM motion model that will be in a link in the description in the article that inner Reflections AI wrote shout out to Inner Reflections AI all of her resources if you need to learn how to use animate diff and do video to video with it she makes the best articles that she posts on citi.com link to her profile and her articles will be in the description below if you have any questions or you just need basic knowledge about this go read her article she is a valuable contributor to the community with everything regarding anime diff video to video and just workflows in general go check out her profile links in the description and leave her a nice comment because she does a lot so this needs to be the animate LCM model mine when I downloaded it from um the link in her article was called sd15 t2v beta I just didn't change the name but that's what that is your beta schedule needs to be LCM square root linear okay so boom motion scale should stay at one you have one animate diff motion Laura if you would like to use it I typically don't use them on video to video outputs but it's there if you wanted all you would have to do is change your strength up to one and choose what animate if motion Laura you want I'm going to change that back to zero cuz that's not for me and in Your Efficient loader here this is where you choose your model and your V now something important to note here I am using a photon LCM specific model that was trained by civi Community member machine delusions AKA Phil shout out to him he is another Community member that provides a lot of great resources link to this model will be in the description of the video below if you use the photon LCM model you do not need to use the LCM Laura because there is an LCM Laura that again I did not rename just yet but this will be linked in the description below as well if you are not using the photon LCM model you need to have the LCM Laura loaded in here and you need to play with your string but I would recommend starting at 1 and one and working your way down this does add a lot of contrast and can overb burn or oversaturate the images so you're going to have to play with these based on your video and figure out what works best for you but if you use the photon LCM model which provides amazing results you do not have to use that LCM Laura and it will definitely oversaturate and overburn your image that is box two your your choosing your model your v as well as your animate diff options but most of that part should be set for you you just need to make sure you download your animate LCM motion model and your animate LCM Laura then underneath that we have group three which are all of these Yellow Boxes which are our control Nets now I have five control Nets in here because personally I never use canny for anything and it's I just don't so if you want to add canny feel free but we have line art soft Edge depth and open pose as well as one for QR code monster slash control GI now of course to use any of these control Nets you need to have the control net models downloaded and installed into your control net folder folder I typically use two control Nets at a time unless I'm doing one of the QR code optical illusion videos in which case I'll usually only have this one turned on now in all of these control net boxes um we have these fast bypassers and you might have if you don't have the RG3 node pack you're going to have to install that in your comfy UI manager and all you have to do to enable and disable these things is click the enable button Bo boom when they're pink the group is disabled when it's when you can see the colors they're enabled so as if you want to use soft Edge and line art boom enable disable right now we like using open pose and depth so we're going to leave that on if we wanted to just do QR code Monster boom and then we pick QR code monster if we want to do our control gift then we select our control gift but again we're going to leave on open pose and depth now that's group three control Nets and again control net values you're going to have to mess with that I can't tell you what the best values for your video is that is all up to you okay so group four your IP image adapter now for those of you that don't know the IP adapter is essentially reference images that you feed it and it will try to build the animation in the video off of the images that you put in your IP adapter all you got to do is either drag an image from your browser into one of these boxes or you click choose file to upload and you select an image from anywhere on your computer now you can have either four separate images you could have two images for your character and two images for your background or you could just have four of the same image which I have here now these nodes that they're connected to the prepare image for clip Vision you have a crop position selector which will allow you to tell the IP adapter what part of that particular photo you want it to crop into and focus on to nail the style I in general like to just keep it on pad that way it takes the style from the entire image but if you're trying to do something very specific play play around with the cropping and see what kind of interesting results you can get the weights I typically leave at 1 and your weight up here for your apply IP adapter from encoded this is the weight you're going to want to mess with um the LCM workflow tends to be able to push the IP adapter much harder so you can make the weight from anywhere from like 7 up to like 1.15 or 1.2 and you can even add a little bit of noise into it but again these are all things that you just got to play with so that you can figure out what fits your style and then of course for the load clip vision and the IP adapter model you're going to need the IP adapter plus stable diffusion 1.5 bin file you can download this from the IP adapter GitHub and you are going to need the sd15 pytorch model. bin that goes into your clip Vision folder in comfy this file will be linked um in a Google Drive link that I will put in the description below so everything should be there and you should be able to just download everything that you need um then you also have your IP adapter crop previews this is going to give you a preview of exactly what stable diffusion is going to look at as it's referencing your photo if you crop into the top the center the left the right um you'll see the crop and what it actually looks like in here uh this little control that is a spelling error see that's why we do these things this control net stack slf freu this is just where the control Nets are hooked up into and then I personally like the freu node I feel like it helps smooth out the anim a little and make it more consistent so that we just don't touch then box number five we have your prompt green box positive prompt and you can prompt travel this if you like there is a certain Syntax for that um we will go over that right now so let's say at frame 50 I wanted the prop to change to to let's just say a cat well the syntax here for your prompt quotations the frame number all videos start at frame zero quotations semicolon space quotations your prompt with all the different tokens separated by a comma quotations if you're only using one prompt you end it at the quotations and you do not put a comma at the end of it if you put a comma at the end of it you are going to get an error and it will not run if you are prompt traveling if at frame 50 50 we want to switch to a cat well then we put 50 in the quotations and a cat you need to have a comma there if this is the last prompt no comment at the end so just remember that rewind that little part as many times as you need to watch it to understand what what I just said and like let that sink in a there have been many of times where I've gotten stupid errors for stupid little spelling or syntax mistakes in my batch prompt scheduler that have taken me longer than I would like to admit to figure out and find exactly what the issue was so batch prompt scheduler this is your positive prompt the pretext anything written in the pretext prompt box will come before everything written in your batch prompt scheduler everything in your apt text prompt box will come after everything in your batch prompt scheduler that means if you have a a 10 different prompt prompt travel whatever is in the pretext and the app text text will come before and after each one of those prompts so they're all getting these same kind of um descriptors okay everything down here don't worry about it and then the red box of course is your negative prompt if you want to use any embeddings you need to use this Sy syntax embeddings or textual inversions it's embedding semicolon and then the file name all the tokens separated by commas and then all the stuff that you do not want to see in your render and hopefully it doesn't end up there so that is number five that is your positive and negative prompt now number six your K sampler and your highres fix AKA your upscaler so the case sampler this is something to note because we are doing this based off of animate LCM the LCM workflow requires very low steps and very low CFG you're going to need to play with these numbers and find the right mixture for your video however a good place to start is a CFG of between 1 to two with steps of 8 to 12 the D noise you're going to want to leave at one your sampler needs to be on the LCM and your scheduler should be this sgm uniform for the best results with the animate LCM at least from what I have seen so far um and then of course you have your seed you can either randomize it or you can um stick to the stable seed I personally always like staying on a stable seed because it just makes it easy to iterate and really get the animation to what I want it to be that is your case sampler and then we have your highres fix script so this is the upscaler um what I have it set to and what will be uploaded with it the settings that are in it are the settings that have given me the best upscale results so far so I'm rendering my low reses video at 512 by 896 after it goes through my highres fix I get a highres video at 768 by 1344 and then I can take that and upscale it again in automatic in a batch image to image or run it through another upscaler and comfy to get it to a full 1080 resolution so I would recommend not messing with that but again all of this is open to be iterated on and hopefully you can make it better than it currently is and then you can share it back with me that's why I'm putting this out there then number seven we have the reactor face swapper everything that is in here are the settings I would recommend um all you got to do is drag and upload an image of the subject face that you want to use in here now word of caution some people are having a very difficult time installing the reactor node if it is giving you any trouble I would recommend probably just erasing this group and just ditching it it might not be worth the time I didn't have well I had to like try to install it and uninstall it and install it maybe like five times before it actually installed and it's been working ever since but a lot of people seem to have a lot of issues with it if you happen to be one of them I would say just NX it and forget about the face swapper but it's there again all you have to do upload an image of your subject's face right there and I wouldn't touch any of these and all of the um face models should be in there if you have these nodes installed and then we have our video combine so this is where you get your output I have this set to a frame rate of 30 and yeah so I mean this is what the output of this video was so we had our dancer girl right here we had our four reference imag of this kind of like 80s looking sci-fi athlete girl and this was using open pose and depth and I think that looks pretty good and it Loops perfectly thanks to the magic of anime diff and then as I'm trying to get a project going and iterating what I will do here and what this box for number eight the preview Gallery I will control and drag my video combine out right here and just copy it over into here that way as I am testing things and reiterating I can compare all my versions of the video so that is the animate LCM vidto vid workflow this is going to be available to be downloaded on my cai.com profile the that link of course will be in the description and I really hope that you guys can make some cool stuff with this and when you make cool stuff with this please tag at hello Civ on social so we can share your videos and yeah we will see you guys in the twitch stream thank you so much for watching I'm Tyler this is cai.com follow us at hello civi on all social media platforms and we're out go make something cool cool peace

Channel: Civitai

Views: 21,616

Keywords: Civitai.com, civitai, animatediff, animatediff tutorial, animatelcm, animatelcm tutorial, ai animation tutorial

Id: 6P5225s83wY

Length: 25min 59sec (1559 seconds)

Published: Tue Feb 20 2024