ComfyUI + AnimateDiff + ControlNets Workflow

greetings and welcome to House of dim in this video I'm going to show you my entire workflow for creating animations with comy UI I'll be using Venture resolve to extract frames from video convert those frames into something usable using comy UI then turning that into an animation with control Nets and animate diff there's a few different control Nets I'll be using we'll go over the results of each of those links to my workflows will be in the description let's delve into it all right so to get starter we're going to get the frames that we want out of a video using denture resolve since it takes so long using a animated diff anyway we're going to make sure we're stuck to 24 frames per second and we are we're going to use a vertical resolution hit save and make sure we're on the uh edit Tab and you you should be able to do all this with the free version of di resolve import our media and I found a video of uh um I don't have her name but I'll put in the bottom of someone who does dancing um videos yeah it's going to ask you if you want to change your frame rate cuz I recorded the video just using OBS and and uh we don't want to change it want to keep it 24 frames per second so we got that imported we're going to drop it down here and now we just need to edit it to what we want this isn't this isn't an in-depth tutorial on a adventure resolve but I'll just kind of show you quickly if you need to uh cut to a part just move the the line here control shift left bracket and that'll cut that and then if you need to cut the end control shift right bracket and then to get your picture to get your video centered you'll open up the inspector up here play with the controls here your zooms positions whatnot you'll see you zoom way in or move around and there's there's plenty of denture resolve tutorials on YouTube if you want to uh dig into it more but that for this this all we need so we want to make sure we're we got the video all lined up where we want it something like that zoom in a bit and just play it and see and I think that'll work for what I want to do and if you find a video without text that'll be better I'm going to just do something to kind of hide that but that's you can ignore this part just kind of blur the text out so it doesn't it's not so prominent so once you got the video how you want it you go to deliver down here you'll want to make sure you have a folder set up and I just created a folder I call it open pose pose input set this to JPEG and then br browse your folder that you created mine's called pose input give it a title I'll say uh woman dancing make sure you got vertical resolution checked here it should be since you set it in your project settings add render que and let it render and I think this is the part that would take longer if you're using the free version CU it doesn't use your GPU resources and once you do that your folder will should be filled with a bunch of jpeg now we'll move on to uh converting these two um lines and and depth maths and all that okay we're back in Cy UI um for this one we have two two workflows the first one I use to extract what I need out of the uh out of the frames we just exported from denture resolve um and we're going to create a a death map HD lines open pose and a segments and and you could do this in your regular workflow but it's kind of a I don't know I feel like you're adding a needed process into what what's already going to take a while to do so um I do it separately the only thing you need to change here is your your path to your poses and probably your image loow cap I'm going to do you could try to run the whole thing but depending on how many you did you you may cause some issues so for the purpose of this video I'm just going to do 128 and you can choose a different starting point but we'll just start at the beginning um the other thing here is the paths I have a I'm using save image extended and so I can have a different B folder for each of these so we have the dance depth dance lines dance pose and dance six so we're going to start this and it's going to take a little bit so it'll probably be skipping okay so it's finished all up we have a you'll see previews of everything there's our depth map our lines our pose and our segments and in your comfy UI folder you should have folders each of those so now we'll go into the uh animated diff workflow okay here's the entire workflow here and I'll go over all of this so we'll start over here I'm using dream shaper um I a seed already selected my uh promps we go up here and this is all the control nit information for each one of these you'll need to go through and choose the folder that we had created earlier this first one is is depth so we'll select that folder and hit upload and then you'll just go through each one you got your lines your pose and your SE so right here we have our start frame we're going to start at zero the number of frames we're going to do on this we're just going to do 32 so I can show you a little quicker and then um select every in so this this if you leave this at one it'll it'll render every frame but if you you know you can skip like I have it two so it's going to skip one frame if that makesense and I like to skip one frame on these because it it it just there's more motion the movement is faster but it's depending on what you got you may not want to do that you have your control net models I'll put links to all this stuff in the description for now the only one have turned on is death we're going to turn the others off and we'll do them one at a time but let's go through the rest of this first down here's your batch size this connects to your [Music] frames and uh uh this is your width and height you need to set this is Anime diff I'm using the version three that just came out not very long ago but you can test around with different ones and just go with whatever you like the most we have the motion lure in here you can do you can chain these together so you can do pan left then zoom in and you can chain more so if if you turn these off and you don't apply any control net this will have a stronger effect right now it doesn't really again it's just one of those things depending on what you have you might want to use it you might not you might not see much of an effect it if you render if you render with this off with all your control Nets off completely you will have an effect and it'll be more noticeable and then you can control the strength here again I'm just going to provide this you guys can toy with it all you want and the other part here that I that I think's important to get get consistency is the uh IP adapter so what you want to do is you want to go render out something you like so I have another workflow I use the same prompt and got this photo and I'm popping it in here and what this does is it every frame that it renders it's going to look at this and give it um you know try to try to try to keep with that theme so I think it's important but you can disable it contrl B if you want to do it without the next step is going to go into the first case sampler and then it's going to render out our draft I call draft so this is the rough draft it's going to upscale by two and uh run it through a case sampler again got my favorite model here my wife we're going to do a reactor face swap we're going to add frames back because I want it smooth at the end you see it's it's choppy here and nice and smooth here and you got to do a little bit of math so we started with 32 frames and we're we're doing uh eight frames a second so that's going to be four uh seconds total and then but I want the end result to be 24 frames per second so we multiply that by three and we get four frames at 24 frames per second and you'll just have to do some math but if you do multiples of eight um on your beginning that should be fine it'll always match up but if you change that you'll need to do the math yourself now let's look through uh each step so here's our rough draft no app sampling um it's soft you know depending on what you're doing that may be fine I want more detail so we upscale it and uh definitely more detail there it changes the the results a little bit but it's not too bad and then the final smooth you know 24 frames per second with the face swap so I'm going to go through and do each one of these I'll turn one off and do the next I'll render them out one at a time and we'll put them side by side and compare and then I'll also do uh all of them together and see what that looks like okay so this is results of all the different control Nets I have up we got the depth Line segs open pose all results kind of speak for themselves I wouldn't worry about open pose too much I know the the way I used it um it didn't know where all the the hands are at there and open pose is probably really great if you use like a 3D program or something like that but um in my in this case it just didn't know what was going on this is everything combined looks good but it's a little soft I think my preference is the HD lines but you know every base preference is different that that's one of the things I like about comfy UI you can kind of do you can toy with stuff like this and try different things so that's it that's my process for creating animations with comfy UI and if you have any issues with workflows or just questions in general you know post them down below I will try to help you I may not get to you right away but I'll I'll get to you as quick as I can again links to everything will be in the description thank you for watching

Channel: House of Dim

Views: 13,663

Keywords: comfyui, stable diffusion, ai art, tutorial, face swap, face replacement, animatediff, controlnet, controlnets, animation

Id: We3UmWWmlQE

Length: 11min 38sec (698 seconds)

Published: Sun Dec 31 2023