ComfyUI + Prompt Travel + AnimateDiff Workflow

greetings and welcome back to House of dim in this workflow I'm going to go over prompt travel with animate diff I'll showcase a few methods for utilizing it and try my best to explain how it all works let's dig into it okay here's the entire workflow we'll go over all of it and we'll start over here right here we have the number of frames we're going to do we're going to do 48 checkpoint I'm using a stream shaper uh we're going to ignore this for now we'll go back over that here in just a minute over here we have the seed we have the batch promp schedule node and the negative prompt so let's look into this this is the most important for prompt travel this text right here is is the pretext it's going to go in front of our scheduled prompts up here and I have portrait one girl blonde hair bangs blue eyes wearing business suit this is the post text it's going to go after the scheduled prompts so if you look here we have a 0632 so at frame zero we're adding sad eyes closed frame 16 we're going grining with teeth eyes open frame 32 we're going to switch to Blue here and I've already I've already ran this so we can look at the results but let's go through this some more first up here I have a control net if you want to use a control net and then we're going to use it here in just a second but for this first run I wanted to keep it simple and just turned it off go down here here's our empty latent image and uh this is your this is where You' set your with and height and here we have anime diff and I'm noticing my testing you want to go with the less stabilized model for animate diff so I'm using a stabilized mid if you go with something higher it's less prone to change on you like you want it to it because I think that's the point of the model is it's trying to keep consistency and in this case we're not worried about that as much and down here I have the motion lers you can use them if you want you can zoom in pan left whatever you want and you can chain these together but for this tutorial I'm not using them continuing on to the right here we have our first case sampler that we go through and and then we upscale it and we send it to a secondary upscaler to make it bigger then down here we go to um face swap if you want to use it if not contrl B I have my wife in there of course and the end here we add frames to make it smooth again so let's look at this closely we'll just look at the end here so at frame zero it sad closed eyes and that you see it for just a brief second cuz it immediately starts going into the grinning with teeth and eyes open right there finally we switch to the blue hair and you that's that's a pretty obvious switch you can see that so that is a basic basic way to use prom travel and you can expand this out as much as you want you know you can go I'm I'm just doing 48 frame so the 30 second frame we turn to blue hair and we keep that for the remainder but you can go you know just play with it do as much as you want so now let's let's move on to kind of a more advanced thing you can do and we'll go over what all this crap means down here okay onto the more advanced method you'll see I got quite a bit more added here before I go into it you want to go up and turn on your control net and I'm using open pose and this is my pose here and I have another tutorial on how I get this rendered out but the the quick version of it is I I used eventually resolve to to crop and and zoom in on the frame I like and and render those out as jpegs then I bring those jpegs into comy UI to be processed so back down to the start we got the batch promp schedule node right here and the pretext we have one girl bangs and the the post text we have sunlight daytime The Prompt text has been converted into an input and so has the pwa and that's just by right clicking and and convert to input we go over here and here's our travel prompt I'm can zoom in on it for the travel prompt we got frame zero and as medieval armor walking in a snowy field and then we got this bit here blue hair back tick pwor a back tick and that that applies down here we'll look at that in a second and then at frame 14 we got the same thing and the reason is so as this works it works you know so between zero between frame Z and 14 it's going to uh start working its way toward the next prompt so I want this transition from medieval armor to Blue business suit to be quick so notice 14 and 16 so in two frames it's going to work its way from Med medieval armor walking a snowy field to the blue business suit walking down rainy Street if you want that to be a more gradual transition you can just remove this whole line here and it will transition from here to here uh you know slowly it's more gradual transition and then I do the same thing here we we keep the blue business suit on frame 30 and then we quickly transition position from 32 to from 30 to 32 to the black latex cats suit and Forest and mountains so then let's look at this part here the pwa now there's there's four four channels for this there's pwb and C and D so you can do multiple different versions of this but we're just going to stick with the one this is the pwa right here you'll see it's connecting to here and so for each frame for each frame we're saying how much strength we want that channel a to have and you'll see from zero to [Music] 13 I keep it at zero just to show you so even though I have this prompt in here blue hair it's not going to render blue hair cuz there's no strength to it then at 14 which is the same time we we put in red hair I got it at one and it'll go up a little bit and I just messed around with it fluctuate a little bit and that should continue you know I got a bunch of numbers in there but that'll continue till we get to the purple hair and then it'll switch to purple hair and it'll stay it'll stay there so you might be like I don't want to to type all this out well you don't have to the developer provided a link to this a key frame string generator you can skip this first part here we're not going to use that and I think the best way to use this you can go out and you can plot out each of those points I was just showing you but I think for me the best way is you know I just kind of try to put it at zero however many frames I got you can see that number down here and this is your strength on the left here how many frames you got just go to that keep it at zero try as close as you can Mark those out and then if you scroll down you'll have your whole string here and it should be all pretty close I didn't get it very close on that case but you get the idea so you can copy this plug in comy UI and mess with it you know whatever you're doing so I'm going to cue this out and we'll we'll pull it up and look at it frame by frame kind of see what's going on but one other thing if you hit print output to true and you open up your console you can see what's going on as it goes through it you'll see this is frame index zero the current prompt the next prompt and then the strength that's going on here so the strength zero and it'll slowly switch this is it transitioning to the next prompt and if we keep going down you'll see the blue hair the red hair the blue hair the red hair blue you know yada yada yada and then when it gets to frame 14 is when my strength comes in on the red hair it's it's 1.0 there but yeah it it could be handy if you're trying to figure out what's going on and why things are looking the way they are so I got it on here I'm fixing playback so we can see what's going on down here I have the current prompt we on and up here is the current frame we on so I'm going to hit play and Let It Go real slow around frame 14 we should switch to red hair and right after that we'll switch to the business suit there we go and as it goes on fix and switch to purple hair and then the cat suit around 30 there we go and that's it does exactly what we want and then it'll just finish since I didn't do anything after that it'll stay that way until the remainder of the video so that's my workflow for prompt travel with anime diff as usual the links to everything and the workflow will be down in the description hope you found this helpful if you have any issues please post them and I will try to help thank you again hope you have a great day

Channel: House of Dim

Views: 4,238

Keywords: comfyui, stable diffusion, ai art, tutorial, face swap, face replacement, animatediff, controlnet, controlnets, animation, prompt travel, prompt scheduling

Id: ykJ4_cuR_0E

Length: 8min 30sec (510 seconds)

Published: Sun Jan 14 2024