ComfyUI AnimateDiff Prompt Travel: ControlNets and Video to Video!!!

so this is going to be a tutorial focused on inner Reflections workflow which is a vidto vid workflow that's powered by control net this is going to be awesome I really do hope that you can follow along because I'm telling you if you get this up and running you're going to be doing some awesome stuff I have my resources folder here as usual with my tutorials and I have two sections a video folder that has my input video and a workflow file this workflow file again is by Inner Reflections but we are going to be making some tweaks to it now we're missing some items here so we're going to make sure that we install them that's going to be right here so as usual we click our manager install missing nodes and it's going to give us our two sections here so we install we install and to apply we make sure we restart comfy UI so to restart we're just going to go inside of our command line we're going to contrl C and you see it says stopped server and we're just going to run it again by doing python main.py no there's a lot of things that it's going to have to download so just give it its time and be patient all right so once all that is installed we're just going to refresh our webp page and you'll note that we are no longer missing our nodes so now this is a much larger node Network than what we're casually used to with the tutorial series but just follow along everything's pretty straightforward from what we doing before the only difference for the most part is this added in control net section now this control net section works pretty simple so you have the actual control net that uh section that's applying the control net this is where we load our model into the control net which in this case is the line art model we're not going to be manipulating it we're just going to be getting our feet wet and we have right here a pre-processor the pre-processor is what actually changes the in put images into the the format that they need to be to make our control net work properly so right now we have a realistic line art pre-processor again we're not going to mess around with things too much we'll do that in other tutorials here we have an upscaler that changes the size of our image to whatever the size width and height we have in this section but we're going to have to change a node so I want you guys to follow along closely and don't get too nervous it's pretty simple here we have a load images note I imagine a lot of you are going to get caught up on how do you turn a video into frames so I wanted to show you guys that there's actually a load video note as you see there so VHS load video and this will allow us to do the same thing this node is doing but it's going to automatically convert our video to frames so I'm going to delete this and I'm going to drag the image into my image uh input here on the upscale image now all we have to do is choose our file to load which I'm going to click on my input video there's another thing that I would like to change and that's because if we look at my video I am in fact a male so because of that I want to change our pretext our pretext is our head prompt so in other words if the pretext says Masterpiece best quality it's going to be Masterpiece best quality spring day cherry blossoms then it's going to be Masterpiece best quality summer day vegetation etc etc but it's kind of hard to tell what's going on here as you can see it's getting cut off so what we can do is we can copy all of it cut it out and I'm going to rightclick on pretext or rather just in the node in general and I'm going to scroll down to where it says convert pretext to input I'm going to click that and notice that now we have pretext up here and the section is gone I'm going to drag off of the pretext and I have ttn text here but we could also search text and as you see it pops up there so I'm going to click and now we have a text box on the side where we can put our pretext in it's a lot more clear it makes things more visually appealing to when you're working on it trust me so the reason why I wanted to get into this is because I am not a girl I am a guy so we want to fix that so we don't get too much fluctuations in our animation too much flickering and whatnot so I'm not changing the prompt I'm literally doing a hello world in the programming World which is just taking ground troof running it ensuring that things are working properly I'm going to use hello young for my model so that's the low checkpoint note as you know from the other tutorials and I'm just going to switch my vae to berries mix because I don't have the other one so I'm also going to have to change my anime diff to from temporal one to the model that I have these are just things that you want to get in order just in case you have any errors just remember switch your models because you're loading somebody else's file so now the last thing we're going to need to do to get things up and running is we're going to have to download the associated control net now I recommend downloading all of them but if you're limited on storage space or you know that you just want to use this one download what you need you know so we're going to head to this page it's a hugging face space I believe you'll need to create an account so that you can actually download the models but I'm not sure I'm not for sure on that um the ones that we'll be using or the one that we'll be using as you can see here is the line art. pth so line art. pth we want to click that this little button here that lets us download the file and we want to put that in comfy UI in models inside control net so we're going to save and we also want to download the Yo file associated with the actual control net that we're going to use in this case the line art yo and same exact directory so we'll just give that a minute to download and then we'll get up and running all right so our model finished downloading as usual we want to refresh our page so that comy UI becomes aware of whatever models we put in so we see that now it pops up there the next thing we want to do is as you see we are generating 120 frames our video may be longer than that so it would be a good idea to frame load cap to set that to 120 because that's telling it hey don't load more than 120 frames note you can skip frames as in the first frames in the video as well as select you know every second frame every third frame Etc but we're just going to keep it everything else on the base uh settings so I'm going to click run notice our uh frames have been extracted here from our video and then the control net it's passed to the control net and inside the control Nets previews we can see that it has turned the frames of the video into a line RS kind of style now it's going to use that to drive the animation output so we're going to give it its time it is going to take quite some time so that's something to note in terms of vram with the preset settings it should be around 12 GB so as you see I'm I'm capping out at 11.9 if that's too much you can always lower the amount of re vram usage by lowering your width and height here but you're still probably going to read Around 8 to 12 GB I imagine so yeah we'll just let that run and see what we get so let's take a look at our output here I think this is awesome aome this isn't even a MP4 or webm this is a gif and it still looks this cool the artifacts are from the fact that it's in gift but the cool thing about comfy UI is that we haven't changed anything and it was a fixed uh seed for the generation so if we switch this to that MP4 it's going to just generate from here it's not going to do everything else again so as you can see we are minimizing the arst facts in the video no longer have those weird striations in it and I think this is pretty impressive uh it's most definitely following my face nicely the eyes I'm surprised it got the the shape over my eyelid so uh precisely before I end the video just some quick parameters you want to stay focused on in terms of experimentation obviously your steps your CFG your sampler and your um your Primitives but also here your strength for your control net it's a big one right there so the strength for your control net the input size for your control net frames and I would say that's about it so that is a very fast run through on inner Reflections work regarding control net vid to vid and animate diff I literally just did a hello world so this thing is capable of way more this is me barely even testing it out just getting it up and running just because I wanted to show everybody how to do it stay experimenting please if possible follow inner Reflections on Twitter and YouTube show them some love as well as if possible please tag me and I imagine them in any Creations that you make we love seeing the work that the community is making and you know stay creative blessings guys

Channel: c0nsumption

Views: 16,796

Keywords: ComfyUI, AnimateDiff, Prompt Travel, VFX tricks, Film editing, Motion Graphics, Character Animation, AI technology, AI

Id: Drh8jpjE1yo

Length: 9min 49sec (589 seconds)

Published: Sun Oct 15 2023