ANIMATEDIFF COMFYUI TUTORIAL - USING CONTROLNETS AND MORE.

enigmatic e hey everyone today we're diving into anime diff using comfy UI and I'm going to be honest I was a bit hesitant learning about comfy UI at first because of the images I saw of the interface with nodes and spaghetti lines all over the place it looked kind of scary but seeing the amazing things people were creating with anime diff got me really curious I tried anime diff in stable diffusion automatic 1111 but I was struggling to get the results like the ones I was seeing from other people however once I really got into comfy I real it's not as intimidating as it seems especially when you're using templates that you can load up very easily and it really does open up a world of customization and workflow options and this video will explore the cool customization stuff but don't worry if you just want a quick setup without going too deep into the customization cuz I will go over that too so don't run away just yet trust me this stuff gets fun all right let's get into it all right so we're going to start off on this GitHub page where we get all the comfy UI info there's a lot of information on this page you can read through this if you like it has stuff like you know shortcuts a bunch of information we're going to go over the installation process it's super simple super basic uh I'm not sure how to install it in other kind of computers because I think it has instructions on how to do it with like Apple Mac silicon and I'm not too familiar with this I never own this computer so read through that you can learn it for yourself but I'm going to install it in my local Windows PC uh I think it also has options if you want to do it through collab just to let you know this uh does work with other computers that don't have uh Nvidia GPU I did test it on one of my laptops and I realized that uh it is very slow because it's using the CPU of the laptop if you have a Nvidia GPU that's the ideal situation or use collab or some other option all right so I'm just going to click on direct link to download I'm going to save this so you might want to create a folder that you name comfy UI and then just save it in that folder so this file is compressed so you do need WinRAR or 7zip I believe also works uh and you need need to extract the files from there once those files have been extracted you come into the folder and here is where you're going to have all the stuff that you need in this folder here come into comfy UI so in this models folder you can put your control Nets your checkpoints your luras your embeddings however if you do already have a place where you keep all your luras and checkpoints like if you use stable diffusion automatic 1111 or what fusion and you already have a place where you put all that stuff then you can always redirect the path to those folders so I'm going to show you how to do that in a bit but this option is here if you want to put all your models in here so if we go back to the previous folder here in comfy UI we can come down to where it says extra model paths yl. example so we want to make a copy of this so I'm going to press control+ C and then crl + V and I'm just going to take off example right here and I'm going to open this as a notepad so this is only if you want to redirect the path to for example stable diffusion web UI folders so and this this case we're going to focus on base path right here and I'm going to go to my stable diffusion folder so here I have all my checkpoints but if I go back to my stable diffusion web UI folder you want to get this URL contrl C to copy and you want to paste it right here so the base path and what this will do it will automatically based off this path look for the checkpoints baes for luras it will look for embeddings and control Nets the other path that you do have to change is the control net path because here it says models and then control net but in reality if you come to models and you go to control Nets it's probably empty where you have those control Nets stored is typically if you come to extensions come to control net and then you go to models this is typically where you have your control net so here you're not going to put the whole URL you're just going to put it from extension on so right here copy and then here you're going to paste this and this is what it should look like so we're going to go ahead and save this and then you should already be set with all the things that you've saved with for the most part everything that you need uh except some other models that we're going to be downloading in a bit so now that we got the paths where we wanted to go we want to install another thing called comfy manager and this is going to be super super useful you're going to see why in a bit but for now just trust me go ahead and come to this page I put a link in the description so you want to come here to where it says get clone and it has a link so I'm going to go ahead and highlight that and press contrl C to copy then I'm going to come here to my comfy UI folder I'm going to come here to Comey UI and then I'm going to go into custom nodes right here so I'm going to click up here in the URL and then I'm going to type in CMD and then I'm going to press enter you should have this new window pop up and then I'm going to paste what I just copied that and then I'm going press enter so now that that's been installed you can run comfy UI and then I come here to runor CPU if I don't have an Nvidia GPU again it's very slow but if you do have an Nvidia GPU then you double click on this and it's going to load this window for you so this is the default interface if you look a little bit closer and if you use stable diffusion automatic 1111 you might see a lot of terms that are very familiar obviously checkpoint models uh your prompt for positive up here and then you got your negative uh the size of the image the width and the height the batch size the CFG parameter steps and obviously this is just a basic setup you can get way more complex than this if we are to just load this up right here actually let's go check if our checkpoint was routed correctly and so click on here I should be able to see a list of models yes so everything was routed correctly and it found all my models so um let me just put this and just test it out and I'm going to press Q prompt to generate something so it's going to generate an image for me and that's what it gave me okay so this is the basic stuff so how do we get more complex right how do we get stuff that's a little bit more like Anime diff for example so there is a website that I would recommend here it it tells you a lot of different setup on this website a lot of examples like you know how to set up luras but then you also have this other website where it gives you a bunch of examples of anime diff and it shows you what it looks like if you are using these parameters the great thing about this is like you don't have to do this manually you can just copy this image by for example like this image right here and it's doing this right here I'm going to right click and then save image as and I'm going to save it in that folder and then I if I come here to my comfy UI interface and then I bring in that image look at what happens right I get a bunch of uh red nodes telling me a bunch of Errors have occurred right and this is why it was important to install comfy UI manager because what comfy UI manager does it will automatically download the nodes that are required for this specific setup so when whenever you load up a Json file or an image like I just did right now you might find cases where you don't have the specific nodes installed on your system so what we have to do is now is we have to go to manager and then we're going to go to install missing custom node so we press that we want to install it and here are the nodes that we're missing I'm just going to click on both of these right here and I'm going to click install and it's going to install both of them might take a little bit once you install those it tells you to restart comfy UI so that's what I'm going to do I'm going to go ahead and come here and then close this and I'm going to reload it all right so now you don't have anything read so now you have a bunch of new nodes that you probably never worked with or not familiar with and some of these stuff you should already have set up like your checkpoints but there are new nodes that you have to be concerned about which is like these are motion luras that you can download I'm going to put links to all those in the description uh and also anim diff loaders and these are also necessary to make this work because you cannot load it as is unless you have these models saved in your folder so you can always search for these uh I have links to these where you can download these uh we have some right here and then uh there's another website on huggingface where you can access uh models and also motion luras and I'm going to explain what these do in a bit you can download all these from here directly from these websites or you can go into manager and install models and for example look at the models that's being used for this specific setup for example in this case it's using mmore SD and look at uh install M mnore SD press enter and then right here you see the models right here you can always install them from here uh the models sometimes are quite big so you might take you a while to download those and I explain what those do right right now and let me just run it as is so that you can kind of get an idea of what's happening so it's I'm getting errors because I had to put in a VI that I actually have and not just the default um okay so I'm going to go ahead and load this and I just kind of want you to see what just the default does and then I'm going to go over some of the parameters right here and how this is working also it took me a while to figure this out but if you want to cancel uh generation then you come here to VQ and then you press cancel so that's the way you do that all right so it's going to load up the animation right here and then we see a generated animation right and you can mess around with the prompts and everything this is the way that you just set everything up obviously you can get way more intricate with this I would recommend if you want to customize this to your liking uh that you check out uh some of these settings here this setup doesn't have Aura setup so I would check here to look at examples of luras and see what it looks like right so we look at the setup just kind of follow the the similar setup it doesn't look the same but you see the checkpoint and right after the checkpoint you have Aur so the same same process here after a checkpoint you want a Laura uh to find the Laura you can always double click on the screen right here like this and then just put a Laura loader right and then you have Laura if you have everything routed correctly you should be able to see all your luras right here uh for example I'm going to put this one and you just want to pay attention to the names here like they connect to each other for example model to model clip to clip and then you're going to take the route these to where this one originally is routed cuz you want it to continue to go where it was originally going to so just think of it as a path that is going to so here I'm going to put this on clip and I'm going to put this model to this model here and then just have it go go to where it was going originally right and now you have the strength of the model uh the strength of the clip I'm going to keep it one um if you want your videos to be longer and the way this is set up your batch size determines how long the video is going to be in this case I believe it's 32 frames so that's why it's going like this and these are the dimensions that it's using uh motion luras is um is used to the way the camera is moving for example like if I use a this is a zooming out and then it's panning to the left that's why you're seeing movement in this generation it's because we added motion luras right here there's two motion luras being used here at the same time and then you put the strength of those here you can change the motion luras once you download them you can have a bunch of options like pan to the right counterclockwise tilt down tilt up you know you have a bunch of options there and also when you do Aur you don't have to put it in the prompt like you would do in automatic uh 1111 or War fusion um embeddings you do put it in the negative prompt so like bad hands or easy negative or whatever it is that you want to put in there um in this case but you probably do have to put something that triggers the Laura uh you know when you when you sometimes when you're downloading luras it tells you on cti.com that you know use these trigger words and so that's more than anything that's what you have to put what once you generate stuff and you come here to Output this is where you have everything saved your gifts or your videos or these images it will save for example a gif and or a video and then it will also save a PNG file right here and this PNG file has data so that for example if I forgot what I used for this specific generation you can come here and grab something that was already generated and then just drag this PNG file into comfy UI and then you have your setup your workflow already set up for you and that way you don't have to guess okay what was that setup again you can just load it in this way and it's super super fast and easy so this is my setup for generating traveling prompts uh what I mean by traveling prompts is I have this uh prompt window right here where I can include multiple frames with different prompts so for example like I have at frame zero screamy man turning into a zombie and then I want that to change at frame 10 like right here um this is the the format for this and I put like screaming zombie and it just keeps going as long as you like it to go these numbers represent the frame and then obviously the prompt after that so here down where it says pretext is where you put your general information like how you want this to look in general like I put like 3D cartoon high quality Masterpiece because I want this to apply to all of these and so I don't want to type all that information into here so I put it in the pretext and so then now I can just Focus here on very specific things like a young man turning into a zombie so I don't have to just a young man tur into a zombie Masterpiece this long long prompt so I just put all my prompt here most of my prompt here and then just like basic information here so that it applies to all the prompts I will include a Json file to this whole setup this workflow for you in the description so that this you can just kind of plug and play if you like so I'm going to show you a few examples of how this works so let's come here to animate diff loader right uh let's say I go to this basic animate diff loader right here that a lot of people use okay and I just load this I put here batch size 70 so it's going to be 70 frames I did put this other thing in the bottom here that's like an upscaler uh I think UPS scalers do take long so just be aware that I would not use the upscaler until you finally get something that you really like and then use the upscaler so so for now I am not going to using this upscaler in this animation you do see some coherence it does look like the frames are animating into each other but you do see sudden shift and changes uh I personally am not a big fan of unless it's something very specific that I'm trying to do but in this case uh you can probably fix this with prompts or uh some settings to make it flow a little better I just wanted to show you what this specific uh model does but uh the one that I like to use is stabilized uncore mid or High um and let me show you what that does all right so here you see the animation that I originally went with where everything's a little bit more coherent that you see it like one full animation happening it's not like a sudden shift I mean everything kind of Blends into each other a lot better than previously and that's you're using mid stabilized maybe it doesn't work for every case maybe it it works for like portrait kind of um head shots because I've done plenty of these and I really like the way it looks there's another one called temporo diff by the person who did temporo net and temporo kit uh she is like amazing she's always like on top of making these models um you can also go with this with temporo diff you do get a lot of consistency but not as much movement and you'll see what I mean once this loads it does struggle to change as you can see it you can't see the human the man that you were seeing before it's all just like a zombie I guess that's what I got to say is like it just works differently one thing can work perfectly fine for specific projects and some may not so just be aware that these are things that will definitely affect what the outcome is going to be and then you have the option here to go with a gift or you can do video also there's frame rate you can adjust the frame rate however you like for me when I generate this with video um you don't see as much of the Grain and the noise that you get from gift Generations again once you have everything set up the way you want to you like the generation then you can just have it continually run into uh the upscaler so that you can get like better quality and upscaled image I will include the work flows for these however I would just ask you please please do me a favor and like And subscribe and please comment um positive or constructive criticism my videos do not get viewed or do not get pushed by YouTube unless you really interact and you know say hey I really like this video or whatever all right the only other thing I want to talk about and mention is using control Nets you can also use control Nets with animate diff so let's try out using control Nets so here and uh this page that I looked at previously they do have have an example of using control Nets here right here it's using uh open pose and based off that open pose this is the kind of Animation you can get so you can save this image load this into here with this setup we can use control Nets uh in this case uh open pose uh you can always load in open pose animations that you've already you know generated if you've done any kind of animations in the past like from Mort Fusion you can get a bunch of images if you've ever used control net you've probably seen something like this where um you see the outline from the control net this is uh soft Edge or open pose you know the skeleton in this case uh you know it's open pose and you can load up an a folder with a bunch of open pose images and and just use that but typically you would have to generate that somehow but if you want to generate from a video and not have to go through the process of you know getting image sequence or whatever you can also do it that way uh I'm going to show you really quickly but I can save this uh Json file so that you don't have to go through this process but I just want to show you like how you can you know customize this to your liking uh once you really get the hang of how this works so for example I want a video loader so okay so we're going to load our video through this and I want something to generate the control net skeletons so go to uh open pose pre-processor and so uh the way I do this is I look again look at what where these are connected to so uh I'm going to unconnect this batch right here so I'm going to put this here instead of uh load images I'm going to load video okay and then I'm going to Lo put image into open pose and then open pose is going to go straight here how it was before and then frame count uh I'm going to put batch size so whatever however long the video is that's going to be the size of the batch size so if the video has 100 frames it's going to generate 100 frames but uh yeah probably not the best idea um so now I'm going to load the video let me look for a video here so I'm going to do this this thing of this head um I'm going to use stabilized and my flat I do want to use Laura so so this is the great thing about uh this is like you can customize it however you like and it's such it's so great all right so this little window is going to show you a preview of the video once I click generate but uh I also might want to create but I just press contrl C to copy contrl V to paste the node uh I might want to preview also the control net images that are being generated so I'm going to go ahead and do that and and I'm just going to go ahead and generate this and see what comes out right from the beginning uh here in this preview window I'm going to see all the frames that it loaded uh from the video and then after I should see the control net frames here and uh you can see here that uh you can enable head body or face uh and then you have your resolution as well here so and as you can see it did some here but uh obviously when the head is turning around here uh it can't detect a face so we're probably going to run into some issues once the animation is generated however this is just uh an example for you to see how this works so now we're seeing the animation like as I predicted once the head turns and and it's the back of the head then everything kind of flips out and goes kind of crazy because it's in this section right here but for the most part you kind of get the feel of what's Happening Here Also um it doesn't look as clean because the resolution is really small also uh you can do this as a video for me that's a better solution because then you also don't get a bunch of that graininess that you get from gift sometimes uh and you can also just um upscale it later so there are options there and uh I wanted to show one more thing there is also a way that you can do a starting image and an ending image so what you're seeing here for example like this this is the starting image this is the ending image um it shows you here the setup for this and this is kind of the results that you can get from this uh this is just another option that you can do we can test this out here we have the Json file that we can download and I'm going to save it as a Jason file and then comy UI all right so here we have a little bit of a different setup now now we can start like with a frame like this and then end with a frame like that all all right so I set up the prompt and I did everything uh that I need to do you know there's a there's a model that I really like called aniverse and it gives me really decent results so I'm going to use that and then I'm just going to change the dimensions so that it matches the aspect ratio and then let's load this to see what this looks like so you do see that transition happening in this so there's a lot of little cool things you can do with this uh it's not just like generating something in anime diff just like that there's different ways to generate through anime diff like through control Nets starting and ending frames there are just a lot of little cool things you can do you know there's even other control net loaders that look different than these control net loaders there like you don't have to use these specific nodes like there's so many different nodes that do essentially the same thing but sometimes with different options different ways of doing it so you can just explore I would load some of these so that I can get some notes that I'm not familiar with because there's some notes that I wouldn't even know okay I should download this node but if you come here and you just load these up you get that red node that's telling you you don't have that node then I would load it through comfy UI manager so you can have it already loaded to your folder all right everyone thank you so much for watching if you want to contact me directly I do uh talk with people who are on my patreon so I have a Discord where people can talk to me through there directly So yeah thank you so much for watching like always take care God bless and [Music] peace

Channel: enigmatic_e

Views: 36,545

Keywords: animatediff, stable diffusion, ai animation, ai art, comfyui

Id: WHxIrY2wLQE

Length: 24min 54sec (1494 seconds)

Published: Thu Oct 12 2023