ComfyUI : NEW Official ControlNet Models are released! Here is my tutorial on how to use them.

hey everybody this is Scotty weather here we have great news and that the sdxl official control net models are now available and not all of them and they're going to be coming out over time as they come out of the oven but there's quite a few that are released now so I want to get you set up on how to use those now we're going to need a couple things to do this first of all I highly highly recommend you install the manager so I'm going to walk through that super fast like 15 seconds just to show you where to get it and how to install it and I did that in another video but I mistakenly said get fetch instead of get cloned so I'm going to show you the right way and then we're going to grab the models from the sdxl the official hugging face repository and then I'm going to show you how to get the preprocessors which we also need so just three things to install real quick and this video is not going to be all about the installation it's going to be about the use so let's get started hey one of the things about comfy it's a little frustrating is that there's so many custom nodes available no way to manage them so there is a manager that's available that somebody wrote uh Lieutenant Dr data and Love by the way love his stuff so whatever he's doing uh so far has been just fantastic so I'm gonna show you super fast how to install the manager uh which is the button here that you may have seen in other episodes so let's do this super fast basically you're going to go to a git I'll put the link down below and it's just a company manager and you're just going to go to code and you're going to grab this here super simple then you're going to go to your company local installation go to custom nodes and then in here if you just go up and obviously I'm on Windows but if you're on a Mac you can do kind of the same thing except for this part just do the command you'll have to just open a terminal window and CD to this directory and then we we have that copied we copied that URL from the GitHub in there which do get clone and then paste in that URL you'll see that it'll flow in a custom node in here and you're good to go so now we just need to restart company and we'll see what it looks like okay now I reloaded comfy and you can see we now have a manager button here and we're going to use this to install a few of the things that are required for control net now we can do it the hard way and go through all of the associated gits and so on but the manager does that for us so let's just use it if you click on it you're going to see there's all kinds of things here I did a video on this so I'm not going to belabor this point but we're going to go right to install custom nodes and we're going to go right to the top and type in control net hit search and you can see there's all these custom node packages that are available remember we don't want a package that does all the work for us we want a package that gives us the pieces to allow us to create our own workflow so you can go to Civic you can download a workflow that's just a bunch of UI buttons and so on but I'd rather learn how this works which is the whole goal behind this by the way comfy said it best himself this is a back end with just a thrown in Throne on top of it it's all about the process and not about the front end so here we need to grab the preprocessors and when you see the list you're going to see is there are two control net preprocessors by the same guy this is the one you want to use it's the work in process version so we're just going to click install and that's going to install all of the preprocessors and that's pretty much all we need to do here so close this and then you'll have to restart comfy again because it does not refresh automatically well this time you notice when it is installing it's going to go out and do a couple extra steps here and it's going to go grab some of those models for us so your startup time will be a little bit slower this time right now the company is fully loaded we should be able to do add node and see control net preprocessors here along with all the preprocessors that you may need okay so now let's install the sdxl models and you're going to find those over unhugging face now they're called control Laura's but understand they're not Laura's in the sense that you're going to install them or use them like you would a Laura but this has to do more with the architectural implementation so don't get all caught up in that it means that they're smaller and more memory efficient by the way that they're architected so what we're going to to do is we're going to install these as well and again if you use this three dot menu over here you can do clone repository and then you know can you all this just hit copy and then to install them you would go to your comfy installation go to models and go to control net and you can place them in here now as an alternative if you're using automatic 1111 and a lot of other implementations you can actually save yourself some time by using this extra model paths.yaml file we look at mine if you look at this you can see that here's my installation of automatic 1111 and I just put my models into his control net folder there and the comfy installation here will find it so if you install it here you only have to put it in one place to use it in comfy and automatic 1111 I use both so this is a great option for not copying a lot of models around but you're going to use the exact same technique by using the get clone feature but it's already copied to your clipboard once you've got that in place you can see this is what your result will be you'll have everything inside of the control net folder and again I put my other version the old 1.1s inside of their own directory and then I have the new control loras in their own folder as well okay now that we have everything loaded let's see if it's working right now let's show you what these preprocessors actually do so to do this we're going to load an image so if you watch my episode yesterday we took the revision technique which is a new thing you can do with sdxl which allows you to take multiple images and run the clip classifier against them and basically pull out the context of the images to create a new one that's kind of a combination of all of that together so pretty cool technique and again this graph and that graph are all going to be uploaded to the YouTube member area so if you are a sponsor of this Channel or higher I really appreciate your efforts to keep the channel going and you guys mean the world to me so I'm going to upload all these graphs there for you there'd be a link to a Google Drive where you can grab that image download it and just drag it into your copy interface and load those graphs right away once again thank you guys so much all right so here let's take a look what these preprocessors do real quick so let's load one up so let's use the I have a couple that I really like uh the first one that's more uh probably the most popular one is the candy Edge detector um I like the head one which is more of a soft detector and then there's other ones like the depth map for example um a normal map and depth map are different things by the way normal shows light Direction on a Surface where depth is showing more or less height information so be aware they are not the same thing I know it gets confusing if you've ever used blender before a lot of people know that distinction but if you don't uh that's uh something you know now uh they're not the same thing let's do the edge detector here and by the way if you are loading in by the way if you're loading one of these preprocessors for the first time the model will download and it could take some time so if you check your console window you'll see that it's pulling it down only has to do that once so if you're wondering why things are taking so long that'd be the reason so if we run this real quick you can see what the depth map looks like and we can see that the face is marked here and anything white is closer to the camera anything dark is farther away but you see there's not a lot of detail here which is why people like a combination of say candy and depth because you can use more than one control net so fingers for example candy is very good at outlines for individual digits but depth map shows which ones are closer to the camera so the combination of those two work very well on hands for example let's do another one let's grab the candy one this one's very quick yeah and again you can use these settings here to determine the sensitivity to what types of lines you want whether you want a rough really just kind of rough sketch of it or if you want a lot of detail use the one that you prefer for this demonstration I think we're going to stick with depth because that's pretty much my favorite one personally I'm sure everyone has their favorite but that one I like the most because it doesn't have a lot of detail so it kind of opens the door for interpretation plus if you have things like buildings for example and you like the outline or shape of the building but you want to see it as a Roman Coliseum or you want us to use a Sci-Fi set wherever it is it has the rough shape and the perspective is held in place by the depth map so I think that's pretty cool loaded in the rest of our graph now so let's start with our standard checkpoint which will be sdxl I know this text is small sorry about that not much I can do about it there is no font thing in here to load a larger font unfortunately I've done what I can to make the magnification as big as possible let's go ahead and load in our control net next so we'll do loaders and then control that model and again you should see the giant list here if you've done it well note that it does use those folders I put them into which is a nice way to organize things now there are two versions of each of the ones that are available right now the 128 and the 256 depending on the memory that your computer has in it just make sure you pick the one that's appropriate for the preprocessor so this control net model is expecting to see this type of image you're going to see in a second it's going to ask us for an image it does not mean this one it means this one so if you already have a depth map that you love and you want to use it you do not need to pre-process it again you can just load it in with this so let's go ahead and do that now if you don't know what the next step is a great thing about comfy is it knows the process so if you drag this out it shows you there are really only two control net apply and control net apply Advanced we're going to use Advanced because there's some things about this that I want to show so notice that it's expecting an image again it's not this it's expecting this so let's go ahead and just load that here and rearrange my graphs but we'll keep this depth map handy just because it's kind of interesting or just roughly water things in here what I think is also very interesting right now is you notice we're dealing with the conditioning of this we're not dealing with the imagery which you may think well I'm dealing with an image therefore this will be the image processing part it's actually in the conditioning part which I think is pretty interesting so let's go ahead and do our standard prompts so we'll load an encoder for our positive and we'll load in an encoder for our negative we'll color them I always try to make this look as nice as possible so when I do upload it to the member area you guys will have a graph it is same at least I might even go through and rename some of these nodes if I'm feeling frisky that day okay and what we want to do now is we want to take these and load them in here so here's our positive here's our negative so we're not going to use the ones from a regular clip we're going to use these into the control net here there you go now we have not satisfied this control net that is expecting again from the depth loader is expecting a depth map which disappeared I don't know why if we wanted to do another control net we would load another control net model and then we would duplicate this and then hook it right in in a chain so you can actually use control net into control net and control net all day long I just make sure that you're loading the appropriate control net model and preprocessor or image that satisfies this control net for each one of those individual steps okay now we'll just roll it through a regular K sampler pull it out there we go and again we need our model and we will need the vae in a second so I'm just going to drag that up here and put a reroute node so we can grab it when we need it I do that to avoid lines crossing all over the place I'll make the graph look a little nicer before I end it out tonight but we'll do something here now it's all all happy like you have some sort of a big bus of lines across the top which would absolutely drive me nuts we're going to talk about graph neatness in another episode because I use another thing called a pipe which is a lot nicer to look at but this is at least showing you the individual flow so forgive the lines for today it's a learning experience that's what we'll chalk it up to all right in this case we need a latent so we're just going to grab a regular empty latent and remember with sdxl it has to be a 1024 or larger on the long Edge so we're gonna go this way here just for giggles and we do need a negative as well so I'll cover negative right so we have all of our things uh set here now for your settings here again you do what you'd like to do I'm going to go with like say 35 again this doesn't make it better what it does to point but then there's the diminishing returns or it might take you 10 times longer but you're only getting a very few pixels changed so I think 35 is a good place to start here I tend to like a little bit lower CFG again the lower it means that you're going to have more freedom for prompt uh kind of but it's interpreting The Prompt as versus much higher here will give you more accurate prop but it will also cost you a bit from a contract standpoint they tend to get a bit sharper a little bit noisier so I think seven is a nice place to be and then for a sampler you can use Euler or you can pick whatever you'd like I tend to like write today we're going to use the DPM plus plus 3M simple differential equation one for a GPU that's a mouthful so let's just use the u3 the DPM pp3m sde and then the scheduler let's do something exciting let's use exponential again play with these because you might find something you really like each sampler is a different mathematical model against how you want to pull information out of this latent so we're just going to go with that scheduler also matters quite a bit so let's go ahead and then decode this thing and here we need our happy vae our happy happy tree Bae and let's soups grab this Dot and pull this to a save image so we have one uh for Giggles all right so we need to prompt this thing uh let's go with uh what does she look like let's drag her over here by the way um I would like to see all these kind of like it's almost a rough um graphical user interface here we'll make our controls all handy so we can zoom in so you can actually see the text as well as the beginning image the pre-processed image here and then we'll have our final we'll take up a bunch of space right here so let's let's get in here all right so let's talk a little bit about the settings before we run our prompt which we'll move over here these strengths here are very interesting so the strength is obviously how much it's going to apply over the over the course of the entire image so this is 100 right now but what's more interesting is to start and end so we have no idea what's going to come up when we hit Q prompt but we could say you know what go ahead and do whatever you want to do and tell us say about oh 20 of the way through the image and then you must pay attention to the depth map here or you could say you know what go ahead and and start with the depth map and and follow it but say about 80 of the way through the step count here we're going to go ahead and then just not pay attention to the depth map anymore and you can do whatever you want to do so you could have this adhere the entire time span or you can have it here just at the beginning of just the end or in the middle so this is really kind of interesting and I prefer this one over the simpler one just because we have this control and obviously if it's too much you can always lower the strength a little bit as well so let's just leave it like this just for giggles and see what happens now let's give it a prompt uh let's go with the let's go with our standard cyborg things that's what I like so alien cyborg female on an alien ship and let's see what happens all right what we got was pretty terrible so let's uh let's fix it up a bit uh we would like a photo photo of my seabird I'm a cyborg female on an alien ship let's try that but you can see we do have adherence to the outline obviously the face is in the same place the Hat's similar so it is looking at what we have downloaded or this preview is not showing up there we go that's much more exciting I don't know where this preview is not showing up over here let's oh let's pull that one in again we'll just get rid of it and Roll Another preview over here and that should calculate there we go and there you go so that's how the depth map works and again all the control Nets work in their own way but each one has a kind of a Rhyme or Reason to how it behaves and again you can stack them by putting other ones in between here so you have one control net you could put another one inside of here everybody thanks for watching hope you enjoy these models let me know in the comments what you think and once again thanks to everybody who supports the channel I couldn't do this without you and everybody take care stay safe and I'll catch you all next time

Channel: Scott Detweiler

Views: 44,290

Keywords: stable diffusion, stable diffusion xl, ComfyUI, stable diffusion tutorial, AI art, SDXL, comfy tutorial, control net tutorial

Id: DMxnf4WXMsY

Length: 15min 59sec (959 seconds)

Published: Mon Aug 21 2023