ComfyUI ControlNet Tutorial and FREE SDXL Workflows!

hello and welcome to More nerdy rodent geekery today's video is about sdxl control Nets in comfy UI at the moment there are only a couple available namely canny Edge and depth but as more come out the same principles shown here will apply then as well in case you've got no idea what stable diffusion sdxl is all about basically it's a way to generate images from text using AI I'm running comfy UI for sdxl locally here so if you need more information on that do check out the other videos I've done on getting comfy UI installed and running this video is a more advanced one intended for those already using comfy UI and who now want to use control net as well while there are some control net examples already available They Don't Really deal with sdxl specifically but they're still useful for a more General overview okay so first stop where can you get the sdxl control net models from well if you head over to the hugging face diffusers page then you'll see everything which is available at the moment right now there's lots of different files but basically only two models one being canny and the other being depth the one I'm using at the moment the full sized control Nets but you can see here they've also got small size control Nets as well which is a much smaller download downloading is simple you just click the link for whatever you want and that will take you to the model card here's the one for sdxl control net depth for example click files and versions and you'll see all the files available now the one I downloaded was the fp16 safe tensors version which as you can see is a little bit bigger than those other files you want to click download on that and the default location for comfy UI is the control net directory which is underneath models there it is comfy UI models and it says put control Nets and T2i here you're also going to need some control net preprocessors as well as those models so head on over to this GitHub webpage for those as well the control net preprocessors for comfy UI start with the download into the comfy UI custom nodes directory by copying and pasting the commands shown so there I'm just running the git clone obviously I've already run that so it says hey you've already downloaded I did that next you'll need to run the installer but this is where the documentation Falls over as trying to run the install command just won't work here because obviously you need to pass that different things and it's nothing related to this repository at all what they actually mean is to run one of the two files there so either install.sha or install dot bat so in my case if I can type it would be install.shure if you're on Windows you would use the dot bat file there you go you've now got everything you need to run sdxl control Nets fire up comfy UI and you can add these into your workflow in true nerdy style here is one I made earlier this is about as basic as it gets so as you can see we've got the control net image there the control net model that's where we load them I've only got two options in there canny or depth little upscale there to make sure the image is of a reasonable size Kani processor or the depth processor depending on which one I'm doing if I want to use the depth processor I just have to connect that one or I can go back to the canny and then finally we've got the apply control net and the preview image as well so that's it pretty straightforward just eight nodes in total with the only thing you'll need to do is to wire this into your existing workflow via the apply control Nets as you can see it's got a positive and a negative input and the same type on the output there if you just want to go ahead and download everything there I've got the sdxl depth Badger so you can download that PNG file and you will have this comfy UI all already connected but I'll show you how to add it into your own as well here for the first one there we've got load control net model so I'll just show you where these are so that's add node loaders and of course load control net model image that is in the image section with so we've got add node image and then you've got load image upscale image is over there in upscaling upscale image then we've got canny now canny is actually in here so you see we've got two places for pre-processors that's the lot we added and this is the default one in here so pre-processing canny is the canny one and the depth as mentioned is in that slightly different place so pre-processes depth and there we've got the Zoe depth apply control net Advanced that's over in conditioning so we've got add node conditioning apply control net and apply control net Advanced apply control net doesn't have the start and end percentages on it root it's just a utility so you can add node utilities routes and preview images obviously the image again so you can add node image preview image and I'm using the default sdxl workflow here showed in my previous video basically the same way you download from The Comfy Anonymous example site that I've just moved the boxes so that's that workflow underneath as mentioned before all you've got is the positive and negative inputs and outputs so you basically just need to add two wires in and then connect those back let's move this up a bit so we can see that so here we've got the base prompt conditioning so there's the positive we connect the green one in there there's the negative we connect the red one in there then we've got the two positive and negatives and that goes into the K sampler there so we've got the positive and there we've got the negative and now you have wired in your control Nets obviously if you're looking to add sdxl control Nets to a different workflow then you'll need to find the matching nodes but it's just two wires in and two wires out all right that's all very well and good but what can we do with this thing as you can see I've got an example set up there with some text and this text is one thing that works great with canny Edge if we set the strength to one for example and the end percentage to one as well and generate that prompt you'll see the preview appears so there's the canny edge outline where the preview being down there so as you can see that's very very clear if you use the full power control net so to speak the uh the text prompt down here has almost been completely ignored as it's meant to be a picture of an anthropomorphic Badger I mean that's that's not a badger it's just text isn't it so what you want to do if you want to give it a little bit more imagination is drop this strength down I like to use something less than 0.75 for the strength and the end percentage somewhere around 0.6 this time when I generate something like that as well and with the lower strength and end percentage then sdxl can be a bit more creative so you get the text and the badger as well of course you can use whatever style you like such as ice cream or eggshells or maybe bread or Peppers or graffiti anyway Kenny Edge isn't just for text so feel free to explore using non-traditional shapes for badgers for example there's a very non-traditional shape for a badger down there in my prompt I've just got a photo of a badger that looks nothing like a badger but the power of control net will give me this result and as you can see we've got a perfectly normal badger for the depth model all you have to do is change this so you select the depth model and then I wire in the depth reprocessor instead and there he is all sort of Hedgehog like and everything you can use text with the depth model as well however it does come out a little bit blurry on the depth map as you can see there the text still comes out quite all right but it's you know not quite as good as the candy Edge and there we go we've got our badger in the background and the text powering on through there so personally I find the depth model much better for non-text again you can drop the strength and the start and end percentage if you like but generally speaking the depth model has a bit more creativity due to all the gradients in that depth map so it's much better as mentioned for shape so we can change that kitten for example into something much more appropriate down here we've got the prompt which is a photo of a badger so hopefully we can change this kitten into a badger and there we go let's just have a quick look at that output so there we've got the depth map and there we have the matching Badger lovely as you can see it's really easy to add control Nets to comfy UI for sdxl and when the new models come out you should be able to just download those and pop in your pre-processors in much the same way if you thought that was cool but want to know more about comfy UI then check out this next dirty rodent video

Channel: Nerdy Rodent

Views: 12,176

Keywords: nerdy rodent, ai, ai art, best ai tutorials, sdxl 1.0, sdxl controlnet, sdxl controlnets, sdxl depth, sdxl canny, stable diffusion, python ai art, easy ai tutorial, comfyui made easy, linux ai art

Id: reqamcrPYiM

Length: 9min 45sec (585 seconds)

Published: Thu Aug 17 2023