Hello everyone, today I would like to talk
about ControlNet for SD XL. In one of my previous videos, we have seen together how to use
ControlNet with stable diffusion XL on ComfyUI, today I would like to have a look together
on how to use it on automatic 1111. I'm not going into many details about ControlNet,
if you want to know more about it, I made a video. This was based on stable diffusion 1.5, but all
of the settings you have to use are exactly the same as for SD XL. This you see is the main GitHub
page for stable diffusion ControlNet SD XL, and, if you scroll down you actually have all of the
information you need for installing, downloading models, and other ControlNet characteristics you
may want to know. Many are still using the old, the previous version of ControlNet for running
ControlNet with stable division XL and they are running into errors. So it's not working -
and this is because you need to upgrade your ControlNet extension. So, let's do that first.
We go into extensions, and making sure you have ControlNet in here, you check for updates. Now if
all of your extensions have been already updated, in this column, here, you will see "latest". If
has not been updated you will see "new commits" with this little checkbox. In order to update it,
make sure that the checkbox is ticked. Obviously in my case ControlNet has been already updated -
because I did it already - but in your case, if it hasn't been you will have this "new commits" with
the checkbox and then you need to press this apply and restart UI for updating them. If you go into
your terminal, you will see that stable diffusion is actually updating your extensions and then it
will automatically restart it. Once restarted, you will see ControlNet version 1.1.410 which
is the latest version and it's the one we need for stable diffusion XL. So now we are ready for
running ControlNet with stable diffusion XL but we still need the models. Where to find them? We can
find them on Hugging Face. This is the main page, and you will see that all of the ControlNet models
for stable diffusion XL will have "xl" inside the file name. You will see that there are many
models and some are the same... so for example, you have diffuser xl candy full, diffuser xl
candy mid, and small. But you also have kohya control light XL canny. So, you will say, what's
the difference, right? So the difference between diffusers and Kohya is just that they have been
developed by different people and this is just a matter of taste, what you prefer, right. In my
opinion, the diffuser seems better but it's again just my opinion. The difference between full mid
is more just on the size really. So full will be slower but ideally more precise, while small will
be quicker and less precise but I can tell you it works really really well - I actually use small
all the time. So, once you decided what ControlNet you want, maybe you want to try all of them,
you just need to download them so you press this little button here, and then once you downloaded
them you need to move them inside your stable diffusion webui folder. You can put them in two
different folders really. The first one is models and controlnet and as you can see here I
have my xl control net models; otherwise, you can go into stable diffusion webui,
extensions, stable decision webui controlnet models and then you can add them here. Now, I
prefer having them in the other folder because here I have my stable diffusion 1.5 but it's
completely up to you - it doesn't really matter. So, once you have moved all of your controlnet
models inside one of these two folders you go back into your stable division web UI and then
you can press this refresh button in order to see them in this control net model drop-down
list. And indeed I have them here at the top now. I will say let's try one of them I will
get my mask for ControlNet from PoseMy.Art. I like this picture here so I'm going to export it.
For sdxl we can use a resolution of 1024 by 1024. and I'm going to export candy. Then, for my prompt
description, I'm going to use mysterious sdxl: this is a checkpoint in CivitAI trained on stable
diffusion XL. I would like to reproduce this guy or something similar obviously would be different
because I'm using a different mask. I'll paste the generation data in here. Press this for moving
all of the settings across my tab, then if we scroll down we have some recommendations from
the guy who made this model and is saying to use the adetailer for getting better results. To
use high CFG scale for Fantasy, low if we want realistic photos. So in my case I want realistic
so I use maybe a lower CFG scale so I'll go back into stable diffusion and I am going to reduce
this to three. Then for the adetailer we can open this tab here, enable the extension
in here. I'm going to use face yolo and then we just need to copy these settings for best results copy and paste the positive and the negative
prompt. I'm going to leave the other settings as they are, and then I'm going to ControlNet,
which is what we want to test actually. So let's go down, we upload our mask I'm going to use low
vram I'm going to use Pixel Perfect. Pixel Perfect is for the resolution of the preprocessor. I'm
going to use candy because I'm using a candy mask though I don't need a preprocessor because
the mask is already a pre-processed image so I'm going to swap this to none, but I'm going
to use the candy model for stable diffusion XL the small one and I've also changed the control
mode for my prompt is more important instead of balanced because otherwise you will get something
like this which is not very nice because you are giving too much importance to this candy mask
but we want to have something more that looks like less than a mannequin but more like a person
right and then I want to send this Dimensions up in here because I would like to have the output
of the same dimension of the input image of the input mask which is 1024 by 1024 and then I want
to use a random seed so I'll change this to -1 I'm going to press generate and this is
what I've got which looks pretty good to me so we can actually see the ControlNet for stable
diffusion XL is working also on automatic 1111 which I find it user friendly, more user-friendly
compared to ComfyUI. so that's it for today I hope this was useful and thank you very much for
watching and see you in the next video bye
