ControlNet in Stable Diffusion Automatic 1111 (Tutorial)

did you know you can generate this 3D anime and stable diffusion and make it say hi or make her sit on the bed maybe on top of the mountain or on the street in some alleyway even change your dress well let me show you how [Music] welcome back to the channel in this video I will cover what control net for stable diffusion automatic double one double one is I will keep this short and simple as possible because things can get complicated so watch the full video what exactly is control net in simple terms consider an extension over your stable diffusion to give you more control on generating images with human Expressions poses and other subjects like animals and even landscapes so this interface is stable diffusion automatic double one double one if you aren't familiar with this I suggest you stop return to my channel and look at all the previous stable diffusion videos like installation civit Ai and upscaling to understand this you need to be familiar with this interface if you are let's continue we start with how to install control net it's a simple extension install this link you see copy it go back to stable diffusion and install the extension apply and restart you can confirm the installation by going into extensions foreign up the first page for the models you need to download all the files in that pth format and save these models in the following folder of stable diffusion you can also see these yaml files when you install the extension they should already be in the models folder of control knit however if they are not for some reason you have to manually download them we also need to download the T2i adapter models you can download all of them but I only have two downloaded which are style and color after downloading all the models you should close stable diffusion from the command prompt and relaunch the web UI now you would be able to see the control net option click on it to expand and let me explain the basics before we start playing around with it for the generation to use your control net you should click enable if it is unticked the generation happens without control net I prefer to tick Pixel Perfect and allow preview with the allow preview option you can actually see what the model has drawn before sending it to generate the image there is a preprocessor and there is a model it's best to match both of them for example in control type select open pose now the preprocessor will show you only open pose options and the model will automatically be selected as open pose control weight is the weight given to control net for the generations let's keep that at one there is a starting control step and an ending control step these settings go from 0 to 1 and are calculated in percentages say your image generation has a sampling step of a hundred starting control Step at 0 means it will start from the beginning and ending control Step at 0.5 will end pushing the control net to your image at 50 steps that's at 50 percent of the steps the remaining 50 steps will be generated without using control net we then have the preprocessor resolution and I would keep this at the same value as your input it makes things simpler the resize mode will matter only if the ratios or the input and preprocessor resolutions are different in control mode you have three options one is the balanced mode the other gives more importance to your prompt and the last gives more prominence to control net we will play around with each one of them so let's start with open pose foreign settings if you go to stable diffusion you will find the clip settings some of the models I showcased in this video require the setting to be at 2 and Summit 1. for the Batman demo I am using a checkpoint called Western animation all the checkpoints Laura upscalers Etc I use throughout the video is freely available on civet AI or hugging face let's start by putting the positive and negative prompts steps around 50. height should be 768. copying and pasting the seed before I hit generate let me tell you I am using a Lora model called Jim Lee offset with this checkpoint so this is a nan upscaled simple Batman animation Style cartoon artwork let's go ahead and use control knit and try to change its pose expand control net I have already generated and edited some AI images for this video let's choose pose 3. throughout the video I will be using Pixel Perfect and allow preview options enabled the images used for control net are in the same ratio as those generated in stable diffusion it's advisable always to keep the same ratio in control type I am selecting open pose I just want the direction of the Batman to be similar to the image however I don't want face detection so here I will select open pose later I will explain what the other open pose functions do with examples we then hit the explosion icon to run the preprocessor to check out the control net preview here you can see it is correctly detected the angle with no hands or face just the body and neck we'll keep the control mode balanced resize mode to crop and resize so we have exactly what we need let's compare this to the original you can see the original image on the left and the control net image on the right and if we superimpose the control net image you can see how this worked let's take another example and choose pose 2 this time the pose is almost similar but the angle of the face is a bit different same settings perfect let's compare the image on the right is the one we just created you can see that the posture is slightly changed you can make very minute and accurate changes using control net with open pose let's return to stable diffusion and see if we can make Batman in another posture and make a fist trying pose 4. let's do one last one with the plain open pose for Batman and then compare the two new images foreign for this one I just wanted Batman to raise his hand and make a fist hence I did not choose open Poe's hand because if we choose that then control net would map the fingers and I don't want that I will show you how that works in the following images excellent let's compare both images the one with the Batman's fist looks cool let's superimpose the control net mapping so playing around with control net can give you accurate results it's fun and a very powerful tool moving on to a 3D anime character with some complex open pose positions and the candy model I am using a checkpoint model called rev animated for this next batch of images it's a fantastic model and you should check it out in Civic AI also to get these 3D anime I am using a Laura called 3D mm foreign I am keeping the sampling steps at 30 and the same resolution format as last time I am using a custom upscaler called Forex Ultra sharp for high-risk fix I suggest you try the subscaler I am pleased with the results it produces let's generate the base image awesome now I will show you the open pose hand function and control knit I generated a new AI image with a female and a waving by posture here I want to tell you for control net to map the fingers correctly the base model you use to Train control net matters the image I used for Batman was not good enough for this one so I had to generate a new one and find the best one that suited my needs so when using control knit keep this in mind when you choose open pose hand check how accurately it is mapped to hand I don't want the face to be imposed only the hand and the posture let's try this on our 3D anime girl and see the results perfecto let's compare this has done the job perfectly look at the hands and the fingers what was happening with the previous hand image was the fingers were getting deformed it was like after 100 Generations one was perfect but other images with the same seed had issues I wanted to use the same character and the same seed and make different poses so the solution I found was simple change the base image to a better one and then I added gloves to the prompt and removed fingerless gloves which solved the problem and the result was perfect fingers now I want to make the character sit on any surface with the hands on the side so for this I am again only selecting an open pose without using an open Pole's hand and I will tell you why in just a bit let's generate the images first here I am changing the prompt and adding sitting on a luxury bed in house at night lovely let's generate one more and change the scene then we compare foreign on a big rock at the top of a Snowy Mountain now here we have the same pose with two different scenes since we did not map her fingers and control net The Prompt automatically adjusted her left hand fingers looking closely even her right hand fingers are slightly changed and adjusted on the Rock scene this is one example where you would choose not to use an open pose hand and just stick to an open pose thank you here I am showcasing an example of an open pose full open pose full detects the face body posture and fingers the face does not apply to the anime character as the setting was on balance but I did want the fingers to be mapped let's generate and check out the image you can see the left hand in the original image was hidden because of this control net could only map the right hand fingers from the base image so her right hand fingers and the entire body posture are precisely as intended let's do one more thing with this character I will show you how to change your clothing let's select this model and I designed her clothing via mid-journey while making a video about Ai and fashion if you want to learn about that you can check out the video from the channel I will now use Kenny here you can see how candy works it is sketch the character outline and her clothing without going into the detailed mapping of her hands and legs it is also sketched a part of the environment using the same prompt let's see the results Kenny only sketches the outlines and control net maps that I found it works best to have the clothing replaced in a model character however there are other uses of Kenny as well just experiment with it on animals and different styles of Art and you will get the hang of it as to when to use Kenny let's compare and overlay the control net mapping it is perfectly modeled the character with its clothing designed as per the base image you can play around further and change the hair color gloves Etc Kenny is a powerful tool in this example and cases where you would want to change the clothing style for the following demo I am using the dream shaper checkpoint what I am going to do here is generate a castle on a vast landscape and Via control net change the mountains to a specific range of mountains that I want using lion art and soft Edge with this Castle example you will also see where and when to use control mode and control net we will have a wide aspect ratio for this image so that the resolution would be 768 by 512. foreign Castle this landscape image is also AI generated I will generate a new image with the snowy mountains and the castle the castle won't be the same though I am using the default line art and running the preprocessor here you can see the line Arc has sketched up a landscape now I will push this mapping to the image generation using control knit and see what happens I want the castle in this landscape the most important thing I want is the mountain range the path in the center and other landscape features are unimportant to me the image by itself is good but it's a complete mess and not what I actually want I want the castle to be more dominant in the image this is an excellent example to show you when to use the control mode till now I have been using balance let's try changing the control mode to my prompt is more important and see the result this is precisely what I wanted let's generate one more image using the same setting but instead of line art let's try soft Edge and then we will compare both images and overlay the control net mapping thank you the soft Edge gets the mountains right but leaves out a lot of details in the landscape actually the whole landscape will change foreign let's compare this is the original image generated this one is with line art and balance mode if I overlay the image you can see the landscape is very accurate let's check out the same image with more importance to the prompt and control net and now we have our Castle the landscape is almost accurate precisely what I was going for I just wanted the mountain ranges also pay attention to the detailed ruined Bridges between the castle on the left and right let me overlay the control net image and you can check out the AI has done an excellent job on the image however when I change the control mode to my prompt is more important you can see a slight difference in the mountain ranges and the overlay is not precisely matching like in the balance mode although I personally find the line art more aesthetically pleasing the soft Edge is more accurate with the mountain ranges it is basically a matter of personal preference both do an excellent job though next let's look at mlsd and how it can be helpful for architectural prompts for this demo I am using a checkpoint called deliberate and I will put a straightforward prompt a modern living room teak wood and leather decor just changing the settings a bit and will generate an image now I will select a living room design of my own then using mlsd via control type I can map the room's layout and generate the same room architecture with my prompt mlsd is good for finding straight lines and edges the value threshold is the value at which something will be considered a line a low setting will result in more lines generated about the mlsd distance threshold it's something like the camera's focal length it sets the minimal distance between the observed data points a higher setting should reduce the number of lines generated whereas a lower setting would generate more lines for my image I need maximum lines and details as possible so I am going in with a lower setting foreign this is the original image that the simple prompt generated and using my image and mlsd running the preprocessor we got these mapped lines via control net and using that I could generate the following image with teak wood and leather design so mlsd is a fantastic tool that is Handy for architectural design for the last demo I am using rev animated checkpoint again I would be using a simple prompt again here bird flying Mount Fiji the prop generated a nice simple image I want to add two birds here one on the left and one on the right using the same prompt so I would be using multi-mode and control knit I will be honest here this process is not as straightforward as the other methods I have shown so far for this I need the birds in a specific location within the generated image I won't be changing the prompt but I had to do some editing in Photoshop and have two images of birds one on the left and one on the right these images are AI generated only but I had to crop use a bunch of tools and ensure that the birds are precisely where I want them in the image including their size and keeping the same aspect ratio I am using for generating The Prompt image how do we do this when you expand control net you can see I got control net unit 0 1 and 2. you can have as many as you want depending on the hardware and this can be changed by going into settings control net multi mode by default I had three on but here I am only going to use and enable two for both images I am going to use line art foreign something went wrong and the images are entirely messed up in control mode Let's first try by selecting control knit is more important maybe it gives priority to the two birds in the way I want and take note I am doing this for both images you can have different settings for each image okay this is better you can see that the right wing of the bird on the right is not proper and even one leg is missing this is because the method I used lion art was not definitive enough in mapping a proper outline now that can be fixed by either choosing another control type or simply changing to a clearer bird image probably a vector that can be drawn by control net clearly but there is another problem here no Mount Fiji which was in my original prompt and I want Mount Fiji in the image so after trying a few things what I'm finally doing is changing the control mode to my prompt is more important for both images and wow the result is fantastic absolutely perfect and I got my Mount Fiji not one but three in the background let's enlarge the image and overlay both control net images to see exactly what happened so this is the final image that we got let's overlay the first control net image the bird's posture is not exactly a match for this one but the position and size are accurate this is probably because of the prompt getting more weightage let's overlap the second image this one is the same overlapping both images stable diffusion is powerful as it is with all the Lora checkpoints wild cards Etc adding control net on top of this gives you specific controls which to be honest are missing in all of its competition control net is vast and the more you play with it the more you learn I showcase just five examples and it took me a lot of time to understand each setting along with the control tag models believe me I tried my best to keep this as short as possible and explain just the basics whatever I showed in the video is just the tip of the iceberg I hope you learned something from it and as always until next time if you have watched till here thanks please like And subscribe and hit the Bell notification to receive a notification as soon as we upload a new video [Music]

Channel: ControlAltAI

Views: 12,505

Keywords: controlnet, stable diffusion, controlnet tutorial, controlnet automatic1111, stable diffusion tutorial, stable diffusion controlnet, controlnet stable diffusion, controlnet 1.1, controlnet a1111, automatic1111 controlnet, controlnet install, controlnet guide, controlnet ai, stable diffusion ai, stable diffusion art, a1111 tutorial, controlnet extension, automatic1111, controlnet pose, controlnet face, controlnet howto, controlnet help, a1111 controlnet, controlnet openpose

Id: JdCyGKVgHKI

Length: 31min 46sec (1906 seconds)

Published: Fri Jul 21 2023