NEW ControlNet for Stable diffusion RELEASED! THIS IS MIND BLOWING!

what I'm about to show you is truly amazing and it's the best change in AI art that I have seen yet I promise this is not a clickbait it will knock your socks off what if you had this image and you could turn it into this and this and this and this and this and this and this all keeping the same composition or pose I'll show you how so all the links are going to be in the description below but we're going to start here at hugging face because these files are the largest so we're gonna start with them so they can download while we do the other stuff and there are a lot of models there that you can use and you can use all of them and play with them but I would recommend that you start with the control that canny the depth map or the Midas the open pose and the scribble so I would pick those four canny when you go into that just press download and you do the same with depth map open pose and scribble and if you just want to try one of them try the candy or the depth map depending on how much space you have now once you've done that you're going to start a command prompt and you do that by typing CMD into your windows search bar you can also go into your folder here and type CMD if you do it from here you would end up in your directory that you are in in my case my stable Fusion web UI and then you're going to type pip install opencv Dash python that will install some prerequisites for you after that we're gonna go into our stable Fusion we're going to find this tab here extensions and here we can install extensions and then you're gonna copy and paste the link for the GitHub control net which is this one in here install from URL then press install and it says installed and then as it says here use install tab to restart we're gonna go into the install Tab and then we need to apply and restart our UI okay so all our models or the four models that we picked have download and we're going to move them you can either command copy to copy but since they're huge I just want to command X that would cut them and then we would go into our stable Fusion web UI folder you need to go into extensions web UI control net models and move them here with command V now they are installed into your stable Fusion control that so we can close this and we are going to go back into our stable fusion and now while we're in text image or image to image you can see here control that we're going to start with text images to generate a starting image look at the pencil sketch of ballerina so here we have a pencil sketch of a ballerina we're going to send this to image to image that will be our input so let's say that this is an image that you have sketched or drawn by yourself and now we're going to change this we want ballerina dancing in a colorful space nebula swirling with saturated colors I'm gonna scroll down and down here you can see control net now if you can't see any models here which is the case for me because I started my stable Fusion my controller before I put in the models you can just press the button here refresh models and if that is not working for you you can go back into extensions and apply and restart the UI and if you check your controller model you should now have the models that we have installed now there are some variations to the models here and I'm going to show you quickly here on another repos gear Tab and with the candy model you will get sort of a sketched paper sketch depending on your input here's the input image and it's a sort of a crude sketch and then it works from this to create this and the same with the dog here the scribble one is basically if you have an image like this our pencil sketch is not as crude as this but scribble could work well for us as well but basically Ms paint use the scribble one then we downloaded the pose and this is basically analyzing the pose of an image so say you have the character here and it analyzes the pose and then recreates the pose so that's just a pose and not the rest of the image and the last one we used was the depth map and then it takes the image and creates the Midas depth image so then you're gonna we have the image to image up here we're gonna drop it in here as well and we're gonna use the canyon for our first one use the preprocessor candy here as well so make sure that you use the same pre-processor as your model the weight is a value between 0 and 2. I recommend that you start with one and you can play with values between zero and one I wouldn't recommend going above if you go below you're gonna have more stylistic results moving away from your initial image so if you go low I say zero for example it's going to be closer to the prompt than the image is still going to be close to the image if you take one for example it's a fine it's a good compromise between the two it's going to be close to the image and still resemble the prompt we're going to change this to just resize now we also need to enable the control net now scribble mode or reverse color ignore this for anything that is not the scribble model it doesn't provide anything for you unless you try to be super artistic and you know play with it however you want for most people you're going to want to check in low vram 8 gigs or below as always in stable diffusion the denoising strength it is how much the image changes from your input so zero no change one a lot of change so we're going to keep this two well 0.85 for now let's generate and see what happens so as you can see here we got two images and the second one here is what control Nets candy model got out of this input and it's it's analyzing the image and takes the sketch here and it makes it you know lines that the computer can read and then it translates into R real image which is basically the prompt it's a ballerina dancing in a colorful space nebula and we have the exact same pose as in the first image so this is truly amazing and this is game changing and this will not only change the game for average users but for professional use as well where you have complete complete control of the final image now if we would lower the weight here to zero we would probably see a little more stylistic results but you can see that we are losing some of the posts we still have a similar pose but it's not exactly the same we've lost the arm and we've lost the leg here but it gets a little more stylistic but that's also well I recommend you to stick with weight one for now let's try this with the other models that we downloaded with the depth map now with the depth map here you can see that you have the outline of the image and you also have it's a great tone depending on how the image looks like now the pose is almost perfect based on our input let's try this with the open pose now the first time that you make an image there might be a slight delay that's because stable Fusion is downloading some of the files and you can check in your command window actually see here that has there has been downloads 200 megabytes 141 megabytes and here you can see the pose now this wasn't perfect but you got the hand here uh we got the hand out there didn't go there like the input we have the face and we have pretty short legs but we're still very close to the initial image now for scribble mode you can use text to image and I actually recommend that you can use text image for all of them so we're gonna start we're gonna paint the dog actually I want the penguin I'm just changing this to Karis 30. and again you have to enable control net and just remember since we're scribbling now you need to enable the scribble mode you can do it without but scribble mode makes it a little better right here scribble model scribble and then you can either you can paint in Ms paint or photoshop or whatever and just put the image in here or you can open the drawing canvas and that's what we are going to do we're going to draw here our little penguin I'm gonna try the body here I'm gonna put some sort of feet here and penguins have arms like this right something like that and you know a penguin head or a human with very short legs okay we have our penguin and we are going to generate this now we took our penguin sketch very literal so we got the back of a penguin sorta so I'm gonna try and give it a little nose here and two eyes see if that changes anything and it actually did so now we have a penguin and as you can see here the pose is correct um our penguin is fantastic let's try and see if we can is there an eraser here no there is not but we can take let's try and take the arm like this and just leave the other part in and see what happens I gave us three arms that's not optimal but hey it worked and it worked well so there you have it control net now that this really is a game changer and now we could have done the ballerina in text image here as well you can play around with it try texture image try image to image and see what works best for you this is a brand new tool so there's bound to be some experimentation ahead thanks for watching I hope you learned something today I have a lot of content on AI art stable Fusion mid journey and AI in general so check out my channel and the recommended video here as always have a good one see ya

Channel: Sebastian Kamph

Views: 495,670

Keywords:

Id: vFZgPyCJflE

Length: 11min 4sec (664 seconds)

Published: Wed Feb 15 2023