Change Image Style With Multi-ControlNet in ComfyUI 🔥

hello everyone today I would like to talk about multicontrol net within comi I know that many of you find automatic 111 more user friendly and I do as well but we confi we actually have more control over the generated image so sometimes it can be more useful to use that if you want to achieve better results or more professional results so that's why today I'm going to use that for changing a style from a realistic style to an anime style and I'm also going to show you a little trick for removing the background using control net let's have a look at it so this is the workflow I would like to show you today it doesn't consist of many notes it's actually showing you just how to use control net really I'm going to share my workflow if you don't have some notes you can download them it's going to be super easy if you have confi manager which you can find in this GI p page here and you can install it following this simple installation process so you will have to go into your confi folder then confi and then custom notes and then here you can type CMD for opening the common window and then you just need to get clone so you just need to copy this second line here pasted in here like this and then you need to press enter after that you need to restore it confi and you should be able to see this manager button in here so if you click that you can then click on install missing custom notes and here you will have a list of of notes you don't actually have if you're not able to install some of them through this process you can still go into manager and then you go into confi notes info and then you search for the name of the box I use so for example in this case I have cr multicontrol nestack so you can go here you type CR control net stack actually it's multi control Net stock in here you have the reference the title and all of the information the description for downloading them you just need to press on this reference which will open the G page and then you need to if you scroll down you will just need to get clone again the URL of this page if you don't have confu manager and you don't want to install it you can Google the name of the of the Box I've used and and then you can find the ga page and install it again using the common window for this tutorial I'm going to use one of the images from pexels for who doesn't know it's a website where you can find free images in free videos so it's quite useful when you are testing or trying different things with stable diffusion and I actually use this picture here then just to be more precise the packages I use for building this workflow are more or less this one you can see here in my folder the most important I would say are confi comy roll costume notes from which I got the CR multicontrol net stack DW posst for the contet preprocessor for open post it's a little bit more efficient than the you know the basic open post preprocessor so i' I've used this one I've also have this efficiency notes confi for now I just use the efficient loader and the K sampler but it's usually very useful for the high resolution fits for the XY plot and it works also with adate diff which is which makes the difference and then for having everything a little bit more organized I have also this package was not sweet kuui which has different actually different features but the the most important I use them is to have the straight lines between each node the control net models I'm going to use are exactly the same I downloaded for stable diffusion 1.5 for automatic 1111 and you can find them on a g face so now let's have a look on how I built this workflow the first note is the load image so in here you just need to upload or drag and drop the image you want to use for your contet model and then given I'm uploading an image an actual image a realistic picture I need a pre-processor for generating a a mask from this image so that the diffusion model can use that mask for generating the image you know there are a lot of mask you can generate and each of them allows you to generate a different picture based on different characteristics of the input image so sometimes you want to use only one type of pre-processor only one type of control model at the time you may want to use more control net models Al together for controlling the depth of the image the color of the image or the shape of a person of an object and what I want to do now is to generate a lot of mask in order to analyze them closely and understand which one I want to use if I want to use one of them two of them three of them or none of them so in order to do that in this control net pre-processor I have included all the pre-processors for different control net models so I have the pre-processor for realistic line art Kenny soft headge scribble normal map depth and open post and then to each of them I have attached a preview image so that I can see the mask created through this pre-processor so I think this is a good starting point for understanding which type of control net we want to use oh I also have tile here so what do I want to achieve actually I would like to transform this picture into an anime picture right so if I look at this control net I'm pretty much sure I'm not going to use this tile one because otherwise it will stick too much to the realistic original realistic picture I don't think I'm going to use scribble either and probably I'm not going to use keny or maybe yes I don't know so let's have a look at it so what I did I have added the CR multic control net stack and with this we can actually control which control net model we are going to use so let me bring this back to default which will be like this let's remove this and this for a second and let's assume that we want to use line art so what you have to do you simply need to connect the line art pre-processor to image one in this CR multicontrol Net stock let's zoom in a little bit we need to switch these two on and we need to choose the line art mod model again I downloaded this from the hugging face website and then I mve them inside here so confi models control net and I have my models in here and then this is the control net strength which corresponds to control net weight in automatic 1111 so how much weight we want to give to this control net one means that you are giving the full weight so if you look at this mask probably the output image is going to have this the these exact details which probably we don't want because we want the final image to change a little bit compared to the realistic input image right so maybe we want to reduce the weight of this of this model to 0.7 so like that we are using only line art it doesn't matter that we have all of these mask here these are just as a reference for us to understand what we want in order to use them you will need to connect the preprocessor to this multicontrol net stack let's assume that we want to use open pose as well I have open pose down here so I need to zoom out I'll grab this here and then like this I'll connect it to image two and then if we want to actually activate it we need to switch the second one on choose the open post model and change the strength if you want to I'm not going to change it I'm going to leave it like one so full weight on open pose and 0.7 online art then let's see where this is connected to this is connected to my efficient loader so from control Net stock to control Net stock this is pretty easy then in here you need to choose the the main settings you have the checkpoint name I'm using cardos anime from CV is this one I'm going to use this variational out encoder you can use whichever you want to which I downloaded from hugging face as well I'm going to use clip skip actually equal to minus 2 I'm not going to use a Laura model so I'm not going to change this to anything I'm going to choose my my prompt which is coherent with this cardos anime so if you look at the the generation they down see you will you will see which are the best settings for for this model so that's what I used the same for the negative prompt and then I left the other settings as they are the only change I made I have included this CR aspect ration this is not needed it's just for having an automatic aspect ratio so I chose the 5 and 12 by 768 and I connected the the first two integer to this empty and WID and height so make sure that WID is connected to the width which is the first one and then the height is connected to the he so it's going to be the second one so this one again this is not needed because when you will have the efficient loader you will have actually the WID and the height here let me see if I can show you an example of that yeah you can yeah you can see that in here you have with and height included in the efficient loader so it's really up to you and then I'm connecting this efficient loader to the cas sampler so the model to the model the positive condition into the positive negative condition into negative Laden image to Laden image the variational out encoder to the optional variational out encoder and that's it now on this case sampler what can we do so we can decide seed I'm going to use a random seed minus one let's zoom in a little bit you can decide how many Sly step to use I'm using 20 the CFG scale 7 you have the sampler DPM PP carus you have the the noise and I think that these are the most important ones and then you also have the preview of the image you're are going to generate so given you have the preview but you are not saving it I have included the save image in here and and then just to compare the initial image to the final results I've just included this load image here with the image I uploaded before okay cool this is most of it so once we are ready with all of our settings we can just press on Q prompt and this is my result Now using again line art with 0.7 and open pose now let's have a look to our output so it seems pretty good but I'm noticing something weird I can see a person here and I really don't want that person but why am I seeing that person like I would assume it's from line art right because it's very very detailed so if you look at the mask there is actually something here but I'm not sure how the model can tell that this is actually a person really uh even because we are using a not a full weight weing 0.7 so I would assume it's from open POS then let's have a look at the open and post mask and we can indeed see that the the pre-processor here is detecting someone here on the left side so there are two options for avoiding having this person in background and in general to have a different background we can use depth which is this one right however if we use open pose and DEP together we'll probably end up with the same issue right because you're still using this mask which includes this person in background so or you don't use open pose or you reduce the weight of open pose or we can use the depth map in another way let's have a look how it's quite useful when you want to use the depth map in combination with also liner for example and you want to remove the background let me zoom out and you will see I have here a remove background section I have a convert image to mask and an in paint pre-processor initially just to show you like what I want to achieve right which is this one where you have the person masked in front based on the depth mask but we don't want this right we want the opposite the inversion of that so we would like the mask to be applied on the person not on the background so we can invert the mask using invert mask again using the in paint preprocess cessor and then we'll get the opposite of what we had before so we are masking the person and not the background then on this picture this new picture we can use the DW pre-processor so the pre-processor for the open post and we'll get a new open post mask which is this one then instead of connecting our previous open post mask which was including the person in background we can connect this mask to our control net stack so let's do out again I'm going to drag this in here image two make sure that the model is linked to the exact number so in this case it's image two because I have open pose in control net two if it was one I should have put it in the one so now based on this new mask here I'm going to cute the promp again and we can see that the new image we are getting doesn't have anyone on in the background and I actually have a three because this is what I described in my Pro prom description I have trees in background quite cool isn't it this is how you transform your images in this case I'm using just through control Nets but feel free to use how many you want in this control Net stock you have only three image you can images you can link but what you can do you can clone this cont control Net stock and you can connect it like this so you can connect the two of them and then you Conn add the last one to the efficient loader and in this case you can add so three plus other three so they are six control net overall so using more than one control net is usually very useful mainly when you want to create a video not just an image then for creating videos now we have more advanced technique for example anime diff or warp fusion super good for generating more stable videos with not much the flickering but if you want to generate a very simple video a quick one you can simply use image to image using different control net models for generating the image and then maybe applying the flickering using D Vinci or adobi so yeah that's it for today I hope this was useful thank you very much for watching and see you at the next video bye

Channel: Laura Carnevali

Views: 5,230

Keywords: stable diffusion, diffusion, ai art, diffusion model, generative ai, generative art, stability ai, ai artist, imagen, install stable diffusion, stable diffusion with python, stable diffusion hugging face, stable diffusion github, stable diffusion v1.5, stable diffusion tutorial, stable diffusion xl, refiner, refinement stage, new sd, sdxl, sdxl base, controlnet, adetailer, sd extensions, realistic, openpose, depth mask, mask, canny, lineart, lsd, hed, segmentation, improve hands, scribble

Id: foCrow4IgaY

Length: 17min 1sec (1021 seconds)

Published: Thu Oct 26 2023