Style Transfer and ControlNet with InstantStyle IPAdapter V2 in ComfyUI | SDXL Stable Diffusion

hello everyone in this video I'm going to talk about a cool project I found the project is called instant style basically what we do is that we upload an image here we give it a promp and then it will automatically apply the style from the image to that prompt so it will look something like this I'm just clicking on the example down below and this is the style The PRP is a cat and then we have a cat in this particular sty if we go down we have another example here you can see the style here again the same promp a cat and we have it in that particular sty if you want to look at this project I will have a link in the description below where you can go to huggingface doco and then you will be able to play around add your own images and just generate in today's video I will show you how we can do the exact same thing inside of confy UI okay so here I am inside of confy UI now in order to complete this project you will require three custom nodes the first one is confy UI Essentials so you can go to your confy UI manager click on install custom NES search for Essentials and then you will have it install it by clicking on the install button on the right side the next one that you will need is the confy IP adapter plus most of you already already have this but just in case again go back into the confy Y install custom node and search for IP adapter and then make sure you're installing the confi IP adapter plus version now the last custom n that you will require is the control net custom node so let's go into confy y manager install custom node once again at the top search for control net and you will get quite a your results look for the one that says control net auxiliary pre-processors that's the one that I will be using but you can also use the advanced control net now for this one I will be using the ID number seven one all right once you have all the custom nde installed everything should be good by the way I will have a link in the description below where you can download this exact workflow you can simply drag the workflow inside of confy Y or you can go to load and select the workflow it will open if after opening the workflow you still find that you have missing custom nodes then go into the manager and then click on install missing custom nodes by the way the IP adapter will require you to download the different models so make sure to download the models and the control net as well I will be using the sdxl canny control net but if you have others you can experiment with it it will also work with SD 1.5 checkpoints okay so I've cleared my canvas so that we can start fresh let's click on the load default to just get the default worklow for my checkpoint I'm going to change it to an sdxl lightning full steps checkpoint just so that I can get faster generation the latent I will change this to 1024 by 1024 now for the steps I have to go for steps CFG is going to be one sampler name will be Ula and then scheduler will be sgm uniform everything else can stay the same I'm going to go back to the project the instant style project and then I'm going to copy this prompt here go back paste it as my positive prompt so I'm going to copy the negative prompt paste it inside the negative prompt section and I'm just going to get a random s by disabling the save node and clicking on Q prompt you will see that this seed will change see it change click again it change again so that's like a trick that you can use if you need to generate a random seat without actually generating an image now I'm going to change the control before Generation to fix okay I'm going to click on the Q promp it's completed and well it's pretty good already because um this checkpoint is actually quite good now let's see how we can add style to basically the output image now we'll have to go to the lad checkpoint move a little bit just make a little bit of space and we'll need to add the unified IP adapter unified load up so the model your load checkpoint the model will go into model here and then this output model we'll go into IP adapter I'm going to use the simplified one it will get the job done this IP adapter will go into IP adapter we'll need an image this is going to be the style that we want to apply to our final app with so just take the image drag out and select load image now I'm going to copy this example image here I'm just taking a screenshot you can also save it and load it inside of confy UI now the IP adapter needs to connect to the K soundplug so we'll take model connect it to K sampler here and it will look like this so we're basically Bridging the blue checkpoint make a bridge goes to the IP adapter which goes into the key samp everything else is the same now one thing that is important is the IP adapter options we have the weight that's going to influence how much much of the image that we want to use and affect the final output have start at and at usually the initial generation is the most important part so if you want to give the model a little bit of flexibility you can start at zero and end at let's say 80% the remaining 20% will not use the IP adapter and will just use the normal checkpoint so you can have a little bit of variety like this now the weight type is similar to these options that we have here so in the in the project the instant install project there's a section that says install mode and it can do install block the layout and then the original Ip adapter in our case the weight type standard is going to be original then we have style transfer which is basically going to take the sty from the image and apply it to the output and then we have the prompt more important which is going to focus more on the prompt now there's also IP adapter Advan which allows in the weight type which allows for a strong sty transfer as well as a composition it will basically take this composition and put the cat usually in the middle okay so this is for the style transfer I'm going to change my weight type here to install transfer then because I'm using an sdxl model I will change this to standard it it works pretty good and then everything else looks good so I'm going to click on the Q promp all right it's done and it appli the style let me open it in a new tab and let's get the original as well so this is the one with the style and this is the one without thear so it's pretty cool it's almost the same cat and that's pretty much it when it comes to applying a particular star to any image so going back to the project here there is an advanced section where we can add another image like a second image so we have the style at the top and then another image this one will be used as reference for the the pose and it will apply a cany control net so we can take a look at this example here where we have this as the style we have this as an initial image and it will apply this style to this image like so and for that we are going to use control Nets now control net will go in between the positive negative conditioning and then the key sampler so we'll make some space and double click at the apply control net the apply control net gives you the option to add the positive prompt to the apply control net and then it goes into the key sampler there's also the advanced apply control net which allows you to pass a negative prompt to the control net I'm going to keep things simple and use the apply control net the conditioning output will go into the positive prompt the positive prompt from the clip text and code will go into to conditioning now we are missing the control net model so take control net drag out select control net loader and then select the cony model that you have valuable make sure that it is the sdxl version if you are using an sdxl model now lastly we need the image so we'll take the image drag out select another load image and again I'm going to go back back to the project and I'm going to just take this as an example so this is going to be the sty and then this picture is going to be the control net let's also get the negative promp same thing the positive prompt now at this stage because we're using the cany model we need to make a basically do an edge detection process using the cany algorithm so so we'll take this image and pass it over to a py edge preprocessor now because we are doing a 1024 I'm going to change the value to 1024 we can output a quick preview here but this image will go into the apply control net so let me clean up the workflow a little bit maybe bring all of these down so what we have here is that we have the normal workflow in the middle then we are adding the IP adapter here and then below these nodes they are the control net nodes now we can disable the output and click on Q promp just to see what the cany model will do and basically it will try to restrict the image to this particular outline I'm going to reactivate the final output and then click on the Q prompt all right it's completed and and you can see we have the sty from this and then we have basically the the shape the glasses controlling the image with the control net section now in case you're wondering yes it does work with SD 1.5 and if you're using the workflow from the description below so the version I've shared in the description below you will see that next to the L checkpoint there is a width and a height and if you're doing sdxl you want to change this to 1024 by 1024 this is going to be your 1X one or square shape aspect ratio then input your control net image your style image change this positive prop here and then you should be good now if you're doing SD 1.5 then if you are doing a an image that is vertical so in this case it portrait image then instead of changing everything you would change the width and the height to 768 by 768 just to get the optimal resolution for SD 1.5 now even though we have it as a square 1 by one so 768 by 768 it will automatically resize and maintain the aspect ratio will pass the final output to the can refine Edge and it will also send that over to the cas ampler as empty latent so as you can see even though the style is a square the control net is Portrait the final image is also portrait and um it basically kept the the same pose like this and the same overall shape and then we have this particular style here if you are doing U SD 1.5 but you want to do like um a control net image that is square then you will do 512 here by 512 or you can do 768 as well depends on the checkpoint if it can generate images at that particular resolution of course do not forget to change the K sampler if you're using um the sdxl lightning usually you would use a l CFG and then low step count if you want more speed you can actually just move this lad checkpoint a little bit to the left drag the model out and then select Lowa loader and from the Lowa loader you would select a an LCM checkpoint depending on uh what is the primary base checkpoint that you're using so SCCM 1.5 or sdxl and then make sure to connect the clip the model will go to the IP adapter section like so the clip will go to your positive and negative and that's it you've connected a Lowa to it of course if you don't want to use the LCM Lowa you can use any other luras that you would like to use in order to affect the final image now you can also use uh face ID so if you want to change the face I will have a link to the workflow in the description below and then you can make these type of images basically uh of course you need to change the pr by use the face I've used my own face but I've kept the promp as a woman and then I was using control net of a a woman there but basically get the point there you can keep expanding the workflow depending on your on your need and U yeah that's it I will have a link in the description below so you can use the workload so that was how we can create this instant style inside of confy UI if you enjoyed this video learned something new give this video a thumbs up share it if you know anyone who would benefit from this video and if You' like to support the channel I will have a link to my patreon page where you can get additional benefits from there with that thank you for watching I will see you in the next one

Channel: Code Crafters Corner

Views: 1,075

Keywords: comfyui, Code Crafters Corner, CodeCraftersCorner, ComfyUI, Stable Diffusion, InstantStyle, Instant Style, Transfer Style, ControlNet, FaceIDV2

Id: lWx4bontDhk

Length: 15min 48sec (948 seconds)

Published: Fri May 03 2024