ComfyUI: Style Transfer using CoAdapter & ControlNet (Tutorial)

hi I am Mali welcome to the Channel I want to transfer my own style to this sweater by using multiple control net and the composable style adapter I can transfer and fine-tune a specific style take another example of this handbag not only that depending on the settings you can manipulate it further and add A Creative Touch to the generated output let me show you how to create the workflow and some hacks in comfy UI for stable [Music] diffusion a big shout out to all the members who joined we appreciate your support this is an advanced tutorial if you are new to comfy I suggest you get acquainted with topics like comfy Basics control net and installing the comfy manager you can find tutorials on all of them within the stable diffusion playlist on the channel page for the tutorial you must install some custom nodes and download co- adapter control net and style models the custom nodes can be installed from The Comfy manager the control net auxiliary pre-processors node is needed for control net I would be using the was node suit with python gos custom scripts install the image resize by palent and comfy math custom nodes the link for this page will be in the description below instead of the text to image adapter models I would use only the co- adapter models for the tutorial the reason is that these composable adapters with various conditions are aware of each other and work together to deliver a more powerful ful composability the style model for stable diffusion 1.5 is only available in co- adapter therefore control net models would also be co- adapter they work really well when all of them are co- adapters since there is no sdxl release for the style model all models for control net and checkpoint used will be based on stable diffusion 1.5 download these three models the color and sketch are control net model while the style is the only style model available for comfy and goes in a separate folder the color and sketch model files go inside the control net folder I have subfolders in my main control net folder for better management I would be putting the files in a subfolder called T2i adapter if you go back you can see a folder called style models the co- adapter style model file should be downloaded here the last Model is the clip Vision model which can be installed from The Comfy manager go to install models and search for clip Vision install the stable diffusion version 1.5 model and restart comfy let's start everything from scratch add the load checkpoint VA and the clip text and code nodes for the prompts I will now start creating nodes for the control net sketch group for this tutorial the main focus is to apply styles to existing images for the pre-processor I will add a single node which allows me to select a single pre-processor from the entire list the image resize node is very critical here to get varied results in the last workflow example I will demonstrate how applicable the node is for this workflow this is a custom node and you should search for image resize without space it is essential you select the basic apply control net instead of the advanced node this is because of the conditioning only the positive conditioning has to pass through the control net multiple control Nets would connect to the apply style node and any negative conditioning here messes up the style output I have tested this extensively the negative conditioning should go directly to the case sampler only connect the remaining inputs in the control net node now I am going to create the nodes required for style transfer style is not a control net function I am duplicating the image nodes from control net as we would use an image input to transfer the style wherever using an image input make sure to have the resize image node next to it for cropping there are two nodes for style the style model loader and apply the apply style should always be last I will explain this later when I set up multiple control net units currently I have only the sketch control net the positive conditioning output from that node should connect to the conditioning input here add the clip fish and loader and and in code notes the image resize connects to the clip Vision in code node adding the K sampler nodes use the basic node and not the advanced one I would consider the output from this group as the raw image since it uses SD 1.5 and is low resolution upscaling nodes would be added later to improve the image quality and details there is a trick that I used to organize the output for the saved images convert the file name prefix to input and connect it with a primitive node any code here starts and ends with the perent percentage symbol this is the code for the date you can change the order in whatever format you like the date to be in a forward slash would mean a subfolder this will make a subfolder within your output folder the folder name would be the date format you can create as many subfolders as you want just keep putting the forward slash after each code or text now I will Define the file name for the the output I want the file name to start with the workflow category I also like the seed to be included in the file name you can pull the value from any node for example I have a node called K sampler so I will put percent K sampler. seed and close it with the percent symbol again if you want the steps value to be included then instead of seed type steps this this way the output file name and the subfolder can be customized as required the realistic Vision version 5 checkpoint is excellent for this workflow I don't recommend the Bas SD 1.5 this checkpoint sufficed for most things I wanted to do during the tutorial later I will show you why and when you should change the checkpoint leave the positive prompt blank as of now I am inputting a simple negative prompt you don't need to complicate The Prompt here as styling is done from image to image we need the sketch of the first image before passing it through the style node for sketch use the pet pre-processor during testing I noticed it's best to keep the resolution at 1024 or 2048 we will then downscale and crop it using the image breiz node the reason I am using a higher resolution is because this pre-processor doesn't Define the sketch lines properly at lower resolutions this will drastically affect the output the subject outline is way way better in higher resolutions if you get an error stick to 1024 or lower also note that not all pre-processors support this resolution depth for example won't go higher than 1024 so keep that in mind when using other control net units in this workflow select crop to ratio for action the larger side should be 1024 keep the resize mode as any leave the rest of the settings at default values for now select the co- adapter sketch as the control net model all control net models in the workflow should be co- adapter only connect the positive prompt to the control net conditioning input ensure both the image resize node settings are the same later I will show you how the crop pad position value affects the output choose the correct SD 1.5 version clip Vision model okay the positive prompt will go to control net and this node will be connected to style conditioning input connect the style model node with the Positive input on the case sampler the negative prompt directly connects to the K sampler's negative input ensure you correctly connect the VA that is from the load vae node and not from the checkpoint output connect the model to the case sampler as well first let's try and check if the workflow works I want to see the sketch pre-processor preview before I generate the image the style group nodes are disabled change the ratio to three is to two for all image resize nodes that's centered let's enable the style noes and check out the preview first that will do since the ratio is 3 is to 2 the width should be 768 remember this is non sdxl so stay within the 512 into 768 limit now let's test with some random Generations this is how you should typically proceed keep generating from random seeds once you are set on the composure use a fixed seed and play around with the sampler and Schuler all the schedulers and Samplers combinations will give you different results I cannot recommend anything specific because it depends on the source and style image for demonstration purposes I am putting a custom seed let me change to an uncommon sampler and scheduler and see what happens the other value that you can change is the control net strength changing the value here will not affect the style hence all the control net should be before the style this would give you a high level of control over how much weight each control net gets without affecting the style transfer technically the ideal range is from 0.5 to 1.5 in practice however I mostly found myself using values between 1 and 1.5 and on rare occasions around two there is an improvement in the sweater design change the control net value with an increment of 0.1 or 0.2 at a time sometimes it's hard for the AI to define the object and you may get weird results at this point I recommend that you define the object and shorten the positive prompt also sometimes it may just give you something better let's put the word sweater in the prompt and see what happens putting in a positive single word prompt actually made it look like a sweater I will now add another control net for the color I will duplicate the control net sketch nodes as the workflow is the same always start with the default value of one and then change accordingly select the co- adapter color change the pre-processor to color as well let's check the preview for the color control net first looks good let's proceed to test and see how the colors affect the generation nothing's changed oh I forgot to connect the color control net node okay the first sketch control net will connect to the color control net node and that will connect to the style node input so to recap this is how I have set it up the positive prompt will combine with the sketch control net group this info will be passed onto the color control net and then these parameters get passed on to the style group after this the raw image nodes will process and decode the final image let's test and see not only has it applied the color but it has also changed the generation keep this in mind a lot of small factors are going to affect the generation however this is a low resolution image let's see if we can fix that I don't just want to upscale the image I want to add details to it as well when doing that it's preferred to do it in latent space rather than in pixel space so let's add another K sampler and nodes for adding details and upscaling I will rename the node to K sampler upscale we'll explain in a bit why search and add latent upscale node here I am going to change the file name to include latent upscale so that I can recognize it clearly also change the node name from case sampler to case sampler upscale this is why I changed the name of the second case sampler if you don't do that it will take the seed from the first case sampler connect the first case sampler latent output to the upscale latent input node now I am going to extend the model VA and the conditioning rout and connect them to the second Cas sampler node so here I am going to do some math nodes to automate the upscale basically the process will allow you just to put 2X or 4X automatically determining the raw image resolution and pushing that upscale resolution to the upscale latent node the only disadvantage is that you won't be able to do batch generation as it can accept only a single height and width and will give errors when doing batch Generations you need to deactivate the math nodes and manually put in the upscale resolution for this workflow personally I prefer single Generations convert the width and height to input search and add image size to number node I will connect the vae decode from the raw Image Group to this node it will get the width and the height of the generated raw image Now search and add a node called number to text twice this will convert the height and width to a text readable format add a text concatenate node connect a and C inputs for the B input add a primitive node here put space the letter X then space again connect the output to a show text node let's check oops okay I did not connect the height let me correct and test again now it's working to add the scale multiplier go to add node math then integer and add the integer binary operations node twice select multiply and convert A and B values to input connect the width and height integer to the a input add a single primitive connecting both the B inputs this is the scale factor here all you have to do is change the value to 2X or 4X and it will Laten upscale automatically the first binary operation gets the width and the scale factor it then multiplies A and B and gives that value to the latent upscale node the same process happens for the height in the second binary operation connect the integer binary nodes to the width and height inputs of the latent upscale node respectively I have put a value of two if you only want to add details and not up scale change this value to one I do not recommend going higher than two for this step keep the K sampler upscale settings like the raw image K sampler except for the seed since since this will denoise the image and add details a different seed will give different results here the process would be to do a random generation and if the details have artifacts increase the Deno value once you get the desired Deno value you can then change the seed and regenerate for variations typically a Deno value of 0.3 Works however in latent upscale it won't that value is not high enough to denoise the entire image see what happens with a denoise value of 0.3 it was not able to denoise the image entirely a value higher than 0.52 sometimes 0.55 works but the details will change the image slightly it's a small tra trade off for the sharpness if however you don't want to add details disable the node and only use the raw image let's try a value of 0.52 way better but there are still artifacts let's try 0.57 excellent let me change the seed and increase the denoising value to 0.65 perfect now to upscale again but without adding any details for the high resolution upscale I will duplicate all the math nodes add a note called upscale model loader I will be using the 4X Ultra sharp upscaler this can be installed by The Comfy manager from the install models option now add the image upscale with model node add the image scale node the image up scale output should connect to a vae encode further connecting to a vae decode node modify the final save image file name as desired connect the vae decode output in add detail group to the image size to number input also connect it to the upscale image using model node connect both these nodes as well change the width and height to input and connect the integer binary op operations output respectively finally use the reroot vae extension to connect to the encode and decode inputs Let's test the highres up scale before I move on to show some examples you can also add a background remover to isolate the sweater for the background removal add a note called image REM BG turn the transparency on or off for this workflow I prefer it to be off with a black background via this node you can also add a white blue or green background turn on postprocessing and Alpha matting what I am doing now is creating an extended pipeline from the output VI decode nodes from the raw image add details and the high-res up scale groups now the workflow is complete and organized if you want to deactivate highes and add details you can directly and easily connect the raw image to remove the background if required as shown previously connect a primitive node to the save image input and let's test the background removal let's move on to some practical examples and tricks you need to know in this example I am using a bag image as a source it cannot apply the style to the bag properly let's increase the sketch control net value to 1.5 that did not have any effect let's try and reduce it to 1.2 and see if it has any effect okay nothing let's add the word bag in the positive prompt and see if it corrects the issue that worked let's see what happens if you eliminate the color control net completely the color here Works differently and it doesn't mean it will make the whole image in a specific color tone the color will blend with the style color and give a combined effect color control net can be used to fine-tune the image style let's check out this motorbike example I am putting a one-word positive prompt control net color is disabled let's add the color and see what happens this is properly defined even with a control net sketch value of one however when you want a creative outcome you will need to reduce the bike definition as a subject let's remove the motorbike from The Prompt and see what happens this has overlaid the style perfectly however the headlight of the motorbike is not defined let's fine-tune the image by increasing the sketch control net to 1.2 perfect let's add detail and upscale beautiful now now let me show you another hack I will change the latent resolution to 768 x 512 wow even though all the input images are cropped at a ratio of 1 is to one you can still manipulate the latent space ratio this is because on both K Samplers raw image generation adding details and upscaling happen in latent space instead of pixel space the AI is still denoising and would compensate for the ratio without Distortion so play around with the latent ratio and see if you get good results I had a lot of trouble transferring the style accurately to the vase as you can see the results in the Raw image are nowhere close to the selected style image let's see if making the prompt blank does anything there is no accurate style also it seems to have distortions let's add details and see if it can be fixed okay this is what I am talking about I kept randomizing this a lot and it was like one out of 50 images I got it right let me change the seed and show you the closest result I got then I thought I would try it out with a different style image the wood texture gave excellent results so apparently I learned the style texture and design matters with the object material in the source image even though this face is nothing like the source image this was the most accurate I could get you may come across some images which are extremely difficult to transfer the style however in my testing it was only this one which could not be fixed by any means I know I did try another vase image and it was working fine this is the last example for this tutorial even though the source image is realistic the style transfer output is a vector I guess this is because of the style image let's try increasing the control net weight to see how it affects the output it is the same only a different kind of vector I will change the style image to a carbon fiber material and see what happens oh that gave the car a realistic metallic smooth look even though I cannot see the carbon fiber pattern anywhere the color control net is disabled let me activate it and see what happens something doesn't seem right the color color transfer on the car is a bit weird however in the background it is smooth during testing I learned a trick the sketch cropping is set to 0.5 if I change it to one the sketch actually moves slightly up towards the center this may seem mild but see what happens to the output did you see what happened there let's change the checkpoint I am changing the vae to the one recommended on civit AI for this checkpoint looks muted funny thing though I reverted back to the other vae and was surprised at the results the contrast and colors are way way better this was purely accidental and not intended at all I got to thinking that if the sketch cropping made a change we can also try changing the color crop ratio to get a specific portion of the color and spread it across the entire image for a better blend let's see if that works well that worked very well so these are some of the tips and tricks you can use while using the style Co adapter I hope this tutorial was helpful until next time [Music] [Music]

Channel: Control+Alt+AI

Views: 4,216

Keywords: coadapter, controlnet comfyui, controlnet, comfyui, comfyui tutorial, stable diffusion, stable diffusion tutorial, style transfer stable diffusion, comfyui workflow, comfyui controlnet, comfyui nodes, img2img style transfer stable diffusion, image style transfer, style transfer adapter for controlnet, style transfer controlnet, comfyui upscale, comfyui node workflow, comfy ui, comfyui hi-res, comfyui controlnet workflow, t2i adapter controlnet, controlnet style transfer

Id: 58X92pypai8

Length: 45min 12sec (2712 seconds)

Published: Sun Dec 17 2023