ComfyUI ControlNet Tutorial (Control LoRA)

hello there I am Seth and welcome to the channel I took this rough sketch of an astronaut and generated this image you can also take these two AI images for example and using multiple control net models combine elements to get a completely new image also imagine how complicated it would be to color this black and white photo well you can do all this with control net and for UI and yeah it is kind of complicated but I will simplify it and show you my own graphs and workflow in this tutorial I will focus on four control Laura models for comfy UI officially released by stability AI these are not Laura in the actual sense the models are efficiently fine-tuned and are way smaller in size the these models require some unique nodes and comfy and their settings also have a learning curve but they are quite powerful and have a more comprehensive range of settings than automatic 1111 if you want to learn the basics of comfy then I have made an extensive 101 tutorial that explains each and every node and the connecting noodles in detail I will leave the link to that video in the description this tutorial is a bit Advanced and I will try my best to explain and simplify the graphs and the workflow I use in comfy the Json files for the same including the images generated will be available in the members only post on YouTube so to start off you need the comfy UI manager I did explain the installation in the previous video but I will explain again go to this web page click on code and copy this URL then open up your comfy install folder and go to custom nodes over here right click and open in terminal type in get clone and paste the link you just copied this will clone the GitHub repository of the comfy manager in this folder open up comfy and make sure that it's updated also ensure that your custom nodes are up to date before proceeding you can do that by clicking on update all in custom nodes search for control nit for this tutorial you need to install the auxiliary pre-processors for control net after it installs close comfy via the browser and command prompt you should first download the control Lura models before starting up comfy again for the models go to this hugging face page here you can see two control luras ranks the 128 ones are smaller and less precise download those if you are very low on resources both these models are way more optimized than the default models which are around 4 GB each for the tutorial I would be using the rank 256 models the recolor model requires two extra custom nodes which I will get to later in the tutorial download all four models as I will cover the settings and the graph workflow for all of them the models have to be downloaded in this specific folder take note of that I have created subfolders for better organization so I am putting all four models in the control Laura subfolder but you can just download them at this location after installing the comfy auxiliary custom node I get this registered path in the command prompt just ignore this as it is not a problem this goes away when you disable the custom node this started appearing only after updating everything to the latest version this specific warning has to do with the DW pose pre-processor model I am not covering the DW pose at all in this tutorial but I will explain to you what the issue is and how to fix it the DW pose is more advanced than the open pose control nit and comfy to make it work you need to install the on next runtime either for the CPU or GPU go to the python embedded folder in comfy rightclick and open in terminal type the following to install on next for the CPU if it is already installed for the CPU and you want to install it for the GPU instead you can uninstall it by typing the following then type this to install for the GPU the problem is that it does not work on the GPU if you have Cuda 12 and above which is required for torch 2 if you go to their website the latest version 1.16 supports Cuda 11.8 only so if you want Torch 2 you have no option but to run the DW pose on the CPU unless they release an update there is no way to run it on the GPU you can verify that it is installed on the CPU correctly inside the custom nodes folder there is a folder called comfy control netox open the requirements file in notepad and add this text on the last line then save and close now run the install.bat file which should show requirement already satisfied you are good to go and can run comfy ignoring that warning on start okay I have divided the workflow into three parts all of them are unique and have different nodes in the first workflow I will start from scratch and for the second and third workflows I will adjust nodes as needed let's start by adding a load image node the image output should be connected to a control net pre-processor node for the first workflow I would showcase the Midas depth map control nit model Midas has a depth and a normal map both of them are different the depth map encodes the distance of the elements in the image from the camera's Viewpoint black means very far off and white means very close the normal map on the other hand is used to measure light Direction on a Surface in other words it shows how the light bounces off and interacts within the image using the light map in conjunction with the depth map can add more realistic lighting and surface details to the image for this workflow I will only use the depth map for now whenever using sdxl it is best to use the Default Resolution of 1024 now I will add a checkpoint not mode I am using the real Vis XL checkpoint this model can be downloaded from hugging face or civit Ai and is still in training it's about 30% completed and so far I found it very good for photo realism now we'll add the positive and negative promp nodes to duplicate hold down the alt key while selecting and dragging the node connect the model output to the case sampler and let's select the advanced sampler here in case you load a checkpoint that requires a different vae you can do the same by adding a node from loaders load vae you can select the model from the list and connect the output to the vae decoder directly for this checkpoint I don't need a separate vae for control net add a load control net model node remember I said earlier that you could organize the control net models into folders they show up in comfy it becomes really easy to select when you have a very long list one thing you should note is that the control net model should always match the control net pre-processor if these two nodes do not match you get weird and distorted results the control net output should be connected to the control net apply node and I prefer the advanced one here as well the strength value defines the control net weight that you want in your image the start value percentage is when you want the control net model to kick in during the generation process for example a value of 0.5 would mean it starts using the control net models at 50 % of the steps defined in the case sampler in the same way you can also Define the end steps this comes in very handy when using multiple control net models you can Define one model to end at a specific step and the other to take over from that step the positive and negative outputs connect to the control net inputs you then connect the control net positive and negative to the case sample inputs the image output on the pre-processor will connect to the apply control net input there are multiple ways you can connect the latent image input on the case sampler one way is to load an empty latent image and feed it to the case sampler however for control net I do not recommend that this is because the empty latent image is not in encoded with any data what we actually want to do is push the data from the loaded image to an encoder and then pass it onto the sampler in order to do that correctly search for a node called image scale to Total pixels and connect it with the load image output leave the setting as it is and connect this node to a vae encode now connect the latent output from this node to the latent image input in the sampler connect the vae output on the checkpoint with the input let me explain what I did here the load image has pixel data the technical term for it is called pixel space the vae is called a variational auto encoder it is basically responsible for transitioning the data in the images between the pixel and latent space so basically the vae en code node takes the data in the pixel space and transforms it into a compressed encoded form capturing the essential features of the image the output of this process is known as a latent image when we connect the latent output from the sampler to a vae decode node it does the exact reverse it takes data from this latent space reconstructs it and transforms it into pixel space the process output is an image which is the final generated image we see to finish the graph connect the checkpoint vae output to the input in the decode node the graph is now complete let's add positive negative prompts change the sampler settings and generate the image nice I will show you how to upscale the image as well I use the ultimate SD upscale from coyote you can find it in custom nodes if you don't have it after installation save your workflow and restart comfy from the command prompt first delete the connections to the preview and save image nodes drag out the connection from the decode node image output and search for ultimate SD upscale I have made a full tutorial on how to use this node in comfy the tutorial covers text to image and image to image upscale workflows I will leave the tutorial link Link in the description below here I will take the final generation and do an imageo image upscale without adding any extra details connect the relevant inputs from the checkpoint and the [Music] [Music] prompts the ultra sharp and the full hearty upscaler both work well here connect both the preview and save image nodes to the upscaler node and generate the image now we have a 2X upscaled image here are the results for comparison focus on the black circle on all three images and you can see how the building structure is maintained via the pass through method now if you focus on the the white circle you can see the image via the pass through method maintains the color tone and temperature whereas the empty latent image is way brighter also notice the hair lighting within the black circle the pass through is more close to the original than the empty latent this is why you should use a pass through for controling net instead of an empty latent unless you want a wholly restructured image these results will vary to a greater or milder extent depending on the checkpoint the prompt and the workflow Al together let's try multicontrol nit using Kenny Edge as the second model I will rearrange and duplicate some nodes I want a pirate style dress and hat candy edge control net is perfect for outlining these elements here the first control net positive and negative conditions will pass through the second control net node the second control net node will connect to the sampler node you can connect as many control net nodes as you want [Music] the Kenny edges come out very sharp to balance out the Kenny effect I will reduce the strength to 0.1 also I want the second control net to start applying from 50% of the steps I am not changing the first control nit apply node as I only want certain elements from Kenny without drastically changing the image adding pirate cosplay to the prompt to explain the first image is still being passed through from the encode node to the sampler the second image is purely canny if you want you can connect the second image to the encode node what you can also do is pass both the images through two Samplers that will complicate things further it has Blended the elements exceptionally well for recolor you need to install some extra custom nodes the stability comfy UI nodes will give you the color blend node for the color correct node you will need comfy UI postprocessing nodes I would delete some nodes as you don't need them for this workflow there are three new nodes needed I will add two for now which are color correct and color blend I will disconnect some nodes as they need to be reconnected differently connect the load image and color correct nodes the color correct image will go through the pass through method so it should be connected to image scale to Total pixels what will happen here is the black and white image can be color corrected in terms of temperature and tone after adjusting the settings the color corrected image will then pass onto the sampler via the encode this node will have three connections one will go to the black and white layer input in color blend second is the pass through which will connect to the pixels input in vae incode lastly you would want to see the preview of the color corrected image so connect it to the preview node connect the vae decode node with the color layer input and color blend the color blend node Blends the colors and generates the final image connect the node to the ultimate SD upscale image input for upscaling for recolor only use the Bas sdxl checkpoint from hugging face I will change the value back to two later on I am disconnecting the node to show you what exactly you can do with the color correct settings I will change each setting and show you the output this is very important as the color accuracy depends on what you do here a positive temperature will make the color warmer and a negative temperature will give it cooler tones if you are seeing too much red in the final image make it cooler the Hue will not show in the preview but is basically a tint setting and applies to the final image I found it best to leave it at zero a positive value will increase and a negative value will decrease the brightness contrast is where the images might pop I rarely have used the minus value if you want the overall image to be washed out use a negative contrast value saturation is also apply to the final image if the colors have too much bleed change the saturation to a negative value higher the gamma darker the overall image this setting is sensitive try 0.8 to 1.2 values then go higher or lower I forgot to mention earlier you have to connect this node to the apply control net input as well this is the default negative prompt from stability AI keep the positive prompt short and to the point for this image I am using minimalistic geometric Vector art [Music] colorful the coloring style will change with the seed sometimes you will get better and more accurate colors what I do is choose a random seed and keep it fixed then play around with the color correct settings if you cannot get what you want change the seed it is recommended that you follow the default settings steps at 30 the sampler should be Oiler ancestral and the schuer should be normal CFG should also be at 7even you can play around but I highly recommend you start with these values and then mess around with a proper positive and negative prompt it does an excellent job if I did not include colorful in the prompt this comes out as a monochrome color another thing I find very useful is to have a batch process with that I can generate multiple images with different color options disconnect the color blend node from the sampler input search for repeat Laten batch and add that node connect the output to the sampler input now connect the vae en code link output to the samples input let's generate a batch of eight images with batch you get a wide range of colors this is not only useful for different color options but when you try to color a real photo you need batch processing many times a couple of images from the batch will be accurate it depends on the seed let me show you another example for any photo use this prompt color photograph detailed DSLR natural skin highly detailed in between add a few descriptive words about the image for example I have added female floral dress and Floral Pattern wall in the prompt The Prompt will have an effect on such images I cannot advise you to be less or more descriptive as it depends on the settings and the image itself so try it [Music] out the original image was quite dull so I enhanced it you can see how the image has a cool bluish tone the fault setting was producing too much red here's a comparison the fourth image is the best that is what I used for the opening scene let's move on to the final workflow for sketch I have to delete irrelevant nodes it is recommended that you use a latent empty image for the sampler the sketch control net model can be used to color the drawing input mostly white on black images it works on hand drawings too add a scribble pre-processor node I will use the S animated XL checkpoint here [Music] for this control nit start with a lower than default strength 0.7 should be fine you can then increase or decrease as per requirement I want to stylize the final image a bit for this I am reducing the end percentage to 0.4 beautiful I hope this tutorial was helpful and you get to learn something new and comfy until next [Music] time

Channel: Control+Alt+AI

Views: 4,699

Keywords: comfyui, stable diffusion, control lora sdxl, comfyui control lora, controlnet lora, how to install stable diffusion comfyui controlnet, comfyui controlnet extension, controlnet, comfyui tutorial, sdxl, stable diffusion controlnet, sdxl controlnet, comfy ui, comfyui nodes, controlnet extension installation comfyui, comfy ui stable diffusion, sdxl controlnet comfyui, comfy controlnet, controlnet comfyui, comfyui workflow, comfyui canny, comfyui controlnet, comfyui node workflow

Id: hTSt0yCY-oE

Length: 39min 29sec (2369 seconds)

Published: Sat Nov 04 2023