ComfyUI: Image to Line Art Workflow Tutorial

hi I am Mali and welcome to the channel you know taking a sketch and using generative AI to create an image is easy with control net but taking any image AI or an actual photo and reversing it to a line art is extremely difficult take this image for example it's complicated if I had to do a rough sketch it would be something like this look at this interior bedroom it's inspired by a kinetic wall design and the Fibonacci sequence but with my workflow it takes me a few seconds to do this not only that but I can take this AI generated comic and first remove the background then convert it to a line art let me show you how I do this and some hacks in comfy UI for stable diffusion before I start let me tell you that this is a very Advanced workflow tutorial the Json file will be available for paid YouTube channel members however I strongly recommend that you watch the video as it is crucial to understand the workflow there are a lot of tips and tricks used that might be useful in other workflows and are not exclusive to line art if you are new you really need to understand the basics of comfy UI along with control net my colleague and I have made some extensive tutorials regarding comfy under the stable diffusion playlist please go check them out this tutorial was requested by a user on the channel the workflow is entirely custom and very robust I have tested hundreds of images from various categories and it works feel free to ask any questions you may have I would be more than happy to help you get this workflow running all the relevant links will be in the description I will start with the essential requirements that you need for this tutorial afterward I will move on to placing the nodes explaining a little bit about why I am using them and then I will connect the nodes and show you the workflow process finally I will end up with some practical examples and difficulties you may encounter and how to use the workflow effectively to achieve the desired result in the comfy manager go to install custom nodes search and install the following custom nodes if you don't have them the control net auxiliary pre-processors this is required for control net the W node Suite I use it quite extensively I used some of the nodes from python gos custom scripts then you will need the ultimate SD upscale for upscaling along with control net I have used the IP adapter models comfy math but I have just used one node from it I used it to calculate the sdxl resolution The sdxl Prompt Styler I will be using the line art style from here however this is very handy for applying other styles to sdxl workflows as well you will also need to install custom models from the manager and download some manually install the sdxl base and vae safe tensor models also install the sdxl clip Vision model needed for the IP adapter one of the nodes used is the blip image analyzer when you cue The Prompt for the first time it should automatically create a folder called blip in models it then automatically downloads a 1.96 GB file in the subfolder checkpoint if for some reason you do not see this file then you can manually download it go to the custom folder inside the W node folder right click on the config Json file and open it in notepad download the pth file from this link and put it in the following folder I also used a node called image remove background if the node is used for the first time it should automatically create a new folder in models and download this on next file you need to download the official control net Laura models this was covered in the comfy control Laura tutorial video I will be using the rank 256 canny and depth models these models go in the following folder I will be using one custom upscaler model which is called 4X full Hardy Remer download and put it in the following folder you will need one IP adapter sdxl safe tensor model download the file in the following folder if any of the mentioned folders are missing create them and put the files respectively based on the feedback from the last comfy UI video I have improved and organized the workflow I would be grouping and coloring sections let's start by adding the basic checkpoint load and the vae nodes use the sdxl 1.0 checkpoint and vae here let's name this group as a models now I will be adding the image and resolution nodes start by adding a load image node in math under Graphics select the nearest sdxl res solution node basically this node is used to get the size of the loaded image it doesn't matter what ratio the image is the node will get the width and height of the loaded image and pass it onto the IP adapter and control net double click and search for the constrain image node I will name this group image and resolution there will only be three nodes in this group this node will constrain the image to a desired Max or minimum resolution it is for resizing and cropping I will take the width and the height from the nearest sdxl node and crop it if required to do that convert the max width and height to input now let's make the prompt and blip nodes search for blip and add both nodes the blip analyze node is similar to clip interrogate from automatic 1111 do not change this text when connected with an image it will query what does the background consist of and get an approximate prompt from the image we will use that instead of guessing and creating our own prompt for the loaded image so keep this text as it is and don't change it this node will use the default model checkpoint from the model / blip folder which I showed earlier search and add the show text node this will be connected to the blip analyze node if you don't do this it will just take and pass on the prompt however this node will show us the prompt the blip model takes from the image the text string node allows you to write five short lines of text the prompt from the blip analyze node is not enough we need to add some additional prompts to get the desired results so the prompt from the text string and the blip analyze node will be combined into into one sentence it will then pass on further I will use the text concatenate note to combine both texts for the prompts I am adding three nodes in parallel the first one is a text multi-line node this will be the positive prompt node which we can edit whatever prompt the text concatenate gives if we want to make changes we use this node for it I will explain later how this will exactly work the second node here will be a duplicate of the show text node it will show the prompt from the text concatenate the third Noe will be a simple primitive node which I will use for the negative prompt the comb find prompt will be passed through the sdxl prompt Styler for the pass through rightclick and convert the text positive and negative to input add the final clip text en code node which will connect to control net this node should be the sdxl version since we use the sdxl checkpoint except for crop width and height convert everything to input that's all the nodes for prompts and blip let's work on the IP adapter nodes this group will contain four nodes the IP adapter apply an upscale image the IP adapter model and clip Vision notes there will be two control net models with a Twist to start let's add the control net loader nodes the first will be the control Lura depth and the second will be cany I will connect line art with canny as there is no other compatible pre processor for sdxl and comfy it works completely fine I will use the Zoe depth model for the depth pre-processor add two control net apply models and use the advanced version Now search for a Noe called image invert the line art processes the image in black with white lines that is the exact opposite of what we need so by connecting the image invert node to the pre-processor we will get a white background with black lines and this will be the final control net image that goes to the sample along with the IP adapter model I will also pass this image without the control net apply to the vae en code in image generation let's do the image generation nodes now adding the vae en code first I did explain in a previous video the function of vae and code versus decode and why we use them the second node here is the repeat latent batch this will allow you to generate in batches this primitive node is for the sampler seed you don't require it as you can change it from the sampler I find it easier to edit it from a separate node now add the case sampler Advance node and then the VA decode for connecting the nodes let's start with the load image the first thing to do is to connect it to both nodes both nodes serve a different purpose here we don't want to calculate the image's width height or ratio every time we load any image the sdxl resolution node takes this info and is passed to the constrain node the sdxl node will also pass on this info to the positive and negative clip and code prompts whereas the constrain node will pass on the constrain width and height to the control net IP adapter and image generation models next connect the constrain node with control net I am using a rroo to split the image between both the control net pre-processors I almost forgot before making the connection ensure that crop if required is yes connect Zoe with the first control net and add a preview node the line art output will connect to the second control net and invert image inputs also add a preview node add a preview node for the invert image as well change the pre-processor resolutions to 1024 enable course and line art to get clear and sharp outlines connect both control net outputs to the apply control net inputs respectively the depth strength should be 0.7 and the line art should be at 0.25 you will not be required to change these values at all for the desired effect I will let you know which values can be altered to make changes in the generation you can see the connection flow of the image to control net make a second connection from the image to the blip analyzer also connect the blip model output and input here we do not connect the constrain node because we just need the node to analyze the original image for the prompt connect the blip string to show text connect the show text string to text a input type comma space black and white comma and space again for the text B type colorless comma and space here type in line art comma Line work and full stop connect in order to text inputs b c and d connect the text concatenate to show text this will be the positive prompt that we will connect to the prompt Styler during a live workflow example I will show you how to edit this prompt by connecting the above positive prompt node with the prompt Styler now connect the negative prompt to The Prompt styler the negative keywords I use are shadows gray fill blurry Watermark text dark background thick color shading gradient and transparency there won't be any need to change the negative prompt as this works most of the things you need to eliminate from an image can be done differently use the negative prompt to Omit elements from the generation as a last resort for the style select line art The Prompt Styler outputs connect to the respective positive and negative text G and L inputs to recap the image connects to the blip which generates the prompt we added our prompt and combined it the positive and the negative connect to the prompt Styler which splits to the sdxl clip in code notes connect clip vision and IP adapter model to the apply IP adapter the load image connects to the line art pre-processor which connects to the invert image node this output has two connections one to the VA en code and the other to the upscale image input in the IP adapter group the upscale image will connect to the apply IP adapter node the near sdxl resolution output will also connect to the sdxl clip text and code notes first drag out the width and reroot now do the same for the height take the width and make four Connections in total to the positive and negative width and Target width do the same for the height connect the model from the load checkpoint to the apply IP adapter input create a rout for the vae and connect the two relevant nodes in the image generation group the model output from the IP adapter should connect to the model input in the case sampler the image generation nodes connection is simple the vae encode connects to the repeat latent batch that connects to the K sampler which further connects to the VA decode in the case sampler convert the noise seed to input and connect it with the Primitive node the K sampler settings are significant to this workflow the add noise should be enabled if you want consistency with seed if you disable this the same seed will produce different results steps keep it at 20 and don't change it CFG here the default would be at 10 instead of seven or 8 the CFG is one value you can play around with in cases where the prompt is edited and you want more weight to it increase CFG also note that the CFG is slightly restrained because of the control net and IP adapter the IP adapter weight plays a more prominent role than CFG I still have to add a whole section of post-processing effects once those are added you will find that CFG is basically a subtle change change the sampler name to Oiler ancestral this works best for line art the Schuler stick to Caris return with leftover noise enable it if you are adding a second sampler I kept it enabled and tested all the images to follow the tutorial I suggest you keep the same setting connect the positive and negative clip text conditioning to the first apply control net then connect the first and the second apply control net and finally connect the second control net to the case sampler connect the model clip output to both clip text and code inputs in the IP adapter group The upscale image width and height should be 1024 and the crop should be centered the image given to the IP adapter should always be sdxl resolution since the load image may have different aspect ratios by this node we are making sure that the out put to the apply IP adapter is sdxl resolution any cropping if done would be centered this however will not affect the output resolution of the final image that's why I have a different node setup in the image and Resolution Group the default value I have set for the IP adapter weight is 0.25 altering this value will give you deeper blacks more details or vice versa it depends on image to image I will show some examples where I alter this value I suggest you keep the weight type to channel penalty for line art this method is experimental and the results may change in the future after an update for consistent results go with the original weight type the workflow is not completed there are some pending nodes but let's connect a save image node to test if everything works as intended now that everything is set let's test the workflow with this shoe concept art this is the final combined positive prompt it includes the blip prompt and the text string the first image is the depth map the second is the line art and the third is the inverted line art which gives the desired results the shoe design is spot-on but a lot of extra details were added this is probably because of the blip prompt this can be fixed easily what we need to do is remove this text from The Prompt so select all the text copy it and then paste it into the positive prompt note note above edit the prompt here and connect this string to the sdxl prompt styler and Q prompt okay the candle has been removed but some details still remain let's change the prompt one more time let's remove on a table and see what happens by the way add noise is enabled and the seed is fixed this is why I am getting consistent results perfect let's add an upscaler to to the workflow add the ultimate SD upscale node use the same seed as the K sampler the CFG scale doesn't matter since the denoy strength would be 0.3 a very low value the sampler name and scheduler should be the same as the K sampler mode type keep it linear and change the tile resolution to 1024 connect the upscale model and choose the 4X full Hardy remery drag the control net output with the positive and negative input connect the vae decode to the image input now connect the model from the checkpoint to the upscaler finally connect the vae via the rout rename the second save as final upscale let's hit the Q prompt and check out the upscale quality excellent in this example I am choosing a DSLR photo of a Joker Lego I will first try the image without upscaling to deactivate right click on the group and select set group nodes to never this will deactivate all nodes in that group as you can see blip picks up a lot of color in the prompt however I do not recommend changing the prompt to remove the color there is a better method there is some green I will remove the color by adding a filter drag the output from ultimate SD upscaler and search for a node called image filter the only value you need to change here is the saturation change it to zero we will use this filter only to remove the color and nothing else let's name this final upscale black and white the color is gone but there is a gray shade there is another node to remove that same process but this time search for image level adjustment here the black levels will make the lines darker and the white levels will make the whole image lighter values below 100 start to fade the image off but that depends on the image let's try 100 and see it did remove the gray it's not 100% perfect but it's very good let's choose a fashion design I have included a workflow node that will randomize the seed and process images and batches to do that go to repeat latent batch and enter the number of images you want to generate as you can see it has randomized the seed and generated 10 designs this node is also useful when you don't want to fine-tuned the images for instance while testing some images had thick back shading in some areas I found it beneficial to always generate in batches and fine-tune one from the batch this workflow works with any category of images this is one example where I fine-tune the image with multiple nodes let's choose an interior design I am increasing the IP adapter weight to 0.5 to get more details the line art for this image comes out a bit blurry in the bouquet areas that can be fixed by increasing the edge enhancement value also I want the black lines to be more prominent and the gray shading to be reduced notice the blurry bookshelf and the trees on the left this is with Edge enhancement blurry sharp less Gray let's say you only want the subject and no background basically this is just one node called image REM bii short for remove background to only remove the background this works out of the box for line art the load image output should be connected to the input of this node for this to work you have to connect the output with the control net inputs the output should also connect to the blip analyze node after the background removal the image does not require fine-tuning control net and the IP adapter will process an outlined image neat and clean lines this will also work with a photograph let me show you another example beautiful I hope you found this tutorial helpful and learn something new in comfy until next [Music] time

Channel: Control+Alt+AI

Views: 3,537

Keywords: stable diffusion, comfyui, comfyui tutorial, controlnet, comfyui nodes, comfy ui, comfyui stable diffusion, line art ai, image to line art ai, nodes comfyui, comfyui workflow, comfyui ipadapter, comfyui controlnet preprocessor, comfyui controlnet workflow, comfyui controlnet, comfyui easy workflow, comfyui upscale, controlnet line art comfyui, stable diffusion controlnet image to line art, stable diffusion change image to line art, line art to image stable diffusion

Id: XodTkkpdSyo

Length: 43min 52sec (2632 seconds)

Published: Sun Nov 26 2023