Master Inpainting on Large Images with Stable Diffusion & ComfyUI

Hello friends it's about 10 days that I am trying to design a workf for my in paintings the first version is ready I had many problems and I have tested many different solutions in this video I want to share all my experience step by step but before that let me explain something to you here the image that you can see the dimension of this image is 3,000 pixel in 3,000 pixel totally about 9 megapixel and you can see here you can see that I can do in painting on a huge image with acceptable quality and it's possible now because previously I had a problem uh for with in painting the main problem was about the latent size and uh every time that I decided to do in painting on huge image I had uh many different problems so uh let's see that how did I solve this problem before going to the details I want to do a fast review here in this part I'm loading the checkpoint and my Laura models here this is my IP adapter okay and here there is a switch that I can switch between the normal check point or combine that with IP adapter here um in this part I am loading my image and uh by clip space I am trying to define the in painting area and after that this note this note is very very important it can help me to define a region for my musket area based on a specific Dimension the name is confy UI art Venture but you should install um this extension from another Branch to see these Dimensions here here there are some different notes that can help me to remove the object from the scene okay or sometime blue rink them I will explain to you but only I am getting one output from this area and again I am selecting the output from this box um after that I'm using using the focus in paint for using the focus in paint you should install the confy UI in paint notes this is another extension and actually this is um the main part of my workflow and also you should download some models I will place all the links and explain in the video okay and this is doing the in painting for me here this is my sampler and and uh I can select you can see for example here there is a four number batch and from here I can select one of them this is my upscaling method okay and after that there are some parts that uh sometimes I can turn them on or turn them off and finally here I can uh mix the result with the original image so totally this is the work follow I am getting the image Define a crop re Define a mask area it will be crop and uh I'm doing the process scaling up and after that I will mix the result with uh original image it's all now I'm trying to explain details and important parts let's start with the first area loading the checkpoint uh for using the sdxl prompt Styler uh you should connect the output to a clip text encode node and normally when you are creating a clip text incode it doesn't have any input and only you can write the text so right click on that and click on convert text to input after that you can connect the output of the sdxl par St to your clip text incode okay uh here I'm loading it my sdxl and after that I'm using the Lura loader model only for loading the sdxl lightening you can use four step two step or eight step different models I will place the link for download it can help me to do the calculation faster after that I am using the freu V2 and self attention guidance about the freu V2 it's very important to use exactly these numbers for when you are using uh sdxl because uh for example these numbers will be different for St 1.5 or other checkpoints to keep my workflow Simple and Clean I am using two basic pip from Impact pack extension for transferring my checkpoint data in this area I am loading my IP adapter this note is extremely helpful for me ip adapter model helper that is coming from Inspire pack you can see here there is a preset and you can select different preset after that it will automatically load the clip Vision model an IP adapter model and you can send them to the IP adapter node and your clip Vision in code model using this note prepare image for clip vision is very important before sending your image for encoding clip Vision can analyze and understand what is our input image okay so this clip vision is very important uh but also there is a limitation for the clip Vision models because they cannot accept larger image than 224 in 224 pixel okay so this is the maximum size that the clip Vision can accept as input so preparing image for clip vision is very important and this note can do that for you in this example you can see that this image is very large and rectangular and when it's going into this node we can adjust the crop position and for example if I select the top after that it will keep only top part of the image and this is the exactly this is the area that your computer can understand from that image so uh if your image is rectangular adjusting the crop position also is very important here I am loading my image with a simple load image node right click and select open in MK editor now now you can select the area that you want to do the in painting okay in musk editor with left click you can Define the area with right click you can erase that with pressing the control and scroll you can zoom in and pressing the control and left click you can drag on the picture and when you're pressing save to note you can save your change on the image you should connect the image to the image and mask to the mask this node is coming from confy UI art Venture extension but originally you can't see and adjust these two size uh to do that you should install this extension from another branch and I will place the link in the description and what it can do for you now I want to add some preview from the out output and after that we can understand that what is it doing mask preview and for overl preview image and Q prompt okay you can see that from overlay image again we can get the original image from in paint mask we are getting a mask but with a crop region and from Ina image we can get the image but in crop region so these are the informations that it can send for us and the information that are stored in the crop region can only send to a specific note from this extension it's very important to check this option in paint mask through okay and if you turn turn it off and cu the prompt you can see that the image that is coming out doesn't have any crop region another note that you can see in this group is see our data bus in it's coming from conal studio and with this not you can create pipe from any input in this area I have have several notes that can help me to remove the object from image here you can see that I have marked the girl and the lion on the image and these are different results for removing them from the image most of these notes are coming from confy UI in paint notes extension and to get more information about these notes it's better to check the GitHub page and you can find many use information the only notes that is different from out there is llama in painting mixlab that is coming from confy UI mixlab nodes also I know that uh there are some different llama in painting notes from different extensions uh and also this note can provide some good quality for you before starting the remove process you should add this note okay carefully pay attention to the input and to the output of these notes it's very important because uh you cannot send directly the image that is coming out from this note um I don't know what is the reason but uh you should uh do this process it's based on my experience for uh feeding the image into the erasing models and also you can see that uh I have resized the image by half uh before processing because it can save your memory and also the image that is coming out from these nodes is going to the case sampler so if you want to save your memory you should do that before sending the image to the case sampler I have added another upscale image node and um with this way I can make sure that the output image will be same size as the half-sized image uh what is the reason some of these notes are uh making some change in the resolution about one or two pixel and it's very danger for us because uh when we want to work with in painting Solutions all the time we have to stay at the same resolution in this area I have added a switch from confir r St to switch between the models so in this case I can switch between the IP adapter or my original prompt also in the upper part you can see edit basic pip from Impact and it can help me to edit my basic pipe it's very important because you can see the model is connecting to that and it's coming from basic pipe so here I can change the model in my pipe this area is heart of the workflow because I have loaded my in paint model they are coming from confy UI in paint note the first note is VA incode and in pain conditioning please pay attention to the pipes and another note is applying focus in paint the model should connect here here and after that it should connect to the case sampler and it need a patch okay and here you for the head always you should select the in paint head model in paint head and for the patch you can select three different model in paint V26 in paint Focus or Focus llama but all the time I get the best result with this patch to download these models go to the GitHub page of the confy Imp paint node scroll and after that you can download the models from this link and you should place the downloaded files in this folder by adding this node repeat latent batch between your vae and your sampler you can add some latent batch for example here you can see that I get three different result for my in painting here in the sampler I have used a normal case sampler uh but if you want to use SX a lightening as luro you should set your scheduler to sgm uniform uh try to keep your CFG as low as possible and the step count with about 12 or maybe 16 can work because we are using in paint and also sometimes IP adapter so we need more steps uh after that I have to use this node remove noise mask it's coming from Impact pack I should add the latent output to this node and after that I have added another note with the name of IM pack pack preview breach and it can help me to check the latent very very fast it is not using any vae and it's just using a preview method finally here I have added a note latent from batch and with this note I can select what latent should be sent for up scaling for example the zero one two okay the number is starting from zero for doing the up scale I am using ultimate s the up scale and here you can see I have added a vaed code and after that I sent the selected latent as a image to the ultimate SD and here I have another note from basic people and this pip is coming from uh some different way because uh I figure out that using self attention guidance for upscaling cannot work fine for me and I decided only use the free U and my Lowa SXL lightening so here this is the basic pip that is going to the the uper scaler and for my up scaler model I'm using this model uh pay attention to the numbers the step is about 8 the CFG is one sampler is uler and schedular is sgm uniform my D noise is set to about uh 0.3 and uh for the mod type I I have selected the chest because I get B result uh and finally the amount of mask blue and tile padding are set to 64 and 128 this settings can work fine for me and is useful now here you can see the output of the up scaler let me open that in the new tab and exact L the output image is 248 in 2048 so uh now we have image with this size and we should combine the result with the original image okay and here this note can help me to do this combination now I want to explain how it's working okay uh this is a CR data bus that I told you at the first of the video and exactly it's coming from this note okay so number one number two number three number four one 2 3 4 okay they are coming here so uh now here this the image that is coming out from this note is my uh uper scaled image this note is original image let me add preview image node and this was the information about the crop area or crop region and let me cue my prompt so you can see and figure out okay this was my original image with my musket area and this was my in painted model so this note is merging this image in the correct position and the same mask as our original input and making the final composition so this is the reason that you can save your memory during working on large image for in painting in this workf I have added another part for fine-tuning the mask for composition and let me show you when you can use that uh before that let me explain something for the up scale I have increased the CFG amount so you can see it is uh creating the image with completely different colors and when I am combining the result easily you can see a border between your inpainted area and your previous image now I want to show you how you can f tune and edit the mask for this composition uh here I am using uh the preview Bridge note that is coming from Impact it's very important you should exactly use this note because you can draw your mask directly on the preview image but before that let me unmute these notes okay and also unmute my preview image okay so at first I can right click on the image open in some detector let me clear that okay you can click on your subject and press detect okay it can detect your object but inside of the object is not detected but no problem cck save on Note after again open in MK editor after that you can paint the undetected part and for example at the shadow area okay and click on save to note so now we have the mask you can see that uh I am using a grow mask with one pixel and mask blue with one pixel and after that I splitted the image with Alpha and sent the image to the switch and also I sent the MK to the switch uh and for this switch we have selected the number one okay and this is the preview image that is coming from our original area exactly here okay in painted image this preview image is coming from here and now I'm using the image composition comp image composite mask so uh it need three input destination source that is this image and MK that is coming from the Mosk blur and when I am cued prompt okay you can see that this fan is mixed with this image because it was our original image and finally if you want to see this image in our final composition just I have to select the second input and execute my prompt okay uh and you can see that uh exactly it's working based on the musket area here there is another step for fine-tuning the mask and make better alphate for you let me first switch the index on the number two and active these notes is it's all and again Q prompt it's trying to process your mask and for example here you can see that we have some smooth mask and in this area and in final result sometimes we can get a better result it's working very very great with some object like ha and four so um it was the it was my workflow for doing in painting on high resolution image I hope that you have enjoyed the video if you want you can download the workf from my Patron also I have placed a complete version of my comy UI with all checkpoints and setting it's about 98 GB you you can download that and exactly run this workflow

Channel: Arch_AI_3D

Views: 2,089

Keywords: AI, stable diffusion, AI architecture, ComfyUI, ControlNet, architecture, artificialintelligence, Tutorial, inpainting, openai, AI art inpainting, Generative art tutorial, mage editing for AI art, ai art, ai robots

Id: 3N_C5I0cDao

Length: 25min 2sec (1502 seconds)

Published: Mon Feb 26 2024