ComfyUI x Fooocus Inpainting & Outpainting (SDXL)

hi everyone welcome back to data leveling in today's video we'll be exploring a simple and effective way to perform in painting and out painting to an image using confy UI so in painting is basically to remove add or replace an object in a Mas area of an image while blending it in smoothly and out painting is to extend the image to fill up areas that weren't there previously with artificially generated pixels based on the ages of the image so this kind of look like the generative Fu function in Photoshop except the fact that it's completely free okay let's start with the dependencies and as usual all the links will be provided in the description we will head over to the confy UI manager custom not section search for inen and install the confy UI inin noes next thing we want to install is IP adapter from your custom notes search for IP adapter and install the confy UI IP adapter plus notes lastly we want to search for impact and install the confy UI impact pack notes for the models we'll head over to the confy UI in pain notes repository scroll down and click on this link once we are there we will install the focus inaine hit and the inaine version 26 Focus patch let's continue scrolling down the repository and install both the llama and met models once we are done we will create an inpaint folder in your confy UI models folder and place all the models there next you'll head over to the IP adapter repository scroll down and install the clip viit model you can rename it to anything that's easy for you to remember you then scroll down further and install the IP adapter plus sdxl vit H model once you done the clip VI model go goes into the clip Vision folder and the IP adapter model goes into the IP adapter folder if you don't have the IP adapter folder just create it don't forget to restart your confy UI and let's get started with a simple example of adding something to an image I'll be using this no image I found on the internet so usually when we perform in painting you'll have to use the Imp painting version of the sdxl model for better results I will show an example with the Juggernaut XLE painting model I am using some basic configurations here since we are changing the entire Mass region of the image our the noise have to be set to 1.0 let's say we want to add a bear to this image you know to make it look cool we will have to mask out the area of the image we want the bear to appear in then send it for and coding using the buil-in note vae en code for in painting okay looks not too bad but we can notice some Sims going on over here how we can fix this Sim outline is by growing the M size as we can see the Sims are not as obvious anymore this way of doing it solve The Sims issue 95% of the time when adding an object now let's try using the focus in painting patch the first thing is to change back your model to your original sdxl model without the Imp painting as the focus patch is able to use on any XXL model let's go to add notes go to custom notes in pain and select load focus in pain for the hit you will use the focus inin hit file the patch will be the inin version 26 patch file you then create an apply focus in pain note link up the models and the later so what this does is that it applies the focus patch to the sdxl checkpoint and transforms it into an sdxl in pain model model and this can be used on any sdxl model this means that we do not need to keep any more imp painting versions for each XD XL model OKAY the result looks good now let's go with an object removal example similar to before we will mask out the object that we want to remove let's try it first by using the current setup let's say I changed the promt to something like snow floor as we can see we do not get the desirable results to have a cleaner object removal you will have to go through some pre-processing steps for the image these are the image pre-processing methods that you can find under the inin custom Noe that we installed earlier let's take a look at what each of them does the first way is we could impin the image directly using a model we could choose either the Llama which then stands for large mask inventing or the M which stands for mask aware Transformer as we can see both models perform well for object removal and generalization of the background if you want to learn more about how it works backend or there are benchmarks you can check out both the papers for the fuel mask area note neutral is basically fueling the mask with a solid gray color and tar and Nava Strokes fill the mask with colors from the surrounding borders lastly we also have the blur Mass area note that adds blur to the image to keep the average color in the mass region for object removal I prefer to use the Llama model as the results is usually satisfying so when we sample we could reduce the den noise as we do not have to change it completely from the Llama applied image to reduce the D noise we will use the buil-in in pane model conditioning node you will then link up the notes and the key thing here is that the latent from the vae and code for imp painting note goes to the apply Focus inaine note and the latent from the inaine model conditioning Noe goes to the cas sampler this is because the conditioning one does not work with the focus patch directly and this is the result we get natural and blends well with the background now there one note in the inin custom node called vae encode and inin conditioning it is designed to sort of have a mixture of both but only encode the image once instead of twice how we use it is similar to before except that the latent inaine has to go into the apply Focus inaine node and the latent samples goes into the case sampler Noe so basically the latent in paint output is the same as the one we get from the vae en code for in painting and the latent samples is the one we get from the inbane model conditioning as we can see here the result is the same but this note design does not have the grow mask by input that we mentioned earlier that was used for fixing Sims therefore I will recommend to use the two notes version instead of this note even if it means encoding the image twice now let's move on to out painting So out painting Works similar to how in painting works except that we add padding outside of the def fine bottles treat that newly added portion as mask and use it for in painting the same way we did previously what we have to do first is to pet our image using the pet image for out painting note the four sides are to set how much you want to extend the image and Feathering creates a soft edge around the paded mask to blend images well for out painting the pre-processing is similar to the object removal as we want the surrounding content to be used [Music] so using the same setup we can see that it Blends nicely with the current image sometimes we might see this where the unmask region is affected slightly this I believe is due to the VA encoding process where the image is too big and it does not decode out properly how we can fix this is by using an image composite Mass node The Source image is the newly generated image the destination is the pre-processed image and The Mask we use the same as before now if we compare it we can see that only the mask area is being affected let's try with another example this is an AI generated image of a girl in a cafe let's try to extend the image okay we can see some decent results but let's bring it further with IP adapter in the custom noes IP adapter search for apply IP adapter we will then link the model to the apply IP adapter node and the output model to the apply Focus node for the image we want the original image without any pre-processing as for the clip Vision we will select the clip Vision model we installed earlier and for the IP adapter model we will also use the IP adapter model we installed earlier we can leave the other parameters as default as we can see the results are more natural the newly generated lights and items on the shelves are consistent to those already in the image and the chairs and tables also blend smoothly okay let's try an example where we change the background of an image let's bypass the IP adapter as we will not be using it here now let's say we are working with a product I found this bait image from Amazon let's use Sam detector to make our lives easier I'm not sure about you guys but when I buy a bait sheet for a queen size bait I usually don't buy the bait frame together with it so let's mask out only the bait sheet and the cute cover but remember the thing we want to change is the background not the bait sheet so to do that we could use an invert Mass node that selects everything but the mass region this is more efficient as compared to masking out the entire background let's use neutral fuel here and since we want a totally new generated background we have to update the latent output to what we did earlier when adding an object and the denoise to be set to one okay the results looks not too bad let's try out a few different thems for our bait sheet great now let's do an end to endend example where we apply every technique that we learned earlier together this is an image of a fashion model let's start by changing his shirt one thing about comy UI is that the masking tool is not the best experience so you could also mask it with external tools if you would like let's try out a r jacket one thing I noticed is that when we are adding or replacing objects to the image the image composite mask usually have the seams so I generally only use it for object removal or out painting okay this black leather jacket looks cool so let's use this image next thing we want to change is the background copy the image from clip space and paste it to the load image as my image is a white background we have to be very careful when masking as sometimes we can get those unwanted white portions in the image let's try out a few different backgrounds okay this street one looks not bad let's copy it over now let's extend the image to cover more of the street don't forget to update the Mastery route let's also use the IP adapter and we can choose to remove the proms or keep it all right we see some weird Sims here let me show you how we can refine it we copy the image to the clip space and paste it to the load image node but this time we mask out those weird looking lines and for the image we will select the original image and let's set the D noise to around 0.7 when we see this error it means we forget to update the Mastery route okay the weird looking lines are gone now of course we can keep iteratively change like for example this tree that kind of look out of place by REM masking it but I will just leave it as it is for this example now let's say we don't want to use this model's face anymore so let's change it so we must out the face and let's do my favorite character Mr Breen we can keep randomizing the seat to get a better one great the in painting Concepts that we have learned is sufficient so the next part is just a bonus section where we apply what we learned previously from my IP adapter video where we create a consistent phas note that you have to watch my confy UI IP adapter video first where I go through in depth on how to create a consistent pH for an a model otherwise you will be very lost and not understand what I'm talking about I will proceed here with the assumption that you have watch that video and have all the models required to perform this step together with inside phase installed so if let's say I have a specific pH of Mr Bean that I want to use we can use the IP adapter face ID model we will do the same as before let's create the apply IP adapter face ID note for the face ID IP adapter we we will be using the IP adapter face ID plus V2 sdxl model and for the other IP adapter we will be using the IP adapter plus face sdxl vit model for the regular IP adapter let's set the weight to 0.4 and for the face ID IP adapter set face ID V2 to true in weight to be 1.5 and for the regular WID let's set it to 0.5 the inside face will be set to CPU now let's load up our joury English image create a prepare for inside face node and link up the image output to both IP adapters for the models we go through the face id1 first then the regular IP adapter then to the focus in painting we will also connect our mask into the attention mask input of the regular IP adapter this is to tell the model that we only want this part of the output to be used by the IP adapter let's remove the prompt and let's take a look at the results there we go now that is what I want a natural looking inpainted face all right I guess that's the end of it and of course if you have better ways or methods to perform in painting or out painting do share it in the comments below so that we can all level up our skills together if you learn something from this video do help to leave a like And subscribe for more content like this it will really help the channel grows and serves as motivation for me as well if you face any difficulties following the videos do also leave a comment and I will try my best to help you and remember don't stop leveling up

Channel: Data Leveling

Views: 7,357

Keywords: Inpaint, Outpaint, IMG2IMG, stable diffusion tutorial, fooocus

Id: ufzN6dSEfrw

Length: 16min 44sec (1004 seconds)

Published: Tue Feb 13 2024