Better Inpaint with Controlnet, ComfyUI Workflow Howto

[Music] DJ AFK here back with another video for comfy UI uh today we will be taking a look at a workflow for uh better in painting uh how to keep the original character pose without doing any extra work uh sometimes those poses get really messed up this way we will uh come up with a way to combat that so further Ado let's get to it all right we start out here with a demonstration of the end result and then we will go ahead and rebuild this workflow from scratch that's what we have is two people sitting on a couch uh there is separation between them I don't want to get too close to the other person here so let's go ahead and mask someone out uh doesn't matter if we mask too much of the photos so let's just cut her out of this picture sorry girl you got to go all right we will save to node so now we have this mask right here what we're going to do is is take the masked portion and run it through an open pose uh control net which will give us her pose and size and shape not bad uh up here is just the original image with uh both parties down here is the inpainted image and we drew a blonde also sitting on a couch we pretty much just wanted to change her person and her hair color did not say anything about her clothes but it did change uh shirt color and all that over here on the right I just have a overlay so that we can kind of see what happened uh now her her face is in a slightly different direction than the old but it's not not too far off she's just a little bit shorter as well you can see right here that's okay we got pretty pretty much the same pose and the same size and shape awesome how did we do that now we take the original image and mask we send that down the open pose uh pre-processor we then create this pose from it and we apply that pose uh not too strong though so we're using 6 for both strength and end at 60% and we just send that as normal over to the K sampler and in paint using the uh set latent noise mask I don't like the regular um latent for in paint it doesn't seem to work the same way and then we have our final result let's go ahead and from the beginning Let's Make a Scene we have two people separately sitting on a couch because we we don't want them to be too close for demonstration purposes go ahead and do one of these up see what happens like these two are a little bit too close I could carefully in paint around her but I'm lazy don't want to do that so let's uh find a photo where they are a little bit further apart and there we go we have our photo that we can manipulate uh we can go ahead and copy this to our comfy UI inputs folder then it's easier to load it onto the new one or you can just drag it onto the screen so let's make new workflow we want to load a checkpoint we're going to use dream crafter this isn't LCM model I like using LCM models for uh video demos they are very fast move all this stuff off to the side and our Cas sampler down here and so we're going to want a fixed seed just in case and let's change this to a preview node all right so we do need instead of an empty latent image going to use a loader load image now this has the image and the mask on it so we can use both we'll use this as our starting point all right so to avoid confusion let's go ahead and make a group for control net and find our open pose open pose pose all right so we can't just send the initial image into this uh why that is is that we will get both individuals go ahead and cue that up what we need to do is mask or uh invert the mask and get just this area in here let's go ahead and draw a mask before we get too far all right once again you got to go we'll leave just a little bit of an extra room so that the models can do what they want save to node now we have our mask over our picture we do need to invert the mask it's not going to let me do that right there so let's invert mask and we'll set that the vae en code for in paint nope we need the other VA en code I don't like this one we can compare them but uh uh not the not the greatest here need our vae we have our mask set and our pixels don't really need to grow The Mask because we we did overdraw it now we'll take this latent and see what we come up with now we need our vae once again and we have a photo just our subject we want to cut out so that is what we're going to send the open pose let's see what shape she has there we go now that we have our pose uh let's go ahead and add all the other control net pieces need to load control net uh open pose is in here right there and we need to apply our control net um we don't have all of the features we need to apply it Advanced there we go before we forget we do want to set the strength down just a little bit both the strength and the end uh you can play with the parameters here they as long as they're somewhere near the middle you should be fine uh if you turn it all the way up a lot of times you'll end up with a really sharp line and a very different area for your mask on the final image that's not really what we want we do want to take uh let's see need our control net model we want to take our positive back to the apply control net first and then to the K sampler and our open pose image that we have in the preview window here is this is what we are going to apply then we need our latent image uh what we need to do for that is actually encode go ahead and attach our vae our latent image needs to get set to a a noise mask here uh it is not in the menu let's see set latent noise mask we'll take our encoded latent image and this is using the original image encode and then set the mask based upon the mask that we drew we'll then take that latent put it into the case sampler decode it and see what our image becomes one thing that we forgot to do is to give it a proper prompt so we uh we ended up replacing her with a um a bottle with a leg interesting all right so we want one girl blond sitting on a couch let's try that again we are running entirely too many steps that's why this is uh taking a little longer we want 10 steps uh CFG about two and Oiler a much better yeah she does have a really similar pose uh let's see we can run the comparison again here's our original image and our new image not too bad now in the original workflow we did have a comparison that is actually called an image blend we'll take the original image and we will take our decoded image and blend them at a factor of 0.5 or 50/50 and this is just so that we can overlay them and kind of see our results make that a similar size here and there you have it we have a really really similar set of photos here now you can go in and say uh profile view that way we could try to get this person to be looking sideways uh sometimes when you try to fight what the model wants to do it will run things like this where her face got kind of sad but it is closer to the original little bit this guy looks happy not quite as happy as me but yeah let me know what you think comments below remember leave a thumbs up and uh post your Generations let me see what they look like till then stay safe

Channel: djAFK

Views: 1,779

Keywords: comfyui, inpaint, comfyui inpainting, controlnet, controlnet inpainting, comfyui inpaint controlnet, stable diffusion, ai replace person, learn comfyui, ai art, artificial intelligence, deep learning, neural network, photo editing, comfyui tips, neural tech, unique filters, smart interface, seamless design, ai enhancements, interactive editing

Id: F2OoICVYXQY

Length: 15min 12sec (912 seconds)

Published: Tue Mar 19 2024