Regional Loras - Automatic ControlNet, ComfyUI Workflow Howto

[Music] DJ AFK here back with another video for comfy UI in a prior video we had done some in painting and automatically replace a person using control net with the same pose uh we are going to take that idea expand upon it replace multiple people using different luras on the same photo and without further Ado let's get started as you can see from this spaghetti mess there is a lot going on here so let's see if we can try to break this down one piece at a time at the very top I just have a a quick group bypasser so that we can start with one image and then move on to everything else that's going on if we look at the input section this is where all of our selections are made for the overall composition let's start here at the top we need to select our checkpoint if you need to set a clip skip you'd like to choose a different vae you can switch back and forth between the baked in and your own we have a Lura stacker for things that will apply to everything within the composition uh we are just using an LCM Laura and a regular model non LCM here that will greatly speed up the process of the image generation uh we are going to make a 768 wide by 512 photo and we have four different inputs for text now this is designed for two people to be in a photo and then those two people to be replaced by different luras so we have a man a woman at a table and in the background we have in a Tavern we describe person a which will be on the left a girl with long blonde hair and person B which will be on the right a man with short dark hair now it doesn't always produce those types of results in the photos so you may have to play with a couple of seeds before you get one that does uh kind of work that way uh when we get into all of the various person outputs uh we'll go over why let's go ahead and we have our feed for the image here good and cue that up uh now we will be working kind of left to right and I like to keep the controls I will need next to the images that are generated now this doesn't look like a horrible photo except there is a third person in here and this will kind of mess up some of the automatic uh segmentation and detection here let's try another seed and we possibly have some people in the background there otherwise not bad at all we have pretty clear view of both people and we definitely have a third person in the that seed all right no third person anywhere to be found now he also doesn't have short dark hair but not a problem we can see in the show text when it puts all of our prompts together we do have a in order a man and a woman at a table a girl with long blonde hair got that so far a man with short dark hair not so much in a Tavern why we have these all separated is so that we remove the initial text on the next outputs but we keep the background append so that it will uh put proper shadows and Lighting on them different things like that uh what we're going to do with this image is run a person YOLO model which will segment the entirety of the Body for us so we don't need to manually imp paint on this image uh it'll just detect where the person is grow The Mask a slight bit uh feather that out and send it to control net let's enable person a we have K sampler for person a we have control net for person a and we have an output for person a that that's just where we uh display the the picture here what we're going to do is select the person now different segmentation models are going to select the people in different orders you can play with ascending descending order things like that I just like to change this number it starts at zero we're going for two people so it's either zero or one you got a 50/50 shot so change take start if you need to and let's cue this up now since we have a fixed seed it's just going to skip generating this image and move right along to our next image but this silhouette here in white the person a is in white is what it detected as person a and they do have pretty much have to be on the left uh the way this is designed um there are some pieces is that will not work quite right if it detects the person on the right and you attempt to just deal with it that way so go ahead and select this back the other direction if that's the case so that we get the person on the left we have them masked then we have a control net detecting what position they are sitting we have selected a custom Laura uh this is part of a collection I've made for Viva La dirt leag people for epic MPC man so this is epic Brit profile view so we have replaced the entirety of our description a girl with long blonde hair we've replaced that with epic Brit profile view so I put the trigger word and then described what I want her to do which is looking sideways so we've replaced this blonde girl that's a generic person out of juggernaut with epic Brit and that is a pretty good resemblance there now you can see it's not exactly refined very well this is a first pass uh believe eight steps eight or 10 but that's okay now that we have person a replaced we take the remaining segments and send them them over here to person B now if you have more than two people in the photo you may need to change the take start on this one but hopefully you only have two people in the photo let's go ahead and enable person B and before we do you can see in the show text that epic Brit profile view has been replaced from a girl with long blonde hair so this is just kind of showing you the output so that in case you made a typo you can correct that go ahead and enable person b control net B and output B with this one we are replacing a man with short dark hair with uh epic Balin uh let's uh let's actually do epic baradon so high sorcerer baradon Barun there we go and if we cue that up it should detect him on the right and it does it properly segments out this person here which is the same as the first photo we haven't done anything with this area and then it detects his pose and puts epic baradon there it kind of kind of missed the top of his head but this Laura does look a little bit overbaked uh let's see if we can turn that down just a little bit see if that helps us not really but you can see the point there so now that we have two characters from completely different luras that didn't get mangled with each other uh let's clean up their faces so we're going to take all of the model pipes so the the model and the clip of luras and we're going to send them down here to uh face detailers this is just a generic detailer uh we are segmenting from the uh a different YOLO model here this is the face YOLO so it's going to get the whole face and make a preview image from that and then make a a k ampler pass go ahead and enable both of those and see what happens it takes the image that is over here in our final pass and it replaced the wrong person so we so we got a nice mangled U photo here so we tried to replace uh barun's face here with um the non-npc character of Brit so to correct that let's go ahead and change this tick start to one and that should replace the two of them all right so now we are actually rep replacing Brit's face with Brit and Baron's face with baradon and it looks like it did a pretty decent job so the detailer did detect that she was looking sideways and that he was looking mostly forward so it fixed a little bit of that uh hopefully the luras that you use don't completely miss the the head up here but we weren't supposed to start with a tall man with um long blonde hair it was supposed to be a man with short dark hair there are a few other things that could be done with this workflow now you could obviously take and uh duplicate these to make a person see and so on that would get really complicated um you'd be starting and stopping a lot to do that uh the input instead of having this as a k sampler could just be used as an image loader and go ahead and take a pre-existing image and do the automatic imp painting with it I hope this helps uh this is version two of this particular workflow go ahead and save it as is and I will post this new version along with the video at the civy in the link below go ahead and leave me a comment let me know if you have any trouble until then stay safe

Channel: djAFK

Views: 1,791

Keywords: comfyui, inpaint, comfyui inpainting, controlnet, controlnet inpainting, comfyui inpaint controlnet, stable diffusion, ai replace person, multiple loras, comfyui multiple loras, regional loras, comfyui regional lora, learn comfyui, animatediff comfyui, ai art, regional loras system, photo restoration, controlnet algorithm, comfyui technology, digital inpainting, comfyui benefits, comfyui features, stable diffusion techniques, innovative algorithms, inpainting methods

Id: ECb6hjd_KpQ

Length: 12min 45sec (765 seconds)

Published: Sun Apr 07 2024