Generative Fill for ComfyUI - Bringing Photoshop into Stable Diffusion

in this video we'll take a look at a technique for completing images that have got something missing inside of them or replacing something inside an image we're going to take a look at a technique which you might have come across inside of Photoshop and there it's called generative fill there's something similar inside of stable diffusion and inside of comfort UI so this is going to be very much an introduction to a bunch of techniques that have been a a really deep area of research in in generative AI and uh in image processing so what you're seeing here is the first glimpse of a bunch of techniques that can become really powerful what we've got here is a an image which I brought in from Photoshop and I just basically saved it as a PNG with the part of the person's face cut out you can see roughly speaking what it looked like uh beforehand when you look at the images on the side there and what we want to do is to replace it with another person's face so beautiful woman's head tilted down as though writing that is the prompt and if we run that we'll get something that looks interesting I'm sure there we are and uh I find sometimes with this technique that it produces weird results like this and what we can do is just increase the grow mask by just give it a bit bit more room so that it increases uh this area of transparency and I can actually demonstrate this is transparent because I can change the color and you'll see the color changing in the background over here and over here so it's completely transparent and that's what a mask is it's it's an area of transparency inside an image now if we take a look at this guy here this is a little bit more successful we've got the woman looking down at her computer but this technique really does depend on a number of factors the first factor is your prompt if you really get your prompt right that can work wonders let's change this actually to a blue wall so that let's see if we can just remove the woman's head completely and uh let's try that so the prompt is important the second thing that's really important is the sampler the sampler has got a number of factors which really uh impact the result the most important one is the D noise so let's go back one seed to where we were and run this again and what you'll see is that the D noise generally speaking in my experience tends to work a little bit better when it's not exactly at one so let's go back again and reduce the D noise to not point seven which is a fairly low score in this particular workflow you can see that it it Blends in maybe a little bit too much there so we can increase it to 0.86 let's try a new seed and see what happens is put a shelf on the wall but you see that it's kind of made woman's head Disappear Completely this is the kind of thing that you can do you can remove objects using the prompt or you can put objects into the area here let's try um blueish glowing or and let's see what happens if we try to remove to replace her head with a glowing orb okay that's not bad so you can do all sorts of crazy stuff with this technique the technique is similar as I say the generative fill you select an area and you select an area by creating transparency then you bring the image in and then you give the prompt either to remove something or to replace it completely uh by something new you make sure the D noise is not 100 and it's not zero you don't want it to be zero you probably want it to be between about 0.6 and one that seems to work well if you take it right down to to to to close to zero you'll just get you know gray and you can grow the mask the area of transparency by increasing this number here let's set this up so that you know how to set it up from scratch normally we have a latent image which comes here and we would choose let's see empty latent image 512x 512 that's the normal let's get rid of that let's drag out an image and let's drag out a node or uh a piece of spaghetti and we'll choose vae and code for in paint which you won't be able to see clearly but I'll click choose that and we'll zoom in and you can see now exactly what I've chosen vae encode for painting so we're going to have an encode coming in from the foldo and we can actually delete the photo and just do that from scratch as well and we'll have a vae coming out from here as well so we still need to save the image and save the image is going to be as normal VA code we select the vae here and then we select the image here and boom there we are now we can you can go ahead and drag out a line from the pixels and we can choose load image and this loads an image of a train now that train image has no transparency so nothing's going to happen so let me just choose an image that has got some transparency because I created it that way so I created this image which had a sheep on it I don't know if I can show you what the image looked like before it had a sheep on it and uh what we what I did was that I remove the Sheep entirely and what we'll do is we'll change the prompt here we can get rid of the hands and the negative and uh just why not go with a bluish glowing orb again let's see what happens with that let's increase the grow mask by but we need to connect the vae here somewhere so let's connect it up to this I'm using dream shaper and we can connect the mask up to here so that will uh recognize the mask and let's run this see what happens awesome so it's given us a little bit of an orb we're going try again to see what sort of results we get this time we've got a glowing orb there now it tends to produce results that are Sometimes good sometimes not so good so you have to sometimes run the prompt quite a few times before you get the result that you want um if I was to say Farmhouse which would be a bit more reasonable surrounded by grass what that does is that it ask us to get a farmhouse surrounded by this sort of environment that we see here so let's see what happens there and you can see it's put The Farmhouse way back there but we haven't got a good match between the the the ground here where we've got long grass and the area here so we can try again and sometimes you need to play around if a few times with the with the prompt to see what results you get sometimes um it's just it's a question of increasing the the number of steps let's increase the steps to 25 let's go back to this seat here let's see if we can get something interesting okay a little bit better we've got a little bit of a f farmhouse in the background and we've got it's trying to blend the grass in here and it's trying to blend this area of greenery so you can see it's fairly intelligent but you can give it too much to do so small selections tend to work better than large selections and it is trying its best to make everything fit in make everything make sense um and the prompt definitely does help Greenery let's see if that helps with the results and obviously as you work through the seeds you'll find some that work better than others that wasn't so successful okay so that looked a little bit more successful The Farmhouse is very very small indeed it's kind of like almost in the valley okay so it tries its best it reminds me a little bit of photoshop's generative fill in the very early days where it wasn't quite satisfactory but where you have the right settings the right idea the right selection it can actually create some very interesting uh inputs it can remove objects it can replace basically fill in however you want the item to be filled in so it's a very powerful technique there are some models it has to be said that work better than other models and if you change the model that can radically change the sort of image that you get and there are specific models for this type of uh process which is usually called inpainting inside of inside of confy UI so that's a useful technique it's not one that I use a lot but it is an introduction to a powerful family of techniques inside of comfy UI and some of those techniques you definitely will probably want to learn as you get more advanced inside of comfy UI so this is quick reminder that I've got a couple of courses which you might be interested in one is the learn generative AI with sdxl stable diffusion and comi the other one is the advanced version of this one if you want to take a part of these courses I'll have a link in the description you may find a discount there hope to see you there learning all that we need to learn about comy UI and all this kind of fun stuff when working with image replacement or partial image replacement there's a very powerful Technique we can use to work with images that we've created we can right click and choose copy and we're going to choose copy clip space and we'll go to the load image and we'll say hey go ahead and paste so now we've got an image in the load image node and this image uh is connected up to the vae en code for in painting now what we can do is to start working with this guy here but how do we do that how do we actually get this guy ready for in painting well I've applied a small edit here to to this image and how did I do that we went ahead and right clicked and chose open in mask editor and it's had a mask editor what I can do is just a paint and we can choose the thickness of the paintbrush here so we'll choose a paintbrush just want to address this area here which is a little bit weak just reduce that in size like so and then just paint there maybe paint a little bit here as well and then save to node we've now got a an image that I work worked with to remove part of the image in the masking process now we've got a mask uh output going to the vae en code before in painting so we could try could try to run this and see what happens okay so you can see now we've got uh this output here I'm going to go here and say okay let me just work on one thing at a time so we'll choose open mask open image and mask editor and we'll select this area here let me just use my mouse wheel to resize the brush save to note I know that that's an area of greenery so I'm going to go ahead and say look um Greenery grass and Greenery let's just say that and this should work with the original image that we output here and you can see it's it's done a very interesting kind of Road area here where we've got uh a road running off there I want to try something a little bit different let's see what happens okay okay that looks a little bit well that looks a little bit good um is a little bit dark I will probably want to try that with maybe a lower seed uh maybe a lower D noise level let's see what happens when we do [Music] that okay it's looking a little bit better and this area here I could also edit that area separately in in a similar way so you can see how when you've got an image which has been output right click and copy clip space come here and paste clip space and if I wanted to do any further editing I could right click and choose uh open a mask editor and I can do a little bit more editing if I wanted to so let's just go ahead and do a little bit of editing here [Music] and say grassy [Applause] area see what result we get okay that wasn't it at [Music] all sometimes it gets a little bit creative like that okay that's a little bit better so you can see how we work with this particular function it's copy uh copy clip space and paste clip space using the mask editor inside of uh inside of cover UI means that we can actually edit any image at all and use the uh use the inpainting option for encoding to allow us to edit that specific area whilst leaving the rest of the area uh rest of the image untouched very useful when you have small parts of an image that you want to edit and you just want those parts of the image to change whatever everything else Remains the Same now this is not the be all and end all of uh generative fill or in painting inside of comfy UI but it is an indication of just how powerful the software is how flexible and Powerful the software is wish you [Music] mine

Channel: Pixovert

Views: 3,765

Keywords: COMFYUI, STABLE DIFFUSION, Generative fill, stable diffusion vs photoshop

Id: BAhmHXSmHQ8

Length: 14min 31sec (871 seconds)

Published: Sun Dec 03 2023