SDXL ComfyUI Stability Workflow - What I use internally at Stability for my AI Art

hey everybody Scott weather here and today we're going to walk through how to do sdxl inside of comfy so this is kind of the core to our let's say official workflow but one that a lot of us use and this is our Point of Departure so there's a lot of other things that we're doing in here but this is the same from person to person so I kind of want to pass this one on to you as I say this is the one you should start with and then get creative I'm going to show you how to make the basic one first and then we're going to convolute it so over this course of this video we're going to do some things that you're not going to expect and they are things you cannot do inside of automatic 1111 yet I'm sure that along this road there will be the ability to do different things with the refinement passes we're going to show off so let's get to it okay so this is going to be the core graph that I pretty much use whenever I'm doing any sort of quality assurance it's stability so this is kind of the the simple one they can get very complicated but I want to show you at least how to start out and this is going to get a little complicated near the end of this video as well so even if you're a person who uses company an sdxl I will show you some things that you may not know so first of all we need to load in our checkpoint so we're just going to right click add node and loaders and checkpoint now no I may have a lot of nodes here you don't have I wouldn't worry about those we're not going to use a lot of those today but again you can grab these from Civic has most of these available so we have our checkpoint here so let's go ahead and load in our sdxl once we have our base loaded in there's also a refiner we're going to deal with that in a minute and we're going to complicated our lives with that yet so here I'm going to just change the color of this to Blue just so I have some sanity with the graph it's going to get complicated right now we need to condition the clip coming out of here and sdxl has its own conditioners for that and I find it easiest if you just double click and choose sdxl it's easy to search for and again we have two right here we're going to want the one that's the regular one not the refiner version yet so we're going to just drag the clip into this and then here we have the width and height and then we have a Target width and height so this width and height I usually put this to a 4096. yeah and then we have a clip G and clip element these are our positive prompts and for the sake of Simplicity I'm just going to make these the same but I also don't want to type these a whole bunch of times because we are going to use the positive and negative prompt uh repeatedly so I want to do is right click here and I can convert these to node inputs so now there's a node input up here and the space has gone away so same thing for this one and what I want to do is I want to put text into this so the easiest way to do that is with a primitive so if I drag this out I can add node here and under utilities you're going to find a primitive and the Primitive can be a string or an integer or a float or what have you so I have this here for my positive prompt and we're going to use it in both of these areas here now this set of nodes here is my positive prompt so I'm going to highlight both of them with the control key and obviously I'm on a PC so I'm going to say control but if you're in a Mac you know that that's not command key so just convert so here I'm just going ahead and hit green now if I control C to copy these so again if you just highlight them both oops I don't both hit Ctrl C if you do Ctrl shift V it will copy them with the connections in place and we need those now we're going to go ahead and change the color of these to Red so we have our positive and our negative prompt I thought for our positive prompt it's a robot a robot shopping at Walgreens now the negative prompt isn't as important and I know a lot of people go nuts inside the negative prompt I mean there's so many things in there that are just urban legends that you do not need to be putting in there to begin with so you can leave this blank or just put in something that is the opposite of whatever this scene is it's like um rocks I'm gonna be good I don't expect to see any rocks inside of a scene with a robot shopping at Walgreens so why not just put that in there so the the idea here is kind of the the opposite feeling or concept of what you're trying to do at least that's the way in my mind I like to look at it and I think that just works easier than coming up with this list of things like extra fingers which don't work how you think they work anyway all right so now we have our conditioning we need to do is we need to sample from this and we're going to use an advanced sampler so we can again you can right click and try and find it but again I just find it easier if I just type in K sample and I don't want this one I want the complicated Advanced one it's not that bad but it does have some more capability which we do want to use so here's my positive and my negative my model and then we're going to need an empty latent that's kind of the base noise for this entire thing so again you can do add node latent and then you can add empty latent again with this model it was trained at 10 24 so you're going to want to use uh that size or larger I think we're going to have quality issues if we make a much smaller image so I'll just keep it right around this size obviously it handles different aspect ratios as well oh that's beautiful and then obviously from this linked here we need to decode it and we see vae decoder here I just drag it out and let go and usually there's a pretty good guess as to what what you're going to need here use this vae and of course then we're going to go ahead and preview the image now remember save and preview are different preview is just going to show it to us and say it's going to show it to us as well as later so whichever one works best for you I think I'll just use the save one for giggles again so you notice it has filing prefixes set to come to UI and you can change this to whatever you'd like but I'm going to leave it right like it is and that should be it now let's talk for a minute here about the sampler because we're going to run into this in a bit and I want you to kind of be clear about a few things first of all we have a start step and an end step and you notice the end step is set to 10 000 by default you're like well that's just nuts well this this relationship between the number of steps it's going to perform and the start and end steps is an interesting one so if you start fiddling with this end step and the number of steps you're going to find out that you can run into some situations where the sampler kind of throws bizarre things at you so again you might have some sort of like creative moment and go wow this I found some sort of really neat formula in here but in general I would tell you to just leave end steps at ten thousand and focus instead on the step start and then the number of steps so in this case we're just going to do 20 and starting at zero and this is fine and then you can choose whatever sample or you'd like I prefer this DPM plus plus the simple differential equation GPU version so the sde GPU that's what I'm going to use here and then for the seed I'm just going to choose just a regular value here of four and then I'm going to change this to fixed so I don't want to have a whole lot of moving variables when I'm trying to kind of configure something once I've got it configured then I'll start randomizing things but trying to track something down with all random numbers is difficult so this should be a good start graph and there we go a robot shopping at Walgreens and that's a very basic comfy graph for sdxl now now let's make it more interesting so we've got this and it's fine but we can do better and that's where the refiner comes in so to bring the refiner in we're going to load it just like we would another checkpoint so I'm just going to alt click and drag this out and then select it from the drop down and then from here I want to use another sampler to kind of work with it but the the clip conditioning is a little bit different for this one I'm so going to notice before when we we search for the clip conditioners for sdxl if we type in sdxl we have the refiner version so we're going to grab this encoder here and we're going to drag our clip in so this is our positive one and you notice that has an a score this is the aesthetic score and six seems to be kind of a sweet spot for this in our opinion so we usually start here and then for the negative one we're going to use a 2.5 again feel free to fiddle with these but again these are a good Point of Departure and if you're going to make this graph something you're going to reuse a lot I would save these as variables so again you could drag out other Primitives for setting your height and width just make sure they're divisible by 16 for best results so here again I'm going to go ahead and convert this text input for both of these and this is going to be our positive one here so we'll just change that green and we're going to borrow the same positive prompt here and this one here well oops sorry this one here red and again yeah and you can also click this dot to minimize that thanks to somebody who pointed that out last time I never knew that I kept right clicking and collapsing so thank you for that tip right now in this case I have my sampler and I'm going to just duplicate it and what I want to do is I want to I want to do this step after this one so this is our first sampler and then I want to do the refinery and to keep track of these I'm actually going to change this one to blue because it is using the base here and we'll change this our refiner purple and then I can change this one to purple as well so we kind of keep track of which refiner is using or which sample is using which which thing so in this case we have this model so we grab this model here we have our positive negative and then our latent image is going to be the one that comes out of this base we'll just grab that drag it up here now I kind of want to see the difference between these two things so in order to do that I'm going to leave these attached to this base K sampler here so this will be our base image and again if I want to I could change the color of this one to Blue so I can keep track of the fact that this is the base sampled version and then we want to look at what's coming out of this and I'm just going to borrow the same thing here in fact I think I might change this one to the preview because I don't intend to keep this node I mean or this this image here let's just drag this out as a preview image instead of it being a save image that way we don't have to deal with a bunch of junk on our hard drives which I'm sure none of you have the same problem I do with about a thousand things you love and then you have to sort through them again later so in this case I have this one here and it's going to borrow latent off of that and the vae has to come from this this refiner here as well so just drag that over so we'll have our final one in here now this is kind of where the fun starts is is looking at these two Samplers and figuring out the differences between them and where you want one to work so the base sample here we want to make sure we pass any leftover noise to the sampler this is like a best practice and in order to do to have the sampler do something I want to leave some wiggle room so I would probably bring this down to something like maybe 12 or 13 steps I wouldn't do all the work in here and then from here we actually do 20 more steps but let's start at step 12. so again I would use some other nodes to kind of help control these things but for the sake of this demo we're just going to do it the old-fashioned way so we're going to start with the base sampler we're going to start at zero and go to 12 steps and then we're going to pass it to the refiner at 12 steps and then finish to do 20 more so this this should work just great run it and see what happens note that it starts with the last changed item which would be this one and then it's going to go ahead and load the finer in after this step it didn't have to load the base warnings that already had it here's our base image oh and I'm sitting here wondering why it wasn't working and I think I see it right away and that's this negative one here doesn't have the clip so you need a positive and a negative in order to make this function so I'm sitting there staring at it going why isn't it moving there we go so make sure you have both of these connected there we go now it's loading in the refiner it does get to be a little bit of noodly goodness all over the place and there we go so we should end up with a little bit more detail and let the refiner kind of do its job which is obviously to Kind of Perfect the image you're handing it let's see if we get these about the same size I usually overlap them a bit so I can kind of see like the like but yeah see we have a little bit more detail and so on obviously we have more steps as well so that also helps but I think the refiner is very good at fixing some things like faces for example it's not again but it works off again a different model has been trained on different content to help again refine the image all right so this is all well and good but let's add one more weird convolution that I promised you would be finding finding interesting and that is we're going to add one more refining step but we're going to add it before the base step so kind of a initializer or a latent noise conditioner however you want to call it so I'm just going to grab both of these or all three of these and say scoot these over and we're going to use one more of these refinement version so I'm going to copy that and then paste it with that V trick again because most of this is correct except that the latent noise is going to be the empty latent noise we're going to hand that in and then this latent here is what we're going to pass on to the base here instead of it going from the other the other one down below the empty we're going to come from so there we go so it's kind of simple we just tuck that in there now what are we going to do with this where does this get interesting and that is with the steps and the start so we're only going to do three steps let's just bring this down to three and then we're going to start at Step Zero and then here we're going to start at 3 and go up to 12 and then we're going to continue on our array way so this is the first three steps are really roughed in by the refiner instead of by the base so you're going to get a little bit different result up this is just a kind of a fun little concept of conditioning the latent noise before we actually start seriously sampling it obviously we're always experimenting with different things so I mean feel free to play around and see what you can come up with again there's all kinds of crazy ideas and thoughts we try so feel free to play around and see if you can come up with some super secret ingredient that you find interesting so pretty fun uh really interesting example of uh robot at Walgreens so this core graph once this is saved off and the save image obviously went to my hard drive I can just drag and drop that image in and it will reload the entire graph for me so this is a fantastic way to kind of continue working on a project but note that if you don't want these shared you make sure that you strip the metadata out of it like converting it to a JPEG things like that so that you don't accidentally share your entire graph with the world now I'm a big fan of letting people see how this stuff is done obviously so I'm I want you to know how to do it but I also want you to be protected in case you come up with something and you don't want to share with the rest of the world at least now you know you need to avoid sharing the actual PNG because it will have the metadata embedded in it but this is a this is a rough graph a good place to start and then get creative with it so this is the way that I do it at stability and again we we kind of take and convoluted it from here but this three steps uh is a pretty interesting concept again this new this new idea of conditioning the lateness something we're just starting to play with a bit or at least I am personally so it's uh it's fun so let me know what you think in the comments below hey everybody thanks for joining me today uh so this was kind of our core that we use here at stability so everybody kind of does their own thing but this course pretty much the same uh for most people and then we take and run and get weird with it so if there's things or ideas you have or suggestions on your own workflow you find interesting I'd love to hear about in the comments below but if you love the video give it a thumbs up I really appreciate that and of course a lot more videos are coming so let me know where you're getting stuck or if you're having problems and I'll try and walk you through it so thanks again once for joining me everybody take care stay safe and I'll catch you all next time

Channel: Scott Detweiler

Views: 34,333

Keywords: midjourney alternative, sdxl, stable difussion, ai art, comfyUI, automatic1111 alternative, stable diffusion, stable diffusion tutorial, comfyui tutorial, sdxl workflow, stable diffusion sdxl

Id: 2Xe79Nl_6jA

Length: 16min 44sec (1004 seconds)

Published: Fri Jul 28 2023