SDXL Base + Refiner workflow using ComfyUI | AI art generator

hey there and welcome back so let's roll another video related to stable diffusion and confy UI so this time let's focus in XL version SD XL which so far is the best one so previously we were about working with uh sdxl turbo the one that is almost real time and then we do with did some o scaling now let's do some since directly using all the high definition let's say model the X so it's very important I always point out to the to the website know the original reference in this case know the model car they call X SD XL1 and here in stability AI you're going to find all those uh model cars but what is important here you need to to know what you can do so this model have something particular that you can use the base model and then the refiner it's a twep model however it's not compulsory to do the refin instead and actually you have here the know that this is more than enough so probably you will find around some of workf flows using the refiner so it will be up to you to put that one personally speaking I don't use it for me it's more than enough this not for what I'm looking for so maybe some people that need something more professional more details will find some advantage in using the refiner then well you have more information there and just read there now from some details some there always shows give you some interesting tips no information so we're going to use the same promps that they have here so an astronaut right and a green horse so this one will be using the base model no refiner and this one will be using uh the base model plus refiner so standard practice here also recommended by the guy for stability AI it's like you do the base model 80% and then 20% you will do the refinement model so later we're going to see how to to do that is you know and at this point also I want to point out that the documentation that you have probably if you go to automatic 1111 you have much better documentation so I invite you to read here know it's very complete also something that you're going to find this documentation is the is the size that was used to train the model so this model was train with this size so you have to be very careful with that and well many much much information here at the end later I'm going to talk about this um talking about now comy you are the documentation is kind of limited you have this uh website where you have know some some partial information which is okay Sometimes some something is missing so in the video description you have the link so at this point let's move and let's work okay with these prompts that we have here so we open and we have a stable diffusion okay the default workflow also at the end of this video hopefully we're going to work with this okay by the way I don't recommend you here I have different tabs I don't recommend you to open multiple tabs no so you can put it there and then you have the other so I don't recommend you but it's up to you uh so this is the a stable diffusion the quite complex workflow so I'm going to I already went through these groups and so on so this this is just notes but here you see that you have the two steps okay base and refin step so at the end of the day we're going to see that but just to show you that you can do exactly the same without that refining step and using the default workflow which is more than okay so let me go and already have the prone here and is this one let me put it here let me raise the final comma later we're going to talk about that also I have here this number I want to fix the number so it kind of chat remember that it's a little bit random so different random numbers you're going to get different values and also remember to use the library so let's use the uh the Base Library so you can download the Base Library always here you have the download so here you have files and versions and remember that download the when you have different options download the safe tensor in this case this one then sometimes you have like checkpoints and M other stuff always go for the safe tenser uh uh so now let me click here so everything is okay have my values and let's see what we get okay also like here to put the preview so let me go and preview here I don't want to save that image so preview and let me bypass this let's see if it read it okay well didn't read it but let me go bum bu bum bu bu put it again change a little bit the value okay and there you go so see that is randomized I forgot to fix it so different value see that you get different compositions but pretty much it's not that what we're looking now this sdxl is photo realistic you should get quality photos so let me go my get again this one let me change here and let me fix this one and let me go here so so far this is cool I have no problem with that but I'm looking for something photo realistic so what is the issue here what was happening remember this data set was train with a dimension size of s24 so look at that now let me change to the right size so you can see that you have that dependence there in your latent Dimension so now we are using the right Dimension and let's see what we have in our outcome so we have the basic this is the basic uh set up which is more than enough to get impressive images and then if you want to refine whatever you have there previously will address the O scaling so you can use a few o scaling steps and that's okay that will give you fantastic result and this is very impressive already without the need of using that very complex now refining is Stu now going to refinement you already have something really really good so yeah this is it and for me this is my workflow I don't complicate s any more than this and to show you something else about the dimensions of your latent so now let me double my latent here so now I know that my model wasn't trained with this let's see what happens is I get something much larger will I get something better worse so let's see first is that you would realize that it's much much slower so you increase your latting this sampling here now this operation it will be much much uh slower so you are noing your data know using your diffusion model to get here your pixel space and then put your image there so let's wait a little bit also well in here in the pr you can see the time so the previous image look at it you have the times so the one cartoon is work quite fast 24 and now this is taking much much longer so the first generation also will take longer because you are you are loading this library but as soon as you load the library since we go faster so let's wait okay I'm back so that was much slower than I was expecting and running some other processes in the background so but it was a long generation so nevertheless remember that you you you increased your your Laten size will be much slower but this is also your output the output is is impressive very nice but usually this is what you get you're going to get no many uh repeated no figures and objects in your composition and this is just due to the fact that it wasn't train you are using the the wrong Dimension so when you start to see this one you know that there is a problem now with your Laten size likely but can be many things so be careful about that so let me go back to this dimension okay it's taking longer than expected to to get the image just because I'm recording and I have some other processes running on in background but let's see so now doing this to go back so basically remember your this is the SE the randomness so different numbers different outcome so be careful about that because sometimes you don't get a compensation that you you you don't like or you get some some strange artifacts you can fix that just by changing no random so will be something different but okay so for instance here you see that you have here the hoof and probably it's two hoof there so that is anatomically speaking it's not right so there are many ways to fix that okay but this is probably will be just to choose another generation and likely you will get what you want or probably you can change steps or CFG and so on usually this one I think when you have this now run Anatomy they didn't have a a strong influence it's quite difficult to control that so we go back to the issue of the of hands now getting hands with the five fingers and so on it's quite can be quite tricky and later I'm going to show you something so see that I change now this one even if you are in this uh resolution you get kind of something cartoonish okay and let me go back and just to show you the about hand so I was working here now I'm using this uh provider here and just to show you about all this generation with hand so I put my specific prono do this so it is really photo realistic it is amazing everything looks like a photo but then the hands you start to see that you have many artifacts so it's quite tricky to control hand so there are different models very specific for that but what I want to say here is that you are not going to avoid this with negative prompts okay you need to enforce that using your positive promp and okay so here we have our image and so on so nothing nothing is strange here so what I want to show you also here talking about these negative prompts um for instance they can have a very very strong influence in your in your output so let me go and add this here okay so as I say you I prefer to fix my composition here in your positive PR rather than in the negative negative put the basic stuff and to show you that sometimes you will find around some very uh very very elaborated prompts and that's just due to the fact that they're trying to fix everything there so look at this promp extremely elaborator and you start to look prompts they're really really elaborated okay the positive power so they try to get it right here in the positive and then the negative is small stuff it's not like you are going to fix extra eyes or backround faces or SC like that so focus in your positive PR so see that now I added these keywords Majestic photo realistic hyper realistic bouquet epic award winning and it's a much much better composition okay you don't have those two hoop anymore now it's running probably in the surface of the Moon I don't know but it's quite nice so something that you will notice also as you look here at the visor you will see that you have a reflection kind of if it wasn't the moon so that that will tell you that the the data was trained with some images there but any case that can be also fixed so you see that here I'm super happy with this no need to go and do more elaborated sense so let me go here erase that one and let me go to the starting prompt to starting image raising everything so you can see that positive they do have a very very strong influence I'm using also reminder reminder that this is every s know XD XL and there are many variations just to go back here the Jugger now is a fantastic one so you have many versions I like to work with version six now uh version seven which is the lest I still have some problems okay so it's quite ra R and there is another one that I'm going to show you the the last video okay later is also photo realistic but at the like the turbo is the Excel T so see that now we have this one okay I changed there and look at that the proms are so important that even is you add a comma look at that I'm going to add a comma maybe sometimes you're going to find some some PRS that they add that comma or two or three comms that can have an influence loot so let's see what happens with this comma there sometimes can be a big influence sometimes not much sometimes can fix your problems but as I say and I just want to to stress that fix everything in your positive PR so let's see what happens with that comma there and there you go so you see that it Mak it worse it change the little but it make it worse but you see the influence so now also let me change my sampler and let me use the oer ancestor this is the one I like to use so there are many of them it will be up to you to test all of them so I don't want to go into details there so that is just the mathematics just to do know the the noising and everything so different methods different outcomes personally speaking the older ancest and uh normal SCH will give me the the best results so now using older ancestor very nice so you fix everything much better than the previous one and let me erase that comma there that is there and let's see what happens now so we fix everything so these values you will see that is they are recommended now these are the recommended values so you can play with them but likely you're going to converge to the same value so usually you put more steps you're going to get more fantasy in your composition but it is more time consuming I would say that 15 will be the minimum accute minimum here in s sdxl and there you go you saw that there is a large difference with that comma that being said I'm happy with this and I I hope my advice stay with this simple workflow then if you want to do elaborated SS you can add the refiner and let me show you how to add the refiner so here I'm using the base library and and if I go back here remember that there is another step so you have a Laten space and you need to pass that so this is a little bit more complicated so somehow we need two and let me do something by the way let me show you also something about and let me go here back to the documentation this one in Twix there is sometimes you will see this click value and now the the process is click in minus 2 minus one or from the second to last here also let me go and let me put that information and here let me go clip and this you will have this keyword that sometimes a little bit strange people don't understand what is happening and let me show that my advice here R what is in here now so you will see that it has to be with the layers that you are using with your with your prompt but let me add that here so to add that one you go here this one and now you connect here and here okay so a small modification but let's see what happens so now this is telling you that here is where you have those no NE Nets and so on and how you are training everything so here you put minus one every single layer is going to be used okay to get the information here your promp and your Laten space so it will be you going to have more let's say if I can use that term more fantasy in your composition and here you you will have the explanation now the the earlier your stop the less layers N Network have work on your PR that is less fantasy that you can go here the same comment that you can find you might okay and there it's working and look at that this is the default volum in confy UI and this is with minus one so this means that you use all your layers to get this composition and if I change it to minus two okay we're going to get the previous value which seems to be the default in confi but also may be related because some of these modules they relate they're trained like this so they will stop in the second to last layer they're not they don't take all the layers and you increase this value and you will see that this composition there is less fun fantasy know so from layer to layer you are going to take these parameters and you have more fantasy so let me go and put here 10 and let's see what happens okay so it should be something that it will be very different from this and let's wait and there you go so you start to see less fantasy probably you start probably so you see here there a moon and this is what I say that they probably in this case we can use the negative spr to remove this that it was at by the model it's not part of your main composition this is your main composition position and here there is nothing to do you cannot say there erase the astronaut or erase the extra leg in the horse or whatever so here's that Randomness stuff that you see there and let's see if it works now if I put here Moon and let's see if it managed to erase sometimes it will erase but then it will change something else so coming back to this documentation here I recommended you have the comment here that many models have been trained with this minus two so you have to be careful that and usually also sometimes you can get better results with minus two so see that I put no more you have the moon there but then you have even the horse there is not very good so my advice there leave minus two or don't put that there it will be minus two by default or it will use the default no value in the model but this is something that you can control and many times when you when you try to reproduce and results not you see now that for instance you go here in one of these and you start to look for the information for for a specific image they're going to talk about that click value and that's that's what what do means that clip so that being said let me stay with this image and let's add the refiner step okay so let me go here oh oh no I don't want to open here have it there and to to add the refiner is a little bit different at first I need to erase this one okay so let me erase this one and I need to add a new one so you go to k sampler and you need to add the advanc one okay so remember there is a customiz one for the T B the standard one and this is the advance that we need to use for refiner so it works pretty much in the same way okay so I will go here and let me connect to the mobile let me positive negative PR you have your Laten image and then you connect here and nothing will change and let me talk a little bit this options so this is very important enable this one in your base model so basically this is telling add noise in each layer in each step add fantasy this one also it will say that's depend how you stop your process okay so later this will make more no sense let me go back and put and put here saying numbers here fix 20 so let's say that I want to do 20 steps this value let me use the ancestral SCH normal you have different so many people also recommend the caras there so it's up to you and here you have this one so you have 20 steps and these options also are better explained in and let me go back again to automatic 1111 so you have all this stuff better explains that act in Matic 1111 you will find some slid where how to control that those auctions that let me see now they are below here and you will see how to control those steps okay so basically start at zero so if you start at five means the first steps you are not adding any noise any fantasy so there will be some difference so for some specific purposes you would like to do that here in our case we want to do everything and now I want to say here stop at 15 let's say so it will do a total of 20 iteration it start from zero is stop at 15 it will be adding noise and everything and let's see what we get here so we're going to get an image but it's state stoping at 15 and then the other five steps I can connect that to another sampler and in that second sampler I can add another model that second model will be my refiner that is another specific model know to add more details so let's see what we get here and this is what we get so do not okay do not enter in par and panic this is okay so this is your image Latin space this is your your noise so this is what you are getting and now let's add the other step here so I need to add the advance sampler okay that this will correspond to my refiner and pretty much will be this similar connections okay let me also add let me double this one clone and here I need to load the refiner and see that I have that refinement remember that you can download that from the website of XD XL then this one you need to clip this one so let me let's connect this one here so you connect this model the positive my advice is to use exactly the same positive and negative PR so connect the SS there so okay I connected those there and then I need to create another Laten here decoder and a preview of my image I put it here okay so as you see when you start understand now this stad what is happening it's quite easy create these connections and let me see what I'm missing okay so here I'm missing okay clipping also need to here this one okay clip here clipping means connect there and then the buy is connected with see so now I have all my connections and if I click this what is going to happen you're reading now the second model you have already this Laten space ready and what is happening is giving me an error there what was let's see that was okay okay so yeah I need to also here so I have 20 steps I need to start in 15 so you have to be careful and r a maximum 20 let me disable disable uh let me put zero there and fix and let me use the same sampler so be careful here I start at 15 I have a t total of 20 steps that you put the same number here and then it start at 15 it stop at 20 and what is happening condition negative negative and clip clip by by okay so I'm missing a connection [Music] somewhere okay let me see so the stand the error it is here okay so let me see if I can pinpoint that error the clip the mod goes there deleting so let me disconnect this one remove okay so I have here a problem bu enable enable 2020 what was my problem here normal there I don't see the problem there okay okay so I don't see the problem there let me do something that I already have that somewhere here let me go and use my model that I already prepared so basically I was doing those connections and let me go I think should be this one okay yeah this one already prepared so pretty much is all this is spaghetti so look at that Advan you connect with your clip model okay your lat space and so on so let me do s a simpler one let me use a simpler one because that one have an extra layer of complexity okay up this one I this is the base this is the that is the basic basic so this is the one okay so here I have the three layers let me go Q so basically the first one see that 20 my parameters is stuck at 15 and then you pass to the second one which is this one this one is connected to my refiner so look at that my model base model refiner model is start at 50 and go to the last one okay so can put 20 there so it doesn't matter will stop always at 20 uh then I put here a random number doesn't matter this is important I'm fixing this one to get my same composition and this is the extra refiner so this is the noise I get at the beginning and then this stop here this step here this is my standard latent uh latent space like the the default workflow okay so see that I not adding anything so I want to have my base image this one that let me close it I'm going to get it here and then I'm going to compare now this one I stop at 15 and then I will add the refiner then here I have my base composition with no refiner so this is already something super super good and happy and here now I'm passing this lat to this second one and add in the refiner and there you go so I think it's quite clear the difference so as you can see it add a little bit more details not necessarily will be better so probably here in the helmet kind of is trying to put a face it's not very nice here you see that can check here the ho have a second hoof or probably a shadow something like that but clearly in this case you have more details so this example not taking the same now prompt used in by stability diffusion what out here now and using exactly the your same prompt I have a few issues but it's not a big deal hopefully you get an idea what is happening so this is basically the refiner we add more details personally speaking as I say I don't use it very often I don't see the point on honestly or at least for me I I don't want that level of details I can fix everything with my positive prompts and get all those details and and maybe later I can I can do some o scaling I find I found that o scaling is much better than doing this stuff and it is less probably it's less expensive so yeah this is it your workflow here is a little bit messy okay so I still added the the the clip is very helpful then be careful about your dimensions and here also use the same when you stop start the second one the first one you can fix the number and then this one you can do it random doesn't matter and now just to show you the diff What happens is I enable this option so basically add noise and coming back you have some limited documentation so here you can see what what what will happen with uh case sampler B so as you go here sometimes you have a short explanation what is doing that and so on so basically at noise it will add noise here okay so let's see what happens I have random values okay and let's see what we get and this is what you have that is the noise that you get so that's why we disable so remember that that noise is the fantasy that is being added that then somehow you will need to decode that to get something much better but that is what is happening you add that fantasy and so on and then this value okay so usually the refiner you disable this one and this is up to you if you want to disable I always leave it disable Okay I haven't seen much different but that depends in the number of steps so if you disable this one it would say that it's going to den noise your image just in the last step so here you have five five steps so the last one is where is going to do the the noise is you put it here every single step is going to the noise so if you have many steps and you enable this one it might be longer computation but it might get you better results instead in this case that we have an small let's say strange there uh the result is not going to change much now there are just five steps but if you have there 20 steps it might be a big differ no you disable that so it's up to you you can try so my advice here will be use five now five steps difference no it's okay you have more than five or 10 better enable Okay so let me enable here just to show the different so disable enable there you go and now let me disable and the difference basically it will be nothing or very very small very difficult to to get it here there you go basically I think there was no difference okay so it's up to you honestly you have many steps probably the better to enable to avoid no introducing errors okay so this is this workflow the the the we have now and let me go so my buys as I say I don't find any advantage in using that refiner for me it's okay I stay with this one this is perfect fine if you want refiner then you can just add a second step there so you will need to connect your boxes there in this case I have three sample because also I want to see my original image and now let's go to this workflow that seems a little bit complicated and just to remind you that you can put some order here so for instance you can select select here select here with control and then you go convert to group you give it a name like we call it group one and you put everything in a single group and you are putting everything in order so also this one you can conver already put here this one I parameterize the bottle so for instance right click here and let's say convert uh add noise to input okay and now you have this input here and you can create a box and you can parametrize in so let me go at no that will be a primitive and there you will have no the option at noise and S how to choose and then you can connect it with the other so basically what is happening here is that all those groups will will create it and so on and also we have right click add group and we have kind of this post it and you put everything that you put there it is attach so let's visit this one and from where I took this one this is the basic uh let me go ah I close that ah I close that no okay so let me go here and that was uh Co UI okay those are the basic examples so I have it here and here Hest stay here this basic example are more than enough and I'm using this one okay so our first in was the too now I'm using this one and here you have this images and so on so if you drag and drop the images this is very handy you take it and you put it there you get that so it's quite cool this for instance this one is a little bit different okay so you have different workflows just drag and drop the image and you will get it so let's visit what happen here here so what what is happening so you created a group there you right click so nothing new you put the tabs here you L your base and low the refer this is a note you can add notes right right click utils note so you put an a comment there so can be very helpful so read that so you load in the two models and then you connect here okay previously I know the buy was my problem okay so here you click here and you connect to the text here you have the text condition so basically what happened here is that you parameterize the text so all the models needs to get this prompt that are connected to a single one so you connect it there and there some coming there then this one you connect to your sampler so your sampler your links your latting comes from here and here also you parameterize steps so see that remember right click and you can choose what to parameterize no so you go conare and then you create this this entry so you create here then you create for instance let me raise here and I can go here add a new like this and it will be that entry so it's up to you to add your primitive now so previously Was A Primitive so we connect there you have so in this case 25 steps stop at 20 pass this information to the other one the refiner the refiner will start where this one stops okay we'll use the same number and then we'll go to the decoder and you have your image and let me suppress this and let me only preview the image and if you click create okay H I mess around with s and let me go the original one here preview and suppress right click bypass okay no I didn't mess I forgot that remember to select the location so might be different locations and there you go so you start to do your steps first step is the base model so here you have recall the previous one so let me go I have it here so you're going to get something like this then you pass that Laten here and that Laten you're going to apply the refiner and there you have so pretty much I think this these options that you get here you can use it at as default I prefer to do 20 steps but here they use 25 it will be a little bit longer so it will be up to you and there you go so this will be the default case that is always this bottle so it's a nice actually I have to say it's a very nice case you have your bottle there so you can run this one without this one and you will see that pretty much there is no difference it's very small so now just to end let's do the second case that the second prompt that I have this one so let's put this one here a majestic liion jumping whatever so what I mention about uh documentation and let me go ISD XL model car here they recommend like to do 20 80 20% no so 80% will be the base model 20% will be the refiner so you you do the steps now you look at the step we have 25 and then you do your mass you will see that 20 will be about 80% and then the other five will be that 20% so there you go so I have to say that this is this is very darn impressive now okay and let me go also I have here a specific value here so let me go for this one let me put here and I have some comments for okay so for instance this one let me do and this is interesting case no for benchmarking it's only benchmarking let's do this standard value okay and there you go we have even much better so probably this leg is a little bit strange any case so let me go now and I want to increase this to 40 now I have my coming here 40 steps okay and then you stop here at 32 which is about 80% and then that information you pass it there and that's all let me leave it like like that let me open this image there let me put it there to have it as a reference and let's produce so it will be long because now I have 40 steps okay so we have our output here so remember more steps more fantasy we have a different composition so 40 steps 32 doing the base eight doing the refiner refiner refining and remember here is up to you so I recommend it to this disable disable but may be the case that sometimes it might be better to enable this one just to to denoise every single image okay to avoid when you have a lot of lot of steps there so it's up to you but we have this nice composition canot compl play very impressive and let me go back now just to show you the influence of steps and so on but let me go back to the base case have okay and now this was the image and let's chance a little bit still if impress is not to say so let's say that now want to El there so go one at this point it's going to repeat we have a new composition and let's see what happens and there you go so we have our new composition so want do the it Chang also the landscape so kind of n one but okay very cool so this is how it works so the sdxl this is the refiner so as you see now demonstrate here that there is not much but it will be up to you for me this is more than enough okay I don't like to use that refiner I prefer just to go for the O scaling that there I can add some details so we covered that in a previous video so that's all for this video thanks thank you for your attention do not forget to subscribe and see you next time bye

Channel: Wolf Dynamics World

Views: 1,913

Keywords: AI, AI art, stable diffusion, ControlNet, generative art, ai art generator, run ai local, air art local, SDUI, install ai, instal stable diffusion, ComfyUI, ComfyUI manager, SDXL, SDXL turbo, img2img, txt2img, upscaling

Id: 0HM3qoBuhww

Length: 42min 42sec (2562 seconds)

Published: Thu Dec 28 2023