ComfyUI - Make Every One a Masterpiece in Stable Diffusion with Model Merging by Blocks

[Music] pixel now offers a range of courses in generative AI including a beginner's guide to stable diffusion and other courses in generative AI including sdxl and comfy UI if you're interested use the link at the description to enroll and get started on a new Direction so in this video we'll talk about checkpoint merges this is model merging inside of comy UI comy UI has a lot of features that allow us to do model merging and this is a fairly complex workflow we have a testing area we also have a merge area and we've got a selection area where we select the checkpoints and the vae so if you watched some of my other videos on model merging you will know exactly what we're dealing with we're dealing here with model merging and testing but we are dealing here specifically with model block merge which is allowing us to merge very specifically different blocks in different ratios so this is one for the more advanced users uh there are much simpler workflows for model merging if you are someone who is just starting out and I wouldn't recommend this particular technique if you are someone who's a bit of a newbie so we'll take a look uh we'll do a very quick tour of what we're seeing here we've got the model uh title there we've got the checkpoints and here you can select different checkpoints for your outputs the the main checkpoint is called R1 a a r1b is the second one and R2 is the third one and what we do is that we choose our vae here now model R1 a comes in as model one model r1b comes in as model 2 and then the output from this one comes to this guy here model R2 uh and this bad boy is just going to take the uh input from the third model so that's going to be this guy here and we'll take the output from that one connect it here and then we'll put in different ratios for this different merge block here so the output from here will go to a save checkpoint which is going to be by default muted and to also a k sampler and the K sampler will be very familiar to you it just allows you to actually just work with the testing so the testing is here set to batch size of four we can see the four outputs here you're going to be changing settings mainly in terms of which checkpoints you're using and also in terms of the the ratios so here we have a render that's using three checkpoints and the ratios are sort of like the the 0.5 0.5 0.5 ratios what we're looking at is an output which has a lot of color the models all female models blonde hair lot of architecture I've really tried to use the prompting to test the things that I want to test which is the architecture the the colors the realism the photo realism how did the female models look prompt his beautiful architecture we've got ancient princess futuristic city this is the creative part VCA that gives us some photographic detail and we've got blonde hair photo realistic now the blonde hair in some situations will give you this kind of outcome where you don't just have blonde hair but everything is turning kind of the same color as the as as the blonde so what we do here and I've already done it we've gone ahead and changed the ratios we can actually see what the results are because I've already run it so this these are the results with those ratios with a newer setting of ratios we get a different result and there it is you can see we have much better color separation we have I would say somewhat more realistic looking models some somewhat more realistic looking females the AR tecture I think is somewhat drab uninteresting the colors are good so we've moved forward in some areas we've move backwards in some areas I would want to save this so what would we do if I wanted to save this this looks to me interesting we would come here crl M to unmute and then we would just basically run the prompt again and what that would do that time around it will just save the checkpoint and the checkpoint here is saving to an area on my it's one of my drives which has a huge amount of space this is where I save diffusions that I create I save them to a large hard drive is a slow hard drive so they get the name you can change the name to whatever you want to to to have and this will only save when you ask it to save so most of the time you want to keep this inactive and uh so most of the time it's going to be switched off the checkpoints you can create sometimes are fairly large and you don't want to be saving these huge checkpoints each every time so only when you've got a result that looks interesting you would say okay I like this result let me save this one and then afterwards you can really start doing your your hardcore testing these are the new ratios these different ratios gave me the results that we got and as long as I'm getting fairly consistent results with the different ratios if I hit the run the Q uh button with the with a different number we should see what sort of result I'm getting am I getting the same kind of consistent results in terms of color in terms of color contrast in terms of realism in terms of architecture okay once again architecture that I'm not very happy with the the the the female models they kind of look okay I I I like the color separation so I'm getting consistent results that means I've got something that actually mean something it's not just random chance that I got a particular outcome I would then say let me save this this is something I can test again in future there are simpler workflows this one is really for the guys who want to have something that gives more control now I do know that there are far more complex ways of actually doing model merges there are other uh much more complicated let's do a [Applause] search this one here for instance model merge block number that that gives you a very complex uh situation that you you can work with so you can get more complex than this but this is a pretty decent one if you want to check the results that you get with different checkpoints different ratios and you really want to finesse the results that you're getting in terms of the the images that you're creating now as well as doing this we can also add a Lura file and I can very quickly show you what that looks like so here we've added a Lura file and we can actually output the new model with the Lura file actually attach attached to it so the lower file here is creating this kind of modern looking fairly dark fairly austere but pretty interesting results and what we have to produce that is basically a Lura and that Lura is attached right at the bottom there and then we're outputting to the checkpoint and we're also outputting to the K sampler from that from that Lura file so that's another way of working you then end up with this particular Lura whatever you've chosen embedded within the the new checkpoint so it's something that you want to use maybe occasionally when you want to completely embed allur inside a [Music] checkpoint

Channel: Pixovert

Views: 1,485

Keywords: Model Merging, ComfyUI, Stable Diffusion Models, Machine Learning, AI Design, Deep Learning, AI Technology, Model Integration, Artificial Intelligence, Software Development, Stable Diffusion Tutorial, Model Merging Techniques, Tech Trends, AI Modeling, AI in Design, AI Integration, Machine Learning Design, Advanced Design Techniques, AI Creative Process

Id: veGopJPn45Y

Length: 7min 53sec (473 seconds)

Published: Fri Dec 08 2023