ComfyUI - 1 Image LoRA?! Check out this IP Adapter Tutorial for Stable Diffusion

hey everybody I'm back I've been traveling a bunch so I'm back now going to get some more videos out and today I want to talk about something very cool came out it allows you to take images and combine them uh kind of like a onstep Laura and it's very cool and there's some really interesting things you can do with it I think this is one of those tools we're going to be exploring for quite a few months because there's so many ways to adapt it into your workflow so what I'm going to do today is I'm going to show you these two images combined into another image but we're going to do some other things with it like we're going to throw a control knet in there to help guide it we're going to remove the control net we're even going to remove the prompt completely and there's some different things that you can do with this again I'd love to see what you guys create with it it's just another one of those little creative tools and we can combine as many images as we want but we're not using the image in a lot of them we're using the context of them uh so it's kind of like image as a prompt so this is called an image prompt adapter and you may have seen videos or heard about it or IP adapters has been shortened and that's what we're going to cover now again in my way of doing things I like to do them a little more creatively so we're going to walk through the basics first but um then we'll go off the deep end and you know take the red pill all right we're going to start out pretty easy today we're just going to load the default so we're just going to go over here and click load default and there you go and I'm going to spread this out a little bit because I don't like working this crowded during the demos things like this I like to see all the nodes are spaced out even though it's a little ugly I think it's easier to follow if you can actually see the lines instead of having them tucked behind things all right there we go that looks pretty good all right let's talk about the nodes we're going to need for today if you go on manager here click on install custom nodes I have my set to installed here so you can see it uh one we're going to have for today is this comfy UI IP adapter Plus right here you're going to need that one and we're actually going to also use this comfy UI sdxl empty latent because this saves us some time and I want to introduce it to you and then the other one we're going to use is actually going to use some control net in here as well uh because there's this thing can fly out of whack pretty easily and we want to control it this is a great way to do so uh this technique uh that I'm going to show you here I think has so much potential uh and really unique ways to approach uh just putting into your models so I'm going to show you a few of the things that I've discovered so far when I love to hear if you come up with some I think this is one of those like groundbreaking things now initially this is going to seem like revision and we talked about revision earlier uh a technique where you can kind of combine images and it looks like that originally or initially I should say but it's actually more of kind of like a one image Laura uh so that's kind of how I'm talking about it or looking at it so let's set this all up first thing we're going to do is going to erase our prompt completely we're going to use this and we are going to use sdxl but we're not going to use the sdxl um clip here we're just going to use this just for Simplicity sake but again when you're working on your bigger models and your bigger projects you know use the all the tools that you have available but we're not really kind of featuring a bunch of that stuff today so I don't want to over complicate this diagram but I do want to complicate a little bit so let's get rid of that and we're going to type in the word latent here if I can spell it and it's near the bottom here where is it sdxl empty latent image and what this is going to do is it's going to give us all of the resolutions that are safe for SD XL and what I mean by that is sdxl works at these resolutions and square which of course you know our first models were Square only this has a lot more capability especially if you work in the ones that it's safe for so use these if you want the best possible result so we're going to put that right in the latent there and be done with it I don't like my lines to or I wish my lines didn't cross here but they do but I don't want to look I didn't want that to line up like that that's the kind of thing that that can cause problems so I at least want them to cross violently if they're going to cross we'll just do that in fact we'll just go nuts here there that looks terrible let's keep it like that yeah this is the anti OCD today all right what are we here for all right so uh what we want to do is we want to actually inject something into the model stream here and what we're going to inject is a pretty interesting thing it's going to take an image and allow it to work with the prompt and it's going to combine the image and the prompt as it goes through its step-by-step process so this is a little bit different than revision where revision kind of used the sole of the image this is going to use the clip model to help us figure out what's in the image as well as the image itself uh so to do that let's go ahead and load it in so double click and type in IPA like the beer and IPA apply we're going to grab that and bring that in and again like I said it fits right in the middle of the model workflow so just throw it right in the M right in the middle there that's Happy we'll minimize this because we don't need to keep looking at that all right and we'll minimize the the -2 cuz we're not going to use that either we will put a positive in here later but for now we're not going okay so what do we do with this thing now it's it's pretty simple as I said it's going to take an image and then push it into the path of the model here and uh we're going to get some some interesting things out the other side so let's see what we need here IP adapter if you pull that out you only have one option other than reroute here we'll pull that up here and you're going to see there's a bunch of models that come with it at least on the uh link I'll put below for the git and you can see them here we have some that say face in them those are obviously four faces and there is not one for sdxl face at this point um now face would be if you're trying to make an image that looks very similar to someone else now if you don't need the face to be exact and if it's it's close enough then don't worry about using the face model which is what I'm going to do I'm not going to use the face model today because these are not people that I know they're just made up uh so the face and keeping that same face throughout this is not not my goal now we are going to be using sdxl here uh so you'll notice that there's a few options here we have a a a b and V is the one with more tokens so it's a larger and if we use this we must use the clip Vision model that also has VI in the name now you can use this same clip Vision with other ones um that don't have VI but just realize if you get an error it's because you're probably using that model here and not this one so this is the one I grabbed here off of hugging face so that's the one we're going to stick with and it's a pretty standard one so you should already have it on your machine if you don't well now you'll go grab it all right what we going to do here now we're going to load in an image so I'm just going to right click add node image and we're going to load an image and I'm going to change this one I'm going to load this one in here I I've used this one before and I really like this this this image is amazing to me like it's there so much going on that's fantastic uh so I want to use this one I'm just going to drag it in here to the image now right now the weight is set to one so we're going to get something that's kind of similar if we just run the prompt through here in fact let's just do that and not going to change well I'll change this I always put this up to like 30 or 35 and we'll keep this at at Oiler and normal there's no reason to change that but we'll change this to fixed uh just so we stop mixing our variables around and uh whenever you're testing our theory on how things work make sure you change these to fixed so you can actually understand the differences when you're fiding with numbers what's going on otherwise you're getting a random seed every time and it's really hard to track down what's moving so let's just cue this and see what happens and I totally forgot got to change my model there we are we're going to use sdxl base all right and you see we have a pretty similar image which is what we would expect because we did nothing except hey this this image is worth 100% of your goal just go ahead and and an image looks a lot like it and you know what it did it so congratulations you've done nothing all right so now let's improve it and here this is where it gets really cool let's take this image and let's load in another image let's load in this image here and uh I did a bunch of sci-fi stuff we'll be doing a bunch of this I love sci-fi stuff so that's always going to happen what I want to do is I want to use these images so there's only one input for image here we need to batch these together that's simple just drag this out and look for image batch put them together and now they are one and they go up in here now this is not the same thing as merging of images meaning they're not overlaid on top of each other there's no blending mode or anything like that they are simply two completely different universes for what we're doing here in fact they're not even the same shape so what this is going to do is it's going to look at not only the image itself but what's in the image and if you're doing this don't load the same girl eight times for example unless she's in a different scene and you want the aspects of that scene for example in this scene we really don't see anything that's sci-fi there some steampunk hat here but nothing is the smooth type of hard modeling here we find on this or the thrilling headgear or the outfit so these should be of equal weight again we have not added a prompt at all at this point and we're still sitting at one and no noise now noise means if we don't have any noise it's going to pass in basically a black image and say don't worry about the noise part if we add noise we'll get a little bit more detail and a little bit more interest in the image uh so we can do that but let's do it without that first so you can see we've only changed one variable and that is including of this image here let's queue it up okay and now I came back and you can kind of see that we're getting the combination of the steampunk as well as the Sci-Fi thrilling headgear and some of the more hard surface modeling here uh so it's kind of combining the spirit of them let's add one more we add this we're going to need another batch image here and this batch image is going to feed into this one you're basically daisy chaining them together uh let me do that a little smarter way so you can actually see what's going on here these two over with this one there that probably makes a lot more sense and that's what we're getting to I do want to move these closer though so we can actually see you know the spirit of them but just so you know how those are connected they're all daisy chain together what are we going to look for this last one just something it doesn't have a person in It Go for something like flowers here let's use this one I really like that one too all right so we got three completely different images here I think this was actually even done in M Journeys I recall I can't remember a while ago so all again all three of these uh doesn't matter the order they're put in but they're all going to contribute and they obviously have nothing in common although the color palette of this one and the color pet of this one are similar uh and we'll see what's happening now if if you got this already hold your horses cuz we're going to get more complicated here in a second let's see what this looks like by the way every time you add another image in here you are increasing the memory requirement for this so don't go overboard you know add them again if they're unique and present something to the image that's unique and isn't over represented so it's going to say what's the differences between these and include those items and there we go so we have some flowers we have thrilling headgear uh yeah it's all working well but what I really love is I love this pose and I want this pose now there's a couple different ways to do that but the easiest way is going to be using control net uh so let's go ahead and adding some control net so we're just typing control net and we have control net apply we'll just use that as our Point of Departure here uh so this obviously fits over here in the conditioning between the prompt we're not using and here and we're not going to put one in the negative side that isn't really going to matter here this means that the the um control net is more important so let's do the control net here let's roll that out control net loader and I want to load in the canny one we'll load in the canny for the sdxl remember the different control Nets are required for the different models because of their their different shapes and then the image here we need to pre-process whatever one we want to use I want to use this one here and we'll grab the cany edge pre-processor we'll put this in here and drag the image so this will this will find the edges basically and uh let's set this to 1024 because we are using sdxl now remember that we're not dragging in this the combination of all three of these because that Cy Edge is going to look totally bizarre CU it's going to be all three of these overlaid on top of each other and that's not what we want uh so we're just going to do this and for strength uh I'll put it like down to 70% so it's not perfect but it should be this pose let's try it again again we are using the locked um seed so we aren't going to have a totally different image but now it's going to take that same image and lock it in with that control there we go so we have now our canny with the interesting pieces that we've added from these other ones uh so that's working pretty well I'm pretty happy with that yeah all right so what else can we do with this well first of all we're not even using the prompt and we're not using the noise so let's do that let's add a prompt here we'll choose a female astronaut and we'll we want to drop the weight so the prompt has a chance to play because right now it's not we'll put that down to say 70% and let's run that without the noise first you can kind of get the idea of what's going on here and there we go so it's a little bit more interesting it's cleaned up a little bit uh if if we were doing this a little more seriously we in the cany part for example we'd stop near the end to prevent a lot of those lines from leaking through but this is coming together rather nicely so it's adding some of the additional sci-fi detail from the last one we can give the prompt more information here and then give it more weight if we'd like but let's add some noise in here let's add in like say 30% noise somewhere in there and run it one more time in fact I'm just going to copy this over and hook this up this way so we have a before and after I think that's kind of an easy way to look at let's I'll use control B which is bypass uh and that will allow it to continue on without this and we can come back to it in a second so you can see that introduction of noise allowed it to a little have a little bit more freedom as it's interpreting the image again it's still using the control net to hold this pose in this face and the hat frankly U but it's using these Styles here so if we start to eliminate these we'll see that there's a big difference for example if we use this one as the cany control and get rid of this image completely kind of rearrange our family here let's this oops so basically I'm eliminating one of the images so we just had these two and we're using cany on this one let's try again oh we got to buypass that again actually I'm just going to get rid of this since they're not side by side anymore anyway and there we go we have a nice combination of these two images in there along with the prompt driving the Sci-Fi theme forward now the the part of this that I think is more interesting I mean this is all good and I really like this but where I want to use this is I'm going to get rid of these and this is kind of how I'm using it to keep just this and I don't want the cany we're not going to use any control net for this actually sorry if you control net fans are like what it's fine we're just going to use this image as it is but what I'm I'm going to do is I'm going to put a texture in here and I'm going to run it at a really low weight well not really low but say 20% um with some nois is fine and we're going to use the same thing but I'm going to let it render whatever it's going to render so the Sci-Fi theme we saw earlier with the same seed will show up uh but it's going to introduce this which will help Drive the scene so so this will change things I did a video oh man months ago on how to find scenes in textures and this is really kind of coming to fruition here here and working pretty well here let's see what we get there we go so you can see that it's using this color scheme here which I really like well I think this is a fantastic tool and there's so much we can do with it uh so I would love to see what you're going to do again this is one of those things I think for months we're going to be playing with it and adding it to workflows and I'm going to toss it into my workflows even if it's just at a low weight because allows me to use some of those handmade uh handmade textures that I've made before and bring them into this which I think is really cool so it adds that little bit of me into what is otherwise a computer generated image and of course I want to take this and play with it further so I think this is really great and I'd love to see what you guys are going to do with it uh so Post in the comments below and let me know what you think again thank you everybody for supporting this channel you guys are fantastic and again we couldn't do it without you uh so I really appreciate it and because of that this graph will be available in the community area uh for the sponsor level and grader uh so thank you once again for you guys for helping me out take care stay safe and I'll catch youall next time

Channel: Scott Detweiler

Views: 32,979

Keywords: stable diffusion, stable diffusion xl, ComfyUI, stable diffusion tutorial, AI art, SDXL, comfy tutorial, sdxl tutorial, sdxl template, ai art tutorial, ipadapter, comfyui tutorial, image prompt adapter, stable diffusion beginner tutorial

Id: xzGdynQDzsM

Length: 16min 23sec (983 seconds)

Published: Fri Oct 06 2023