Run SDXL Locally With ComfyUI (2024 Stable Diffusion Guide)

so it's been 10 months since I've made a stable diffusion tutorial and in the world of AI 10 months is like forever so so much is changed there's been better tools that have come out better processes to run stable diffusion and to be honest it's actually gotten quite a bit easier to use stable diffusion in this last video 10 months ago I showed you how to set up stable diffusion using the best tool at the time which was automatic 1111 in this video I'm going to show you how to set it up using comfy UI which most people who are using stable diffusion these days seem to find it a better user interface first I'm going to quickly talk about why you might want to use stable diffusion as well as why you might want to use comfy UI if you came to this video and you just want the comfy UI installation tutorial I did timestamp this video so you can jump straight to the installation process so with tools like mid journey and Dolly 3 and Google's imagin and all of these AI art tools available why would you want to use stable diffusion over the Alternatives well probably the reason most people choose to use stable diffusion is that it is free and it is open source it's also completely uncensored so you can generate any images you can imagine and I mean literally anything I've accidentally created nightmare fuel in the past also since you'll be installing stable diffusion locally on your own computer you don't really have to worry about privacy concerns or companies collecting data from you because you can use this without even being connected to the internet if you want and finally stable diffusion is ultra customizable you can use things like control net to get the exact pose you want and there's all sorts of other add-ons that extend the capabilities of stable diffusion so this is automatic 1111 here this is what I've used for the last year and a half to generate images with stable diffusion it does the job but there's definitely some downsides to it now this is what comfy UI looks like it's sort of this visual workflow where you can see the arrows pointing to the next step in the process so you get this full visual of everything that's happening during the processing of your image and once you've used it for a little while you feel like you have a little bit more control over each step in the process now other than the reasons that I just mentioned comfy UI is actually a lot easier of an install than automatic 1111 you'll notice once we get the process started we'll have it set up and running within minutes you can also save entire workflows using UI so if you get a flow of the perfect prompt the perfect negative prompt the perfect model the perfect GFC scale and you have every little element dialed in the way you like it to make the style of images you want you can save that entire workflow reload it anytime you want and just tweak the prompt to get exactly the same style again and with that ability to create and save these workflows you can also use other people's workflows here on civid AI we've got the comfy roll SD XL templates and you can see these are custom workflows a simple intermediate and advanced workflow that somebody else created that we can download and just use to jump start the process now in order to get comfy UI to work really well you're probably going to want to be using a PC with either windows or Linux installed and you're going to get the best results if you're using an Nvidia GPU now you can have it processed through the CPU it is just way slower when you do it that way and if we look at the comfy UI GitHub page page we can scroll all the way down here and you can see that there are instructions to use Apple Mac silicon AKA an M1 or an M2 or now I guess an M3 and there are instructions but I have read over on Reddit that a lot of people have had a lot of issues getting comfy UI installed using these instructions however I got you covered I did find this YouTube video from AI animation called install comfy UI on Mac OS M1 or M2 he uses a tool called home brw to get it installed so if you are a Mac User and you can't follow along to my PC installation instructions I'll link this video below and you can follow along to his Mac install instructions and then pretty much everything after it's installed is going to be completely the same cool now that you know why you might want to use stable diffusion why you might want to use comfy UI and what specs you need to make it work let's walk through the install process like I mentioned it's super easy super quick the longest portion of this entire process is just going to be you downloading the actual files that you need so to install this you're going to go to the GitHub page which you can find over at github.com comfy Anonymous comfy UI don't worry I'll link up below so you can just click the link and come straight here once you're on this GitHub page we're going to scroll down all the way until we get to the installing section there's a big blue link here that says direct link to download you're going to go ahead and click that link and it will download the installation files once you've downloaded the install ation file we'll go ahead and paste it in the folder where we want to install comfy UI in my case I'm installing it on my e Drive here in a folder called Ai and then I'm going to create a new folder called comfy UI we'll jump into this new folder and we will paste the file we just downloaded into this folder now you'll notice the file extension that we just downloaded is a7z file that's a seven zip file I personally use a tool called windzip on my PC to unzip files so if you have windzip installed on your computer that should unzip it for you there's also a tool called szip which is what a 7z file was created with if you go to 7-zip.com to windzip here and just click unzip to here and you can see it will start to extract the files it is is a large zip file with a lot of little files in it so it might take a few minutes to unzip all right so it did take about 5 minutes or so to unzip this because it is a very large file but once it's unzipped you can go ahead and delete that 7z file and now we have this new folder comfy UI Windows portable double click into this and you are presented with two options here you've got run CPU and run Nvidia GPU if you do have an Nvidia GPU on your computer you're going to want to use this version if you're not using an Nvidia GPU you're going to want to use the CPU version which will be a lot slower but it should work let's go ahead and double click on run Nvidia GPU you might get some sort of error saying that Windows is protecting your PC because it doesn't know what this file is you can click more info and then run anyway and then it will start the Run process now one of the beautiful things about this way of installing comfy UI is you don't need to download python or get or any of those tools because this installation process takes takes care of all of that for you and after about 30 seconds we're in comfy UI it's open and we can move stuff around and if you need everything a little more separated so you can follow the lines a little bit better you could drag it out however you want there's just one problem we don't have any models installed yet if I click Q prompt you can see we get this error that the prompt output failed validation that's because if we jump back over to where we installed comfy UI we come into our comfy UI folder here and we go to models and then click on checkpoints you can see it says put checkpoints here we haven't actually downloaded any stable diffusion models we only downloaded the front-end user face to run stable diffusion so now we need to put a model here and I recommend installing stable diffusion XL this is at the time of the recording the newest and most powerful model available in order to download stable diffusion XL you're going to come to this hugging face URL up here which of course I'm going to link in the description so you don't have to remember this or type it all in you're going to click on files and version and scroll down and download this sdxl base 1.0 saafe tensor now it is a 6.94 GB file unless you've got Ultra fast internet it could take a little bit to download but go ahead and download this file by clicking the download button here you're also going to want to download the sdxl 1.0 refiner model card this is part of the sdxl process and again I will link the hugging face page below you're going to click on files and versions scroll down and we're going to download the sdxl refiner 1.0 saafe tensors again 6.08 GB keeping track we're up to over 12 GB of file size for our stable diffusion checkpoints here thirdly you want to download the sdxl vae file I'll link it up below but go ahead and download this one here sdxl va. saaf tensors go ahead and download that I've already downloaded all three now I just need to move them to the proper places so that comfy UI can find them so we'll start with the sdxl base 1.0 saafe tensors that we downloaded this is our base model I'm going to go ahead and cut it from my downloads folder here jump back over to where we installed comfy UI click into comfy UI click into models click into checkpoints and we will paste the checkpoint here all right let's jump back to our downloads now we have our refiner let's go ahead and cut the refiner from the downloads folder jump back to our checkpoints paste this in the same folder here cool so now I've got my base and my refiner both in my models checkpoints folder inside of comfy UI now we've got our vae file so let's go ahead and cut this I'm going to hit back on checkpoints but down here we've got a folder for vae let's go ahead and double click into this you can see it says put vae here so we'll go ahead and paste that there and now we should be ready to run sdxl through comy UI so let's jump back over to our browser where this comfy UI tab is open and let's go ahead and refresh it and you can see down here under checkpoints we now have sdxl base safe tensors and if I click to the right we've also got the refiner we're going to go ahead and leave it on sdxl base so now it's going to load this sdxl base checkpoint here this top box that says clip text and code this is our prompt so the default prompt that's here is beautiful scenery nature glass bottle landscape purple Galaxy bottle and then this one right here it says clip text and code prompt this is our negative prompt here so anything we don't want text or water Mark is what they've got by default down here it says empty latent image this is where we can set our width and height and our batch size so if we want it to generate four Images at a time we can set the batch size to four we just want it to generate one we'll keep it at one this is basically the starting point for our stable diffusion image we've got our seed here if you leave this seed and the control after generate at randomize it will create a new random seed every time you generate you've got your steps here I usually like to go between 30 and 40 steps this is just kind of how long it sort of bakes the image for I guess you've got your CFG here by default it set at 8.0 this is sort of how closely it's going to follow the prompt that you gave here but it does seem that a number between the 7 and 10 range seems to be about that sweet spot but it is something that you can play around with for sampler name we've got Oiler Oiler a and a whole bunch of other Samplers that you can play with I'm going to go ahead and just leave this on the default boiler you have some alternate schedulers here these are other things that you can play with and see how they affect your image and then you've got your D noiser here I'm going to go ahead and leave this just on the default of one for now and then over here this is how it's going to save your image so if I click into this I can actually change the name right now it's just going to name it comfy UI and if I click Q prompt we can see that it will run through all the processes and then generate an image for me by right click on this and click open image you can see here is the image saved to my computer at 5 12x 512 however there's a way to make comfy UI even more powerful you can add more of these nodes and have it do more stuff now the best way the easiest way to add more nodes is to install this thing called comfy UI manager you can find it at github.com well this URL here it'll be linked up in the description we're going to go ahead and right click inside this folder click on open in terminal and then you can see inside of the terminal it already opened to the exact right path for us by right click it will paste in this get clone that we just copied from our GitHub Page by hit enter it will then proceed to install the comfy UI manager I can go ahead and close out of this terminal window here you can see I now have a comfy UI manager folder available inside of my custom nodes folder here now if I jump back over to comfy UI come up to my refresh in the top left up here you can see it now added a manager button that wasn't here before by clicking to this this manager button we have a whole bunch of new options most importantly we have a button to install custom nodes now a node is essentially each little section here our load checkpoint our clip text encoders our empty latent image our sampler our decode and our save image these are all different nodes so by installing more nodes we can add more pieces to the process let's click back into our manager here install custom nodes and for example let's say we wanted install control net I can search for control net and you can see right here we have comfy UI control net auxiliary pre-processors I can go ahead and click install on that once installed it's going to ask me to restart comfy UI so if I click restart and then press okay next to reboot server it's going to go ahead and uh install control net and then restart for us now that comfy UI has rebooted if I right click anywhere you can see I've got a menu for add node if I come down here you can see I now have control net pre-processors and now that we have the control net node installed we also would need the control net model so if I come to my manager here again click on install models let's do a quick search for control net and you can see we've got different control net models that we can install here and for this example let's go ahead and install open pose since we're running sdxl I'll grab the open pose model for sdxl here now that it's installed I can close this we'll close our manager click refresh and now if we want to add control net into the mix you can come up here add a node let's add an image load image node so this is going to be the image we're going to use for our control net where it's going to try to find the pose of it and we can add a node come down to our control net pre-processors let's click on faces and pose estimators and open pose and now we can bring this image into the image of open pose let it detect the hands the body the face you can turn off any of these if you want and then if we want to preview what this open pose is actually doing I can add another node come down to image and preview the image now if I connect this image here now if I pull in an image like this one right here and we Q our prompt it should be able to find the pose from this prompt here you can see it found the face and the way the hands are positioned in this prompt now we want to make sure this pose is fed down into our image generation here so that it follows the same pose so in order to do that we need to add another node to quickly search for a node if I just double click over here it brings up this search box if I type control you can see there's this control net apply so I need to select control net apply and you can see down here where my text prompt is I've got a line going from conditioning into our positive prompt let's go ahead and change our conditioning and pull it into our control net up here and then this side the conditioning output we're going to bring it back around and pull it into our po positive here so now it's going through the text prompt passing through this control net and then going back into our sampler and then for our image input on the apply control net we're going to pull it in from our open pose pose here so I'm going to pull this little image right here down to our image box here now we have one box left which is our control net where it's actually asking again which control net we're using now to get this control net I need to add one more node so I'm going to double click right here I'm going to type control n we're going to grab a control net loader and since the open pose model is the only model we've got right now we're going to go ahead and keep that one selected pull our control net right in here and I know it looks complicated but if you followed the instructions I just showed you you should be able to pull in this pose from this person into our control net image here and let's just put an excited man wearing a colorful shirt on a beach and I know this is starting to look more like a spaghetti Bowl but hopefully you're following along on so far if I hit Q prompt it's going to load our control net models and our poses and you can see down with our image here it sort of modeled him I think it's having a hard time telling that it's got two hands up in the air based on our pose here but it modeled our image and if I cue The Prompt again you'll notice that the pose is going to kind of stay the same every single time we've got this consistent pose now one of the things that makes stable diffusion so amazing is that you can go to a site like civit ai go to the models page and there are literally thousands of available models that other people have custom trained that you can use in your comfy UI be advised there are quite a bit of um not safe for work models in here and if you want to make sure you don't see them make sure that up here you have blur mature content turned on otherwise you will see a lot of mature content on civid AI now the most popular model on civid AI is this Juggernaut XL if you wanted to use this model in inside of comfy UI simply click into it on civid AI come to your download button and download the model safe tensor once again over 6 GB so it could take a minute to download but once you've got the model downloaded come to your comfy UI folder go to your models folder your checkpoints folder and drop the Juggernaut XL safe tensor file right here and now you'll have the Juggernaut safe tensor model available under your load checkpoint model if you watch the video that I made about a year ago about how to inject yourself into an AI model and you created your own checkpoint with your own face in it you can use that model inside of comfy UI as well simply take the model that you created it should be a file that ends in ckpt and is probably going to be a long file name like this if you followed the dream Booth model that I showed you now if I jump back to comfy UI click refresh here you should see our model that we just loaded up as one of the available models models again it's really long so it's uh taking up more than what's available in the box but that's okay so I'm going to enter this prompt using my keyword to generate my face Dominic Cooper person I explain why that's the keyw in a whole bunch of past videos I want to put is excited and standing on a bright colorful Beach this one I'm also going to delete this node right here that brings control net into our sampler and just bring our text prompt straight into our positive the reason I'm doing that is because this open pose is designed for sdxl and not for this model that I created I could install open pose for stable diffusion 1.5 and that would work but I don't want to get too into the weeds I just want to show you that our custom model with our own trained faces should work so let's go ahead and Q prompt and just like that it generated an image of me it sort of ignored the part about standing on a bright colorful Beach so if I up my CFG scale here a bit let's throw it at 10 I'm also going to up the steps here to 40 and let's change our sampler to Oiler a here let's try this one more time and there we go there's me standing on a beach so you can load your own custom models in using that method that I just showed you and one final thing I want to show off other people can create workflows and you can literally copy and paste their workflows into your comfy UI and use them yourself for example here's this comfy roll sdxl templates we can download this here and it basically gives us a bunch of Json files we've got to advance to an intermediate and too simple and if I jump to comfy UI and I open the folder where I just downloaded these Json files if I want to use one of these templates that this other person built all I have to do is click on it drag it and drop it into comy UI and just like that it brings their entire workflow in now these are red because there's some extra custom nodes that I haven't installed yet that need to be installed but once you have all the nodes installed it's as easy as dragging and dropping it in and then it just works so there you have it there's the new way to install stable diffusion and get really customized with it and really really dial in the control of what you can do now this tutorial was a little more dense than some of my other tutorials hopefully you're able to follow along if you want more comfy UI tutorials and you want to go even deeper with it let me know in the comments I'm happy to make more advanced tutorials in the future these types of videos don't normally do super well on my channel but I like having them on my channel so that anybody searching for comfy UI tutorials can find this in the future and if you love nerding out about AI as much as I do make sure you check out future tools. this is the site where I curate all of the coolest AI tools that I come across as well as all of the latest AI news and if you just want to know about the most important news of the week and just a handful of the coolest tools that I come across make sure you join the free newsletter and I will send you just the important stuff to your inbox it's totally free you can find it over at futur tools. and if you love AI tutorials you love AI research you love AI news make sure you subscribe to this YouTube channel and like this video and I'll make sure more videos like this show up in your YouTube feed thank you so much for tuning in hopefully you found this tutorial helpful hopefully by the time you see it it's still relevant and still works AI moves really really fast but there you have it there's comfy UI I'm having fun learning about it hopefully you enjoy it too and I will see you in the next video bye-bye

Channel: Matt Wolfe

Views: 67,125

Keywords: AI, Artificial Intelligence, FutureTools, Futurism, Future Tools, Matt Wolfe, AI News, AI Tools, stable diffusion, sdxl, comfyui, ai art, generative art, stable diffusion tutorial, stable diffusion xl, comfyui tutorial, ai art tutorial, stable diffusion ai, sdxl 1.0, comfyui workflow, sdxl tutorial, comfyui nodes, sdxl template, stable diffusion sdxl, ai art generator, sdxl automatic1111, stable diffusion beginner tutorial, stable diffusion controlnet, ai, art, ai tools

Id: 9k-yb83ZHfc

Length: 22min 27sec (1347 seconds)

Published: Wed Jan 17 2024