Stable Diffusion ComfyUi Control Net Deep Dive

hello and welcome to my channel if you've been here before welcome back my name is Alfonso paluso and I'm an associate teaching professor in the College of architecture at IIT the home of the legendary me vandero and I'm ad junk faculty at Columbia College Chicago in the interior architecture program shout out to all my students I hope you're having a great day it's a really nice early spring late winter day here in Chicago I hope the weather's great where you're at all right today we're going to look at stable diffusion comfy UI control net so we're going to look at taking images like these and turning them into renderings like these so you see here I start with that image and I end up with this image and that's an upscaled image so we'll look at using control knot and we're going to go pretty deep into it a little deeper than this workflow shows we're going to use use uh a a depth map or sometimes it's called a zdepth m map and we're going to use a canny which is an edge detection and we're even going to look at scribble for doing turning some sketches into AI images all right before we jump into it if you haven't subscribed to my YouTube channel what are you waiting for click on the Subscribe button click on the down arrow to receive all the notifications I'm getting pretty close to 14,000 subscribers help me get there that will be a great milestone there is something here for you if you're into to AI 3D modeling parametric modeling real time rendering digital fabrication there is something here for you on my channel also connect with me on Instagram at my first name Alonso uncore my last name paluso I've just reached 12,000 followers so I'm pretty excited about that I've been putting out videos lately that are helping architecture students Along on their journey and they've been pretty well received so don't miss out on any of those all right so where do we start all right let's just start I'm just going to start with a load default so the load default script call it a script it's called a workflow script workflow same thing all right and I'm just going to cue that promp up and we get the stable diffusion default image which is beyy buty beautiful scenery nature glass bottle landscape purple Galaxy bottle all right so what are we going to change up so we're going to use we're going to use the architecture real mix from civid AI so that's a checkpoint we're going to use just to know View um version 1.1 at the time I made this video is not working version 1.1 is not working at the time I made this video I've been having to use V1 repair so just a note um that's the checkpoint I'm using so I'm going to go ahead and change it to V1 repair and we're going to start with a prompt we're going to start with the Contemporary Farmhouse cabin and we'll move through some of these so we'll start with that contemporary Farmhouse cabin and we're going to change some things here we're going to change the steps to 30 and we're going to change the samp to DPM plus plus 2m and we're going to change the schedule schedule scheduler to carus or Caris all right let's go with that so it's going to sit a while at that checkpoint the first time you look loaded in all right there we go so you know it'll give us results it'll give us some you know cabins in the woods but if we have a a particular design in mind like I have this design in mind and I have this Viewpoint angle I want something like this but I want it to be an AI image we can do that using control net so we're going to jump on into it so first thing I'm going to do is I'm going to right click here and I'm going to make a new group and then I'm going to right click on the group and go to edit group and then click on title and I'm going to call this this is going to be my control net so these groups I found to be really helpful so that's going to be my control net and then any any node that is within this group or this group kind of crosses through the node it will be part of that part of the control that group all right so what do we need um so we need to uh load a few things through the manager so I just want to go through and show those things to you so if I go to manager and I go to install custom nodes and I go to filter and I choose installed this is what I've installed so I've installed the comfy UI control net auxiliary preprocessor so that's been installed so that's something that you'll need to install now I've been uninstalling this just because of the conflicted nodes that shows up here and I don't want anything to go wrong when I launch my comy UI so I have been uninstalling it at the end of my sessions but that's what I have I have that control and that auxiliary pre-processor and I have some other things in here that are not relevant but that's something you're going to want to go ahead and install and then also if I go to my install models and I go to filter this installed I have I have the ultra sharp which we're going to need at the end of this video the ultra sharp I have the the T2i adapter I have depth T2i adapter sketch and T2i adapter canny so canny is for Edge detection sketches or scribble we can try that with a sketch and depth gives us the Z depth so those three models are important for this tutorial along with the ultra sharp all right so make sure you have those all right let's jump into it we're going to we're going to load control net load control net load control net model and we're going to start with the canny and then that's that's the default here is canny so we're going to keep canny we're going to get the cany Edge detection so not cany but cany Edge and go ahead and it's looking for an image to plug in and we'll get to bringing that image in in a second I I want to point out if you drag out your wires from these nodes you'll get hints on what you need so there's control net apply and control net apply Advanced I'm going to use control net apply Advanced and that leaves that plugged in so you can see it it's looking for an image it's looking for the control net which we've already plugged in it's looking for a positive and negative prompt okay and that's coming from here from the conditioning and then this is going to go into the K sampler so I'm going to take the K sampler and the nodes to the right of that and I'm going to move those over and then I'm going to plug the negative and positive in from the supply control net so it kind of sneaks its way in between the the prompts and the case sampler okay so now we just have to load the image this is Cy Edge is looking for an image so we're going to load image plug that in okay so um a little bit about resolution so the resolution of this image is not Square so I'm going to change the latent width and height of this I'm not going to leave it at 512 x 512 I'm going to change that to 640 640 by 426 it just rounds that to 424 so if that happens to you no Bigg e there all right so that's the right proportion so yeah let's uh let's Q promp all right not bad so that's that's the easy part I like this I like that I like the way that looks that's the easy part all right so let's look at adding some more control nut options because that's just with that is just with the canny edge detection so I'm going to I'm going to lock our seed in all right so I'm going to go ahead and fix the seed so any changes we make we'll update to this image and we can see how it it works maybe in our favor or not in our favor all right so let's make this bigger okay so we're going to load in a control control net model so load control net model and this time this is going to be our depth so that's our depth and then if I drag that out and release I can choose control net apply Advanced okay this says canny edge here for the depth one I'm going to type in depth and I'm going to choose Midas depth map there's a Zoe depth but I'm going to choose the Midas depth all right Midas depth and my image is going to go into that and my image is going to go into this and the positive and okay so here we go positive and negative so what we're going to do with this this is going to be the one that actually goes into the K sampler so I'm going to plug that in there so positive see if I can get it without zooming in positive and negative okay and then from this previous control net I'm going to take the positive and negative here so we're kind of chaining these together okay we didn't bring out uh a preview image or an image preview let's just drag it out here much simpler okay preview image and let's get one from let's get one from this one too so two preview images so we can see those too all right let's cue The Prompt up all right so this this will happen to you it will put buildings under buildings and that's going to happen so let's just take a look at it so this is our cany our cany image here and I'm just going to make that a little bit bigger that's our cany image and then this is our depth map so the images that are not the image but the area of the image that's bright and white is close to us and the image that's dark and gray is far away from us so you don't necessarily have to use the two of them together you might use just the cany or might use just the depth or you might use them both together it's really up to you okay that seed is locked in so it's not going to change this as I um as I make changes here but maybe I'd like to see it with just the depth and not the cany so to do that let's see the way our the way that our nodes are set up so I'm going to go just from my posit positive and from my negative so I'm bypassing the canny okay so the whole canny part of it this these nodes have nothing to do with it I'm just using the depth okay and let's see what we get there all right so that's the depth so um without the Cy drawing it had a it looks like it had a little more trouble with it but I do like the the the top building it's putting buildings next to it and buildings under it and that's just part of what you're going to get with stable diff Fusion that that's just going to happen no matter what all right so let's let's let's go with just the cany now so if I want just the cany I need to plug in this one and this one so you just kind of you know got to keep track of what how this is all working out so this is going to be just the canny when i q prompt this okay and then if I want to plug my depth map back in I can do that I can plug in this to that positive and this to the negative okay and then I can plug this positive in and this negative in okay and let's see what we get oh wow that is totally awesome totally awesome so uh that's kind of funny so something something is wrong and I know that I I can tell what's wrong here we got some some stuff going on here so this positive has to go into this positive and this negative has to go into this negative so I got some really psychedelic stuff happening there all right I'm going to make the seed randomized and Q prompt a couple more times all right I just want to take a little deeper dive into this and just look at using it with some other some other images all right so let's do that I have some other images here so if I go to this I have this which is just a sketch and then I have this which is this this pixel Tower which you might have if you watch my other videos you might have seen that before so let's go ahead and load [Music] in let's go ahead and load in the pixel okay so that's a totally different um proportion it happens to be a square resolution which is pretty good so I can set this to 512 and I can set this to 512 okay I don't think I have to set anything any where else uh I'm going to change the prompt though so this is going to be our skyscraper prompt okay an ey Lev view of a skyscraper made of steel and glass in the parametric Style located in Chicago with mist and fog in the atmosphere Park and foreground Greenery from Park is growing up the building crowds of people ethereal having like light ambient occlusion soft Shadows you may have noticed that some of these things just don't work at all like crowds of people we haven't seen any of those Greenery from the park growing up the building haven't really seen that so it's just kind of you know has to do with a couple things your checkpoint mainly and how that checkpoint was trained what model it was trained on so some of these things just are a nogo and you just kind of you know you you you see that happening unfortunately all right let's let's jump to this so this is going to be using uh canny and the depth those are both in play the depth map is kind of interesting in this um it just kind of you know it has nothing to do with the building in this case which is which is kind of interesting it's like a foreground and a background so you know it's just kind of experimenting and seeing what that has to do with the actual image that's made um so this is pretty cool so again just just experimenting um and I'll and I'll experiment one with you so you can see that let's go for just the Z depth map so that's that's this one this is the depth map plugged into this Supply by control not here so I have to bypass this one so I'm just going to do that so that's my positive this is my negative so that should be only the only the depth map okay so you see the depth map in this case by itself it worked with the cabin but in this case it's not going to work because the building there is no there is no reference to the building at all in this map so that's not going to work let's let's do just just the cany now so this is going to go here and this is going to go here it's got the negative positive prompt in it so that should just work just like that okay so now our now our building's back in the mix without the depth so you just got to experiment going you know between those which ones do you like which ones don't you like there's also um there's also thresholds for the canny edge and I found that the 100 to 200 work pretty well it's a pretty good range but you could experiment with the those numbers and see what it does to the candy Edge detection drawing all right let's look at scribble let's bring in [Music] our sketch okay and we're going to need another load control net model so you're always going to need this is the preprocessor you're always going to need the pre-processor so I'm I'm looking for sketch in this case it's making this group bigger okay and this is going to use scribble scribble lines there's a scribble x x dog lines you can work try a little bit with both of those there's fake scribble lines so there's three of them I'm just going to go with the scribble lines and I'm drag this control net out release it and choose control net apply Advanced so my image is going to go into here my scribble image is going to go into this one okay positive negative prompts see if I can it's like shoot it's like passing a Hail Mary here and let's just use just just this one so that's going to go into our K sampler comfyi does a pretty good job of plugging these in from a far distance all right let's cue that prompt up all right so that's our scribble and no prompt The Prompt is of the skyscraper so let's get this ey Lev view of a museum from zahad okay I level view of a museum made of concrete and Glass by zaha adid in the parametric Style with M and fog in the atmosphere sky in the background water in the foreground let's put a comma in there water in the foreground with crowds of people ethereal heav light light ambient osion and soft Shadows just want to fix this and put my comma in there okay so some prompt there for it and this is uh using the scribble okay now I would say that I haven't had much luck with the scribble now the control net is set to one so let's look at that the strength of that so we haven't looked at that the strength of that is set to one so that's really putting the emphasis on the control nut I can start to lower that I can put in like 0.25 you might hear my dog read it in the background she's crying because my family decided to adopt a cat from a rescue shelter and that cat was adopted yesterday so the dog and the cat have not met yet yet so the dog is crying to meet the cat so you're going to hear that in the background and I apologize for that so this is the image with the control nut not set to one this is the image with a control net that was set to 25 for the strength so I can change that to 75 and we can cue that up so a little bit more control n uh in there so it's finding that right balance between um the control net and the prompt so the scribble I'm I'm not really digging the scribble so we got a lot going on here so this could get really confusing really fast Let's uh let's bring out a image here a low preview image and let's just see we didn't get to see what the scribble lines look like okay so that you know looks pretty much just like the input image it pretty looks just like the sketch here so let's go with let's go with canny Edge detection and see what that looks like so that's just plugging in um the apply control net from The cany Edge so I'm just emphasizing that you really need to experiment with these all right so that's that's with the cany edge detection so we get something that looks like that it looks it looks a lot different than the sketch actually and um just kind of looking at our control Nets and what's what's going where you can get pretty confused with that okay because that's that's the zdepth that's plugged into that so let's plug in the canes you got to trace where these come from so this is the cany going into the control nut and that's going here okay so that's going to be the canny all right so there's the cany edge and we get something a little bit different there so it's just experimenting with these I just want to make sure I have these right okay cany this is the control net that's going directly in this is The cany Edge right here okay image is going into that strength is one it's set to one okay it just looks a little different than I imagined it might look all right so that's Edge Let's look at let's look at the Z depth okay that's this one that's the Z depth here so let's plug that one in kind of already looked at this one just a minute ago when I thought was canny okay let's look at that one all right pretty cool what if I was to plug all three of them in okay because we didn't really look at that what if you want three control net models well we just got to chain this so that this is going to go into here and this is going to go into here and then this is going to end up going into the K sampler have no idea what kind of result we're going to get okay so this might help being out here since it's the the latest one so we're chaining from this one so I got to get this right got a chain from this one to this one all right so creating a connected chain between this one this one and this one one all right let's check this out let's make this make this a little bit bigger just kind of matching the other one how about proportion um what kind of it's making squares right now and I don't think that scribble image is square so something to think about that scrimo image is not Square all right let's see so I'm sure that's having an effect on this all right I don't we don't like that at all that results of that are just terrible that's just terrible um so before we get out of this I want to look at upscaling but I also want to look at the proportion of this um this sketch image okay so it is a little bit different let's go ahead and put in the proportions that it should be all right so we're going to try 512 so 512 is the width but the height of that is a lot smaller so I'm going to try 300 try the proportions of that not exact all right so that's a little better in terms of proportion the proportion does really affect as we're seeing it does really affect the generated image that comes out so you do want to match the proportion of the input to the output I just took a guess at that one 512 by 304 and we already see the results of that are much better all right let's how do we want to end this let's let's go with um let's take out the the scribble I'm not digging the scribble and let's chain just the two of those together okay so we're going from here to here and then up in the case sample it's chained to this but it's a it's a dead end so it's not going to not going to matter all right pretty cool dig in that let's lock in our seed so we're gon to make that fixed and now we're just going to upscale this bad boy all right this doesn't need to be so big okay so I'm going to make a group add group and right click and change its color edit group color and to make that purple going to rename the group at a group title this is our four four time ultra Sharp upscale all right I'll make this plenty big here okay so the model I just want to go in and to my manager and go to my install custom nodes and just filter this by in installed okay so nothing here about four time ultra sharp okay so where you find that is under the models so install models and we have I'm going to change this filter to installed okay and at the top you see four time ultra sharp so this one where my cursor is moving back and forth on you want to have that installed all right let's jump into the upscale so we need a load upscale load upscale load upscale model okay and you see the four time ultra sharp is there I'm going to drag that out and release and I'm going to need image upscale with model and if I drag this one out um I'm not really getting the one that I want here so for this one I'm going to need upscale image so I'm going to need so that's an upscale image using model I need just an upscale image and this upscale image is important because it allows me to control the size of the upscale so that um zah adid sketch that I made I know the final resolution of that so that is 2048 it's 2048 by got zoom out a little bit here 2048 by 1495 so proportions matter so 20 48 hey drag that out grab a save image all right that's pretty much all I need for the upscale so I want to cue that prompt oh let's see here the image I'm missing the image all right the image comes from the vae decode kind of move this away move that up a little bit all right nothing's it's crossing here let's make this gigantic cuz we're upscaling all right let's cue The Prompt all right that was pretty quick all right so now I have my my full resolution make this bigger all right and that seed is fixed we can randomize that just to end this off and just get some more results with that all right so I hope you enjoyed this video we looked at cany Edge we looked at zdepth we looked at scribble we looked at bringing in an image that was maybe produced from some software we brought in a sketch a little scribble and we took a look at that my head's going to pop up in the upper left if you haven't subscribed click on that if you like this video go ahead and give me a thumbs up tell me why you liked it and I will see you on the next one thanks for watching

Channel: Alphonso Peluso

Views: 1,709

Keywords: Stable Diffusion, ControlNet, ComfyUI, ai, midjourney, ai architect, rendering, real time rendering, architectural rendering, architecture school, architecture student, interior architect, interior architecture student, realistic rendering, photo realistic, best rendering software, generative, generative images, generative renderings, iterative, iterative process, best ai software, artificial intelligence, ai in architecture

Id: xkPKmPkcDss

Length: 34min 11sec (2051 seconds)

Published: Fri Apr 12 2024