AI-Powered Creativity: Innovating Look Development with Stable Diffusion & ComfyUI

welcome today I'll guide you through the process of using stable diffusion to spark creative ideas for your work faced with the common dilemma of determining the visual direction of my completed 3D model I opted to Leverage The Power of AI to streamline the creative process in this tutorial I'll show you how to use confi to generate hundreds of Unique Look development ideas for your 3D rendering in just minutes confi is a noe-based graphical user interface for stable diffusion you can construct an image generation workflow by shining different notes this approach isn't limited to 3D it can be applied to other creative domains such as drawing photography or VFX compositing let's begin to follow this tutorial you will need to install some stuff you will need python git confi one checkpoint model confy manager and since this is not a beginner tutorial it will be good if you add already some basic knowledge on confi I will not be covering the installation of these elements but I will leave links in the on the description for everything you need to install Okay so this is my finished workflow uh when you load the Json file probably you'll see some notes marked as red that's because you need to install the missing notes to do that just go to the manager right here and then click on this button to install the missing custom nodes done that we are ready to start analyzing briefly this workflow so up here I have my base image that I render it in Cinema 4D and I'm feeding it to the control net preprocessors which are these two groups here the KY pre pre-processor is an edge detector it extracts the outlines of an image which is useful for retaining the composition of the original image so the generated image will follow these outlines right here okay here I have the depth pre-processor that guesses the depth information of the reference image and these two are our control net so they are connected to the basic nodes that are needed to generate an image in conf y right here we have the checkpoint loader and these two Market as green are our positive prompt and the red ones are our negative prompt here we have the note that determines the resolution of our image and here we have the first generation that was made using the outlines staish it with Kenny okay since I liked this this image I decided that I wanted to see some control panels in these areas right here and to do that I loaded my first mat pass which has some areas uh with colors and the blue ones are exactly the areas that I want to see uh some control panels and since they are blue I have to choose the blue channel of this image if I wanted to change the ceiling of the spaceship I will choose red for example and the result it's this he added some control panels to those areas and then I decided that I wanted to change the spaceship floor so I loaded another matte image this one this area right here mark this blue again I choose the blue Channel I made a new prompt and the result was this but this image has a lot of weird edges and some errors and to soften those errors and improve the image quality overall I made an image to image pass that basically it takes the original image and adds a little more information just to improve the overall quality quality and this is my final result after that I did an upscale to four times this is now a 4K image of our final result I will not be covering the upscale part in this tutorial because I think that's a subset for another time so let's begin so this is confu y this is the basic workflow but we are going to do one from scratch so let's clear this up and start with another one if you double click on the UI a search bar appears and you can search for the node you want so let's start with the checkpoint loader a checkpoint is a model pre-trained in a particular set of images XT XL base 1.0 is a model released by stability AI for a the XL but there are other ones like this one on cvti you can download it the model sa anwers and put it in your checkpoint folder inside confi then if you refresh UI it will appear on the drop-down list next we need a place to write our prompt so let's search for clip text and codex the XL and as you can see there are two Fields here where you can write some text clip G and clip L but in reality we need just one field so let's convert these two to input convert clip G to input and then convert clip L to input if you click on the tiny circle on the side of text G drag outside the box and release the mouse a menu will appear let's choose head node utils primitive now the Primitive is connected to clip G but we also need to connect it to clip L this is going to be our positive prompt so if we give it a color like green it's easier to identify likewise we need a negative prompt so let's copy and paste these two notes and give them the red color now it's time to start connecting our nodes let's start by connecting the clip from the low checkpoint to the clip text and code now we need a Cas sampler let's start by connecting the model from low checkpoint to the input on the case sampler and likewise our positive and negative prompts next we need a new note called empty latent image this will determine the resolution of our images H in fact I think I'm going to use the XD XL empty latent image just because it has a list of the resolutions that work best with XD XL now we need a VA decode vae Means variational Auto encoder and it encodes the image into latent space and then that same latent space is decoded in the new higher quality image finally we need a way to save our images let's search for save image and connect these two perfect so now we are ready to write our first prompt let's see house by the lake trees in the background daylight uh beautiful sky with clouds dramatic lighting professional landscape phography and since we want a realistic image in the negative prompt let's say we don't want something like drawing or painting or anime or cartoon and we want a sharp image so let's avoid blurriness all right so let's Q prompt the green box around the node means that that node is being processed uh the first time you load the checkpoint it will it will take a while to load but the next time will be quite faster so processing the image yes and here it is we can play around with the settings to improve the image quality the settings I like to to use for XD XL are for example here on clip text and code I like to use 4096 instead of 1024 here too same for the negative prompt in the case sampler I like to increase the number of steps to 30 for the sampler name I like to use this one and caras so let's make a new image is going to be quite different of course because the the seed has changed and there we go okay fantastic now uh the fun part let's start adding our control Nets if you are planning to use more than one control net I suggest that you search for the stack note for control net this one right here so if you push the tiny Circle the drop- down menu appears and let's use control stacker okay for the control net we need to choose our model control net loader yeah and here is the list of uh control net models that I have I like to use this one and if you don't have this one you can go to the manager install models and you can search for it here Kenny and make sure you're choose the one for XD XL and it's this one right here cool after we install again refresh your confi and it will appear right here so we need to load our source image so load image this one cool okay now we need a pre-processor for this image so let's search for cany Ed pre-processor okay we connect the image to the pre-processor and the pre-processor will connect to the control net stacker cool let's have one more node here called preview image connect the cany to the preview image so we can see what will be the interpretation of stable diffusion of our source image so we fre Q prompt right now okay so here it is uh keep in mind that this is only an approximation this is what stable diffusion thinks are image lines are and it will not be perfect if you want some more approximate results I suggest that you make a render on your on your 3D software like a tun Shader or something so that you have a more accurate version of the lines you know uh like this one for example and we are ready to connect our cont net to the rest of the workflow so what do we need here we need to connect our prompt to the control net stack so the negative prompt goes here and the positive goes right here next we need to connect the conditioning conditioning plus to the positive and the other one to the negative see okay and now we need to adjust our prompt so this is a spaceship room class Windows interior lighting cinematic shot you can be as descriptive as you want to but for the sake of Simplicity I will give my PR short so let's see what it gives us the use of control NS will make the generation process a little bit longer so from now on I will Fast Forward the generation process so we have a shorter video okay so this is our first result let's do another one all right really cool mhm oh so as you see we don't have any Windows here so stable diffusion thinks these are windows apparently and it's quite okay because we are just getting some ideas for our texturing and lighting for our 3D scene but if you want to really really have an accurate version like I said before I suggest that you make your own candy rendering in your 3D software and feel that image to control net one thing that is important about this process is that you can play around with control net settings right now the strength and the end percent are at Value one but you can reduce this value and give more freedom to stable diffusion to imagine what the Canon lines will be so let's say that we reduce this to 0 five and same here and as you will see the result the results will be quite different okay he had a lot of new information here before this was a rectangular window and now has a completely different shape let's see another one okay as you can see these are very different results but the the base the Draft lines are still here but for concept purposes is quite cool so we can get a lot of different ideas things that we didn't think before for our 3D but for me I like to give just a little bit of Freedom let's say this value and the image will be a little bit different but we'll still maintain our base lines one thing that's not going so well are these edges right here they are little bit jaggy and in our base render they are straight and well defined but as I said just need to play around with these settings until you get something that you you like you can also feed a dep map to the control net and I'm going to show you how so first let's clean this up a little bit let's select these notes right click and add a group to them let's rename the group this is going to be arcanic and copy this paste them add another group and this is going to be our def okay so what we need here we need to change our model because this one is too cany and I'm I'm going to choose the dep right here if you don't have it you can go to the install models search for it and make sure you choose the one for XD XL install it refresh and there you go so what we need here now uh we need to change the pre-processor let's search for this one right here is cool I like it so we need to connect the image to the pre-processor this goes to preview image and this one goes to the control net and let's see what is the result for the map okay so I don't think it did a really good job I can play around with the settings to get a more a better version but I think the best really is uh to render this pass on your 3D software and then fed it to the control net but let's see how it goes with this one we just need to connect the control net stack to the other one it doesn't really matter the order you connect them they just need to be connected and let's see how it goes okay um since this death map it's not really good uh I'm going to WIP W the group because I don't want it to affect our rendering and let's change the prompt a little bit let's go for let's see uh an abandoned spaceship uh with broken glass windows and let's add some daylight Sunrise coming in all right this is really cool but I don't see any sunlight or a sun race coming in let's try again huh this looks weird but let me give you a nice tip if you really want to reinforce uh a particular as of your prompt you can control and use your app Arrow to increase the weight of the those keywords let's see if if it gives us some sunlight right now all right there you go there is some definitely some race it has some uh weird lines on the window but uh keep in mind that it will staple diffusion will give you a lot of bad results you really need to Sherry Peck the ones that you like you know uh this is a process of trial and error absolutely let's make another one mhm let's say that we want to see a forest showing a forest outside mhm nice so as you see we have a new set of windows in the ceiling uh this wasn't on my base render right but we are here uh just generating new ideas for finding things that we didn't think before and if I wanted to go with this look I will go to back to Cinema 4D and I will adjust my geometry to include these windows but then again you can go back to your control net settings and adjust these values closer to one to be more accurate relating to your base image so let's say that I want to go uh with this look and made some changes without completely change the the overall image so to do that you can click on uh this button here view history and this is my last render by the way you have here the history of your prompt previous prompt and if I like one like this one uh I'm going to go to the case sampler and I'm going to freeze my seat so I'm going to set it to fix so if i q prompt again and nothing happens because I didn't change any settings and the the SE Remains the Same okay so let's do some compositing on this image but first let's clean up uh the workflow a little bit I'm going to push my control net groups up here like so I'm going to create a new group for this notes right here I'm going to call it first pass and we are going to copy some of these notes so I'm going to select the positive prompt the control net stack Cas sampler vaad de code and save image I'm going to press crl C and now if I press shift contrl V it will paste them and keep the connections like so cool now we need to load our image mask let's search for load image mask and we are going to load our M pass that has the floor and since the floor is colored in blue we are going to tell that the channel will be blue so now we need a new node called set latent noise mask okay and we are going to connect our mask into here this one go connects to the latent image and the previous case sampler is going to connect into the samples cool okay so the the negative prompt Remains the Same but now we need a new prompt let's say that we want a water bule on an abandoned spaceship and let's see what it gives us as you can see it's only changing the floor right now and the image looks weird and I know why it's very very sharp and with noise that's because we are using the same SE as the previous casy sampler so to solve that we are going to tell stable diffusion that we want a new s setting it to randomize let's try again all right nice another one the cool thing that it is even trying to get Reflections from the ceiling you see from the windows of course they are not physically correct but congrats stabil diffusion for at least trying okay let's say that we want to change the lensc cape outside the the space ship room we have a problem with that because we already have some Reflections on the ground and that will end too so I'm going to make a group out of this and I'm going to duplicate all these notes paste it here and I'm going to bypass this group for now since I don't have have a mat pass for the windows because this is new information that didn't came out from the 3D software uh I went to photoshop and I did a rough mask for it this one right here so we have red to Mark the position and areas where the windows are let's set this for red and let's say that we want a new landscape I have a prompt for that here let me copy and it's glass windows is inside an abandoned spaceship showing the view outside which is a desert landscape and a deep blue sky on a sunny day the cool thing about XD XL is that you can write your proms as you normally talk you know it's easier so let's let's see what the result is going to be okay nice we have a desert cool but here it's not working because we are using the KY edges from the original image so to fix that we can do a little trick we can copy the candy notes once again let's paste it right here and let's load the N image into here and connect the control net stack to the apply control stack okay but this time we need to set our values to one because we don't want stable diffusion to deviate from the lines we set for uh the this scny so let's see how it goes nice now we have a desert and we have our Sky uh let's say that we want to see a city landscape let's let's change the prompt is a city landcape with TP buildings let's go all right very cool I love it so let's say that we are happy with this result let's lock our SI so we click on load the last generation okay like this set this to fixed and let's see how the floor will react to the new information let's set this group nodes to always we already have our desired prompt and let's Q prompt oh right I forgot we need now to set the the latent I mean uh we need to connect the previous latent from the cas sampler to the set latent noise mask so let's do it so connect here to this one cool and let's let's try [Music] again okay very nice very cool it's working I know what what you're thinking this doesn't look like a spaceship anymore does it well I admit that I got a bit carried away but I just wanted to show you the process and the cap capabilities of this amazing software in a matter of minutes you can generate 100 different ideas to show to your clients before you do the actual work on the 3D software right let's try another one h I think I like this one better so assuming that I want to finish with this image I'm going to click on load and set the SE to fixed nice now as a final step I want to do some image to image to try to soften some of the errors like this details right here that don't make any sense these lines that have chromatic de aeration doesn't look good same here so to do that I'm going up here and I'm going to copy some notes like the prompt we don't need the control net stack but we need the case sampler the V code and the save image notes contrl C to copy let's go down here and paste it whoops wrong we want to do shift contrl V to keep the connections when we paste like so perfect now we need to connect the positive prompt to the case sampler and do the same for the negative prompt as for the latent image we need a new note called VA en code let's connect it and for the image we need to go get at our latest generation so load image and if you go to your output folder inside conf you you are going to have all of your images let's drag this and load into here perfect now let's connect to the VA in code now we need to connect the VA to the checkpoint uh note that it's way up here so if you click on the red circle drag and release we are going to have this option right here re root and this is a very useful note for complex workflows so let's bring it right here and now we can connect to the VA let me just organize this a little bit better this is going to be another another group like so okay yes I always need to keep things a little bit organized nice okay so I think the last last thing that we need is to change our prompt but before I do that let me change this SE to randomize again and I'm going to copy our first prompt paste it here I don't need the weight in the keyword Sunray is coming in so I'm going to delete it and I need to adjust this part because we don't have a forest anymore we have a city landscape with tall buildings and a deep blue sky and we also have a water puddle on the ground okay let's see what the result will be let's Q prompt and as you can see this doesn't make any sense that's because I made a mistake I failed to connect the positive prompt right here or instead I put the negative prompt on the positive let's try again okay the image is a little bit similar to what we had before but still not the same that's because the denoise value is set to one values like one or close to one will generate a completely new image so if you want to let stable diffusion add a little bit of information but without changing too much the original we have to reduce this value like 0 45 okay it's looking more similar to what we had before and okay let's compare it to the previous image okay I see that these details right here have gotten better I see that the shape the chromatic aberation around here has disappeared it's more clear now the city has changed a little bit I like the detail right here like it's like dust or something it's better that than what we had before and this image this line of this window has also gotten better so as a final word I want just want to say that this is highly experimental I'm still learning and every day new tools are appearing that can greatly improve this workflow and I'm committed to learn them as well and uh I hope that you learn something from me and I'm curious to see what you guys going to do with this so if we have any questions don't hesitate and ask me and good luck

Channel: Leonardo Cunha

Views: 3,393

Keywords: Stable Diffusion, ComfyUI, 3D rendering, VFX Compositing, Look Dev

Id: CLPvdbFUNqw

Length: 38min 48sec (2328 seconds)

Published: Sun Jan 07 2024