Hyper Stable Diffusion with Blender & any 3D software in real time - SD Experimental

hello everyone and welcome to another episode of stable diffusion experimental today we'll be looking at a way to integrate stable diffusion into blender or any 3D software really in real time now last week we did use blender and stable diffusion together and I didn't say it was experimental I did say it was for professional creatives so why is it experimental now well let's just say that to Define this setup as janky would be making it a compliment today I'm going to show you two different workflows you can see them on the right hand side of the screen here one is with sdxl lightning and the other one is with Hyper sdxl now hyper sdxl just came out there's no fine-tune models for it it's just the base sdxl model but fine-tuned for generating images with just one step and that normally would just be Jank enough but everything here relies on a note that is called screen share now I've searched far and wide on GitHub for a node that could use the viewport from blender or from any other 3D app out there but I couldn't find any now some of you may be aware of a castom node that lets us use comfyi like nodes inside of blender but one that would be appliable only to blender and two that node suit is not well suited for what I had in mind because it's quite restrictive in what can actually be used as nodes inside of blender so what I wanted to do was having a comi instance running and sourcing images in real time from what I was doing inside of a 3D environment and the best node I could find was this screen share node now same as usual we will be building this node structure together and I will show you and give you the things to download all the custom notes that we're going to use but first let's take a look at what this screen share note does basically the screen share node allows us to share a screen much in the same way you would do during a normal conference call for example so if you click on the button a window will pop up and it will ask you about which screen you would like to share in my case that's blender and if you select that you can allow the screen to be shared and then the window will pop up here inside the node now everything is been captured live and we can also set an area so in my case if I want to select just a tiny square area inside of my blender workspace we could do that and then we would be loading the image directly from the screen now normal this would be great amazing even but the thing is that screen grabing live consumes a ton of resources and when you're adding those resources on top of blender and on top of comp Pui as well and in my case on top of OBS for recording and my camera then the setup becomes a lot jankier than usual and that brings us to the question as to why I am providing you with two different workflows well at first I was testing this with sdxl lightning which requires between four and eight steps of inference but while I was testing it sdxl hyper came out and that only requires one step so I wanted to show you the difference between a fine tune sdxl lightning model that requires a bit more steps and the base model for SX on hyper so now let's build the workflow from scratch and if you want to jump ahead and see the actual testing you can go ahead and Skip all the building process same as always you can find the workflows in description below so if you want just test it out yourself you can just download the files and have your way with it so the first thing that we're going to need is a suite of nodes called mixlab This is a suit of nodes that has a lot of different noes in it but what we want to use is the real time design nodes with its screen share Noe unfortunately we have to install all of it so what we want to do is just go over to the green code button here copy the get up link and then go over to our confy folder double click on it search for custom nodes double click on it right click open in terminal and then type get clone and paste the link to the repository and then press enter now in my case it already exists so it threw an error but in your case it will install all of the nodes then we're going to go searching for the comyy mix lab nodes folder inside of our custom noes folder double click on that right click again open in terminal and then write pip install dasr requirements.txt D- user and then press enter and if everything goes correctly it will install all of the dependencies that you were missing then the next thing we're going to need is an sdxl lightning or the sdxl hyper models let's start with the lightning version for the lightning version you can use a fine tune model just like jugut XEL lightning you can go ahead and download it and then you can see here that you have the actual settings for the case sampler that you're going to use for this sdxl lightning model so in our case that's between four and six steps with CFG between one and two sample is DPM Plus+ SD and if you want to use this model you're going to need to remember those cuz those are the ideal settings and then if we want to use hyper instead we're going to go over to this hugging face page it goes as slow as just one inference step and it's very very fast now the D side is that it's not a fine tune model like Juggernaut is so it doesn't give the best results now if we go down in the page we can see that there's some settings for the confi usage we can use it with either sdxl or STD 1.5 and there's also a text to image workflow if we want to use that as well just for starting so let's grab this workflow which is in Json form you can click on raw and then download it as a Json or you can copy and paste it in like Visual Studio code for example and then save it as a Json then let's go over to files and versions and download the save tensor for the one step sdxl unit now there's two different models one that is 10 GB and one that is 6.94 we want do one with confu Y written on it so that's the 6.94 gigabytes we're going to download that and place it in our models checkpoint folder inside of confi and while the safe tensor file downloads we're going to drag and drop the Json file for the workflow that they provided inside of a confi instance and we are gred with this node Tre now in your case you're going to be missing some nodes so what we want to do is just go over to the manager and then install missing custom nodes here I'm not seeing anything cuz I already installed them but you're going to see just a suit of nodes and you're going to install it and then you're going to have all the nodes you need now let's take a look at the noes that they left here so to use the hyper sdxl unit for one step inference it requires the hyper sdxl One Step unit scheduler and that the noises from 800 steps rather than 999 now what does that mean to you basically nothing what it means is that the model has to go through this node over here here and this node takes care of everything but in order to use this node we have to use a custom case sampler node now a custom case sampler node is basically the same as the usual case sampler node but it's just customizable with some things that are not customizable in the usual case sampler node in our case that's a sampler node and a sigma node the case sampler node basically does the same as the sampler window that you had over here in the basic K sampler node and the sigma node takes care of all those the noisy calculations that are present here in the notes so now that we have everything that we need out of the way we can start building our actual workflows since we already have an example workflow for hyper as the Excel here with us we can just start by switching things up over here so we already know that the basic structure of the case Center must be the same as this one so we're going to leave it just like that I am just going to rearrange the windows a bit so that it looks a bit more tidy I'm going to delete this note and then we're going to start actually building so the model that we actually want to use in the load checkpoint node is the hyp sdxl onestep unit comi we're going to select that and then let's think about what we actually need we want to grab an image from our screen that will allow us to use our viewport in blender or any other 3D program and use that as reference for Our Generations so we don't need an empty latent image what we need instead is an image that gets encoded and that image comes from our screen share node so we're going to search for that and add it over here another thing that we're going to need is a resize image node because we are not entirely sure about the dimensions of the image that has been screen shared so we're going to search for an resize node put it right over here and connect the image output from the screenshot node into the image resize node we're going to select 1024 by 102 4 since this is a sdxl model and then what are we going to need so we want to keep the overall structure of the composition inside of our 3D program so we want to use a combination of control Nets now remember that everything that we are adding here add resources that are needed in order to do all the calculations so we want to be as lightweight as possible we could be using a different combination of different control Nets such as open pose depth Kenny or any really but I find in my testing that the minimum amount of control net that we want is one basically and that's depth and then if we want to add more we can do that by adding open pose or Kenny or whatever but for starters let's start with depth so what we want to do is search for apply control net drag and drop from control net and search for control net loader here we're going to select a light depth model and then we are going to run the image from image res size through a Dept estimator so in our case that can be Zoe or Dept anything I'm going to go for the latter I'm going to connect the image output from the image resize node into the adapt anything node and then the output from the adapt anything node into the apply contol net note now what I usually do is just add a couple of preview image notes just to see what images are being grabbed and just troubleshoot in a way what's happening during the process so I'm going to drag and drop from the image resize image output and search for preview image place it right over here and then I'm going to do the same from the depth anything output these two nodes will give us an estimate of the preview image that we're getting and is being resized and the preview image of the depth that is being calculated then we're going to run our positive conditioning through our apply control net node and then we're going to connect the conditioning output from the apply control net node into our sampler custom node and that's basically it the only thing that we're missing is an actual starting image from which we're working so what we're going to do is we're going to encode the resized image from our screen share node into a Lattin so we're going to search for a vae encode node link up the output from the image resize node to the vae en code node link up the vae from our checkpoint and then link up the latent output to the latent image input inside of the case sampler and that's it our workflow for hyp sdxl is complete but before running it let's do the same but with sdxl lightning instead now if we want to use sdxl lightning instead of hyper what we can do is just delete all the sampler custom nodes that are related to the hyp sdxl instance so in our case that's sampler custom that's case sampler select and hyp sdxl onestep unit scheder going to select them and delete them and I'm going to search for our regular K sampler node going to place it right over here I'm going to connect the model output from the load checkpoint node into the model input in the case sampler and then I'm going to connect the positive conditioning coming out of the control net into the positive input in case sampler the negative prompt into the negative input into case sampler and then our latent next I'm going to connect the latent output from the case sampler into the samples input into our vae decode node then I am going to select our sdxl lightning model in our case that jugernaut XL v9 lightning and then I'm going to set all the parameters here in the case sampler so in our case that's between four and six steps if you remember from before I'm going to put four CFG between 1 and two so let's say 1.3 for sampler we want DPM Plus+ sampler so let's say DPM Plus+ 2 msde and for shed we want to select sgm uniform and that's it we got both workflows up and running now what we're going to need is a scene setting blender and then a prompt that actually describes the scene that we're trying to reproduce use inside of comy Y so I'm going to open blender and here we can have the scene that we set up last week in our blender to com fui workflow it's basically the same I'm using a mixo character with an animation of the character standing up and nailing down and if you want to see how I set that up you can go back and watch that video as well what we basically did is just we grabbed a mixo character and an animation from mixo and then downloaded the fbx and set up our whole scene inside of blender now a thing that is very important here is to set up the scene so that it's being rended with Cycles instead of Eevee we don't want the normal viewport without shaders we want our actual render View and we're going to go over to our render settings inside of the scene inside of the viewport and we're going to select Cycles here if your Hardware is struggling a bit with keeping up with all the samples that are needed for a scene you can just lower here the samples that are being rendered inside of the viewport but why do we need Cycles well that's because as a source image it works better if we have actual volumetric lighting now for testing let's start with the sdxl lightning workflow I already set up a positive prompt that describe the scene that I'm trying to get it is a cinematic photo of a battle war Knight kneeling in a temple in front of a globe emanating light we can see that we have all those elements in our blender scene over here we have the overall structure of the runes we have the Knight kneeling down here we have the orb glowing with light now what I want to do is click on the share screen button here in the screen share node I'm going to select blender and then I'm going to click on allow I am going to set the area I already set it you can try and set it to as close a square as possible CU you cannot set up a fixed square area by default but if you go like close enough that's enough and then I'm going to click on live run now the way that this is all set up and the screen share node works is that you don't have to actually hit Q prompt instead every time that something changes inside of blender let's say you switch your camera view or you click on some things instead of others the live feed from the screen share node refreshes and it uses that as a new load image and as you can see we keep getting new images here in the preview image node we can just try to to move around the camera and once it's done refreshing and generating the previous image it's going to start again and generate a new image so this one is from the previous View and the one that's coming up right now is going to be from The View right now now why was I telling you before about using Cycles is reflected here in this image in particular you can see that the globe here is not present in our generated image and that's because the loaded image from the screen is not detailed enough to see some changes in the lighting in our case the globe and the ring in the background they are both emanating light and the difference in contrast is not high enough for our adap ening node to pick up the difference so it's not being reflected actually in our generated image so we want to have this globe of light either shining a bit more than the background or we can just place it let's say over here and you can say that there's a difference now since the globe is in front of something that has a different D and different lightning then it gets picked up by the actual ding node now keep in mind that this is slower than normal because I am both using blender and stable diffusion but I'm also recording myself doing this but still it's quite fast and this is not Top Notch equipment I mean it's very good equipment but it's nothing crazy like a 4090 I only have a 3080 TI now if we want to add a node that we already know from our Photoshop to comi experiences what we can do is just drag and drop from the image output from the vae the code node and search for Popup you're going to find a link to the popup node as well in the description below if you want to see how that works you can go back and watch some of my Photoshop Toom fi videos but what this does is basically creates a popup that then can be fixed in position inside of another environment let me show you that cuz it's a lot easier to understand if I show you so I'm going to move the image around the generations are going to commence and then a popup will be created here on the window what we can do now is just resize it a bit and then pin it in a place where it's not up structuring anything so let's say I'm going to put it right over here now I'm showing both the compy instance on the right and the blender instance on the left but that's only because I have to show you the workflow as well but if you wanted to maximize the bler window for example you can just keep the confi workflow working in the background and then have just the popup preview inside of the blender instance for example and that's quite good to just preview the images that you're rendering while you're working inside blender so now let's close this popup I am going to stop the generations here from the screen share and let's go over to our sdxl hyper workflow so I'm using the same exact prompt as before but the node tree is going to be a bit different because I am using the hyp sdxl version same as before I'm only using dapt as a control net because I want this to be as lightweight as possible and I'm going to click on share screen I'm going to select blender again allow and then I'm going to set the area here in the center of the viewport and I'm going to click on live run now here as you can see it's picking up the image from the viewport it is picking up the ADT and then it is generating the image the quality of the image is nothing close to the fine tune model in our case jut but it is a bit faster because it's only taking one inference step instead of four now let's add a popup preview node I am going to maximize my blender screen and I'm going to reset the active area because changing the proportions of the window and maximizing it changes the actual active area of the screen share node I'm going to hit on live run I'm going to minimize the confi workflow and here we have our actual viewport with everything that's happening inside of composition and the generation here appearing in the popup on the left now we can work on the image in real time and then it will get updated inside of our popup in something that kind of resembles real time and this is all with just one in step by using hyper sdxl in now in order to get better Anatomy we could add an open pose control net but that would slow things down a bit depending on your Hardware you can do that but here let's say we wanted to add some columns so I'm going to look for a cylinder going to place it going to resize it adding it right over here duplicating it we can see how these things get get implemented in Our Generations let's say I wanted to resize the glowing orb and maybe T down the light a bit we can see that the scene changes in close to real time to me this is amazing let's say I wanted to switch to the sdxl lightning version I'm just going to select again the blender instance I'm going to set the area I'm going to hit live run and there we go we get a much better image much more refined because we're using sdxl lightning now and even if it's a little slower it is very nice and this has its uses we don't need to use shaders we don't need to build materials we can just use blender or any other 3D programs as a source of composition we can just use basic shapes in order to create complex environments this is not refined yes but at least it gives us a way to control things even without any actual abilities in the medium I haven't used anything that you cannot learn in half a day basically I've only used basic shapes cubes cylinders spheres and basic materials such as a white material over here and an emission material over here even the human figure is just downloaded as well as the animation from mixo so all in all I think this is as amazing as is janky it's a great implementation in order to get rough ideas and start from scratch and get good composite results and it shows that we can have a great degree of control over how a final image is actually generated if we Implement already existing tool in our new medium even if this does not substitute the need for renders and render engines and materials or shaders it is a side grade of Swords because getting materials and shaders that are good enough for compositions and just for allowing the clients to actually get a feel of what we're trying to do is often time cumbersome and timec consuming whereas this way we can just generate stuff in a matter of seconds that actually resembles what we want to do now I can already hear The Architects among you saying that this is not good enough for them because they want to be able to show the clients actual materials the real stuff that they're going to use but I believe that taking this from a different perspective from a previous stage can have its uses and for those of you that are doing concept arts or are doing image research for new projects then I think this is actually invaluable because it allows you to set the scene and then just forget about everything else that you were used to do in a 3D environment yes you are trading Precision but you're trading that for close to instant Generations so that's going to be all for today I am actually quite happy that we managed to do a real-time implementation of the previous week's process I hope you like this video and had fun and learned something new if you liked this video please leave a like And subscribe my name is Andre Boni you can find me on Instagram at rushi or on the web at Andre bay.com and same as always I will be seeing you next week

Channel: Andrea Baioni

Views: 63,056

Keywords: generative ai, stable diffusion, comfyui, civitai, text2image, txt2img, img2img, image2image, image generation, artificial intelligence, ai, generative artificial intelligence, sd, tutorial, risunobushi, risunobushi_ai, risunobushi ai, stable diffusion for professional creatives, comfy-ui, andrea baioni, stable diffusion experimental, sdxl, sdxl lightning, blender, adobe mixamo, mixamo, Hyper SD and SDXL Lightning with Blender & any 3D software in real time - SD Experimental

Id: pA_tZCscOFg

Length: 24min 33sec (1473 seconds)

Published: Mon Apr 29 2024