Real-Time AI Rendering with ComfyUI and 3ds Max

hi in this video I'm going to present a workflow connecting 3D Studio Max where we will build a basic 3D model with stable diffusion in order to create an almost realtime AI renderer this will give us the benefit to update 3D model and cameras and get almost instantly AI generated images for the 3D software I'm using 3D Studio Max but this workflow can be applied to any other 3D software where you can build any basic 3D geometry for the AI part I'm using stable diffusion with the interface of comy UI I'm not going to dive into details how to set up comy UI but I'm going to leave those two videos in the description where you can find more details on how to set up the workflow and which videos inspired me to create this specific workflow that I'm going to present you so here we are in U 3D Studio Max and confy UI I'm going to keep everything very sort as so I'm going to fast forward processes that are taking a bit longer and the best uh way to move forward with confy UI I think is uh to install confy UI manager and then after downloading my workflow you can install any the custom missing nodes that I have used in this process so I'm going to build a very basic setup on confy UI adding an efficient loader and an efficient sampler and I'm going to connect that those two um one thing that is uh important for our workflow to be faster is to use this Lowa Cod called diffusion by do model which you can find in the videos that I have on description and we can lower for now the steps maybe to 16 that no to 0.5 and let's leave the rest as it is for now we need the viewport to be transmitted in conf UI so for that I use we can use screen share and we can start sharing our part of the monitor for for instance our viewp now we need to scale that down because uh it might be um a lot of pixels for um confi to handle so for that I use a note called iMat scale to side which will scale down um one side of the iMat here the longest uh to whatever pixel we putting on the length now we can uh PL our emots into our sampler I'm going to use a very basic um prompt here a house on a snowy landscape and I'm going to build that very quick in 33d Max so here's the basic model of a house on a snowy landscape and we can already use the pr in order to see what we get as an uh I think I haven't hit live run which is going to update the viewport so you can see already that it gives us something it is close to what the viewport is but it's not very accurate but it's still very very close we can help it in order to make it better by adding uh control net adapters this is going to be the depth map and the can Edge we need to also apply nodes CL and control net loges the depth for the depth map and the g for the can so we take the bring it here and we connect in and finally to our Cas something so what depth map and canat does we can check it with the preview want we C The Prompt here we see what information we getting out of it so we can find here B can since it's missing some edes it's good now and we can c that and we see that we are already much more accurate and closer to the viewboard so that's the first step the other step is that we really need to play with all the case hler settings and find the ones that are good for our case hit okay so here we are with another scene uh I have worked just a bit more on the surrounding of the building and I have used a very basic prompt a housing tosan with a pool so here just want to showcase a couple of stuff that can um give us the freedom to manipulate um things in the benefit uh of controlling the output and not let the AI doing random stuff while we move around and we create things with our uh 32 Max VI so the first thing is that um we need to be aware of two values cfd value and the noise value um my feeling I I know that CF value goes more with the prompt side when it's higher and it gives like AI the freedom to to affect stuff with the prompt that we put in and um going down and we are connected more to what we see in the viewport in our workflow so the same thing with the the noise if I bring the den noise higher we're going to see that it doesn't really uh take one to one what we have in the viewport but it tries to translate stuff we are still getting the geometry because we have the control adapter but the I have the feeling that if we uh use the lower value we can change stuff on the viewport and this can affect our final image for instance we can start with the with the color of the building and we can make that another even something like green and we see that in instantly changing the way it renders the image so what that is one thing that we can do let's bring it back I think I like that color another thing uh um is that as we saw the color affects the result so for instance you can see that um some tree trees are liten up some others are not um and this is because due to those orange spheres that I have um copied and put them inside the tree just a bit visible so that it changes the color of the tree and this will affect also the final image so it's good to keep in mind that the color in general can be interpretated into something else in uh in stable diffusion so now we see that this tree also has light and but this of course we don't need to do it also only with spheres or just trying to manipulate the textures or we can also put real lights inside and we can see if we have the seene lights on we can see AI is affected uh of it so we can put the light wherever we like and this gives us a different impression and time on the stable diffusion and the final thing uh I would show here is that if I start changing also the HD on the background for instance bring the Sun at the back we are getting a different iMat of course the our omni light now is very very intense so we can breing any bit down and of course we can move also our camera so we can test very fast different views see what works what doesn't and move forward I hope you found that useful feel free to ask anything you want I will try to reply and of course let me know if you have any suggestions to make that workflow even better thanks for watching

Channel: Yannis Efstathiou

Views: 2,213

Keywords:

Id: pNyIp73zva8

Length: 13min 3sec (783 seconds)

Published: Fri Dec 22 2023