How To Create Real-Time Renders with AI (synced to 3D model)

let me show you how we can create realtime renders like this directly from a 3D model using AI I will use a model from Rhino but you can use any 3D model you want to and then we will learn how we can create this workflow using stable diffusion and com UI and use our 3D model directly in real time I will also share a custom script to install stable diffusion on a Cloud Server so you can use it even if you don't have a computer it is going to be a slightly longer video video because I want to share all the details but you can find the timelines in the video descriptions so feel free to skip the parts you want to watch let's start so for this workflow I will try to render this model and this is poly Plaza project from China the script was created by decoded faat thank you for sharing the scripts with me for this model he's sharing most of the famous parametric designed uh buildings or maybe just their facades and how we can recreate them uh not just the models but he also tries to explain what is the logic behind this parametric model so if you're into this topics definitely go and check this page out you can use any type of view in your computer screen it can be either from a rhino in this case from a SketchUp model or from Arch RIT basically any model will work with this workflow I'm going to use confi on Rod instead of using on my own computer uh just to make the process faster and also I want to show you how you can set up confi on Rod if you cannot use it on your own computer for some reason you can easily do it here first of course you have to go sign up for the rod platform you can find the link in the video description after you sign up this is a paid platform basically you pay per hour of usage but it doesn't cost much so you need to go to your wallet and then deposit some money in your account you can deposit as low as $10 and after you do that you can go to pots click on new pot here we see all of the gpus that available for us to rent we have two options secure cloud and the community Cloud we can choose both of them Community cloud is a bit cheaper compared to this one so I will go with this and then make sure the speed is choose extreme you can choose any GPU that you see in this menu but most of the time I use either RTX 490 or RTX 390 I think this one is a really nice one for performance and the price it cost 26 cents per hour so I will go with this one in this menu we have to click this part and change storage we want to use I prepared a script that you can install conf and all the models extensions control nut models IP adapter models that you will need and to install everything we need around 200 GB but just to be safer you can type 300 and this doesn't cost much for the ports we have 3001 and this one for the default these are probably enough but just if you want to use maybe two different confi at the same time or for some reason we cannot connect to this one that's why let's add also 3002 3003 3004 just so like we will have place if we have any issues if this doesn't make any sense right now it's it's fine um you just need to add these ones here as well and then set overrides and for the template we will use this one rod fast stable diffusion template and just type fast and it will appear there is also this confi template Direct within Rod uh for some reason for some extensions is not working in this one uh that's why I prefer using this one and install the confi and all the modeles myself and then continue deploy this will just take couple of seconds to activate the GPU and if you come to my pods you can see now it's starting and once it is ready we will see this connect button available it is running and [Music] con to Jupiter lab option and then at Jupiter notebook will open and I have prepared this two scripts so we can install confi and then download all the modeles extensions that we want to use so just drop them here onto the file menu go ahead and open a terminal to start the process you need to copy and paste this script here I will share all of these files and the scripts in the description in down below so after you past it just click enter this will install all the necessary libraries and the confi the models checkpoint um control net basically everything that's why it's going to take some time uh depending on the internet connection of the GPU we connected and once we see this installation is done enjoy text at the end of the terminal that means it is done you can open a new tab from here and then let's open another terminal and this time to actually run the confi you need to copy and paste this script and then enter the first part you need to do only once once you open a new po but if you already have a P that installed then you don't need to wait all the time like around 30 minutes 1 hour you can just start to config it directly sometimes you might need to paste the code couple of times to get it right but eventually it will work and while it is downloading we can check Which models we are going to use for the realtime render this is the main model stability AI published called sdxl Turbo Turbo Vision X L model reality Edge Ultra spice realistic Vision turbo model which is one of the most popular ones I will also test this if you want to add a new model that is not here for example let's say this one click on this file and we have this download link here but instead of directly clicking on it right click and copy link address and then we need to go and install it into the correct folder which is inside the conf UI and then the models and and checkpoints folder I don't know really why but maybe there are uh lots of files inside checkpoints for some reason I cannot go inside checkpoints folder in Jupiter notebook so that's why what we can do is um you can right click and copy P first and then to to move there we can type CD slash and then paste the address so now as we can see we are inside the checkpoints folder let's go and copy that address again and we can use V get comment we get you can place the link between this quotation marks and then um we want to rename the file automatically so to do that you can type Dash all and let's copy this name you can type turbo. Save tensors which is the file extension as we can see here and then enter and you can see the progress in this part and it will be saved as what we named in this part but for most of the models you don't need to do this because I already installed many models and we can check this by going to the checkpoints folder again and if you type LS we can see all of the ones we installed so let's go back to our original terminal where we started the confi if you see this message at the end of the terminal it means the confi is started and it is active but since it is not locally running we cannot just click to this link we need to go back to my pods open this part and click on connect since we started on 3001 Port we need to connect this one and if you click we will see the conf is running so this is the default workflow you see when you open the first time the confi if you check these are all of the models we downloaded and let's just test it once make sure it is working we get our image so I have already prepared the workflow for this real time render I will directly place it here and then I will try to explain on top of that one and if you don't know if you have an image that you generate inside comi and you want to use the same workflow with that image and the same parameters the same settings everything you can just hold the image and leave it into your canvas then you will get all the workflow like this so you can directly change the base image or the prompt and then um use it yourself right now we have all the extensions all the custom nodes installed correctly that's why we don't get any read errors about the missing notes but if you see an error like this then all we need to do is go to manager in this manager we have many options to install a new custom Noe install the missing custom noes or install the models uh this one is really handy because when you get a new workflow probably you don't know which notes is used inside so you don't know the names of also but if you click on this one uh the manager will directly say okay this workflow is using this custom noes but you don't have these ones and it will list them here for now I don't have any missing ones but if you do they will appear like this in this table and you can just click on install after the installation is done you can restart the confi and then refesh refesh the page then it should appear sometimes it cannot get the right version in the first time so if you still see the missing custom nodes after the first one just go back to manager go back here and install it one more time then it should work it is not really a super long super complex workflow but uh one of the notes is new and it is the more exciting ones which is this one it's called screen share sh from mixlab nodes normally as you know for control net and this part is for the control net we need to feed an image normally we are doing this by using this load image component and you can just choose an image from here up upload an image and then we can connect it to the control net but in this one what we are doing is instead of loading an image we we want to get a view from our 3D model which is Rhino in this case we have couple of options to choose first you can share screen if you want you can also direct get view from your camera so from your computer's web webcam I will choose share screen and you can either choose a specific tab like normally you are sharing your screen in in a meeting or you can choose a specific window or your entire screen in this case I think uh choosing a window makes more sense so you can fix it to the Rhino so even if you change your screen will not change the view popup shows we are right now sharing our screen just say hide now we see our entire Rhino uh window or here but we don't care about this parts we only want to see our 3D model that's why I will click on set area and then we can specify Y which part we want to get let's choose something like this or maybe if we choose something more Square it will be better so something like this so right now we are only capturing this part after we captured this part I'm using this image resize component and which you can find by typing resize and this one what is happening in this part is I want to resize the image before feeding to the control net because for 1.5 models the default size is 5 12 pixels and right now I'm going to use the XL model turbo model so the default size is 10 24 pixel if you feed a super large image you will get weird results like these ones and also if you be a really small image it will not also perform really well so so just to automate this process I'm doing this resize and I said the smaller size is 1024 so it will adjust the other larger side according to this aspect ratio and from this resize mode you can choose any option so if it's smaller it will make it bigger if it's bigger it's going to make it smaller and if you go to the beginning from this load checkpoint note we can choose which checkpoint model so which AI model BAS basically we want to use let's start with the base model which is stable Fusion XL turbo this one and we have two text box basically for our prompt and this one is for the positive prompt right now if you want to make it more obvious we can change the color to Green for example and this one is for negative one we can change this color to red but for Turbo models negative prompt is not working so even if you typee something for the negative prompt during the generation this negative prompt will not be taken into the consideration so this will be just empty for now normally if you're not using control net this is the whole process but we want to feed our model that's why we have this additional control net part for this 3D model I think the dep one makes more sense that's why I'm using the this control net XL models the the dep one as you can see we install many different contol nut models for both 1.5 and XL models directly so you can try different ones and see which one performs better I have initially seted this as a Midas St map but now we have a new one which is I think U working better dep anything so let's change change this one to this depth anything and we can see both of them how it performs and then we have apply control net where we are feeding our positive prompt first to the control net here so basically from here to this one and then it is going back to K sampler for the latent image we are using an empty latent image component and normal this one is looking like this we can change the image width and height from these bars but I want to automate this one as well so to do this you can right click and convert width and height to input so instead of playing with this bar here it becomes a input that we can feed a number automatically and also for the height and I'm using this get image size component to do this so if you type get image size you can we can feed our resized image directly for to this component and then connect this ones to WID and height so we will directly generate the same image size as our input I will delete this ones because I already have here and the biggest difference for the turbo models is the number of steps as you may know uh normally a good range for the step is around um 20 30 30 to 40 steps depending on the sampler and the mod models but for this one technically you can even generate a nice image with only two or three steps let's go with three but of course if you use something higher around like 8 um the quality tends to be better so let's go with four for the first one and for the CFG normally um around 8 10 12 is a good number uh but for Turbo model uh you need to keep the CFG scale a bit lower so around 2 three even 1.5 maybe so I will just organize my view a bit better something like this maybe and we can see automatically how it's adjusting in this part and maybe I would change the view to render I think it's a good starting point in our prompt we have Auto parametric building in the city fall late afternoon Sky Lim gray night quality Etc so let's just hit generate and see what we have I will test both of the dep maps to see which one gives better results and then we can remove one of them because for the control n and stable diffusion uh this image we are feeding so uh this image we are feeding to stable diffusion it doesn't mean anything we are not using this image directly we are using the dep control net the only important image is this one in this case so this should be a nicer one if we want to get all the details as you can see right now the details on top of the model like this elements and the vertical frames for the windows are almost totally missing so probably we will not get a super detailed image but we can fix this later and we get an image like this for the first time which is not really a nice one let's try another dep map the lest one and maybe increase the step number let's see how this one will perform we can see the frames a bit better but still it's not super nice so and the image is not super nice let's try a better turbo models for example turbo Vision XL the new version and see how this one will perform and after we find nice settings and nice parameters for our model and then we can go to the real time render process this one is much better is somewhat getting there let's increase this steps number a bit more we found a better um settings and parameters right now I didn't change mod settings we are using right now four steps and this letter step map which is quite nice in this case captures most of the details compared to other ones so so I will eras the other ones and only have this one and for the prompt um it's kind of same but I only added single tower building we can set the seat for increment and keep testing some different options sometimes it tends to put another building next to it I'm not really sure uh why it does this to have a better control and maybe more accuracy from the prompt we can add another control net and combine both of them right now we are using this uh dep model what we can do is try to get um edges of the model so we have more details and more information about about this small details on top of the building so I will just copy this apply control net and load control net model but for this time instead of depth I will choose Kenny let's get a Kenny preprocessor and I will fit our image here then let's copy the image preview as well and feed the image here now instead of directly feeding this conditioning to the positive prompt to case sampler first we will feed it to the second control net and then we will connect this one to this one so first from the positive prompt we are going to first controler to dep one and then from here we are going to second one to the Kenny and then we are connecting back to case sampler let me add a group here too so it's more organized let's run it again so now you see we are also feeding this information and this is quite nice but maybe we want to see more details even about this uh frame on the on the windows so we have two threshold settings for the Kenny if we low these values it will capture more details from the Bas image and if you increase this F it will capture less details that's the idea so let's set up these ones to something like this and if if I run it again you will see the difference in the image as you can see we have way more details and we we want to capture these ones maybe we can even reduce it a bit more to 0.5 to1 we solve this issue of adding another building next to it this was the previous one the new one let's run it again with even more details so now I will add more details to The Prompt about the building and maybe less remove the f um white polycarbonate material glass window with Metal Frames also remove this late afternoon or let's keep it for now let's run it I'm happy with the we can play with our prompt and get a better result change to mood Etc but now let's go and try to explore how we can uh play with our model inside Rhino and get instant views from the model right now it is not constantly updating from this view so even if I update this part to something like this uh if I don't run it you don't see anything to run it constantly and get the views all the time we can click this live run option or open this extra options and open the auto que button let's click on live run and now even if you don't click Q prompt it will automatically render all the time so after we make some changes in our model we can directly see the views here but it might be hard to do this if you have one window that's why there is a nice component that we can use which is called floating video right now is connected to save image but we can also connect to this one and if you click this picture and picture now we have this floating window with a prompt I want to copy our prompt here and place to this one and let's now go back to Rhino so now we can directly play with our model and get instant views to here so ideally what we can do is change some prompts here and then we will see the effect directly and also really good thing is since our model is from grass offer we can directly play with the model inside grass offer and see the changes I will keep the view stable like this for now and then try to to change the chome right now we are using six steps that's why it's a bit more slower and remember we not using a super high quality GPU but if you if you increase the performance of the GPU and get a new one you can do this way faster but let's try to reduce the number of steps to three for example and see how it will go now it is uh way faster but the quality is not so nice so I prefer waiting a bit more to get a better image so let's try to use for example black material and change to Dark Knight View and if you want to make it more stable not to change the Im so much but slightly we can try to change this seat to a fix fun so we will always use the same seat number okay I really like this image composition how the like background and building itself the reflections on the Windows see it as fixed right now and now I will play with some of the changes and maybe like let's try to make this window frames this w white Parts a bit larger of course because I moved the frame the model the view is also updated but now it's back to a better position we can reduce the number of this uh dividation for example have a larger window frames and as we can see this is a really nice way to see and imagine the building in in different forms because the form we adjusted from the grass offer and the view directly change but the and environment and the overall mood is almost the same but we can see the how changes in the building shape can affect the the VIP and the mood of the final shape for example let's make them way dense this uh frames and the viations on the on the facade and see how it will look like so right now we are only using this VI as a input and our prompt so from image to image we expect it to change more or less but even without any additional settings it is still more or less similar WIP so we can focus more on the changes on the building but we can try to but even improve it with using reference images I will remove this one and I will open some space by moving this ones a bit and we will use the reference images to do this with uh IP adapter note so we need IP adapter model and then I will feed the the bundles and the C Vision model quickly uh these ones are also coming directly with the in installation script automatically so you don't need to do anything else we are using the Excel model so I will choose this IP adapter sdxl save tensor one there are some couple of options these are for specifically for person faces these are uh the the plus version more or less similar results so you can try and see um different ones um let's go with this one for now plus means it will have more effect on top of the final image and of course we need to feed our uh reference image and to connect it the workflow so right now as you can see this is not connected to anywhere we need to feed the model first to IP adapter from our checkpoint so instead of going to the case sampler directly first we feed it to the model and then from the model back to case sampler so right now it's will also use the this reference image but right now it's using this one I will find one of the nice images we get for example to use as a reference let's say this one I will download this image and then from here we can place to this part so now we will use this image as a reference so ideally the logic is even if you change the model shape or the model view by mve inside the Rhino the end result is going to be more or less similar to the this image so we will always have the same wipe maybe we can reduce the strength of bit so it's not super powerful so even if you change the prompt it it will still be somewhat similar so now let's go back to Rino modle and have our window back of maybe changing the parameters we can try to move the model and see how it will change now I I want to change and see how it will perform instead of the turbo model but with the normal Exar model in reality Excel model and of course then we need to change the step number to let's say 40 and CFG scale has change to 90 I will keep the same reference image for for now and now let's test this Ultra spice one I don't know how that one will perform but let's see and as we can see like there is almost no quality difference for me if you don't look super um carefully and it's way faster so this makes more sense especially in this workflow I want to try one more thing uh with this workflow right now we are feeding as a reference image one of the generations one of the images we just generated but as you know this is a real building that is built poly Plaza so I want to get an image of the original building and then um use as a reference image maybe this way it will be closer to the actual building let's choose this one and we can save the image so this is the image image size is really large 2,000 to 3,000 pixel I will resize it from from this option resize image to Let's reduce the height 2,000 pixel or maybe 1024 and save it now I will place it for the reference image and let's see how it will turn out like this is way closer and way nicer I think the original design especially let's go and adjust the parameters a bit so I think we covered most of the idea and what's going on in in the workflow how it is working what is the logic behind it and how we can capture uh view from Rhino as you can imagine this doesn't need to be Rhino specifically it can be any 3D modeling software or any type of other view that you can capture from your window basically it can be also video or it can be directly your webcam so it is really flexible right now we use the depth map and the the Kenny Edge map but later on I turned off the depth map because even only with the keny it performs quite nice and this is way faster normally generate this dep map it takes some time so you can either choose to use them together or use only Kenny if you you don't want to use uh a part like this you can just choose all of them and press contrl B to turn it on or turn it off and we have this part where we can set up a reference image basically you can upload any type of image you want in this window and use it as a reference image and you can play with the string of this reference image in this bar and that's all basically and this is only one of the use cases for an exterior building view this can be also super powerful with interior views to generate um different layout options to visualize different materials different textures super quickly directly inside the model and here are the final images so I hope hope you like this video and this might be useful for you in your workflow I will share all the resources including the rod installation scrpt workflow that we created and also the the 3D model that you can download and play on top you can find the link in the video description and the resources will be available on my patreon page so if you want to just download the same ones and use them as it is you can find find them there but otherwise feel free to create your own workflows as well so I hope you liked it and see you in the next video

Channel: Design Input

Views: 9,467

Keywords: architecture, render, AI, rhino, 3d, cad, modeling, unreal engine, tutorial, design, parametric, computation, parametricism, 3d print, planfinder, ai, arch, automatic, generator, Learn, Architecture, Visualization, lumion 2023, ray tracing, Interior Design, midjourney, stable diffusion, sketch rendering, sketch to render, ai art, artificial intelligence, ai image generator, midjourney ai, residential project, house design, home design, 3d modeling, 3d model, architecture student, ai design

Id: GmXkrgqvWvk

Length: 37min 41sec (2261 seconds)

Published: Thu Feb 15 2024