NEW ControlNet Applications - A1111 Stable Diffusion Tutorial

hello everyone in this video I will show you some of the coolest application of control net with stable diffusion and to be more precise I'm talking about some interesting and trendy ideas on how to use control net in the text to image context so we're not going to use control net with image to image or other extensions like the forum for example alright so without further Ado let's get started [Music] first of all for those of you who don't know what control net is well briefly it's a neural network used to guide and control the diffusion process in stable diffusion and if you're familiar with how stable diffusion models work you know that the text are the prompt we use in the text to image generation as well as the input image we use in the image to image generation these are used to guide the diffusion process as shown in this image and you can find numerous explanatory videos and articles on this topic I will leave some reading materials in the description so feel free to check it out if you want So before we start with the first application we will be using the auto 1111 web UI so you need to make sure that you have it installed as well as the control net extension and you can check my previous stable diffusion video to see how you can easily install the web UI and for the control net extension I'll quickly show you now how to install it and by the way I'm currently using the latest version of the web UI which is the version 1.6 so to install control net simply go to the extensions Tab and then look for the SD web UI control net and then click on install on the right side of the table and after installing you need to make sure to reload the web UI so that you can use control net in text to image also if you want to get similar results as me you need to set the eclipse skip parameter to two in the stable diffusion settings tab and don't forget to apply the settings okay so now you should find the control net tab in the text to image page and to be able to use it in image generation one last thing you need to do is to install some control net models and to do that you should simply go to the control Net v 1.1 repository on hanging face and there you can find the different control net models so feel free to download the models you want to try for this tutorial however you need the depth mlsd IP adapter and seg or segmentation models and make sure to put all the control net models in the control net folder within the models directory and apart from these models we need two more models for the QR code image generation so the first one is by monster labs and you can find it also on hagging face and the second one is by Diane Timmer also on hanging face and after you've downloaded all the models press on the refresh button and then you're ready to go okay so the first application I want to show you is interior design and interior styling I think that this application is very fun and can come in handy when needed so let's say you want to change the style of your office your bedroom or even your gaming room by using control net with stable diffusion you can come up with great and amazing ideas a specialty style wise and this is how you can do it so first let's get an image of a living room next you need to upload it to control net and make sure to check vram if you have six gigs of RAM or less check a low preview and choose the segmentation control type now click on the explosion Emoji to run the preprocessor and get the segmentation map now save the segmentation map because I will show you later on how to use it to add remove or modify objects in this picture and for now you can leave the remaining parameters as they are let's try now to change the room Style by using short and precise description as a prompt and as you can see we have a really great result as a starting point we can enhance it using higher rest fix or tweaking a little bit more the control net parameters as well as the text to image parameters but I'll leave that for you now let me show you how we can use the segmentation map to add or modify the detected objects so now we need to open the segmentation map in an image editor like Photoshop for example here you can use the brush with white stroke to modify the objects as you can see we can easily duplicate objects or change their sizes and to add objects that are not already present in this image we need to search for the color codes of the preprocessor we used so here we have the ofade 20K model and here is the color code of the different objects we can use so for example let's add a vase with some flower next to the sofa in this picture to use the color code for the vase and draw a simple outline and then we can do the same thing for the flowers let's now head back to the web UI and upload the new segmentation map to the control net and let's give it a try we don't need the preprocessor now because we already have a segmentation map ready so set the preprocessor to none and click on generate alright so that's a really great result you can be as much imaginative as you want using this technique now before we move on to the next application let me show you how to change the style of the room using other control net models and you can then pick which one works best for you so let's first try the mlsd model which is used for architecture and geometrical purposes because it detects the straight lines in the image okay I like this result we obtained different style while keeping the main composition of the object in the room the next one we're gonna try is the depth model it is mostly used for images that has clear distinction between the foreground and the background so that it would be able to detect the depth in the image okay so we obtained the decent result as well so the last Model I want to show you is the ipaductor and this behave a little bit differently as it keeps the overall style of the image while trying to change a little bit the composition and it can also be used without a prompt as we did here all right decent results but you can also add a prompt to specify certain needs for the adaptation and try to adjust the weights to give it more imaginative power I really think that this model has a lot to offer and you can obtain really great results with the short or no prompt at all all right now enough with the decoration let's move on to the next application which is the logo gives this application was trending about two months ago and it is a really good application of the Canon controller type so first what we need to do is to upload a black and white logo image with white as the background then create a simple prompt here we have an aerial view of a tropical island with coconut trees and few Yachts around and feel free to use negative prompts as well but this application works better with short prompts so remember to keep the description short and then for the control net settings remember to pick the candy control type and reset the different settings or weight settings to default and now let's generate the first batch of images so we're going to create a gif out of this Nike logo and for that we need multiple images so for each prompt you can generate maybe four to six images and here I'm using a good seed that I found after multiple tries and that's why I advise you to do so feel free first to try different seeds and different settings and once you find the perfect seed that you're happy with then you can generate a bunch of images and as you can see here the results are pretty cool of course you can use the hi-res fix if you want your images to be of high quality I'll show you the difference between the first image here and the the one generated with high risk fix alright this is much better image and as you can see here we can also see the Yachts added near the Nike logo let's now try a different prompt so here we have a prompt for a Countryside Road all right these are decent results that we can use to build our GIF for the third prompt we're going to create images for a modern city with skyscrapers alright for the next prompt we're going to create images for snow remounting and then of course you can add as many different images with different styles as you want it will only make the final GIF much better so now I'm going to generate images with different prompts so here we have an image or an aerial view of a beautiful Oasis then we have an image of a historical building in a city we have an image of an aerial view of a multi-color flower field okay so now we have generated multiple images and to incorporate these images in a gif we can use online tools like easygive to create a gif out of these frames but I will show you that later on since we're going to be generating other gifs so stay tuned for that now let's move on to the stylized QR code image generation so first you need a QR code and you simply upload it to the control net tab for the control net type select all and choose the QR code control net model in the model list for the preprocessor we don't need any preprocessor so keep it at none and here I'm using a prompt describing an advanced electrical circuit but feel free to use the prompt that you like and let's generate so this first result is not great but you can see the QR code embedded in the image to get a better result we need to use the iOS fix so here I chose the upscaler ESR gun for X with anime 6B the denoising strength of 0.7 and as you can see we have a much better result here feel free of course to adjust the settings of the sampler of the Cog scales to produce the result that you're most happy with and here is the next one it's pretty cool alright so this application is pretty cool and fun to try but now let's move on to the one I like the most so for this last application you need to find a gif for black and white optical illusion so here we have two black and white spiral gifs and for each GIF we need to split it into frames so to do that you need to go to easygif.com and there you can use the split tool to split the gifs into frames so you simply upload the gif and then click on split to frames so once you have the frames you need to upload the first frame into the control net Tab and by using the same control net type and the QR code monster model we can generate images that has the overall aspect of the optical illusion but that adds an artistic style to it so here I'm generating an image of Medusa with re-inspired hair and you can be as much precise as you want for this generation process so don't get discouraged with the first try you have you always need to improve on the first image you obtain so let's generate and as you can see now you get the idea of this application so the spiral overall is present in the image but it is embedded with the image that we want to produce so once you have a prompts and see that you're happy with you need to keep those settings and then head over to the control net Tab and use a batch instead of a single image and there you simply need to copy the path to the frames folder and finally you simply click on generate so now let's see the results so here we have the different frames generated and let me show you now how to create a gif out of these frames so we just need to use the easygift.com website and here we need to use the GIF maker tool and you can easily select all the frames and click on upload you need to wait a little bit for the different images to upload then you need just to modify the delay time and this would set the delay between each image so if you choose a higher delay time it would generate a slower give and in our case we're using a gif for an optical illusion so a more directly quicker GIF would look better so here I'm using 10 milliseconds as the integrate time and that's it so now we have our GIF ready simply click on Save if you want to download the GIF and that's how you do it so now let me show you some other GIFs that I have generated with different prompts and different styles thank you [Music] foreign okay so that's it for this video I hope you find it helpful and informative and see you in the next one

Channel: Machine Learning With Hamza

Views: 2,244

Keywords: python, machine learning, ML, AI, computer vision, NLP, tensorflow, pytorch, data science, data analysis, EDA, machine learning project, learn machine learning, stable diffusion, train stable diffusion, generate my photo with ai, Controlnet, control net, controlnet applications, new controlnet, ip-adapter, canny, segmentation, depth, logo gif animation, qr code controlnet, artistic qr code, qi generated qr code, stable diffusion qr code, new stable diffusion trends

Id: 4GPH1fE-6dA

Length: 13min 25sec (805 seconds)

Published: Thu Sep 21 2023