Style Transfer Adapter for ControlNet (img2img)

hey everyone today I'm going to be talking about some new adapters that came out for control net specifically I want to talk about the style transfer adapter I did a video a while back where I combined style transfer with EB synth and got some interesting results and I had a lot of people asking me how I did the transfer style but now stable diffusion allows for this through control net and these adapters if you're not familiar with how to download stable diffusion or control net I will add some links in my description of videos where I talk about how to install that so this is what it does you get the original image and you put in a style image and it creates something very cool like this yeah so you got the original here and then this is what it does so as you can see it's following the outline and that's going to make sense in a little bit once we go into stable diffusion here's another one uh that I I haven't gotten to work for some reason I just maybe I have to change something but basically what it does it creates a color palette and from that color palette it creates another image using those colors so like right here this is the original image it reads all these colors here and then it creates this so you put an image here it kind of pixelizes it and then it applies it to the new image the way you get these is I'm going to put the link in the description it's in the hug and face website and the ones I want to download it's the ones that are like the newest where it says four days ago downloaded the style adapter right here and then I also downloaded the color one again I can't get it to work yet but I downloaded those two once you download it you want to go into your stable diffusion folder you want to go to extensions you want to go into control Nets models and then you save it into here once that's done you come into stable diffusion you go into extensions check for updates if you're behind then apply and restart UI and then come back into image to image you can do this with text to image as well but it's a very similar process except you don't have this window here that is used for image damage so so yeah so the first thing I want to do is I'll put an image in here flip this guy running to make sure that I have the right dimensions here you want to come down to control net you want to start off with something that's going to create the outline of this image and I'll explain to you why in a bit so you want to make sure you enable use canny canny here and then control canny right here for the model and then let's go to a second control net if you don't have multiple control Nets here the way you do that you come into settings come into control net and then right here it says multi control net Max models amount then you put the amount that you want then you apply settings reload UI so we come back here we come to our second control net press enable go into let's go to head here and then for the last one we're going to enable this we're going to go to clip vision and then we're going to go to style T2 I adapter style in here we're going to put the style that we want in my case I'm going to try out Starry Night by Van Gogh if you really want this style to show up in the final image you want to make sure you bump this up all the way and I'm going to show you the reasons why so as you can see it's it's heavily being influenced by this if you put prompts it will affect the final output but for now I'm just going just straight up just directly when you put the noise straight really high it pretty much ignores the original image however when we have canny enabled and head enabled it creates these outlines and so now stay with the fusion is not being Guided by this image when we have it all the way up here but it's being Guided by these outlines now if you come down to canny and you bring down the threshold and you generate your image it's going to create a way more details in its outline what also would affect the final output is the weight and the guidance start the guidance start basically tells you when it will start to be influenced by this so if you put it halfway it's going to be influenced by half the amount if you start to bring the the noising strength down it's gonna go back to being led by this original image so you're going to start to see less of this in the final image see it's going more towards here but you still see a little bit of the colors from the style typically the higher you go the more style you're going to see in the final output and uh what happens when you bring up the CFG scale oh yeah it kind of makes it look a little more like this but it also makes the image a lot more blurry unfortunately so you want to make sure you have this at a low amount yeah I mean it looks pretty cool it's definitely influenced by the painting uh this is without Kenny it's just getting this outline uh there's so much you can alter here it's so crazy it's it looks like it's so much fun because then you can mess around with the settings let's say I introduced candy back in but I want this to have like a way lower weight let's say 0.5 yeah so it gated a little bit more detail detail that this was this one was missing yeah this is very cool it doesn't look like him uh like the original image but let me see um let me see if the prompt changes it at all yeah once you introduce prompts we do start to see some changes and it starts to be influenced by that prompt if you put the guidance strength to let's say 0.2 you're probably going to see a little bit less of this style on here yeah so slightly less and if you push it a little bit more let's say let's push it hard like let's say 0.5 let's see what we get you see it's it's reducing more and more but uh yeah it doesn't look bad it starts to align a little bit more with these lines and whatever style you have whatever checkpoint you're using and also the prompts that's pretty cool I added a different style down here another image right here to kind of follow and uh yeah I've actually followed it pretty well yeah so this is some of the results you can get with these Styles um I kind of wish it was more of these colors it's getting like the shapes right but not really the colors but uh yeah I'm gonna be messing around with it more I would just say play around with the setting some more you can play around with the weights for the outlines and how much you want each of these to influence the final image and yeah it's a very cool tool I still need to play around with it and obviously how can we apply this to video I can already tell you this like this will look like a total mess at the end because it's going to be so flickery it's going to be so crazy so I did a test already and this is the results that I got I think for certain clips not this one because there's just too much movement but for certain Clips you can combine this with EB synth the problem that I used to have before is that I couldn't get a style that was consistent with stable diffusion here it's not consistent either but they look similar like some of these look similar to each other and if you can get the ones that look very similar to each other and run them through EB synth you can get some really good results I have some videos on combining stable diffusion with EB synth so that you can check out that should appear above I will definitely make a video trying to use this with EV synth but for now I just wanted to introduce the tool first and then we can mess around with TV synth later it's jittery that's what I expected but um I think it's still cool that this is available to us um we don't know what kind of advancements are going to happen in the next couple of weeks or months so having this tool available if anything for just like if you want to just do one image then you know this is this is your answer to that um for video if we're not there yet um but maybe this is a step forward for that so as you know all this Tech is moving so fast and uh I will come back to this and try to mess around with it see if I can make something that looks consistent and still have a style so we'll see I'll work on it but uh yeah all right everyone take care God bless peace [Music]

Channel: enigmatic_e

Views: 24,813

Keywords:

Id: T1fYCLNfm-o

Length: 9min 22sec (562 seconds)

Published: Tue Mar 07 2023