CONSISTENT VID2VID WITH ANIMATEDIFF AND COMFYUI

enigmatic e hey everyone today I'm going to be going over how to get some heavy style and consistency with comfy UI if you're not familiar with comfy UI I do have a tutorial on how to get started the link should be above right here and you can go check that out first if you're not familiar with how this works so I'm going to include the workflow in the description so you can find it there but I really want to explain the parameters because then later uh a lot of people will be asking me questions and I just can't answer all the questions so I'm going to try to explain things the best I could so that there's not as many questions once you get this workflow but before we get started check this out as you can probably guess I spend a lot of time online and I'm always looking for ways to protect my data and my privacy that's why I'm so excited to be partnering with nordvpn nordvpn is a leading VPN provider that helps you stay safe and secure online with nordvpn you can encrypt your internet traffic protect your privacy and access restricted content and right now nordvpn is having a Black Friday deal you can get up to 65% off your nordvpn subscription for the 2-year plan and up to 60% off the one-year Plan Plus for my viewers you get four free months of nordvpn I highly recommend you checking it out and taking advantage of the special offer you can use my affiliate link below in the description to get the best possible deal so don't miss out all right so this is the workflow and uh I'm going to go over all the parameters right now I know it looks like it's kind of a lot but hopefully uh once I explain it it's not too overwhelming so all right we come here at the beginning um up here is where you load your videos so we have a few parameters here we have frame load cap basically how many frames you want to actually generate so if you put 10 it's going to generate 10 I recommend if you're experimenting you keep this around 10 15 if you want uh just to see what your animation is going to look like and you don't have to run the whole entire thing especially if you got a long video you don't want to like generate 100 frames and then find out that it looks like crap right so uh yeah so I would start like with 10 uh skip frames this is pretty much where the generations are going to start so like let's say your video has 100 frames but you don't want to generate the whole thing you want it to generate from like half of that video so instead of going uh zero you go 50 so that it does from 50 to 100 if you leave it at zero it's going to do everything like it's going to generate everything so for both of these and then you got select every nth frame which is like if you put one it's going to generate every single frame of the video If you put two it's going to generate every other frame if you put three every third frame and so on and so forth pretty self-explanatory you load a video in here and we're going to we're going to experiment a little bit to see the results we get with this but for now I'm just kind of just going over the settings right you got your width and height these numbers are going to depend really on your GPU you know if you don't have a a strong enough GPU then you're going to run into issues going to have like Addam memory errors you have to just be aware of you know the the aspect ratio of your video and then try to put the numbers that align with that aspect ratio like for reals I typically do like 720 by 1280 but if this is too much for you then you can obviously go lower as you just got to make sure that these are the same aspect ratio as the original videos I think checkpoints are pretty self-explanatory you get a checkpoint whatever it is that you want to use so I do have to let you know that I created this workflow working with 1.5 models so if you are using an Excel model then there's a lot of things in this workflow that have to change and it's also not going to give you the same results so I do recommend you only use 1.5 models for this workflow I have here two luras that I'm using you don't have to use two luras if two luras is too much or if you just want to you know bypass the luras you can just come here put zero and then zero here it's going to bypass this Laura uh sometimes when you have the luras with the high strength you your luras uh will make your image look really bad so just be aware that you might have to lower these maybe Z 0.5 or 0.8 if you're having like issues with the image then play around with these parameters because that might be the reason why you're getting like a really bad image so skip frame will really depend on what kind of checkpoint you're using because when you come to citi.com and you see some information about the model you see right here it says Skip One and this other one you don't really see that here however if you come down to some of the images like for example this one it'll give you a bunch of information often times and like right here skip too yeah so whatever it recommends then I would put that here I would put NE 1 or -2 also if you notice that your image is looking not so contrasted this might be the reason why it doesn't looks so good because of this so just be aware of that coming here to animate diff loader for me stabilized High has been the best one for me I've been getting the best results for this specific kind of uh animation at least for like character consistency I will include a link to where you can download this specific one in the description below so that you can download it now you all you got to do is put it into your whatever folder you have your anime diff and then model and then put it in there so that you can access it here also uh free U also really improved my image quality it really made a difference I'm actually will show you an example right here so I'm going to test this out without the free U so you can see the difference with and without it so that we can kind of understand what's going to be happening and uh right here we have this image I'm trying to make this image of like Snoop Dog uh I'm I using this guy as like a reference um I have some control Nets there and uh so like if I run it like this you know these are the ugly results we're getting probably because a little bit of the luras um but then we can always mess around with the parameters to make it look cleaner and then we're going to try with freu and see how much of a difference it makes too so here I'm going to just bring up the steps a little bit higher let's see what that does okay it improves it slightly gives a little bit more more little details there let me bring the CFG to five and then see what that it's cleaning it up a little bit more what if I bring it the noising a little bit down as well okay it added some details there right all right so you know it looks it looks pretty good I mean it can be better but what happens if I put the free u in there and I got the freu parameter letters from the preu GitHub page and right here it tells me like for 1.5 model to put these numbers in there so that's what's in the node that is in this workflow and then let's just put this in here let's see how much it changes it it cleaned it up drastically and just look at look at the difference yeah free U is super super important another thing that I have here there are two things that I have here that are really helping me with the consistency obviously the Laura um the freu and then the IP adapter and uh I want to give props to the guy who developed this actually he goes by uh Lon Vision on YouTube definitely follow him he goes more into depth with you know how this works check out his tutorials he goes like really in depth and he also shows like when he does like updates to this IP adapter is just amazing and like I just want to give recognition to this person because they they did an amazing job with that like it it makes things so much e easier and I'm going to show you what it does it's pretty much uh you know I like to think of it as kind of like a Laura like an instant Laura or references an image so let me just set this up really quick so that you can see what's happening I have this set up right here it it's going to be in the workflow as well so that you can see it and I'm going to put like I'm going to use this reference image that I already have here so you put an image um here you have the weight and the noise let's just test it out this how it is and just see how it changes things right all right so you see that it g gave it a little bit of the tint of this the color that you see here what if I put the weight all the way to let's say one and let's see how much of a difference it makes you see now it's way stronger what if I bring the noising up and's see how much that changes okay so this is good for like getting style or like for example this like the colors that you're seeing here you're starting to see it here and I'm I'm I'm telling you this because we're going to piece it all together later on with this workflow so I'm just kind of showing you with one image how this is what's what's happening here you can do this as like colors but you can also do it with faces so um let me see all right so I have this image here but I think let me see you have to see what it's actually going to generate all right so yeah it looks at his face so okay so it's not making much of a difference when you bring the weight down it's going to follow more more of the prompt if you put this higher it's going to try to follow the image a lot more there's also different models that you can use here in the IP adapter section you can try this thing called face where it will look at the face and try to match the face more so if you try that it's matching a little bit more of what's Happening here since his face is bright here it's making his face even brighter here also the face is not it's not facing the same way so obviously that has a lot to do with it too okay think this is the best example I could find right here like it literally got this guy's head hair uh piece right here like you can mess around with uh linear and it's going to give you different results yeah linear Channel penalty yeah this is really really cool so for these I think you do it's better to have like close-ups of the face so that it can really see the face but you have some other options you have for some XL models as well let's proba like something like this right something that doesn't even look like the person and just see what kind of results we get from this right so you're starting to see some some of the influence from this image on here like a little bit of that gold right there um this is really where it gets kind of fun and interesting because if you start using this on a video then you can really get that really strong style like you're doing something like that's metallic and then you put like an image as a reference that's metallic like right here um then you start to see some really interesting stuff try to get some stronger weight in there let's see if it doesn't ruin it you see now it's really really taking over you put this a little bit higher and then you change some of these parameters like the weight type you're also going to get different results with this so this is how was I was able to accomplish something like this with that exact same image and look at how like metallic this looks it looks so clean so this is again this is a useful tool that you can use so I have that in here I have the foru uh Noe in here and uh I also have a face restore this is not necessary you can actually remove this if you want you can keep it sometimes it gives me decent results not not always let's let's just test out everything so you can see what's really happening let's say I put this here and let's see what kind of different results I'm going to get it smooth it face out quite a bit and if I bring this down it's going to smooth it even more and it's probably going to make it not look like him the more I add this so yeah and then the higher I bring it the less the effect you're going to see okay so let me try without it okay so that's that's it does clean it up a little bit so it's not necessary if you want to use it it's there if not you can always just take it off here I have control Nets depending on G GPU you would have to see if your computer can handle three control Nets I typically use three and you have to also be aware of what footage you have so like in this case in this footage here there's not a lot of details but I do want the AI to generate something like if I want him to wear a tank top I want it to kind of make a tank top here and uh and not follow these little wrinkles that you see around his neck from his hoodie or whatever he's wearing this jacket so in that case I don't want a lot of like I don't want the the art line or even the soft Edge to be too strong so I do have it the pretty low I have it at Z 030 I also have um the depth pretty low I have it around the same amount 030 0.27 um but I do have the open pose really strong because the open pose it's not tracking it's not paying attention to the shape of everything it's just tracking the body and the way the body is moving so it's really just getting like a skeleton and not really the details so this one I do have it boost up so that it can follow the body a lot easier for example like if you do have footage where you want it to look like for example let's just say it's somebody who's dressed up as a pirate and you want to you want the actual generation to be like a pirate and you want all those little details then in that case I would boost up a lot of the you know if you have line art as a control net I would boost up the line art um I would boost up the soft Edge I would boost up things where it it really does match the person like very closely but if not then I would try to bring these low also be careful to not bring these too low because then uh you will be lacking a little bit of consistency because it's going to have less to go off of so the lower the control net the less control you have of the final Generations keep that in mind all right so let's just test this out right let's let's let's do something with this this is a square so I'm going to have to let me just make this 720 by 720 I'm going to use the Snoop Dog Laura that I trained so so now I'm just going to come here and I'm just going to cap it at 10 so that I don't have to be waiting for too long and this is the animation I'm going to be using I'm going to be using this video um doesn't look really look like Snoop Dog but you know that's the for me that's kind of the point to kind of stylize something so let's just try that let's see what kind of results we get all right so this is this is the results I got um I don't think Snoop would ever wear something like this I mean maybe the the tank top yeah but like uh yeah not probably not something like this but the only reason you're seeing this it's because like the pants are like skin tight so that's why it's really kind of getting that but obviously with the settings you can probably I could have make that go away with the negative prompts and stuff but uh as you can see the face is not so clear because it's like a far shot if I had a closeup the face would look a lot cleaner um there is a technique that uh like a guy who I know who's been messing around with uh comfy UI he found a way to actually upscale the face and put it back into to the body and make it look so clean but uh that person is going to be sharing that workflow at some point maybe um and uh I asked him if it was cool if I shared it and he said yeah but at the moment he's asking me to wait so when the time comes I will definitely share it with you guys also I I you know I'm boosted up the uh the resolution pretty high up there um if I lower it let me just let me actually lower it to see what kind of results I get if I were to lower it um but I did mess around with the denoise I brought the denoise a little bit down uh the CFG is kind of low the steps I have it at 30 so just this is something that I would play around with a little bit the den noising because sometimes when you have it too strong then it kind of veers away from the actual uh original video because obviously it does have the help of the control Nets but um stuff starts to appear stuff that you might not want to appear starts to appear like outside of the body so just be aware of that so we're just going to see what this does all right so um yeah I mean it it lost a little bit that detail but it still looks fine even at 720 by 720 I actually gave them pants now which is interesting uh but yeah it's going to look different to me it still looks really cool it doesn't even look like really like Snoop Dog but um I think it looks close enough let me actually test it with a close-up of somebody all right so obviously you know there's a lot wrong with this still because it's it's going based off of this also I can probably put up the the noising but I just wanted to kind of show you what's happening obviously you can see that there's like a lot of details and obviously the control Nets are probably uh giving it little details that we don't really want on there but probably the biggest thing is the noising uh strength right here we could bring it up probably more but as you can see like obviously the face just looks a lot cleaner when you have a closeup like this does look like Snoop Dog for the most part and then you're seeing a lot of these colors from the IP P adapter right here like a lot of these purples here because of this which is also like really really cool thing that you can uh influence the image with besides that over here I have some uh some nodes for like upscaling I have it disabled right now you could enable them by pressing crl M when you select them but I'm not really upscaling with that I'm if I do need to upscale I do use uh topas AI it's super easy to use and it's uh way faster to upscale the reason why I really would recommend this is because if you're going like really low resolution and your your image is going to look very blurry and then you bring it here to topas aai and then you upscale it and it's going to look way much cleaner and more presentable for like social media or for whatever I I use it like 90% of the time that I do anything so I'm going to just show you what it can do I'm upscaling it by two so it's going to be 1440 by 1440 40 it's going to look a lot cleaner all right so now it upscaled it and if we come up close you're going to see like how much it actually cleaned it up you still see a little bit of blurriness but it's it's still much cleaner the line and this is really zoomed in so this is just an amazing way to upscale it really quickly if you're interested in getting this I am an affiliate of top as AI so there is a link in the description uh if you want to support me through that just go ahead and purchase it like I use this all the time a lot of people use this it is very useful and if you don't use it for just this you use it for so many other things there's other options like motion to blur stabilization uh frame interpolations obviously I'm going to give you the option to do it through here as well so so that's pretty much it if there's any other topics that you want me to look into then I can also uh try to look into that but I'm always learning and discovering new things with comfy UI there's always something new coming out that just improves just a quality of things so like I will constantly be trying to improve and try to make things better and I will try to make uh workflows that I can share with you guys I am uploading a lot of workflows uh first on my patreon so if if you want to get a hold of like the latest workflows that I'm creating the more current ones that I've been working on then you can go to my patreon and access them there also um I I'm always like uploading stuff there so just uh yeah if you want to you want to support me through patreon you can go check out check out patreon again there's uh Link in the description don't forget to like comment and subscribe because that's going to really boost this video it's going to help me and my channel grow so please you know if you guys can do me that favor do me that solid I appreciate it all right everyone thank you so much for watching like always take care God bless peace [Music] he

Channel: enigmatic_e

Views: 28,737

Keywords: animatediff, stable diffusion, ai animation, ai art, comfyui

Id: P4IdHKHrb48

Length: 20min 10sec (1210 seconds)

Published: Mon Nov 20 2023