VIGGLE + COMFYUI - CRAZY COMBO

all right so in my last video I talked about vigle and I had AI warper the guy who I mentioned uh his video actually went viral by the way he watched my video and he told me man you got to talk about how to use vigle and comfy UI to get the best results and so that's what I'm going to talk about today okay so I had to interrupt myself because some of you may not even know what I'm talking about so there's a thing called vigle that allows you to put it a video input and also an image as a reference and it will animate that image I talk about it in this video right above so you can check that out to see how that works I will also be using comfy UI and if you're not familiar with kyui I have a video on that as well so you can check the link to that in the description or above as well all right back to what I was talking about I'm going to be sharing two workflows today one is by AI warper it's the one that you're seeing right now this is pretty much AI warper workflow I just cleaned it up and made some minor changes to it but it's basically the workflow that he gave me and he wanted people to know the autoc cropping section of this workflow he got from a person that goes by anime isra so shout out to anime isra as well and I'm also going to give you a workflow that I did and this one is very simple it's like way more simplified this I think works well for up close shots the other one does too this is just if you want a very simple setup you have this as well that you can mess around with so you have that option but I'm sure a lot of you are going to want this one because this one has so much to offer all right so let's take a look at what's happening here right on this area we have the full final output right here down here we have the cropping part it's basically just autoc cropping and following the subject as you can see here this is a very wide shot and normally if you try to generate a shot like this this will look pretty much like mashed potatoes like all mushy and no details but when you crop in then you can add all those details and make it look super super clean right here you have the output after it's been run through this workflow and on the left you have the animation that came straight from vigle and as you can see the one from vigle looks quite rough uh and then the right it looks like extremely clean so this is a very powerful workflow this is going to be like a game changer for a lot of people however I think the biggest limitation is the quality of vle Animation because the quality can vary right and even though comfy UI is fixing a lot of issues if the input's really really bad then there's nothing comfy UI can do about it you'll have this workflow this is pretty straightforward you put in your mask in this section right here you load up your video this is an alpha mat right here and I'm going to teach you how to create one of these in a little bit using Da Vinci resolve the free version of the Vinci and down here we have the vigle animation with the background from the original video and here in the prompt section you have two options you can use this prompt window right here or you can use a batch prompt schedule I believe AI warper used this for his video like this is his original workflow and if you come down here this is how he scheduled his video so there are Parts where he scheduled it to be the front face and then Parts where he did side and then rear um he says he's not sure if this really helped but he did it and and maybe it did help when the subject was turning around to schedule it like this but he can't say 100% if it really helped or not cuz he didn't do the testing and I haven't really done the testing either but I didn't really add a prompt schedule I just used one frame like you see here for me it worked fine but you can always mess around with it and see what kind of results you get so let me show you the original input that I used it's this guy dancing he's turning around another thing AI warper told me like hey if you don't do a person turning around then people are going to call you out on it and so I was like okay well let me find one person is turning around a lot and uh also is moving really fast so this is why I chose this clip and then vigle gave me this as you can see the image quality is not good but it's tracking a lot of the body parts very well so that's what you're looking for it it's okay if it doesn't get all the details right it's okay if it looks kind of blurry a little bit but if the movements are very clear like you can see where the hands are going you see where the feet are going and also there's not a tons of ghosting around the character what I mean is like a bunch of little like Parts where it's a little bit transpar parent you don't really want a clip like that I mean sometimes you can't help it but if you can avoid it then that's going to be a good thing for you to run through comt UI so this looks really clean image quality not great but the movement is really good so this was a great clip to use okay so we have to create an alpha mat and you might be thinking why do I need an alpha mat what's what's the purpose of this thing here and it serves actually many purposes one is that this is how it tracks the subject so here I'm showing you an example of how the cropping works with the alpha mat as you can see you see a closeup then you see a far shot right and the reason you're seeing that is because I added a white block right here just for demonstration sake I added this white block to this Alpha mat so basically what it's doing is instead of cropping just this area right here where the white is so it seees this white block here and it says okay well this has to be bigger then so it starts to crop a bigger section right here um and that's really not what you want and this is why sometimes if you have an alphat that's not so clean then you might run into certain issues like this we are going to go into Da Vinci to create an alpha mat I'm going to show you how to do that and there are ways to create Alpha mats inside of comfy there are notes for that but for me they're not always reliable so I'm not going to teach you it if you really want to dig deeper you can probably find some workflows somewhere online like maybe even searching YouTube uh where they have this workflows if you can't I one online and you really want a workflow with it so that you can add it to this then let me know maybe I'll make a video about that and also the alpha mat allows you to render something with a background and it does not affect the background just the subject so dentry is a free editing software you don't have to do it in here you can do it in any editing software but I'm using this one because there is a free version and you should be able to do what I'm about to show you in the free version of Da Vinci all right so we're going to come here we're going to go to editing bring a green screen footage in here and it's going to ask change project frame rate go ahead and put change so then we're going to bring our footage down here to the timeline then we're going to come down here where it says Fusion we're going to click on this media end node and then we're going to press control space so that we can get the select tool window to pop up and we're going to type in Delta Kier and now we select the Delta Kier we come here to the right and then you're going to come here where it says color background you're going to select this little eyedropper tool you're going to press it and then you're going to drag towards the screen right here all right so it took away the green uh let's make it one screen so that we can see the whole thing right now it's split into two so we're going to come here and it's going to make it one screen you can move up and down if you scroll up and down and then if you press control you can zoom in and out and then middle Mouse to move it around like this and from here you may think okay we're done but not really cuz if we come here to view mode and we go to mat you might notice it's not completely white so how do we fix that we want to make sure that this is completely white and your mask might also have some white outside here as well so let's go over here to where it says mat and then there's this thing called threshold right here and you can mess around with this threshold right here and right now it's not doing what we want so you have to come to this other side and there you go boom so you see we still have this text here that might cause some problems so there are two things you can do you can try to still mess with this threshold right here until it goes away and hopefully it does not affect your subject in any way so you can do that or or you can come here to color and then we're just going to cut this out so you come here to where it says window and then we're going to select this Square I already lined it up earlier align this so that it's right on top of where the text is at and then you're going to come here you're going to right click and you're going to click on ADD Alpha output you're going to come here to this little blue square and you're going to pull it out and then put it right there in the alpha and so it should be gone and then if you come back to edit it should be gone obviously it all depends depends on whether your subject gets in front of the text or not right so yeah now I have a clean mat and then we can export this by going to delivery naming it save it wherever you want to save it make sure you have all the right formats this default format should work add render to q and then you press render all and then it should render it for you and so now you should have an alpha mat there's one more thing you can do if you want to obviously this is up to you so we have our original clip right all right we put that there and we we come here back to fusion and we come back to view mode and go back to final results and then we come back to edit you should be able to see your animation on top of the footage you might notice that your character is not fully covering the original person in the footage so you can see a little bit out here if you're fine with that then we can just leave it as is and just put it into comfy but if you do want to remove the subject there is a way to do it in D Vinci but you need the studio version to do that there is a free method kind of a quick and easy way to do it and that's through Runway you can create a free account in Runway the only catch is that you can only export in 720p for the free version if you if you buy credits then you can export it at high quality up to 4K this is optional by the way you don't want you don't have to do it this way you can find other methods of doing it if you like to me this is really quick so you go here to edit videos you go to in painting and you bring in your footage I I already had my footage in there and all you really got to do is just come here to adjust your brush size however like I might need it a little bit bigger but you want to cover your subject like this oh shoot so I have to leave it press or else it's going to to the job already so I have do leave press and then I release the mouse and then with only one frame that I do this it's going to imp paint the rest but it doesn't do it perfectly as you will see right now see it's doing a pretty decent job job there so far right here is you see Hands you see a lot of ghosting unfortunately but if you think about it your subject maybe blocking a lot of this ghosting so it might be that it's still very useful for you maybe not this is just an option you when you export it uh if you have the free version then it's going to tell you you to export it 720 and then if you don't care too much about the quality of the background you just really care about the quality of the subject then you can always do it this way and Export it or if you want to buy credits then you can obviously go higher and get a higher quality background but you can export it and when you export it uh you come here to your assets and it's going to tell you that it's exporting and it tells you percentage once it's done you export it put it back into Da Vinci and then you're not going to see the guy in the back as much anymore you might have to mess around with the scale if it's at 720 and your video footage is higher quality then you might have to like it might turn out like this and you might have to just scale it up and again you export it and once you export it then you're ready for comfy UI come to comfy UI load up your mask in here load up your video footage in here make sure you have all your models like once you get this workflow it's going to show you the models that's being used for this if you don't have them just look for them on coty.com or uh just Google search some of these models we oh another thing is uh Laura do really help uh I did test it out with using uh IP adapters and I'll show you the results I got it's really bad footage but I try to make him like the Armored Titan from Attack on Titan and uh it looks really bad the original one but then I ran it through my workflow and it made it way better but this is without anything this is without a Laura and this is without an IP adapter this is with allora oh that the armor Titan this is with uh IP adapter that I used and this is another IP adapter that I used so you kind of get an idea of what is possible when using IP adapters versus um using luras for me luras I feel like work the best but sometimes you might not have a Laura that you want so you might want to try an IP adapter if you're not getting the results that you're looking for as you may notice these workflows do not have control Nets except the control gift control net and people are always looking for this model it's it's kind of confusing to find but I'll I'll include a link in description where you can find that um besides that you should be able to find the rest through manager and if you're familiar with comfy you should already know how to find nodes and stuff so yeah so that's it for today um again I want to give a super special thank you to AI warper um he was super helpful and he was super generous with the information as vgo just keeps improving it's going to make this process even better so yeah all right thank you guys so much for watching make sure you like comment subscribe please and like always take care God bless peace [Music] oh [Music]

Channel: enigmatic_e

Views: 18,730

Keywords: comfyui, animatediff, stable diffusion, ai animation, viggle

Id: SzBfub2Bq7Y

Length: 14min 7sec (847 seconds)

Published: Fri Apr 12 2024