Make INSANE AI videos: FULL Workflow

do you want to make dope AI video animations like [Music] this so this is YouTube what up what up I am Tyler better known as JBS or as most people on Instagram know me as jbooks docreative I am a freelance creative director videographer and editor and for the past 8 months I have been obsessed with all things AI video and animation and this is my YouTube channel in this video we're going to dive into animate diff and comfy UI if you don't know what that is no worries it's all good I have a couple really great resources that I'm going to recommend if you are a complete beginner this video is going to be if you are more intermediate and advance or at least a little familiar with anime diff and how to navigate comfy UI now if you are a total beginner and you have no idea what anime diff or comfy UI actually are the links in the description below are going to take you to two places one is going to be a really great article written by Inner Reflections Ai and that article will explain how to get started with a very barebones basic videoto video workflow that my workflow was built on so once you go there and you understand that come back here the second link is going to be to my good friend enigmatic E YouTube channel where he has a video tutorial on how to get comfy UI installed and running on your machine go there do that come back to this video now between both of these resources it should be more than enough to get you up and running and at least feeling a lot less oh about comfy UI because let me tell you when I started doing this and working in this specific program a couple months ago I had never worked with a node-based workflow before and man it stressing me out every time I would open up comfy and look at the node graph I got anxiety so I I'm right there with you if that's how you're feeling however inner Reflections article and edig Matic E YouTube channel really helped me out those two resources put them together you guys will be up and going in no time and you'll be creating videos but enough talking let's hop into the workflow to download this workflow that we're about to walk through you're going to have to go to citi.com and you're going to have to create an account you cannot download the workflow unless you have an account on the website this is going to be the page where you download it from Once you create an account everything is free there's no there's no catch of any of this you're just going to go ahead and click this download button make sure you're on the most current version at the time that this video is posted this will say version 3.0 and in when you download it it's going to be a PNG file all you're going to have to do is open up comfy UI and actually drag the PNG file that you downloaded into your browser window and then all of this will load it's not a Json file it is a PNG file the PNG file is the workflow now the first time you load this you will probably have a bunch of red nodes especially if if you're used to comfy and it's going to look super scary and it's going to tell you there's a bunch of Errors don't worry go install the comfy UI manager follow the links I put down below learn how to install it install the comfy UI manager once you have your comfy UI manager you're going to click this manager button on the right hand side on your comfy UI menu and you're going to click install missing custom nodes you click that and it will literally download and install every everything you need to run this with the exception of the image save node and to download and install the image save node you're going to click install custom nodes and you're going to search for w and you're going to install the was node Suite you're going to want to get to at least beginner level and get this stuff installed using the links in the description before you attempt my workflow disclaimer back back to the video all right everyone so this is the JBS creative and machine learner vidto vid animate diff workflow version 3 I'm very excited about what this has evolved into for everything AI video and animation especially the stuff that I do with the dance videos this workflow can do almost anything that I've been able to think of not only does it do video to video it can also do text to video it can do the QR code image where you're basically feeding it a black and white Alpha mask and it creates the image in the shape of whatever is white in that Alpha mask you have a four image IP adapter so you can use up to four reference images to influence your animation you have the prompt scheduler so you can do multiple prompts and travel them and key frame them you have up to five lauras you have all your control Nets in your low resolution version it has an upscaler that then goes into another control net pass so that when your video upscales to the higher resolution since it's running through another diffusion process we are not losing detail and your animation doesn't get out of control because you're still applying the control Nets to it if you don't know what any of that or a portion of that means no worries we're going to go through this block by block right now so just a little bit of housekeeping before we dive into this like I said if you're complete beginner please go visit the resources that I have listed in the description both the article written by Inner Reflections AI as well as the how to set up comfy UI and install the comfy UI manager by enigmatic e if your comfy UI does not look like this with the straight lines but you would like it to well that I can show you right now so if you look at your little comfy UI menu that will be stuck to your screen over here on the right and you click the gear icon this is going to bring up your options depending on what nodes you have installed from your comfy manager your option list might be different than mine that being said we all have this thing link render mode mine right now are set to straight if we set them to spline this may look a little bit more like what you are used to seeing in comfy UI and it looks like our no tree was just attacked by the Spaghetti Monster now even though looking at this stresses me out and gives me extreme anxiety when it comes to hooking up nodes and reconfiguring and moving things around this is actually more ideal cuz you can see very clearly where each individual strand is going to when I am just in work mode and I'm not moving nodes around and I am just making a video or an animation I will always keep this on straight this is this gives me the nice clean pipeline it is a pleasure to to work in I love it linear some people like this I don't really care for it it's like a cleaned up version of the the splines um I would actually prefer this over the splines I think but again the splines are good just for hooking stuff up but when it comes time to work and you're just zeroed in and trying to dial in an animation do yourself a favor and set that to straight so you too can have a comfy UI that is as clean as your bedroom hopefully all right y'all so starting at the very beginning here box number one if you notice every group here is numbered now you can do stuff in whatever order you want but I did want to make this a little bit more approachable for people that get stressed out using comfy UI the way that I did in the beginning so box number one your low reses inputs AKA your base video so this upscale image node right here you don't got to touch or change nothing about that the width and the height are the resolution that you would like this video to come out to now if you really want to you could run this in 720p I don't like to do that I always run my animations in a 512 resolution for the low resolution or the first pass most of what I make is specifically Geared for Instagram and social media so I'm working in a 9x6 vertical format which almost everyone is now my favorite resolution is 5 12 by 8.96 so this will be the Default Resolution when you download the workflow now the next box the load video this is where you are going to choose the video you would like to upload simply click choose video to upload and then browse your folders and figure out what video you want to upload um the video does have to be under 100 megabytes keep that in mind sometimes my videos are larger because I exports everything in really high quality I have to go back and Export it at just a little lower which doesn't really affect anything but you just need to get it under 100 megabytes which for most people should not be an issue whatsoever also in this box the force rate you shouldn't have to touch the force size leave that undisabled the frame load cap this is important this is how many frames you actually want animate diff to render at the end of your animation if you leave this on zero and you put in a 30se second long video it is going to render the entire 30se second long video if you change this to 50 it is only going to spit out the first 50 frames now best practice before I run an entire video I always do a 50 frame test why 50 why not 20 50 frames on my 4090 PC takes me roughly three and a half to 5 minutes to render 50 frames is enough movement in the test to tell me if my control net settings and the model are good and the animation looks good I don't think I've ever gotten a good 50 frame test rendered the whole animation out and then the the animation was bad the 50 frames has always told me what I needed so that is frame load cap AKA how many frames of an animation you want the video to render skip first frames if you want to test a different part of the video well what I would do normally is open up the video in Premiere and look at it by frame count and see like what frame number is what part and if I there's a certain part of the video I want to test maybe the movement is really complicated or whatever and I want to test you know um frame 200 to 200 50 I would put skip the first 200 frames it'll start on frame 2011 and a frame load cap of 50 so it's going to render out 50 frames from the 21st frame of the video make sense I think makes sense now select every in this is how many frames do you want it to continuously render one means it is going to render every single individual frame of the video if you change select every in to two then it's going to render every second frame of the video and so on and so forth if you're doing stuff like dance it is best I mean just animation in general it is always a best practice to render every frame if your GPU and your computer can handle it um it's going to give you the most accurate real to motion video in your animation that you can get it's going to show you a video of your a preview of your video don't mind me this was this this video I made was stupid you'll see at the end of it but this is just me flexing after working out underneath that the last two in the load video box and our number one is our load checkpoint this is going to be connected to the checkpoint folder in your comfy UI directory this is where you store your models and your checkpoints that you would download from a place like civii this is the list of models that I have downloaded that I use use for this video in particular we use Boton V1 you shouldn't have to change anything else in this box and your load V node you're going to you're going to need to put download and put a v in the v folder directory of your comy UI folder the V is essentially like the contrast color correction like it just makes the images better but you need to have a v loaded or nothing will work that is group one your low res and base video in put all right group 2A your animate diff nodes these are very important because this is what is making your animation come to life your anime diff loader this is the important one this is where you're going to select your anime diff motion module now I would just suggest right off the bat going with animate diff V3 this just came out um right before Christmas it gives you incredibly smooth an animation and a lot of detail in the animations so in everything you see from me now you're seeing anime diff V3 before that I was using temporal diff or the motion module V2 um your uniform context options not going to lie don't know 100% what these do but I know that I've never had to change the numbers and they work for me just fine just the way they are the anime diff Laura loaders um these are your motion Lura so if you want to motivate a zoom in motion or a zoom out motion or a rolling clockwise or rolling counterclockwise pan up pan right pan left pan down if you want those type of motions in the animation you're going to select those and you're going to give them a strength a strength of one is insane a strength of 05 pretty good place to start if I use them I usually start at like 2 to 35 but I very rarely actually use them even though these say V2 these actually work and are compatible with animate diff V3 so have no fear this is your animate diff nodes for almost every video that I do I don't have to touch these I don't mess with them unless I'm doing something very specific in my animation all right guys box 2B The Prompt one of the most important parts of this whole thing so we're not going to go over um prompt traveling too much here we're not really going to talk about prompt scheduling I'm just going to give you a few notes to make sure that nothing in this box actually breaks when you go to run your animation so the green box the batch prompt schedule this is where the meat of your prompt is going to go the syntax here is extremely important I cannot stress that enough the syntax means the way that it is typed out and written in the box so notice quotations frame number quotations semicolon quotations prompt quotations comma it is the same no matter how many prompts you have in there but whatever the last prompt is in your batch prompt scheduler you need to erase the comma at the end of it if you accidentally leave a comma there you're going to get an error when you go to run your animation so that is super important while we're not going to jump into prompt traveling right now or key framing the number in the quotations here is the frame of the video that this prompt will take place because we are using a batch prompt scheduler you need to have at least two prompts with two key frames in here for this whole thing to run and the animation to go through zero I just put frame 400 just cuz 400 is a random number and I copied and pasted the same prompt made sure to erase the comma at the end of the last prompt to the left of it we have our pretext node the pretext means that whatever is in here is going to be placed in front of whatever your prompt is I have sharp in quotations along with consistent color scheme that means before all of this the meat and potatoes of my prompt is going to be sharp consistent color scheme and then shirtless the strongest man in the universe blah blah blah the app text you could probably guess it these are words that are coming at the end after whatever your prompt and your batch prompt scheduler is so these two nodes you're going to put in all your descriptors or just like the little detailed things highly texturized Masterpiece best quality detailed all that stuff that goes in these two and then the batch prompt scheduler is only the specifics of your prompt now the red box is your negative prompt this is where you would put put any of your embeddings or textural inversions if you are using any embeddings or textural inversions you have to use the syntax that I have here with embedding semicolon and then the name of that embedding file if you don't know what those are we'll talk about those in a future video and then after those I have just my regular negative prompt blurry nipples NSFW nude naked simplified low resolution just things you don't want to see in your animation we are going to come back and talk about the IP adapter right after we talk about luras all right so box number three the luras I have five Laura nodes in here and anything purple that you see means that I have those nodes bypassed it just means that everything is going to travel right through them and Skip them that's because right now on this specific video I only have one Laura activated and that is this retro anime Laura my best practice for Laura so I don't get too lost in the weeds with trying to figure stuff out is I leave the strength clip value at zero and I put the strength model value at 0.35 just having to slide this value around gives me a result much similar to when I use Laura in just generating images with automatic 1111 or with how that how we used to use um lauras in warp fusion if you're familiar with that um if I wanted to use a second Laura all I would do to un bypass this is rightclick the node and click bypass again and it activates I want to turn that back off so right click and bypass don't have to worry about it you have five luras that you can put in your animation now the IP adapter this one can be a little difficult for people to get set up sometimes I'm going to walk you through it hopefully you can get it so the IP adapter allows you to use up to the four reference images to inject those reference images into your animation and it is incredibly powerful for getting your animation to actually be exactly what you want it to be this particular video in this example I was trying to turn myself into a combination of gigachad and a very angry Super Saiyan Goku because why not right so um even though you can use up to four Images if I'm only using two images because we don't Bypass or turn any of these off I actually just put the same images twice so we have two Giga Chads and we have two Goku the nodes that these are connected to these prepare image for clip Vision nodes these are cropping your IP adapter images so notice you can select top bottom left right center or pad top will make the IP adapter crop into the top part of the photo the center part of the photo the bottom part of the photo we don't want that and or the left or the right pad will take the entire image um I placed these preview image nodes to the side here so that once you hit Q prompt and it runs through your IP adapter it will show you exactly what the crop looks like so you can see what part of the image you're pulling the style from that way if you see and you're like oh snap it's only pulling his head but you actually want the body you can cancel your queue and then you can change your crop position and get that how you want now um I never I i' I've I've never personally used the sharpening at this point so I always leave that at zero the encode IP adapter image all of these are hooked up into this these weights stay at one I don't mess with them the less variables I have to mess with the better if this IP adapter plus thing is changed to false everything will break it's not going to run make sure that is turned to true and that runs up into your apply IP adapter from encoded node this one is important this is where you're selecting the actual weight and strength of the IP adapter that's getting injected into your animation I personally like starting with a weight of 7 I've went as low as a weight of. 3 because my image felt very oversaturated and kind of burnt in these numbers are not set in stone you're going to have to play with this for yourself and figure out what works best for you with what you're trying to accomplish I personally always leave it ending at the full diffusion that's what that does so I don't touch any of this it is connected to these two smaller nodes that are going to be collapsed when you open open up the workflow the load IP adapter model you're going to want to make sure that you have the IP adapter plus stable diffusion 1.5 model in there if that is not in there it's not going to work if you're using um stable defusion XL and hotot XL then you're going to need the sdxl model I personally like 1.5 that's what I'm using there we can collapse that you can get that from the IP adapter GitHub page and then we have this load clip Vision this one is a little difficult and gave me a little trouble because for some reason when they uploaded this model to hugging face they literally named it model. bin so um this clip Vision model needs to go in the clip Vision folder of your comfy UI directory if this is not in there it will break it will not run in the description of the video there will be a link to a Google drive with that specific clip vision model that you need to run this workflow in it go there and download it at the very top of our node tree we have our control net group so you will see that I have created a different group for each individual control net each one of these groups has the control net model loader where you will select the control net you just want to use the one that the group is named after to keep everything consistent and clean you have your net pre-processor which for something like open pose has the specific options and settings for that control net like detect hands detect body detect face and then you have the control net stacker where you actually choose the strength of the control net as well as what part of the diffusion process or of the steps you would like the control net to end at for most of my control Nets the most I let them run to is is 85% of the steps or the diffusion process with animate diff you do not need to run your control Nets very high to get incredible results a matter of fact if you run them too high they're actually going to restrict your animation and make it a lot less creative and a lot stiffer so for this particular video because I'm not doing anything overly complicated and I'm just doing very simple poses with my body I used open pose and line art I use open pose to get my character figure red and then I use line art to get little details about stuff and to kind of glue everything together a little bit more um and yeah same thing as the Laura whatever control Nets you do not want to use right click the group bypass group nodes that turns them off whatever control Nets you do want to use rightclick the group and set group nodes to all ways that is different than the luras which are individual nodes since you are bypassing and enabling a whole group it is bypass group nodes and set group nodes to always those are your control Nets just a side note that for the QR code monster and control GIF control Nets there are no pre-processor nodes for those control Nets they are not needed so that box only has the control net model select as well as the control net stacker where you would select the strength and whatnot um I put those two in the same one because they both don't need the pre-processor and I very rarely use them together now moving from our control Nets down to our number five group The sampler or the K sampler so this is where you're going to find your steps AKA how many iterations of the diffusion process do you want animate diff to go through before it actually gives you your animation your CF G which is basically kind of like how much do you want this whole thing to adhere to your prompt higher does not always mean better I personally like using 30 steps with a CFG of 10 sometimes I lower that down to seven this is where you select your sampler I personally like the DPM p2m sde I Can't Tell You Why all I can tell you is that it looks better this this is where you select your scheduler I personally like um Caris or caras or whatever however that's pronounced I think it gives a lot more detail and the animations come out nice crisp and contrasted and then you have your denoy strength which is how strong do you really want this I like running my animations really high because I like my animations to be animations so I run it at one um if you go too low on your D noise or your steps or your CFG in anime diff you will get blurry outputs so this is basically what I run all my video videos with and sometimes I change the CFG to to um seven so that is your sampler this primitive node right here is your seed um by default I have this changed to fixed so the seed is always the same so I can reiterate over um like if I'm as I'm tweaking my animation and my video to get what I want it's working off the same seed which means I can actually see the changes um iteratively and not get like a full random video every time you can change this to randomize however and every time you generate boom you'll get something completely different based on your prompt so that is your number five box your sampler all right now the case sampler runs into our number 5A group The reactor face swap now this is if you have a video that you are doing where um your subject is looking at the camera most most at a time and it's fairly stable and you would like to put the subject's actual face or like your face or just a face a specific face onto the subject well this is the reactor face swap node so here we have a face restore model that we have connecting to it which is also going to um help clean it up but the face swap node right here uh these are the default settings in the in the node and that will be in the workflow so I didn't change any of that the load image where you load the picture that you would like to put onto your animation hopefully on the face this works really great sometimes if there's a whole lot of movement then sometimes it just like breaks everything this face model I believe came with this node when I downloaded it as well so as long as you install these nodes everything should be there for you and you can see that my face for my selfie right here which we're not going to talk about that it actually really got put on this an on this gigachad Super Saiyan animation of me really well before we get to the outputs though let's talk about these buttons real quick the bypass and enable face swap you are not always going to use the face swapper I have these easy buttons in some of the larger groups so that you don't always have to right click and bypass um these buttons are linked to these two little nodes right here and it's super simple this this one is for bypass this one's for enable click the action button it bypasses the entire group click the action button it turns the entire group on easy so let's move on to number six our low reses outputs so this is where the first version of your video is going to get place this is the 512 by 896 resolution that we put in in the very beginning um I always work in a frame rate of 30 I know this I always work in a frame rate of 30 because the native frame rate for Instagram is 30 FPS so I try to keep everything as close to the native for social media as possible that way it doesn't have to do any compression or alter anything because the platform's compressions are terrible so frame rate of 30 this format is h264 or an mp4 and there's nothing I ever really change in here the only thing you would have to change is the frame rate if you want to do you know a frame rate of like 24 for more realistic motion or just less frames but here is your Lowes output this is what I call the the um preview or the test gallery or the right now it looks more like the peanut gallery but when I um am running my test and I'm trying to figure out what I want to what I actually want want my final animation to look like every time I get an output here I am holding down alt and dragging and it creates a copy and I literally just drag copies into this group right here so I can compare all the things I've gotten and since we're here just so you can see here's what the animation looked like prior to me putting the face swap on it it just looked like you know gigachad with super staying hair but once I actually put the face swap on it um man it it it really put my face on there I mean I think I think that looks pretty good right no no no okay anyways let's move on to the upscaler so this upscaler is going to be the settings in your upscaler are going to be very individual for your video I can tell you though that these are the settings that I use on most of my upscales the upscale method is set to bilinear scale by 1.5 I'm using another 30 steps this time with a CFG of only seven with a d noise of 7even not one like above 7 the same sampler except my scheduler is exponential now this is going into 7A which are your upscale inputs and these with height boxes are what resolution you want the upscaled video to come out to so I like getting this at a resolution of 768 by 1344 and then after I get my final video I will go and I will upscale it in either topaz or run it through another process in automatic 1111 which I will detail in my post-process video that I will make in the next week or two so future video on the postprocess and upscaling but this is the upscaled video you will get that leads into another face swapper if you are using the face swapper same thing same everything don't change anything just same picture same values same everything and from underneath that we go into the second round of control Nets now rule of thumb use the exact same control Nets that you used above and you're either going to want to use the exact same values you used in your first pass or lessen the values by like 0.1 in the strength and it's also very important that your resolution in these pre-processors says the resolution that you're running the upscale in so 768 otherwise it's going to come out all kinds of messed up if you're running these at a lower resolution but you're getting an upscaled video so again for these um I didn't want to put individual buttons for all of them you can just right click and set the group nodes to always or right click and bypass the groups that you are not using at the most I only ever use one to two control Nets so I only ever have one to two control net groups active next to that you will find the bypass and enable buttons for the actual upscaler in general so when I am running my 50 frame test my entire control net group looks like this it's all purple I don't want to I don't run an upscale test until I have the first 50 frames that I'm happy with I'm like okay I think I want to run this now let me test the upscale and then I'll do 50 frames on the upscale but yeah your final um your final group here after you after you run everything you're going to get your upscaled video I have it set to prores this will not give you a video output in the notes you'll have to go into your output folder to see the move file it will save as and there is an image save node here and this image save node here has some code in it that will place it in a nice little folder neatly in your output folder shout out to pixel Pusher for helping me with that one um some people have errors with the image save node even after they get all of the other node groups installed when they run this workflow the image save node in your comfy UI manager if you go to install custom nodes you want to type in w in the search bar search and it is the was node Suite or was node Suite however you pronounce it that is you will probably have to manually install that one and then once you install that and restart art com for UI and boot it back up this image save node will work so that is the fix for the image save node and I think that is the workflow that should assuming you have base level comfy UI knowledge that should get you up and running and help you understand what you're looking at here a little bit more you can always come to the twitch streams every Thursday at 3:00 p.m. Pacific that I do on twitch.tv/ civii because I work for coty.com and every Thursday we create videos in this and I walk through projects and I answer questions so you can come get help from me directly in the twitch stream and that's the workflow I'm going to send this to the sign off right now if you stuck around this long I sincerely hope that this video was helpful this is going to be the first of many hopefully these videos can continue to get better please like And subscribe if you guys want more about videography editing and AI video and animation I am Jay boogs peace

Channel: jboogx.creative

Views: 17,244

Keywords: jboogx.creeative, animatediff, animatediff tutorial, ai video tutorial, ai dance videos, how to make ai videos, comfyui, comfy workflow, comfyui tutorial, vid2vid

Id: OVOb2qFT5U8

Length: 38min 14sec (2294 seconds)

Published: Mon Jan 08 2024