Txt2Vid Made Easy with ComfyUI & AnimateDiff

one of the really neat ways to use stable diffusion models is to transform textual prompts to short form videos this tutorial will show you the basic steps you need to do within comfy UI using animate diff along with prom scheduling let's take a look uh first and foremost you want to make sure that you have comy UI manager installed onto your machine I'll have a link in the description to that repository with all the instructions to install that but once that's set up you're going to want ahead and install three custom nodes so first you're going to want to go ahead and install the um Fizz nodes which allows for the batch prompt scheduling uh then you're going to want to install the animate diff evolved uh which will allow you to create these uh motions and then lastly you'll want to install the video helper Suite uh which combines all the images um and then creates the animation so so let's go ahead and close this and then start with the workflow here so I'm going to go ahead and load the uh control net model or I'm sorry the checkpoint and then we'll go ahead and and load the vae uh loaders load vae all right so then let's go ahead and add the animate diff we're going to go ahead and use the context along with the motion uh modules so let's go ahead and start taking a look at the so I'm just using the dream shaper model I'm going to use a staple diff Fusion 1.5 VA here uh some models have VA baked in you can just use that as well too so I'm going to go ahead and connect this to animate diff so uh we'll uh that will impact the outputs here the context we're going to go ahead and add that here you can change the context length uh to uh basically how much from my understanding how much of all the total images to take into account whenever it generates the next image um the motion modules there's lots of these available online I'm going to use the latest V3 as stable diffusion 1.5 motion module and leave uh these others as default I don't know if this has an impact on V3 or not um but yes so we'll go ahead and use all those and so I'm going to do the batch uh prompt schedule so that will be this one here this is by the phys nodes Uh custom node here and this is going to be connected to the clip and then we're so this will basically act as our positive prompt and then I'm going to go ahead and use just a basic clip text and code for the negative prompt okay and then this will go into a k sampler um so let's go ahead and get that loaded so we'll have our positive our negative and then for the model we're instead of going to that dream shaper model it's now passed through animate diff and now we're going to use this modified model here okay so now we're going to go ahead and add a note and I believe this is a uh animate diff node actually and it is a um noise layers now motion it's the big batch there we go empty latent image big batch so we're going to connect that to the latent space and then once it gets out of the Laten space it's going to go to a vae and decode it and so we're going to use this vae that we pulled in at the start there and so I like to always have a preview image just to see what's going on uh you know just kind of alongside it's just interesting to see it and then we're going to basically Stitch everything together here with the video combine okay so then that's going to go right into there um and then we'll go over these options in just a second so we have the basic skeleton here so let me just go ahead and uh paste in some of my notes here so I have the schedule and basically these are the key frames that we're going to be doing so I'm going to do a 45 uh key frames uh 45 images rather uh for this entire animation uh so at frame zero it will be this and at frame 30 it will be this and then I'm not going to Define 45 but basically it should just continue doing this and then loop back over so there's this prepend and apend uh Parts here of this batch prompt schedule so basically if we take a look at this uh this is everything that'll be added before the prompt here and then appends of course anything afterwards so let me go ahead and get the prepended text and I'm not going to use the append text at all but very important here is commas I can't tell you how many times I screwed up on the commas on such uh thing uh but basically uh you're going to want to make sure if you're prepending is that there's either a comma right at the end or a comma right at the start either or uh otherwise if you don't have a comma it will be natural lighting foggy forest all one um to or one section here um and then same with the appen so I was playing with the appen so I always made try put commas at the end uh but since we are not going to be appending anything for this one um you can just go ahead and uh make sure that you drop the commas within the quotes off uh this is like a Json format so you do want to make sure that you have a comma at the end of each line um with exception to the last line here so this will be pre prepended to uh key frames uh 0 through 30 uh and so on so we going to want to go ahead and do that uh the start frame will be zero and now as far as Max frames you can Define it here and that's okay to do and you can also Define that here too but instead of trying to keep track of values across a couple different nodes it's a lot easier to just actually Create A Primitive node where you hardcode the value say at the start and then connect it to both of these throughout so to to do that I'm going to go ahead and put Max frames as an input and I'm going to change this batch size to an input so what I'm going to do now is create a new primitive node and then I'm going to just go ahead and connect that to both of these so there we go so I want to say I'm going to go ahead and create 45 frames so now the batch prompt schedule will know this as well as the Laten space whenever it goes uh to generate those images will know that as well too now next up U for the negative prompt since I'm using dream shaper uh it works well with um both of these embeddings uh bad dream and fast negative V2 uh now if you're wondering why I didn't include the negative prompt in the batch prompt schedule uh it looked like there might have been some sort of issue on the GitHub repository I could be off on that but it was kind of a little hard to follow but basically it would be something to the effect of like a negative and then I believe there needs to be a line break uh and then with the terms but I saw some issues in the console whenever I was using embedding so I just like to do the negative prompt uh separately here so we're going to just use this as positive prompt and then this will be as the negative prompt here okay so let's go ahead and take a look here and and everything else looks like it is good to go for the moment and again this is just a basic setup um that you can definitely play with prompt weights and I want to do that in a future video here but this should be able to generate a basic uh animation and then for the video combine I'm going to go ahead and do a h264 um so it's MP4 it's will look a little bit nicer than just a gif I'll leave the frame count at8 we can play with that uh but I'm just going to go ahead and cue this prompt now and let it go through its process so I'll go ahead and fast forward in the video uh but it will take you know a couple of minutes to process this I'm using a RTX 360 is a graphics card uh for the 45 frames oh I actually just forgot to do one other thing here so let me go ahead and cancel this and I'm going actually change the sampler uh I believe the dream shape works better with the DPM PP 2m I believe it was and then the Caris uh for the scheduler so we'll go ahead and uh set that and also one other thing I forgot to add is we also need to hardcode the seed so let me go ahead and do the Primitive node seed and then I'm just going to go ahead and put in a random seed value so this keeps the seed value fixed for all the uh images that are generated so therefore they all look consistent and starting out with the uh the same noise uh whenever they are being generated okay so now I think we're good all right I'm going to go ahead and cue The Prompt here let it go through its process then fast forward here uh to whenever it is going through the decoding process okay so that just finished it is now being decoded uh by the vae so we should see the preview images as well as the video popup here so let's take a look so very cool so we got our Mountain scene and here's all of our preview images uh kind of coming together here into a pyramid uh with snow so very awesome uh so there's a few things here that are going on um so first up is if you wanted to nice see I did a mistake there I meant to actually do the H2 where I don't know if there's a big difference but we can change the format uh we can also play with the frame rate without having to go through the whole process this is really like one of the cool things I love about comfy UI is that like you can make these iterative changes without having to redo everything because like if you don't do anything earlier in the prompt um or any of the other nodes it won't process them all again so here's kind of a look uh at a slightly quicker frame rate and then we can even do it a little bit quicker until we get to the right Aesthetics we like and it's is saving each of these files onto our uh disk because we have the saved output set to True awesome so that's a basic setup of uh text the video uh with comfy UI now I just wanted to talk really quickly about uh interpolation so I added another custom node and I was playing with this earlier and it's the comfy UI frame interpolation and so I added this film vfi so here's kind of so interpolation is kind of like the blurring of two frames it will make things look a little bit smoother basically uh so I'm going to go ahead and connect that to the frames and then go ahead and create another video combine video combine rather uh and then just connect that directly to here so we're going to make two outputs again it doesn't have to process everything over again but this time I'm going to go ahead and use the um and I'm going to use actually identical settings between these two so you can kind of see what the difference is whenever it passes through this process all right so that's going okay so here are two identical animations and you can see how this one seems like way smoother um compared to this one one it's not that this one's not bad but you see some jitteriness um but it seems like a little bit smoother uh as it goes through its process so that's that's the really uh cool things that you can also do to kind of smooth it out if you will if you find that your animations are looking a little bit too jittery uh we'll also be going over like you incorporating control net into this as well there's lots of really that you can unpack on this um but this is just really scratching the surface but I just wanted to share that another thing that you could do too is that since this is a 512 x 512 it's not the highest of equality uh but you can pass it through a Laten scale uh or and upscale it through the Laten space uh basically sample it again or you can just use a a straight up upscaler uh and make it larger as well too so uh all that you know is things that you can kind of patch together to create this so very cool um thank you so much for watching if you have any questions though about this um you know please drop a line in the comment section below I'd love to you know try to help you out on this again the workflow is included in the description so you can go ahead and download this and load it onto your own machine um if you like the video please consider hitting that thumbs up and also uh if you could subscribe to the channel thank you so much for watching I'll see you in the next one take care

Channel: Prompting Pixels

Views: 2,507

Keywords: stable diffusion

Id: KIOmQr8_BPE

Length: 14min 28sec (868 seconds)

Published: Fri Jan 26 2024