Mastering ComfyUI: Getting Started with Video to Video!

um hello everyone and welcome back to Dreaming AI my name is n and today we are going to learn how to use video to video with comfy UI uh I've been searching for a straightforward way to perform video to video with comfy UI for quite some time now I found some custom nodes in the wuite that I couldn't quite Master them and even the lack of a preview that made the experience less comfy why like made me opt for a different solution so I de decided to create some simple custom nodes that allow those who want to delve into this field to focus Less on how to decompose and recompose videos and more on a result uh let me start by saying that I'm not very experienced in videoo video techniques and I've tried to find the best method to help you achieve a decent result while aiming to inspire you to do even better the custom nodes I've written can be downloaded along with my Suite Link in the description for those who have the previous version simply follow the steps to update so let's launch comfy UI and immediately look for the first node load video uh this node aims to be the video version of the standard load image with some additional functions that I'll explain now first of all to load a video you can simply drag and drop it onto the node in the top part or use the dedicated button this way the video will be loaded into comfy UI and a preview in a loop Boer it's important to note that it only accepts MP4 videos as input I made this choice because there was no point in struggling to make the node support more formats when there are applications like file converter Link in the description that can convert even gifs into this format with a simple Windows context menu and now let to delve into the details of the node the node consists of the F in settings the field where you can choose between the current video and previously loaded videos just like load image and then there's the local URL field which provides the The Comfy UI URL to access the video and this Feld this for visualization purposes Omni and doesn't actually serve you next is the frame rate field which can be modified to original half and quarter by changing this field we can reduce the frame rate to half or a quarter of the original video obviously uh reducing processing times especially if you want a rough preview of the entire video I'm also convinced that in the future by reducing the frame rate and then using other AI based methods to re interpolate the frames to the original frame rate you can achieve greater coherence between the different scenes in the video one of these techniques has been applied in a new node that I'll explain later the next field is resize by which can have values none height or width you'll use this field only if you want to resize videos that are too large or too small for you choosing which dimension to modify vertical or horizontal um the video's proportion will always be maintained the size field will tell you what value to apply the selected dimension in resize by the last two configurable fields are images limit and batch size the first one tells comfy UI how many frames to use for the video um I created this field to allow us to run all tests on a few frames to make the work faster once your satisfy Victor results you can finally render the entire video by setting the field to zero the second and final field identifies how many batches the rendering work will be divided into within comfy UI as much vrm as your graphics card may have even a 30-second video with decent Dimensions is impossible to work with without setting a batch because of the vram required I suggest starting with batches of 15 20 units to figure out which value works best for you now let's talk about the outputs of this node Imaging this is the most important output as it contains our video divided into fres ENT ladent I included this as a replacement for a latent empty image because I didn't want to manually set the height WID and batch size every time uh this output already contains all the correct settings metad data this contains data to provide to the video saver node to make it work correctly width and he these are as you might guess the width and height of the final video in case you want to input them into other nodes that require these perimeters uh the second node I've created is the output node which I've named save video it's equivalent to save image and is responsible for combining all the frames coming from the sampler or any source that returns a list of images and merging them while maintaining the frame rate Chosen and load video there are three available options save video this allows you to save the video in the comfy UI folder output and sweep videos save frame this saves all the frames of the video in PNG format in the same folder as the video lastly there's the compression mother option which lets you choose the quality of the PNG frames and consequently the video quality higher compression means lower image quality next I created a node called load frames from folder which given a path to a folder full of sequentially numbered frame images creates the image flow needed for video saver to recreate the video there's a node in the was Suite that seems to do the same thing but since I couldn't get it to work I recreated this one the final node in this series is the frame interpolator which as I mentioned earlier can help create video frames using AI um for example if you have a video running at 10 frames per second and set a multiplier to three the output video thanks to AI magic and a technique called rifle will reach 30 frames per second uh this node takes a batch of images and metadata as input which are used to calculate the output in the output we have practically the same Fields but modified based on the processing caused by the multi mullier all right um now that I've introduced you to these new nodes as promised let's create a workflow that uses them I decided to develop one of the most basic techniques for a simple video toide transformation together with you as I mentioned before my aim is to inspire you rather than do something very complex and perfect as I'm not capable of it not having been very involved in this practice so far but with these not modes I can finally start practicing um first I downloaded this GIF from Min so let's convert it to MP4 and upload it since it's a gif converted to MP4 it has very few frames and it's also small in terms of resolution so I'll keep the original frame rate and won't use batches for the resize I want the whiff to be set to 512 pixels which is the resolution where most stable diffusion models work well to speed up testing I'll set the images limit to two let's also select the model that will perform in painting um some models do it better than others and since we'll be doing you know in painting on on dozens if not hundreds of images it's better to choose the right one um I'll use revi animated which you can find on civid AI for this model it's recommended to use an external Dae let's load the two props and combine them as usual for the body position I want to use control net I'll explain the details of control net in another video so if you're not familiar with it know that it's an incredible tool that allows us to take control of what will be generated using submodels in this example I'll use the KY model which you can download from hubbing Hub along with its CA file placing it in the ki/ model slol n folder will make it detectable by kuui I'll leave the link in the description now let's load the control net model with the load control net model node next let's pass the resulting images from our video loader through a dedicated pre-processor uh for our control net model called canny Edge pre-processor taken from comfy wise control net auxiliary pre-processors I all the final node of the control net Network called the ply control net Advanced this node takes his input the Kenny model the images coming out of the pre-processor and the two props that will be used used as output for applying the control nut result now that we've got this far let's also load our K sampler Advanced as I want to select the steps from which it should start applying the in painting we'll connect the model chosen at the beginning our VA encoder um into which we will input the vae and the images coming from the load video and our two prompts coming from control met great now attic Cas sampar output will connect the VA decode to which will attach the same vae as before in the output will connect the frame interpolator which will take the metadata from load video and finally what's connect the save video and the frame interpolator to each other excellent our structure is ready now all that's left is to set the parameters that have resulted from the countless tests I conducted while preparing this video uh in the cany edge let's set the parameters needed for it to detect the image correctly let's also add a preview so you can see the result in apply control net we'll set the parameters that determine how much impact the modification will have on the final image the same goes for K sampler where we also use a feature that kuui custom scripts ads which is the ability to select the steps and doo value in this way perfect everything is ready for our video to video transformation let's start the operation in weight as expected we have only two frames as requested and as you can see Controla has done its job very well in detecting the drawing inside our image um I could have used the open post model as well but for this example Cy works just fine the two images seem quite consistent with each other as I mentioned before um I'm not aiming for Perfection okay now let's runder the entire video by setting images limit to zero here's the complete result I think it's not bad considering how simple the technique we just used is of course there will be specific parameters and a lot of experimentation for each video but I hope this doesn't discourage you and that's all for today I hope you like the nodes I've created and that they inspire you to create incredible Creations please consider liking and subscribing if you found this tutorial useful also if you have any questions please let me know in the comments below I'll be happy to help you out as much as I can and as always keep dreaming

Channel: DreamingAI

Views: 24,465

Keywords: video2video, inpaint, videotovideo, inpaining, rifle, revanimated, ComfyUI, programming, python, advanced, text generation, AI, stable diffusion, artificial intellingence, dreamingai, ai news, best free ai, best ai model, dreamingai tutorials

Id: nMyiuiWjiEc

Length: 13min 52sec (832 seconds)

Published: Fri Oct 06 2023