ComfyUI Animations - Easy with ControlNet and TemporalNet

[Music] want to create animations like this in comfy UI stay tuned check out this tutorial this short animation was made in three steps one pre-processing the original video to prepare for comfy UI I rotoscope the main character with runwayml I exported the character with black background and extract the frames with ffmpeg you can use of course DaVinci Resolve Premiere or any other editing software 2. making the AI animation with comfy UI this is what this tutorial is about you will see how a smooth animation can be made for that you only need to set a workflow which combines control net and temperament three editing the video comfy UI will create a sequence of pngs these can be made at MP4 video then it is just mattered to edit the video rotoscoping and using effects and resolve Premiere or Runway [Music] let's get started with comfy UI we start with the template we will start making a regular text to image workflow in this case we will start with the Laura to adjust the model Style [Music] now we will start preparing a workflow batch image to image for that we need to use the batch image load this is a custom node not in the standard installation of comfy UI I leave the link to his nodes in the description below [Music] you will also need to write down your positive and negative prompt for this demo I will use a pre-made prompt already prepared select your model and your Laura we create a test folder in the input folder of the comfy UI installation in this folder we place three or four of the frames to test the workflow works [Music] [Music] when testing there is no need to save the images we place a preview image node only foreign we need to set the seat to a fixed value for this example we will also reduce the CFG to 3. in the batch load image node we point to our test folder when the node is ready we test the workflow and check everything is alright now we will start adding our control net nodes some space is needed so I am slightly reorganizing the layout [Music] for control net we need the node apply control net the positive and negative prompts need to be connected to the node [Music] the outgoing positive and negative are connected to the case sampler this node requires we connect to a control net model which we load to a control net loader in this first control net we are going to use the line art model to apply the control net we need to feed a line art image to do that we need to add the corresponding preprocessor the preprocessors are not installed by default control net auxiliary custom nodes are needed I will leave the link in the description down below but these can be installed using comfy UI manager when we test our workflow we see there is an error we forgot to connect one prompt to the control net when running complex workflows it is good to test every small change to make sure everything works as we want We Run The Prompt again and check that now everything is correct and the Right image is generated for the second control net we are going to use open pose we connect again the positive and negative prompts consecutively next we load the control net open pose as a preprocessor we choose to use in this demo DW pose in my opinion this preprocessor detects better faces and hands than the standard open pose preprocessor we run again that the workflow works well [Music] the line art and open pose control Nets have detected the images correctly and we can also see how the final image is based in the original image for the last control net we will use temporal net this control net is not in the standard Library needs to be installed manually in the control net folder instructions will be found in the description below for temporal net we do not use any preprocessor we just simply connect the starting image to the control net we run a final test to see that all the workflow Works without problems [Music] now that the workflow works fine we will prepare to run the complete batch of images to save the output frames we will add the save image node from was node Suite [Music] we will still use our test frames that the files are saved where we want and with the name we want [Music] foreign we now create a new folder called frames in this folder we are going to copy all the frames from the original video we are ready to start we rename now the input folder to get the frames from the new created frames folder in the same way we rename the output folder where we want the stable diffusion frames be created we run now the whole batch of images in this demo is 167 frames depending on the duration of the video the quality of the image and the rendering settings in your machine the processing will last more or less while the images are being generated you can check them in your output folder in the case that you do not like them you can cancel the queue delete the output folder and refine your settings take into account that when you restart the queue it will start from the last frame you processed if you want to start from the beginning it is better to run first one frame from the test folder and the restart the complete batch foreign s are ready you can use them for post-processing sometimes you may want to manually polish some individual frames with Photoshop or with the video editor you can improve some of the imperfections sometimes it can be useful to rerun some of the frames in comfy UI this really depends on the results of your video but we will not cover it in this tutorial [Music] this has been my tutorial to make animations with comfy UI I hope it has been useful and you have enjoyed the video thanks for watching [Music]

Channel: Koala Nation

Views: 10,605

Keywords: ComfyUI, Stable Diffusion, Animation, AI, Automatic1111, Midjourney, Video editing, RunwayML, Video Animation, Artificial Intelligence, comfyui, stable diffusion, midjourney, tutorial, controlnet, comfyui stable diffusion, runpod, automatic1111, jupyter, jupyter notebook, jupyter lab, google colab, colab stable diffusion, confyui animation, comfyui nodes, comfyui tutorial, SD, SDXL, SD tutorial, SDXL tutorial

Id: Spc_F57FmK8

Length: 7min 59sec (479 seconds)

Published: Tue Sep 19 2023