The coolest Vid2Vid with RAVE and AnimateDiff version 3. New technique for AI videos with ComfyUI

today I am going to show you how to create cool AI videos starting from any clip you have combining Rave with anime diff you can take any of your videos and transform yourself into a neon cyberpunk dog or convert a car on the desert into a warship on the sea Rave is a new method for video editing and stable diffusion after the transformation we will combine it with anime diff for a consistent result for this video we will use the new version 3 ra is a videoo video method recently created that can be used in comfy UI with this method you can change objects and change styles by using prompts no need of Ip adapters or masking tricks this video has two parts first I will explain how to get the starting workflow install custom nodes and the models that are used in the tutorial in part two I will show how to construct the workflow with Rave and animate diff in the description you can find the links to the workflow nodes and models you you will also have the chapter list for quick access to each step let's start the starting workflow we are using can be found in the comfyi custom node of Rave in GitHub navigate to the workflow folder and open Rave basic workflow Json file download it by clicking in the arrow icon on the right come back to comfy UI and drag the file you have downloaded over comfy if you do not have the required nodes an error like this will appear as usual install the custom nodes by going to the manager and then clicking over install custom nodes to run Rave we need to have at least these custom nodes in addition we will need anime diff evolved and advanced control Nets I also list other nodes which are used in this workflow or could be useful to improve the layout after you have downloaded the custom nodes you will need to download the relevant models in this tutorial we will make two examples in the first one we will use the realistic Vision checkpoint for the second we will use jaggernaut for control net we will need the depth control net which is the one used in the starting workflow this model can be installed directly from the manager in install models we will also use the control GIF it is a control net very useful to get low flickering animations with anime diff in the second example we will use of the loose control depth map control net this new control net allows us to get even more creative Transformations we are going to use animate diff version 3 this version works with a domain adapter a motion module and a sparse control encoder however we only need the adapter and the motion module download and copy the version 3 motion model for anime diff download and copy the version 3 SD 1.5 of the adapter and place it in the Laura models folder after all is installed update the comfy UI and restart select the video you are going to modify in the example a relaxed lady leaning on a fence for testing we are going to set the frames to process to 16 we also select every three frames which we can later recover with frame interpolation we will reduce the size of the images as this video is high resolution you can use many of the image resizing nodes available we connect the frames to the input and the image output to the Zoe depth map pre-processor and the V en code node for this video we will downscale the images by four so the RIS scale factor is 0.25 in the load checkpoint node we are going to use the realistic Vision checkpoint we downloaded and installed the control net is depth make sure that the right right model is in the load control net model the control net is used for both the main prompt on the branch on top and for the control net on the branch below for the UNS sampler node reduce the steps to 12 in this node we inverse the diffusion process we create a noise from which we reconstruct the transformed image with the Rave case sampler therefore in the case sampler we need to use also 12 steps we will not change the rest of the settings but we will fix the seed and use the one I used for the preparation of the workflow now the prompt for this video we are going to transform the lady into a sporty Chihuahua we also want to transform the image to a cyberpunk style while still keeping some realism using the realistic Vision model in the negative prompt Watermark worse quality bad quality let's deactivate the saving option in the video combine node and run the workflow well the Rave workflow is easy to run we have easily got a nice Chihuahua with a lot of neon details the image is nice but we can make it less flickery with control GIF and Anime diff we will do a second pass to the animation for that we can use any of the several lat and upscale nodes connect the lat to the node upscale by a 1.5 factor and use the bilinear mode for the second pass we will use the case sampler Advanced first let's connect the output of the latent upscale node we copy the Ved code and video combine nodes paste using control shift V to keep the V connection connect the case sampler to the V decode for long animations we need to change the apply control net node and use the ones from the advanced control to avoid errors during execution we just make sure that the connections are kept and that we use the same models and the same settings like at the beginning now we copy the advanced control net apply node we connect the conditioning output from the first to the second apply control net node we are going to use now the control GIF control net we need a new loader we select the right model which is control n checkpoint we do not need a lot of guidance so we decrease the strength to 0.3 finally we connect the output conditioning nodes to the K sampler for control GIF contron net I find out that depending on the animation sometimes it's better to use the original frames and sometimes the Rave K sampler frames we will prepare both possibilities by placing rerouting nodes next to the control net however for the Chihuahua example we will use the original image to use version three of anime diff we need to add a Laura model loader node we connect the checkpoint to the input then we choose the version 3 SD 1.5 adapter model you do not need this if you use a different version of animate diff the output model from the Laura node is going to be connected to the animate diff loader with context node we then connect the output of the node to the sampler to run long animations we will need to indicate the context options in its corresponding node which make appear by dragging the pointer from the context options input for animate diff version 3 we then select the motion module we downloaded [Music] before finally we set the case sampler final settings we use the same seed as the Rave case sampler and we set it as fixed we also use the same CFG sampler name and scheduler than in the Rave case [Music] sampler and as a starting step we start at five now we are ready we can run the workflow and see the outcome with animate diff and this is the result not bad test looks okay let's run the animation for longer let's say 966 frames and here we have it we have fully transformed a young woman into a Chihuahua with Rave and animated diff we are able to take a boring clip and create a totally different scene this technique is amazing to create fantastic videos let's make a second example and use a different clip we will try to convert this car into a boat we select every fourth frame and reduce the number of frames to 64 we will keep the original video resolution so we set the RIS scale factor to one we will also change the checkpoint and use juggernaut depth maps provide too much resolution and the object will be difficult to transform instead we use the new loose control depth maps this control net takes the depth map boundaries and guides the images more creatively with the prompt we also change the prompt in this case we want to convert the car into a warship and instead to be in the desert we want to be on the sea for this new animation we will change the control GIF settings we are going to set the strength to one but only apply it from 0 to 0.3 we will also change the input frames for this example we will use the Rave render output frames and ready we can test the new animation we have done a complete transformation of our original footage we have now a clip of a boat sailing on the sea final adjustments and frame interpolation can be used to improve the animation more control Nets or IP adapter can be used to further steer your animation I hope you've likeed the tutorial check out my other tutorials of anime diff And subscribe if you enjoy these videos please consider to support this Channel with a cofi that means a lot and gives me energy to continue making these videos and thanks for [Music] watching [Music] a

Channel: Koala Nation

Views: 4,220

Keywords: stable diffusion animation, ComfyUI, Comfyui animation, Animatediff, comfyui Animatediff, comfyui ip adapter, IP adapter, controlnet, OpenPose, DWPose, Zoe depth maps, comfyui animation, comfyui video, comfyui vid2vid, Animatediff comfyui, Animatediff controlnet, Animatediff IP adapter, lora, Animatediff evolved, controlnet animation, comfyui Animatediff controlnet, comfyui controlnet preprocessor, Unsampler, KSampler, KSampler Advanced, RAVE, Noise diffusion, LooseControl

Id: 4826j---2LU

Length: 11min 39sec (699 seconds)

Published: Fri Jan 19 2024