Enhanced Video to Video Workflow with Animatediff LCM Lora & LCM Sampler

with the powerful combination of animated div LCM Laura and the LCM sampler notes I will demonstrate how to transform a video game scene into an animated Wonderland by using a videoo video technique I use the control net extensions to trace and influence the original video using animate div through my process I experienced a few challenges before achieving the final results and I will reveiew how to avoid them by using the right note connections a couple of you requested a video to video method following the previous tutorial which is what encouraged this video I have already included a step-by-step guide in my earlier upload that explains how to get the LCM Laura models use the sampler LCM alternative and also how to install the extensions after that we will modify this into a videoo video workflow the idea is to use animate St to create a racing car I'll copy and paste The Prompt for this description and let's see what we get by hitting Q prompt to get started all right so getting started the speed and the quality is outstanding from animate diff but the outcome is not what I was hoping for so I'll give this another shot by improving the prompt with a more specific description and I'll hit Q prompt once again to see the results okay so things are improving now the trails and the smoke are nice uh but this is still problematic and we have no control over how the car is moving so I have already outsourced the clip I want to use which is the video game scene I'll trim the clip to the correct length of duration if the size is too large be sure that the video remains in the frame so I have achiev this by using key framing to modify the point of view during the video Once I'm satisfied I will export this as my reference video to control the animation so back inside comfy UI we have already created the animate D process along with the animate LCM Laura so I'll remove my main group here and divide it into smaller sections first let's start with animate div right click add a group and rename this to animate D plus LCM drag to create the group around the notes also right click below add a group and I'll rename this to text to video I'm going to switch the color here to purple all right so I'm going to speed things up a bit and uh arrange this to a more simpler way for you guys to understand and tidy things up just a little bit so I'll scroll up here and let's get started with the control net extensions I'll the right click to add node and add a group I'll name this to control net one also I'll change the color to green and drag this out to create the group selection next I'll double click and search for the apply Advanced control net so instead of using the original apply control net node this is what is going to make it possible to allow you to use the sampler custom LCN node so make sure to have this installed inside your custom nodes as advanced control net if you don't have this install go to the manager install custom nodes search for advanced install the kinking comfy UI Advanced control net close the manager after installing and refresh the server uh moving on I'll double click to search for the line out processor we are going to be using the realistic line ad connect this into the input of the advanced control net drag out the control net here and load the control net model in here I'm going to go for the line up model similar to the processor drag out the processor image again to see a preview and I'll use the VHS combine to know what the processor is doing change the color to Brown I'll change the frame rate to 12 also I'll use the h264 and also input the file path we also need a second control net Group which I'll rename here to control net and also change the color to Green once again again so I'll duplicate the node apply advaned control net and for the processor I'll search for depth and go for depth anything I'll drag this down here and connect this into the input of the advanced control net drag out the control net here and go for the control net loader which I choose the depth model to go along with the processor once again duplicate the VHS node which I'm going to connect with the processor to also have a preview so I'll double click here to search for load video and go for upload I'll select the video here I I exported earlier I'll zoom in a little bit and from here I'm going to change the load cap to 100 for a quick preview and next I'll draw to connect this to the line up processor input and also I'll rear oute this to the depth processor input image I'll let Q prompt test this to see whether everything is working perfectly all right this is all working perfectly the line art is tracing the original video to a simple drawing and the deck processor provides more Freedom by excluding the subject from the background as we can see so I'll quickly change the colors here as well for easy recognition from here the big question is how do we connect control Nets to influence the entire animation so I discovered earlier on using both control net might cause issues so choose the best control net that might work for you and go along with that so all you need to do is to connect the positive prompt to the POS positive Advanced control net and also connect the negative prompt to the negative Advanced control net I'll disconnect the model input from the sampler custom and reconnect it into the apply Advanced control net Noe from here we connect everything back from the advanced control net node into the sampler custom node I'm going to zoom out quickly and let's see if we have all of this connected properly by using control net to influence the animation from the original video all right so this is doing a great job uh this is amazing extremely accurate by animate diff and precise from control net compared to the initial generation without using control net so let's customize this to our own taste usually I come on to Lexica to get my style and I have the inspiration to use the Blade Runner theme to see how this uh works with the animation so I'll would click on search to see the results um from the platform I'm just going to click on this image and I'll copy The Prompt open my Notepad to paste it in here and I have also saved a few more Inspirations um as you guys can see here don't just copy the exact same prompt from the platform go ahead to create something different which you want to see and also you want to use once I'm satisfied with this I'll copy my master prompt back into comfy UI to paste this into my prompt node um I would also put some weight on some of the words for for more emphasis and I'll take out the human features from the negative prompt as well so zooming out uh let's see what we get by hitting Q prompt after putting in our sty all right so um this is done and I'm excited to see what we get this is actually amazing from animate div I especially enjoy the Blade Runner theme how this appears and the reflection on the streets is just incredible whilst it's in the rain we can also see here this only took just a matter of seconds using the LCM sampler and the quality is still impressive taking less time in just a matter of seconds to push this even further I'm using the Laura here known as the neon outlines download and place the Laura into the right directory copy the trigger word here and include it in the prompt back onto the page again I'm going to scroll down to copy the color theme recommendations and I'm going to include this as well inside the prompt so in here I'll duplicate the Laura node I'll go into select the neon outlines we just downloaded I'll reconnect everything from the checkpoint into the Laura to realign with the additional Laura we have earlier from here let's go ahead to Q prompt and let's see the magic from the neon outlines I'm going to skip this just to the results so we don't take too much time all right so there we go again with another amazing result from animate diff there are no limitations one you have created the notes the right way using animate diff and LCM sampler so for the third style which is the cartoon style as you guys can see I used the style here by the artist hge I just clicked on one and from here I just used the Illustrated style I just went into the prompt and then make sure I had the artist name in there and also I changed the low checkpoint model to use the hello Yan 25d I have also included an upscale to this workflow which will be the final workflow I provided in the description so that's how to get the best results and the fastest render time with the video to video technique using the animate LCM Laura and the LCM sampler nodes i' would like to thank everyone for their contributions and support in the comment section if you want to understand more about animate diff and the sampler LCM noes you can watch this video on the screen now and don't forget to smash the thumbs up as always I hope to see all of you in another Power full [Music] video

Channel: goshnii AI

Views: 1,933

Keywords: Animatediff LCM Lora, LCM, comfyui, stable diffusion, Stable diffusion tutorial, animatediff, animatediff comfyui, animatediff tutorial, Animatediff lcm, comfyui animatediff, animatediff lora, animatediff & lcm lora, animatediff with lcm lora comfyui, lcm lora comfyui animatediff, animatediff with lcm lora, lcm lora animatediff, lcm lora, comfyui tutorial, stable diffusion animation, stable diffusion tutorial, lcm, ai animation, video to video ai

Id: c8HI3vyVcso

Length: 9min 42sec (582 seconds)

Published: Tue Mar 12 2024