Stable Diffusion AnimateLCM Speed Up AI Animation Generate Process (Tutorial Guide)

stable diffusion have been developed so fast just last few months we have sdxl model LCM and sdxl turbo models we have did a little trick before to get LCM work with anime diff to create AI animation today we can simplify the workflow with one motion model for animation using LCM model it's the animate LCM motion model we are going to test LCM motion model we are going to test the performance and see how we can update our existing workflow with anime LCM let's check it out look at this anime style generated by anime elom and here we have realistic style generated by anime LCM also this is simply using Ford sampling steps only and we have stable diffusions Loris and checkpoint models a few months ago and right now we have animate LCM model the motion models for animate diff and now let's take a look at the method diagram of the LCM here so first of all we have the image diffusions model working image consistency model for 3D inflation at the same time it works with the videos diffusions and working with the consistency and both of them the image and the video diffusions model processing and going to the last step which is the video consistency models and It produced the video result in this steps and basically using the animate LCM like this generate a realistic result and we can use two to four sampling steps to generate things like this now I will share the links of this animate LCM demo page and they have the research paper available as well for you to check it out if you want to go through the details of the technical side about this research paper you can do that so in the anime LCM demo page here we have saw a lot of controllable even controllable video Generations like this one that's basically using control net and then processing the movement and the motion of the animation and also here we have number of function evaluations is called the NF so here we can see different settings that they use and then producing different details of the result here so you can take a look at that as well so this one is the hugging face demo page of anime LCM now I have to mention this one because the demo gradio UI page they allow you to generate simple animations that is within 2 seconds just for demo purpose so I saw this one actually hugging face is kind of busy servers and you cannot really test the speed of it instead of testing on here let's bring it to our local machines and go to the comfy UI so animate LCM they have create a custom notes for that it's called the comy UI animate LCM so they have installations steps in here you install comy download the anime LCM from hugging face now we cannot download this custom notes in comy UI manager directly at my time of recording maybe because I'm not sure if they have indexing this custom note yet in manager so we have to download this one manually and then unzip file extract in the custom note folders in comfy UI the last part is very important you have to download these two models files the sd15 t2v beta ckpt is the motion models that you have to use that for movement just like animate diff version 2 version 3 and also you have to download the sd15 Laura beta safe tensor files and this is the Laura file for the animate motions so both of them have to be download in here they have provide the hugging face download link and I will share this link in the description as well and then once you download this tool you have to put the Laura files in your model Laura folder and the Motions model is not going to put in the checkpoints model like usual instead you have to put in the animate LCM motions models folder or an element diff custom notes we put that in the custom notes motion models folder but then I'm not going to test this one the animate LCM custom notes because this is only specified for the LCM sampling method for this one and instead I am going to test animate LCM with our good old animate diff evolved custom note we have used that a lot for doing animations and they have updated their custom nodee right now now it support animate LCM motions model and also the same step you have to download that two files previously we just mentioned and all you have to do is in the beta schedule you select either Auto Select or LCM and using fully with LCM you have to also including the LCM Laura in the beginning with connecting with any checkpoint model so in this one it's also the same one like the previous LCM models that we tested using the CFG between 1 to two CFG and the minimum sampling step is going to be four and also they have support the Macintosh computer right now so the Mac User you guys can play around with this without any problems if you're using M1 m 2 and M3 chips and then also they have some features on context options sampling settings outside of the animated diff loader that means they have another new custom node in this update pack they call it the Gen 2 in the new updates of this custom nodes so first of all let's try out a very basic workflow let me clean it out this workflow diagram and we start in fresh one so we're loading up very basic fresh animate diff loader for text to animate and this is coming from here this is the Json files from LCM Json so let's bring up this groups here and we can set up the checkpoint models and we have first of all let's select LCM checkpoint models to run this I'm going to use dream shaper 8 LCM checkpoint models and then in the lorus here we have to select the anime LCM lorus and right now I am renamed this one for easier to see and manage so I renamed the Laura's files name to anime LCM sd15 Laura then we are easy to recognize because by default you download this it doesn't name any meaningful thing with just the sd15 safe tensor file name with it so you have to rename it when you download it for easier to manage and then moving on to the anime diff loader so for the model's name we have to select the anime LCM motion models so in here I rename that one as well easier to manage easier to recognize so this is the SD 15t 2v beta ckp and then we have to select the beta schedule rather you select the Auto Select or LCM so right now in this example I'm going to use the LCM as the beta scheduler now moving on to here we check there's no problem as well and let's click Q prom and try out one time now we're going to see how fast this thing are going to produce one animations so in here we have 6 seconds in sampling steps and total using 13 seconds for creating this animations so again this is 24 frames animations that is using 6 seconds for samplings and 13 seconds for the total Generations including loading the models initialize everything so let's try one we are going to switch the checkpoint models to a non LCM checkpoint models this time I'm going to use the realistic visions and see if there's any effect on the loading time or any issues that is going to pops up so there's no pops up any issues and then the Styles it turns up to the realistic Styles as usual we're using realistic visions and you see this is pretty smooth the movement I cannot say it's very detailed because look we are using a very low sampling steps and CFG with LCM so face the reality we don't expect to have a very high detail and seeing how great that detail is right so let's connect with other Lura here so usually we have ADD details Laura for LCM as usual and this time I want to test if this LCM luras can be compatible with the other existing lores models and looks like it works it turned past to the green spot and then go to the case sampler here and yes we did have some detailing on the backgrounds of the animations so it looks like they are compatible with the existing SD 1.5 Laura's model and they are using 8 seconds for the second one with only realistic visions and then they are using 12 seconds with our last one that is generated with ADD detail l Laura so that means it will take a little bit longer seconds if you add other Laura's models or any features on top of this workflow pipeline so that is reasonable but then let's say this is going to be really fast using few seconds to generate an animations and then as you see this one we just mentioned the animate diff evolve they have new updates so let's check out their new custom note so before that once again I want to remind you guys to update this one element diff evolve you have to update this one if you're using the comfy UI Manager Go update with the comfy UI manager then let's check out their Gen 2 custom notes here so you can rightclick and then go to the animated menu in here once again you can click the gen two for the new custom notes and then gen one they have the animated loader and then the Legacy animated loader that we use in previous versions and right now we will using the new versions and let's see what kind of of new configurations or new settings that they have available for us so in this use evolved sampling you can focusing and select the beta schedules with separate custom notes here so if you're using sdxl you can use the sdxl beta schedule or the hotot XL scheduler and if you're using the animated LCM you can select LCM models and beta schedule as LCM as well so it is very simple just like previously we have with one element loader but then this time the new updates we have separate loader so now we have load element diff model as a separate custom note and then we have apply element model as like a pipeline just like what we usually do in segmentation impact pack that they have set up like a pipeline for this and I think this is a new design for if you can use the ad key frames and affect multiv values and scale multiv values at the same time with different groups of custom notes and I think that is how they let us to use instead of just only one element of loader for everything and that is more flexible for us when we further on to design workflow that is more complex with different steps in District building in different groups of element diff loader and custom nodes to work on the animations so for example the Neo Gen 2 I believe we can connect with this kind of styles so the animate diff we have the load checkpoint and then we have select motion models as we want and then we have the element diff Laura if you using the Motions Laura to pending the camera left and right and then just like the previous one but then everything allows you to separate in different custom notes so you have more flexibility to extend your workflows so for example we have a k sampler connect at the end here and we have a viit encode preview image or using video combines let's say we use a video combines here to create our animations and display on here the video player So This Is the Gen two way of connecting if you still want to use the Gen one of course you can use that but then this is the way of the new method of connecting using Gen 2 and let's bring up the basic animate LCM workflow and we can try with the Gen 2 connections so this is the element diff loader and we can replace this element diff loader with the New Gen 2 custom notes by using this three note base and we can try it on with the existing workflow so basically very simple we select the models connections from Lura and connect with use evolved sampling and then the output of the models connection connect to the K sampler and this is very simple way to replace the older generations of element of custom notes and here we have the new Styles or new generations of the custom notes in here for animated then other thing is remaining the same in this basic text to anima workflow let's click one time and see the performance and if there's any errors on the New Gen 2 so so far no any problems and the animation is here so I have think of what will be the Gen 2 can use for is that maybe in the future we can develop more complex workflow with multiple element loader and a applying multiple element models in one workflow and then you can be flexible and share the low animated model with different groups like that and I think that is how we can expand the way by using the Gen 2 and this is kind of like a brief ideas of what I think of if they have different custom nodes separate instead of using just one big loader so let's check out the animate LCM motions models with existing videos to videos workflow so I have my control net open post workflow that I have me mentioned in the previous videos this is a very lightweight workflow that is loading very less thing and just only with model settings IP adapter and DW pose for making animations and let's see how fast is it if we are applying with the element LCM so in the Laura loader we select the element LCM and then we don't need the older LCM Laura's model and then we can connect the clip vision's yellow line to the conditions of positive and negative prompts then go back to here we have the other settings and let's change the animate diff loader the Legacy one to the new one so I have the Gen one Legacy here so right now we are going to use the Gen one anime diff loader the newer versions now as you can see the newer versions they have more settings in the input point after the ad key frames they have sampling settings scale multiple value and effect multiple values that you can set up so that will be more for you to make elements movement and how you want the Motions are going to be maybe in different time frames things like that so let's connect this models first and run this existing workflow for this sample so again animate this LCM motions model select it in the model's name and then we connect the connections with the other models loader and then we select these content options and looks everything is good to go now let's change to other image for the IP adapter so we can see the difference result and let's lower down the video frames to 60 seconds for testing then let's wait for the prompts and want to see the result how fast this will be okay so it takes 38 seconds for 60 frames in the first sampling and then the second sampling groups are taking 1 minute 49 seconds I think because they have more sampling then you have taken more time but for the first sampling groups it takes 38 seconds and the result is here looks pretty good although it's very low sampling step and let's try out the full versions of the video two videos animations workflow here now I have disabled the Rave groups and then I'm just only using the animate diff here and the good part of this workflow I'm just easily changed the loader models in here we have the Lura models and then the Motions models for the element loader to change us the element LCM then that is all we have to change then we set it back to 60 frames just for testing and let's click run and see how fast this will produce so here's the result of the three animations groups we have the first sampling and then the detailers and the face swap and let's go back to see the detailers first so the detailers group we have used 49 seconds for detailers group to load and then scroll up back to the history we see the segment upscaled and then optimized for persons and then scroll up back to the previous history we see the display here the first sampling steps are using 1 minute and 32 seconds so in this work group in this workflow we have a lot of initializations in the beginning because we have a lot of control net and other models to load up so therefore we have taken longer time in the first sampling groups to initialize everything but then in in the second groups which is the detailers it takes less time like 48 seconds then that is the result of the face swap after detailers and it looks okay pretty cool and then I have a little twist tricks on the background image to make it so steel and steady and a little movement goes with the characters so that is how we use the animate LCM and it is really fast after this testing and I hope you guys have Inspirations from this and test it out by yourself and I will post the models link of the animate LCM in the description below so see you guys in the next videos have a nice day bye

Channel: Future Thinker @Benji

Views: 5,426

Keywords: AnimateLCM, animation workflow, anime-style animations, realistic animations, motion model, creative projects, sampling steps, anime LCM demo, research paper, installation steps, Comfy UI, custom notes, Loras models, Element Diff Evolve, Comfyui, workflow, animatelcm, stable diffusion, stable diffusion tutorial, stable diffusion video

Id: HxlZHsd6xAk

Length: 18min 37sec (1117 seconds)

Published: Sat Feb 10 2024