ComfyUI AnimateDiff Part 1 - Tutorial #stablediffusion #comfyui #animatediff

hey everyone welcome back if you've been following our anime diff tutorials you've already witnessed the power of transforming text props and videos into stunning animations to unlock the full potential of anime diff I am creating a video series that will show all the functionalities of animate diff in very detail in this first video we will show the basic use of animate diff using text to video and describe the different options of the anime diff apply model node so you can get as much as possible from it this video will be very detailed but I hope it is a good guide for you when you want to create animations with anime diff we will follow in great detail the information and instructions provided in the anime diff evolved repository the link is in the description and can be consulted there the repository contains explanations of the installation the features of animate diff and how to use the nodes I hope that the Practical demonstration shown help you in further getting most of the value from anime diff to start and come for you in the anime diff evolve page go to the samples we will use the text to image example drag the workflow image and drop it over the comfy UI canvas the workflow should then appear if you do not have anime diff evolved installed the red message with missing custom nodes will appear to know how to install the custom nodes check out this short video link on the card the animated if workflow works as follows the checkpoint model is loaded with the load checkpoint node we will change it to Rev animated the clip is connected to the prompts same as image generation workflows we Chang The postive Prompt and use one girl solo blonde hair listening to music smiling cyberpunk Anime style very detailed high quality in the negative and not suitable for work they are connected to the case sampler we keep the default settings the checkpoint model is connected to the animated if loader which has a motion module model motion modules transform stable diffusion check pints into animation generators motion modules trained with real world videos are used to do this transformation The Empty latent defines the resolution and the length of the video with SD 1.5 models we will use 512x 512 pixels video resolution the batch size is 16 which determined the number of frames the Primitive is the seed used in the case sampler we will change it for 1 2 3 4 with all these settings we can already render the animation the latent output of the K sampler is connected to a v decoder and this to the video combine node where we can see the final animation we will not save the result change the format to h264 MP4 and run the workflow easily we have created our first anime diff video like when creating images we can change the settings to make a different video let's try with a different prompt add at the beach in the positive change the seed to 1111 and the Schuler to DDM uniform so far so good the basic use of anime defit not very different than generating stable diffusion images in comfy UI as you can see the anime diff loaders show us a large number of possible inputs the motion model and beta schedule are parameters this is a generation one node which is good to apply in most of the cases generation 2 was made to give more flexibility to the workflow in the GitHub you can see what the differences are they are not a lot to be honest but I will use the generation two nodes for the rest of the tutorial add the use evolve sampling node which contains some of the inputs but not all of the generation one node next add the apply animate diff model Advanced node and connect it to the model input of evolved sampling third add the animate diff model loader and connect it to the motion model finally connect the checkpoint to the evolved sampling and this to the case sampler [Music] delete now the Generation 1 animate diff apply loader node the first thing we can do is to change the motion model these are models that have been trained differently in the GitHub of anime diff evolved you have a description of some of the possible motion models but there are even more and more will appear in this first video of the series we will not go through all of them we will look into more detail into the different animation modules in next videos If if you still want to experiment check the documentation of the models to know what are the right settings to use these models need to be installed and the easiest way is to do it via the manager and install models just search for animate diff and you will find many of them we will now test a different model mmsd v15 V2 and see the results as you see a different animation with the same settings has been generated the different models have different characteristics because each model has been trained differently each one needs a specific beta schedule the default option is to use Auto Select which selects automatically the right beta scheduler for the motion module if the beta schedule is changed the animation will be different let's change the Auto Select schedule which is the square root linear to the average of the square root and linear in this case we are lucky and we do not see much difference but there is some the mouth is not the same and it looks a little bit more blurry if you change it for a very different beta schedule it could be that you cannot even run the workflow or it could be that the rendering is totally off so use the right beta schedule depending on the motion module but feel free to experiment put back the beta scheduled to Auto Select and connect the motion Laura input to the anim diff Laura loader there are eight motion Laura options these influence the camera movement according to the strength value the motion luras need also to be installed and you can do that through the manager in the same way we installed the motion modules in this case look for the motion Lura types let's choose the zoom out Laura and test it the render has a zoom out effect see that there is a watermark in the video this is because the original anime diff models and lowas were trained with Shutterstock videos and the watermark appears they tend to disappear when the strength of the movement is reduced let's try with a strength of 0.7 now there is no watermark effect the strength the zooming out effect stands out less with other models there will not be also Watermark effect but the motion luras cannot be utilized for all motion models recently there have been developments to train motion luras however we will explore the use of these tools in other videos of the series motion luras can also be concatenated for example connect a second motion Laura loader and use rolling anticlockwise check out the result a new animation combining both motion luras is created the sum of the strengths of both motion l ases is more than one so the water mark appears again while this is limited to version two models there is potential to use other tools which we will explore in next videos the next input we can play with is the scale multiv Val previously called motion scale it influences the amount of motion generated by the model so when the value is low there will be small to no movement and when the value is high it will be more Dynamic there are two nodes we can use multiv valve Dynamic and multi Val scaled mask let's start with the first and try to increase the movement by applying a float value of 1.2 the generated animation now has more movement both of the girl and the background the float value though cannot be varied a lot if we increase it to 1.6 the animation loses consistency and a lot of strange artifacts appear if we reduce by a little to 0.9 the movement is reduced significantly and even more if we reduce it to a value like [Music] 0.4 to use the multivalve dynamic node with masks let's first obtain the silhouette of the lady let's generate the original animation again but now connected with an image preview [Music] node copy the first image in the clip space add a load image node and paste the image from the clip space add the one former Coco segmenter add color to mask and change the colors to Red 220 green 20 blue 6 set the threshold to zero add a grow mask with blur node expand by 20 pixels change also the blur radius to8 looks good so we apply the mask to the multivalve dynamic node as you see in the result the movement is only applied to the mask the non-masked area remains blurry we can apply a background using the reference image as a latent connect the image to a VA en code node connect the ve and use a repeat latent batch with 16 frames connect then the output lat to the cas sampler for this test increase the float value to 1.4 we get now an animation with a fixed background taken from the starting image and a more Dynamic subject with masks we can also use the multivalve scaled mask this node allows customization of the Scale based in the dark and light tones in the areas of gray masks the degree of movement is defined in the range of the minimum and maximum if we use absolute scale the 100% black areas will have the minimum value movement and the 100% white area is the maximum value in the gray areas the value that will be taken will be the interpolated value determined by the intensity of the gray between the minimum and the maximum when the scale is relative instead of 100% black and 100% white the minimum value is taken in the area of the Mask which is the darkest and the maximum takes the pixels which are the lightest this is easier to understand if we see it with an example let's add a gradient mask set the size of the Mask mask to 512x 512 and the number of frames to one The Mask we have created goes to 100% white on the left to 100% black on the right in the middle the value of the gray is 50% white and 50% black let's set the minimum value to 0.4 and the maximum value to 1.6 and run the workflow the resulting animation looks weird but shows the effect of the mask the left part with more white moves a lot on the right with dark tones it does not move right in the middle with a gray 50% black and 50% white the scale value is the average of 1.6 and 0.4 which is one the next input we can use is effect multiv Val the amount of effect determines what is the influence of the motion model over the sampling process so with a value of it would be as running the case sampler without animate diff let's see this more clear with an example duplicate the cas sampler V decode and video combined nodes use contrl C to copy and the control shift V to paste to keep the connections to this case sampler now we are going to connect directly the checkpoint loader so we render frames without animate diff connect the multivalve dynamic node to the effect multivalve input in the apply animate diff model Advanced node remove the mask change the float value to zero let's run the workflow to generate the animation the case sampler without animate diff has created 16 different images resulting from batching 16 latent let's see now what will be the result for the animate diff loaded case sampler both Samplers have generated the same images with a value of zero in multiv val effect there is no influence of anime diff in the sampling let's increase the value to 0.8 now the animation is more consistent than before but we can still see some flicker let's try now with a float value of 1.2 the animation seems to be more consistent but at the expense of some movement if we compare with the original animation the combination of scale multivalve with effect multivalve is possible connect the previous multivalve scaled Mass to the animate diff apply node change the minimum to one and the maximum to 1.2 the result is nice with quite some movement of the lady we can also use masks to apply the effect multiv Val disconnect first the multiv Val scaled mask from the apply animate diff node then connect the face mask to the multiv Val Dynamic node change the float value to one and run the workflow the effect multiv Val is applied to the mask we have defined but we can see that in the non-mas area the background is changing continuously the background images have been diffused but the motion module has not been applied in that area if we try to apply the encoded image as a latent as we did for the scale multivalve the Mast area gets this new latent but the background still keeps changing therefore we need to find another way if we want to keep consistency on the background let's first reconnect the empty latent duplicate the apply anime diff Advanced node connect the output model to the previous anime diff apply node connect the existing anime diff model to the new node duplicate the multiv valve dynamic node connect it to the new animate diff apply node and now connect the inverted mask from the gro mask with blur node with this we will also render the animation in the background but done separately from the person mask run the workflow and see the results the final animation as you see is similar to the one with no mask however with two masks two motion modules are applied independently so the result is a different animation let's finish this part by testing the scaled node mask disconnect the previous model connection and connect the multivalve scaled mask node with the gradient mask to the apply animated if node reduce the minimum float value to 0.8 and run the workflow the gradient mask is applied in the animation the maximum value is used on the left while the minimum on the right scale and effect values allows us to tune the extent motion module is applied in our animation next we will show how to use animate diff key frames ad key frames allow to schedule scale and effect multiv Val the scheduling is done in time steps this means it is done during the sampling process but better explained with some examples connect to ad key frames node to the apply animated if model node simply Connect Now a multivalve dynamic node with a float value of one and run the workflow to see what happens as expected with the default value of one the animation is the same change now the multiv valve value to 1.2 and run again make a copy of the video combine node to compare the results with the scale effect of 1.2 add a multivalve dynamic node and connect to the scale multivalve input in the apply animated if model node change the float value to 1.2 and run the workflow the result is an animation with a scale value which is too high this is because both float values are multiplied to prove that change the dynamic value of the ad key frame to 0.6 and the one of the apply node to two by multiplying these numbers you should get a total scale value of 1.2 run again the workflow and see the results the new animation is exactly the same as the animation which had only one scale value and was set at 1.2 obviously if we swap the values the resulting scale is also 1.2 and we will create the same animation again delete the multivalve dynamic node connected directly to the apply animate diff node change the float value of the ad key frame node back to 1.2 duplicate the two nodes and connect the new ad key frame node and run the workflow the animation has not changed so we do not have this multiplying effect as we had before now if we change the scale value to 2 and 0.6 the result will not either be a multiplication an average or a sum of the results it works differently to illustrate it set back both scale values to 1.2 add a sampler Advance node and connect the same inputs as in the regular K sampler node make sure the parameters of the case sampler Advanced are the same as the regular case sampler change the scheduler to DDM uniform and check that the leftover noise at the bottom is disabled connect the case sampler to a V Code and video combined node and run the workflow to check the setup works good the case sampler Advanced with the same settings creates the same animation as the case sampler which means connections are okay if You observe the preview in the case Samplers you see that the image changes as the diffusion process progresses what you see there is a preview of the first frame of the animation as the steps of the case sampler increase in the case sampler Advance node we Define the start and end steps of the diffusion process change the end step to four from the video we see that the sampler has only run four steps out of 20 this is 20% of the total diffusion process let's break down the sampling process duplicate the existing case sampler Advanced and check that all the connections are there in this case we need to connect again the model input duplicate the case sampler advanced again this will do the remaining 16 steps what we want is that this branch of advanced case Samplers does the same as the regular case sampler enable the leftover noise in the first Advanced case sampler so it is transferred into the latent that goes to the second sampler connect the latence of both case Samplers and disable the add noise to the second case sampler advanced in the second case sampler we will do from Step 4 to step 20 using 1,000 at the end is the same as completing the total number of steps if we run it we can check this new branch of the workflow works as expected the resulting animation is still the same as we have yet not changed any parameter in anime diff change the start percent of the second ad key frame to 0.2 for 20 total steps total this is four steps change the scale effect is zero the idea is that during the first four steps the render takes a scale of 1.2 while for the following 16 steps until 20 the scale value is zero however we still have some guarantee steps if there was an overlap in the start percentages at least one step would be applied for the moment let's set them to zero now run the work Clow in the preview of the regular case sampler we observe that the image suddenly had a change when reaching step four the final result is a very static video because we have rendered 80% of the animation at a scale of zero in the advanced case sampler we see that the first four steps Define a moving scene but after applying the next 16 steps with scale zero the resulting animation is exactly the same as the one we have generated with the case sampler before we set the guarantee steps to zero however if we indicate a value and the key frame of the left has the same start percent it will be executed the number of guarantee steps indicated as example let's set the first key frame to one the change has no effect in the standard case sampler because the start percentage of the second key frame is 0.2 or four steps on the other hand the second Advanced case sampler does apply these four steps because it's starts at 0.2 however one step is guaranteed the first step on this sampler is at a scale of 1.2 in total both Samplers do five steps with a scale of 1.2 and 15 at zero the difference between renders is small but it can be appreciated Now set the total number of steps in the first ad key frame node to four and make sure that in the second the start percent is zero this is equivalent at indicating that the first 20% will run with a scale of 1 .2 and the rest will be at a scale of zero the new animation is still the same these examples illustrate how the ad key frames nodes influence the diffusion process this demo with a scale value of zero is in the extrem but shows the flexibility of using key frame scheduling imagine you like the character of the animation with a scale of 1.0 but you want some more movement however if we increase the scale to 1.2 the cloes the position and the background will change what we can do is for example set the scale of the first key frame to one and apply it in the first 30% for the remaining 70% we apply a factor of 1.4 what happens is that during the first part of the diffusion process a base image similar to the one made before is created then a higher scale value of 1.4 is applied to the rest of the diffusion process resulting in an animation with more movement as you see by using ad key frames we can obtain a more Dynamic animation reducing the changes in the character and the whole scene to finalize the apply anime diff model Advanced options we see that we can change the start and end percent at which is applied this is very similar at how is applied in control net oras it is done in steps and advanced case sampler if we set the start percent at 0. three the anime diff motion module is only influencing the last 70% because the first steps are more important the resulting animation is mostly a sequence of images however if we start at zero and end at 30% we observe that the influence of the animate diff motion model is higher the image is still flickery but there is still some consist on the face of the lady the control using the start and end of the apply animate diff node can also be done by scheduling the sampler with ad key frames and effect multivalve if we connect two ad key frames with the first at float value of one and a start of zero and a second with a float value of zero and a start percent of 0.3 the resulting animations are the [Music] same obviously with ad key frames even more steps and subdivisions can be applied which makes the workflow more flexible but also more complex anyway today we have seen the basics of anime diff and we have also seen in great detail how to use the different options in the apply animate diff model in the next video of the series we will show how to better control the animations with prompt traveling doing video to video and further controlling the output with control nit an IP adapter we will also delve into the use of context op options and Sample settings to be able to create longer animations and improve the animation quality thanks for watching

Channel: Koala Nation

Views: 3,415

Keywords: stable diffusion animation, ComfyUI, Comfyui animation, Animatediff, comfyui Animatediff, comfyui animation, comfyui video, comfyui vid2vid, Animatediff comfyui, Animatediff controlnet, Animatediff IP adapter, lora, Animatediff evolved, controlnet animation, comfyui Animatediff controlnet, KSampler Advanced, motion lora, stable diffusion, AD, Animatediff keyframe, scale multival, effect multival, beta scheduler, comfyui tutorial, animatediff tutorial, animate diff

Id: E_GnupiwAeY

Length: 27min 8sec (1628 seconds)

Published: Thu Apr 18 2024