ComfyUI AI: What if the new IP adapter weight scheduling meets Animate Diff evolved?

hello and welcome to my new video this time Charlotte's friendly AI voice tells you about my adventures with the new IP adapter weights nodes an animate diff when I was experimenting with the stable video diffusion Model A few months ago I of course also looked at the possibilities of animate diff however my results were perhaps amusing but anything but satisfactory so I no longer bothered with the technical side of diff but then a few weeks ago the creator of the IP adapter plus nodes Mato from the YouTube channel latent Vision published new nodes I'm not usually one for exaggerating but it's appropriate here the new IP adapter weights nodes are simply phenomenal I've spent the last 2 weeks trying to figure out how to use them with animate diff and the level of control and creative application possibilities is so great that I decided to turn it into a video series even the basic workflow presented by Mato is already filled with a variety of control options that produce really cool results even without additional control Nets and other methods however the amount of resources and time required to generate the moving images is quite massive that's why I decided to rely largely on LCM you can install all the necessary LCM nodes and models via the comfy UI manager or download them from hugging face if you don't want to follow my LCM strategy you can of course also use the usual nodes for sampling the setup is as follows first we need the usual checkpoint node at civi you can download stable diffusion SD 1.5 models that already have LCM integrated the model I used the most right now is absolute reality LCM but there are others also from dream shaper if we want to follow the model pipeline then the next node is already the use evolved sampling node from animate diff evolved we then connect this to the context options standard static node and the apply animate diff model Advanced node we can also connect the motion model to the latter via the load animate diff model we then connect the IP adapter unified loader to the model Pipeline and we connect the loader to two IP adapter batch Advanced nodes the last of these then only needs to be connected to the K sampler so far so good as I am using using the possibilities of LCM we need a sampler custom node here we connect the LCM scheduler to this via the sigma inputs and outputs and the sampler LCM cycle node only the vae decode and the load vae node are missing from the sampler and of course the empty latent image node back to the IP adapters here we need two image loaders and if required two prep image for clip Vision nodes I always add them simply because they provide further useful options as far as the source image is concerned we attach these to an image batch multiple node from which it then goes to the new IP adapter weights node we link its weights and weights inverted outputs to the first and second IP adapter batch nodes weights into the first and weights inverted into the second the image 1 and image 2 outputs work in a similar way connect the batch size of empty latent image to Total frames once we have made it an input we then connect the weight strategy output to the next new number node prompt schedule from weight strategy this leads us to the batch prompt schedule node for additional prompt inputs we can then create a text field with a primitive node and connect it to app text now we just need to connect total frames to Max frames from the IP adapter weights node then connect the positive and negative prompts to the sampler and we've now made it in the sampler LCM cycle node you can add more Cycles to the generation without having to see the K sampler's total steps of 10 if you select the setting true for tweak sigmas this can help to optimize the processing of image noise in the case sampler or custom sampler ancestral also adds the influence of this sampling method to the UL steps that is more image noise is added per sampling step which can sometimes lead to more creative results here too you have to try it out in the sampler custom node my tests have shown that the ad noise setting can remain set to True here too the motto is the more noise the better the new IP adapter weights node lives up to its name as the values set here actually have the greatest weight when controlling the content of the generated videos IP adapter weights and promp schedule from weights strategy form a single unit this means that the value set in the weights node have a direct effect on the content display of the images in the finished video values of 1.0 and above give more weight to the IP adapter and values below take more account of the prompt depending on how many Source images are connected to the image batch multiple node and the corresponding prompts in the weight strategy node a value can be assigned to each of them this allows you to specifically influence the interaction between the prompt and the adapter in the weights node itself you can use the frames option to set how many total frames the video should have however this value increases quickly if you add additional frames via add starting frames and add ending frames with these three values alone you already have a huge amount of influence on the video generation in my experiments with this basic workflow only the alternate batches setting worked well the others are probably intended for more complex workflows I was somewhat surprised by the timing option this allows you to influence how the individual images behave when they appear and disappear together with the weight type settings in the IP adapter batch nodes this causes the objects in the image to come towards you if you select the ease in option for example the two batch nodes also give you further options for controlling the source images in order for the source images to become motion pictures they must be brought into motion by the animate diff nodes and the connected models this is where the various animate diff evolved nodes come into play these can also give you a whole range of options for influencing them on the one hand via the selected model itself I prefer to use the animate LCM model as a motion model but I also tried out the others you can see all the results of this series of tests in the uncut version of the video if you like in the use evolved sampling node you can select the different schedulers under beta schedule but you have to make sure that you select stable diffusion 1.5 compatible unfortunately these cool things are not yet available for sdxl so they are not an option for the time being the context option standard static node gives you even more control over the generated video however you should leave context length set to 16 because the motion models of the second version do not yet process longer ones well experimenting with context overlap helped a lot in my attempts to generate a coherent video from the content of the two images from the IP adapters but you get the actual control over the movement via the apply animate diff model Advanced node you can connect motion luras to this and exert direct influence on overall or partial aspects of the animation in numerous ways scale multiv values effect multiv values and add key frames are responsible for this in addition to the direct setting of values the start strength and end of the animation can also be specifically influenced this works via color gradients or rather gray gradients the general rule is black means that there is no movement and white means that the maximum movement is used the things you can do with these masks alone would need a video of their own you can also follow my learning process in the uncut version of the video if you want this brings us to the end of the first part of my series the other parts will be about dealing with the resource problem if you don't have an extremely powerful computer at your disposal your computer will certainly go to its knees with 110 frames that's why I've added two more steps to the workflow one to generate many images in the first step with low resolution a second to improve and upscale these images using perturbed attention guidance and a third to generate longer more stable and high resolution videos from them and another video will be about integrating other methods such as control Nets into the workflow and above all trying them out for me working with the combination of Ip adapters and animate diff evolved is still completely completely new and a really exciting learning process if you found the video interesting and or helpful I would appreciate a like and a subscribe thanks for watching and last but not least have a great day

Channel: Show, don't tell!

Views: 7,313

Keywords:

Id: SMXzoBbS1Hs

Length: 13min 45sec (825 seconds)

Published: Sat May 18 2024