Stable Diffusion Animation Use FreeInit In AnimateDiff For Consistency Improvement

we have an exciting topic to discuss free init technology it's a revolutionary Innovation that aims to bridge the initialization Gap in video diffusion models so let's dive in free init technology is a groundbreaking solution that addresses a critical challenge in video diffusion models these models often struggle with the initialization process which can lead to suboptimal Performance and slow convergence free init technology aim to overcome this hurdle and unlock the full potential of video diffusion models to grasp the significance of free in it we need to understand the initialization Gap in video diffusion models the initialization stage plays a crucial role in setting the foundation for subsequent operations however traditional methods often fall short in achieving an optimal initialization resulting in limited performance and efficiency free init technology introduces is a novel approach to initialization in video diffusion models it leverages a combination of pre-training and fine tuning techniques to bridge the initialization Gap effectively by utilizing pre-trained models and carefully design training strategies free init enables video diffusion models to start from a more initialization Point leading to improved performance and faster convergence the advantages of free init technology are manifold by addressing the initialization Gap it enhances the overall performance of video diffusion models enabling them to generate high quality videos with greater efficiency this breakthrough has significant implications for various applications including video editing content creation and visual effects production free init technology represents a significant step forward in the field of video diffusion models its ability to bridge the initialization Gap opens up new possibilities for video generation and manipulation as researchers continue to explore and refine this technology we can expect even more exciting developments in the future so here's the result and that is going to be just using one times loading of 20 sampling steps and then going back there we have the previous generations that is loading with iteration 6 using the free init features and that is running six time of the 20 step that means we got I did like 120 sampling steps in the previous one and you can see there's totally six times like the prompt execution in 800 seconds and then in here I got 142 seconds so actually how do you enable the free init in anime diff using comy UI and it's really easy so using this workflow as an example here so let's create another group so we got better understanding so first we need an animated loader and then you have your checkpoint models you can do something like that maybe you use Juggernaut checkpoint models and you have context options that is a very standardized one that we use obviously and right here we need a sampling set setting the sample setting so the sample setting when you drag this out in the update animated custom notes it has this sample setting that is from the Gen 2 as you guys have seen my previous videos we have talked about the difference of anime diff gen 1 and gen two so right here we have the gen two and then the Gen one that is the Legacy one that it will fade out someday so we are using the update latest custom notes of this anime diff evolve then once you drag this out the sample settings first you got to see here confirm this is using comfy and then the second thing you got to confirm is the noise type by default these options if you're using default then it's just by using default sampling settings but if you use this free noise that will be enabled the free noise of the free init models the framework to be take place so in here they said the ddpm for and then the noise realizations and this is going to be happened in here so let's check out how do they mention in animated actually you can click to config UI manager if you install this animated diff you can find that in here using just filter the install and let's see mine is right here then what you have is let's go search and type free in it so it's right here so right here they said the free in it and the free noise support free init is under iterations options okay and then the free noise is in Sample settings noise type in the drop down so once you do this both settings and it will enable the free init take place in these sampling steps for your element diff Generations so once you have the noise type set as free noise and then you go to he the iterations options drag this out you will see these two options for you so in here you got to choose this one iterations options free in it so it will pops out this custom notes here so in here then that is all set for basically the free init sampling method with animate diff so in here you can set the iterations how many times you want the sampling to take step and keep looping back and forth like this framework diagram that it mentions it will refinement again and again if you set it like two times you got two times samplings if you have three times then you will do one times sampling and two times refinement in here so that means totally you got three iterations sampling steps so that's how you do animate diff enable free init framework using in your videos to videos or text to videos Generations here and I have tested also that you can use animate LCM or you can use LCM Laura so for example we have Laura loader and you can use LCM by default the SD 1.5 LCM or the anime LCM if you choose right here the anime diff loader you use anime LCM then you will have to choose anime LC c m Laura in here simple as that and if you are not using anime LCM use the one the LCM Laura so for example I have my SD 1.5 LCM Laura here it is also compatible with my settings in the free init sampling settings here then once you connect this and again you connect to your checkpoint models and you are ready to go with our motions model version three this will be basically it yeah that is how we can use that and of course if you enable LCM luras or animate LCM in the loader it will eventually processing faster and even you put more iteration steps in here it will still processing faster so I have another example that I use at LCM to process videos so let's see two result okay so here is both of this Result One is using a very simple animated loader and then one case sampler even this one is using one sampler as well so don't expect they are very high detail and the face is very detailed Etc but then we can see both of them have totally different performance and the output of each animations are going to be different when we take a closer look let's bring it side by side with that too and then we will see so this one again this one is coming from the free init features I'm using uh literation six okay so uh loop it six time basically that will be six time running the sampling step before it to pass to the vae decode and generate all this image frame Stitch it back together as an image and this one is without free in it and this is just a very simple anima loader running 1K sampler and then go to vae decode okay so both of them are using 1K sampler but then you see there is some difference result here as you can see the backgrounds here it doesn't have any clear backgrounds in the one that without using it and the one that using free in need we see the backgrounds actually is showing the Z View and then the beach below the stairs of the building and then you go to here obviously the character's outfit are almost the same quality because we have the golden dress IP adapter image and I know that image are going to be performed very outstanding of the color and the dress style when you look at this like zooming out you cannot tell there's much difference but if you zoom in here you will see there is some difference the golden color on here the golden color on here is more bright than this one the one that without using free Enid that is kind of look dark in here right right and then in this one that is using free in it the color is more shiny and it's more bright and you can see the dress both of them this one got more pattern right on the bottom here that actually showing there's more sampling steps going through on the texture of the character's outfit and also a lot more texture on the backgrounds and also the stairs are very inconsistent rather than here we are not using the free init you'll see there are some twists on here is not stay in one straight line and then obviously there's no backgrounds on the back you're just only seeing a blur backgrounds here but then the color of the outfit is a little darker it's not like a really sharp bright color or a very clear golden color and then the bottom of the dress here you won't see the pattern and the texture of like this is the motion right this is the legs stepping forward and then you see the dress is kind of curved in this direction but in here they don't show too much detail of that one even we are pausing both of them in almost the same image frame I cannot say it's exactly the same one but then yeah you can see there's comparison and overall if you look from zoom out view here there's some difference also on here and here we got less color and details on the stairs of this side yeah so this is the difference between we are using the free in it and not using it check out the full screen and see the difference now here's the difference between both of them and as you can see their color and the shyness and the sharpness of the character and the stairs as well now I have the other example in here and as you can see right here I got the characters from Dragon Ball and try to make that outfit and the background into these animations now this one is the without free in it and then the right one is using the free init now both of them are using same sampling step 20 and I'm using three iterations in free init so 20 step times three times so that is the totally I need to run in this generations for this iterations Loops here and that looks like three times obviously and as you can see right here the settings so there you go I have the full screen here and you guys can compare pause the videos and compare as well and I have two set now this is set a and I got the other set the set B and you guys can take a look and let me know which one is using free in it or which one is using animated diff only so that is it for this videos talking about the animated with free in it and this is a pretty cool technique add on to animated custom note and you can just use that right away without installing additional things on top of animated and that is pretty cool technique to enhance your videos generate quality as well as the whole motions of this but then again it gives and tick you have to spend double or triple times of your sampling step to generate videos so it depends if if you really need that quality then you will have to take time to do that as well so hope you enjoy joy and got some Inspirations and I will see you guys in the next videos have a nice day bye

Channel: Future Thinker @Benji

Views: 6,335

Keywords: FreeInit technology, video diffusion models, initialization gap, performance, convergence, pretraining, fine-tuning, efficiency, video editing, content creation, visual effects production, anime diff, comfy UI, sampling settings, iterations options, enhanced quality, video generation, video manipulation, Stable Diffusion, Stable Diffusion Animation, FreeInit In AnimateDiff

Id: f3cchKVfPT8

Length: 14min 43sec (883 seconds)

Published: Wed Feb 28 2024