Stable Diffusion Animation Consistency Stay Still Background In AnimateDiff Workflow (Update)

so in this video I'm going to talk about the current update of the stable diffusion animated if workflow in comy UI now we have created stay still backgrounds for animations like this I am also showing some good examples and bad examples and also talking about when we should be using stay still backgrounds and when we should not for example in this case we have the camera pending now let's check it out so in the current new updates of the animation workflow that I created I have added stay still backgrounds within the IP adapter groups as you can see the result here shows the lightning cities coming from the IP adapter backgrounds group and the outfits of the characters are coming from this IP adapter group basically some people have asked me how I achieve consistent results with my output and the key is using a very simple and clear image for my styles that is the most important thing when choosing an image for our IP adapter for example in this case the IP adapter backgrounds are using CTS with lightning the image ipab is passed to other groups to create stay still backgrounds which I will discuss later then coming back to the animated if we are connecting the models that go through checkpoint models and the IP adapters this is crucial when you using custom characters and backgrounds for our animations One update in the control net is the use of the till control net for stay still backgrounds we connect it here using the image IPA BG as the output from our IP adapter image the reference image and the mass BG from the segmentations group so we have this till control net to help us maintain the image and replace the background of the mass image of our background part it comes from the mass BG of the segmentation group and it is the inferred mass of the character that will have the fully background mastered with our image frames in this process we have the initial size resize image from the source videos or image frames which is passed to our segmentations to work on the characters and the background Mass this is the main point of using the segmentations here currently I have just disabled the rayen M group and if you guys want to use it that is up to you but for me I'm just demonstrating here and don't need to use it using the IP adapter for Styles has already proven to be very consistent for my animation output in this case we are connecting the latent image from the initial setup which comes from the initial loader groups so this is the result of my first run in this demonstration lightning cities and the Wonder Woman outfit on the characters coming from the two IP adapter groups my image is very simple and nothing complex just a close-up shot close enough to see the whole outfit of what I want and really focusing on the center the second background image is going to be the city with lightning on top and it will stay still as you can see all the buildings and everything are replicated from my IP adapter image and pass to these image notes so these get and set notes are going to be passed to the till control net which is the last part of this control net group that's basically what we use for stay still backgrounds let's try another one I will now show you some good examples and bad examples of Ip adapter images for animations that I have tested so far for example using this Superwoman image with this street view image as the background you will see that there is something not right when using certain images for characters okay let's set the image frame to 60 and have a quick test with our image here now let's check out the other things in this control net as you can see the text prompt doesn't have much in it just very general ultra realistic Styles all kinds of those basic text prompts that's all I need and based on that I rely on the IP adapter for styling and that's basically it it let's run it once and see what's going on now this image from the Superwoman outfit right here is the result and as you can see it's not consistent you see the color of the upper part of the t-shirt is changing it's not consistent because one thing here we have so many elements going on beside the main characters in this image some color cloud on the side of it and then also we are influenced heavily by the third control net here which is the line art now let's skip this line art we have to delete this yellow line and then we bypass these groups of control net and let's try another image now I have another Superwoman image here with a white background and that will be better and let's see the result now we have the result here and now as you can see the outfit is more clear it's a blue t-shirt and then there's like a superwoman sign on the chest areas of the T-shirt but then this kind of costume is too complex now I will try this one a cheerleader outfit that is more clear like colors everything and also using white backgrounds that can be clearly showing for the IP adapter to just focus on the character's outfit now here's the result of our output result for the cheerleaders outfit now as you can see it's very clear and top of the part of the characters the outfit is almost exactly the same as what I referenced in the IP adapter except we have a mini skirt but then in our source videos we have shorts so the shape of that outfit lower part will be keep remaining with the short influenced by The Source videos but then we have the styles of the color and then the shirt is influenced by the IP adapter here and let's put side by side for the backgrounds we want to talk about the backgrounds here the stay still backgrounds as you can see these two sides of the row there are the buildings and trees that are replicating from our IP adapter groups here so in here basically I pass this image to this control net and using till control net model mods so we have just like paste the image on the background Mass area and then we set the movement a little movement in this control net strength let's say 013 and 0.5 something like that you can test around and let's test 0.3 and you will see what happened to the movement in the backgrounds so here's the generated result for the backgrounds set it as 0.3 in this control net and then as you can see there's a lot of moving part and the tree branches and all the wind blowing effect like this so you have to test and twist what kind of values in the control net strength that you want in a stay still background video style like this and The Sweet Spot is 0.6 so far far I tested most of the image sometimes it's 0.5 and then if you go to 0.65 that will be the most image that can be performed good for that situation now sometimes you turn it down uh too low like this one 0.3 it doesn't look realistic and a little flickering so let's try this another image like a stage concert image this kind of image performed really well actually for this dance animation so in here we have the cheerleader costume image and the concert image the concert stage image for our backgrounds and this is really matched to this kind of dance animation videos as you can see we have the stage very stay still and the light and some smoke effect from here is coming from the IP adapter actually and once I set it to 0.6 or or 0.65 in control net the element of the background is uh very consistent and very stay still and it's only allowing a little part of movement like for example in here we have some lightning we have the spotlight effect and the smoke effect and the only text prompt I did in here is the blue color boot because we don't have the boot in our reference image in IP adapter so we have to fill in this one for the missing part that not allows the AI to randomly generate use the lower part of the shoes so let's set it to 0.7 if you want more stay still backgrounds if you want more movement that will be the maximum thing if you put 0.3 and that is too much just like the last example with it so mostly I would do .5 for this kind of animations that is a little movement but not too much so let's try another example let's say when do we not using a stay still background and then also the characters we are using is this one and that is pretty good for this kind of dance animations actually these Source videos are also AI generated Dance videos that are only doing a Face Swap and there's nothing much edited in these Source videos but then as you can see in here I'm generating the full length of this video because I have tested this already with one two seconds and then here are the segmentation frames you see there are multiple characters in here and then some zoom in and zoom out from the cameras let's go pre-fill this one now with this kind of scenario we have cameras pending left and right and zoom in and out and first let's compare with our character's outfit now as I mentioned just mentioned that the character is using an IP adapter it performed really well with clean backgrounds and also like this kind of 3D illustrations image that can be performed really well for IP adapter and then in this case since the camera is pending moving around I'm not going to use the till control net except we are not using this image and we are only loading this background image into our IP adapter for backgrounds and the scenario why I'm using that because as you can see the camera is pending left and right and sometimes zoom in and out of the dancers and for this kind of movement the Motions that you shouldn't use like a steel background that all the background elements are going to be just like in one spot right so in this case we are mostly using just a free flow with not a steel control net for a fixed background behind so as you can see here we got the line art I just want to use the line art to make the movement more stay strong with that and this is the result of my 2C result here actually so as you can see when the character is stepping backward the shoes are actually stepping on the floor it's not like stay still backgrounds if you have characters that are moving around but then the background stage is not consistent aligning with the character's alignment and that looks kind of awkward and is not real realistically right so in this kind of case when the camera is paning in and out left and right then I am not going to use stay still backgrounds for this kind of animations so this is so far what I tested and what I have in the testing result and as you can see also even I'm not using the till control net and this is the example of what I'm using with the till control net and then the camera is pending zoom in and out of the characters and then you see the leg is kind of awkward that when the characters zoom in like this and then the background image stays still like the stage is suddenly getting small when the characters are zooming in right it doesn't look natural in this way so I am not going to use in this kind of scenario if I really want to generate good animations you see this yeah well when the shoes are standing on the floor it looks natural but if the camera is pending into the characters and then your background is still staying the same it doesn't look natural and then the lightning and all the spotlight on the top of the stage you got to move like this one it looks more natural of real camera zooming in and out of some objects right and this is what I got so far have a good animations effect that is suitable able for different situations and also even I'm not using a steel control net for the background like this I still enable to have consistent styles on the background behind it and all the spotlight and then the floor the wooden floor of the stage stay the same Styles and the characters obviously the clothing of the characters it stays the same Styles compared with this one we have different movement zoom in and out but then it doesn't look natural like a camera is moving moving so this is what I suggest if you guys have situations like that for stage still backgrounds it is not going to be always using stay still backgrounds for AI elements like this but if you have something very static very stay still the camera is not moving a lot and it is a good choice to use a till control net to keep your background stay in positions but then if you have dancing videos like this it is not going to be a good choice so so I hope you guys got Inspirations from these videos the current update of this workflow and I will see you guys in the next videos have a nice day bye

Channel: Future Thinker @Benji

Views: 4,306

Keywords: Stable Diffusion Animatediff workflow, ComfyUI, stay-still backgrounds, animation, IP adapter, consistent results, checkpoint models, till control net, resizing images, segmentations, control net strength, movement in backgrounds, tips and tricks, animation workflow, stunning backgrounds, video tutorial, ComfyUI update, animation quality, stable diffusion, ai art, stable diffusion tutorial, animatediff

Id: w5kNZCK1Ebg

Length: 15min 13sec (913 seconds)

Published: Mon Feb 19 2024