AnimateDiff SparseCtrl RGB w/ single image and Scribble control

yo what's up guys many of you have been asking me to try out the sparse control single image generation okay let's check it out today we talked about RGB encoder with first and last frame last time if you missed it don't worry you can catch up by clicking the link at the right upper corner with just a single image as the starting frame it's also capable to produce nice animations from my test and the example videos here the motion scale is not large so very similar to that from stable video diffusion the video interpolation seems useful and might be an alternative for vfi but we will skip it for now additionally I want to talk about the scribble or sketch encoder very powerful look at these animations I have made using single picture not bad right however you might notice that it's hard to control the motion and sometimes there are unwanted artifacts in the background we can use scribble control to D noise and redraw the anim animations to improve the consistency this also allows us to experiment with different styles you can also add control net to guide the animation using some tricks like the open pose here it provides a way to make the generation more controllable we will talk about both later let's get back to the workflow it's modified based on the sparse control workflow version one added with face detailer folder parser and scribble control path if you are new here please check the previous video so let's jump to the sparse control part recently animate diff evolved has updated gen 1 and Gen 2 two families of nodes it's noted that both produce exact same results while Gen 2 separates model and sampling applications and is more flexible so I updated to Gen 2 here for more future tests the nodes are shown blue here for more details you can refer to their web page then we need to change the sparse method to starting uniform is what we used last time and there are two more ending and Center we will try ending later that's all the changes made for this part go to scribble control group to prepare the sparse control input we need fake scribble lines pray processor note and not all the frames are needed the sparse control can function well with just some inputs this is the sparse control index method you need the index of images as the inputs usually a series of numbers to make it simple I just wired up some nodes to automatically create the index you only need to change the value of every inth node here the index will be prepared just some conversions here then this index will be sent into the select image indices node for the scribble prep process and also to spark control index method to use as correct frames correspondingly okay let me walk you through this workflow first to generate input images to utilize the help of open pose later we can pre-process the reference videos and use the first pose picture to generate the starting frame ensuring the pose consistency the same thing is applied to the last pose picture too we will test it later now I don't need IP adapter bypass it time to run we got the input images for starting frame and ending frame separately let's use the one after face detailer then activate the sparse control group and deactivate the image group I will disable LCM this time and bypass the batch image because we just use single picture drag the picture here and deactivate the scribble control for now ready to run we will try without open pose first bypass this control net the animation is smooth but not controllable for the clothes and sometimes background I tried several times to get this to have a better control I suggest to combine with open pose or any other control net you like let's try with open pose this time it's not bad with open pose the animation is smooth we can rerun to get a better color and outfit but this is just demo let's continue to try the image as ending frame still single image drag the second image we just generated using the open pose from last frame this looks even better so you can use this way to change the style with customized motion and postures then let's move to scribble control I used it as image to image before running just mute these three output nodes so it won't rerun change the input video to the one we just generated hit the run if you check the input of index just a series of numbers separated by comma it's done this is another style better lighting and shading also stable and smooth looks good for total 16 frames we used eight of them for controlling if changed to every third we get six every four four that is four one more thing to test is the total frames we used 16 as default and let's try 32 now change it back to starting I will just speed up here the background unfortunately changed but the person is still consistent throughout the animation showing the coherent facial expressions gestures and clothing now redraw using scribble control the red draw is pretty good but didn't fix the background yet maybe we can try even lower counts of input and control strength if you have a better idea or a different perspective just let me know I'm all ears okay that's all for today a lot of tests thanks for watching have fun

Channel: JanRT

Views: 3,440

Keywords: comfyui, animatediff, SparseCtrl, Scribble, stable diffusion, single image, animation, controlnet

Id: tAlzByG4nMs

Length: 8min 10sec (490 seconds)

Published: Tue Feb 06 2024