AnimateDiff Face Interpolation Comfyui vid2vid Tutorial IPAdapter ControlNet

learn how to create fascinating face morphing effects with comfy UI and animate di create this simple trick in stable diffusion let's get started for this tutorial you will need to download assets of this Civic AI page the link is in the description copy them in the input folder of comfy UI on the right you can see the different attachments in the photos file there are the photos that are being used for the morphing there are two zip files with the original frames that are used to create the masks they are in two files because I could not upload them in one unzip and place them in one subfolder in DW pose you will have the skeletons to be used in one of the control Nets so depth maps will be used in the second one the workflow from the tutorial is based in the hot shot Excel guide by Inner reflections with this workflow we make animations which use stable diffusion Excel models in the civit AI page there are more instructions on how to use it you can see the different requirements this workflow uses the IP adapter and frame interpolation for example you will need the Hot Shot XL motion model which is also indicated in the link I will also leave a link in the description there are two models available you can use any of the two but for this tutorial we will use the F-16 model download and copy it in the model's folders of animate diff evolve custom node and of course in the civit AI page I also leave a copy of the workflow you can use it to follow this tutorial or adapt it as you want after dragging the workflow and dropping it in comfy UI you should be able to see the workflow in case you have a message error and a lot of the nodes in red use the install missing nodes feature you can do it by accessing the manager in the menu of comfy UI now let's jump to the morphing animation workflow and see how it works in the first module we are going to load the different models in this group we also have the animate diff node we are going to use a stock video we will load the the frames which we need to create the masks later the animation in this case is guided with the prompt and the control net modules with these nodes and the anime diff node basically create the animation and what do we need to do for morphing the face the first thing is to create a mask of the face for every frame that is processed we want that the animation is applied only to the face we do not want the rest of the body and the background to be changed that is why it is applied to the latent and to guide the style we use the IP adapter with an attention mask for this reason the mask is connected to the IP adapter the morphing effect is created with a face morphing group for this tutorial we are going to use three reference images the new fold Batch's functionality and IP adapter allows a different reference image for each animation frame concatenating as many images as frames and doing interpolation in the transition frames we can create the morphing effect this way the IP adapter can do what we want let's see in more detail how every node works first let's bypass some of the groups for the demonstration we are not going to need to do the upscaling for the moment we will not do the face morphing so we bypass this and the masking nodes group looking at the first module what do we have first is the V we are going to use the base SD XLV in this module we also have the original frames that are being used later for face masking these images also Define the size of the latent in the case that the images are very large we may want to make them smaller we can use for example an image scale node the checkpoint we use is the base model of sdxl but of course you can use any other sdxl model you like this workflow uses the Hot Shot XL motion model and animate diff therefore we also need to use the hot shot Xcel beta scheduler in uniform context options in principle we are not going to change anything hot shot Xcel works with a maximum of eight frames so we keep them the rest too in the prompt we use the clip text and code sdxl in the positive we need to write a prompt which describes the animation in the negative we just indicate we do not want bad quality and worse quality now in the control Nets first we are going to guide the animation with an open pose control net we have already prepared the frames using DW pose we have generated the frames of the skeleton of the whole body hands and face but we are mostly interested in the face in addition we are using Zoe depth maps I do not think these are needed for the morphing effect but normally work nicely with animate keep in mind that the control Nets must be the sdxl because we use sdxl checkpoints these two modules should be enough to run the workflow let's try without the IP adapter and see what happens let's bypass the saving images and for this small test set the cap at 16 this is two rounds of anime diff steps without the mask and without the IP adapter an animation with a singer is created we can see the control net work the hand also the face the shape of the mouth the workflow seems to be okay okay let's now apply the style with IP adapter we activate the group for IP adapter we need to use clip Vision we need to use the right 1.4 sdxl checkpoint use the IP adapter plus sdxl v model clip Vision SD 1.5 is good however depending on the model you have to use another the attention mask is connected but will not yet work because the mask node is not active what we are going to do is to include a reference image let's use the first image let's activate only the first image for the IP adapter we do not need Now to repeat 39 times the batch we will need it later for face morphing let's use one as we do normally with an IP adapter the unfold batch option we could set it as false too but we keep it as true because we will use it later let's run it and see how it looks with just one image with IP adapter we see that the frames take our style in this case a scary lady but as you see the background is also animated without the mask the sampler takes all the information from the reference image and applies it all over therefore what we have to do is to apply the mask first we activate the group of face masks we want to detect the face for that we use the simple detector Sig we use the ultral liic detector using the to be box face model the segmentation model is B is okay the segments we then convert into mask using the seg to mask node with automatic detection you have to play a little with the levels to get what you want in this case in the background of another person if we use it with the standard values it will also detect this face here but we are only interested in the face of the singer in this this case we increased the threshold to 0.85 we have also increased the bbox dilation and the rest we leave the standard values we want to expand and blur the face mask a little bit this way the edges are not so sharp as we see here we do this expanding The Mask by 20 applying a blur radius of 10 and increasing the sigma to 1.7 with a simple detector we are creating a shape as well and we need to place it over the frame so so we need to have a black reference frame to do this first we take the dimensions of the original frames and create a solid mask with these Dimensions their mask is black so this value has to be zero if we have one it will be completely white but because it is the background it is zero we then need to put together both masks we use the mask composite node the destination is the solid mask the source is the mask that corresponds to the [Music] face of the different options select the one that is add the composited mask we use for both the IP adapter and also on the latent what we do with this is that the animation is only applied to the mask area the face in this case so the rest is not affected by the rendering the background and the body will be the same as in the initial photo let's see how it looks looks you can see that the mask is applied to the part of the face and the rest of the image remains intact now we are going to create the morphing effect for that we have created this module to do a short demonstration we are going to take only two of the modules we connect them to our IP adapter the amount of repetitions in the batch depends on the total frames of the original video in our case we have 133 but we have 16 caps so for this test we are going to put here six here we are going to put four or five we do four and we are doing another six I think more or less 16 in this quick example it does it very quickly it morphs from the first to the second image obviously it will be much better when we do more frames we are going to do all our initial images in this case case we are going to put to our third image and activate the rest of the nodes let's also use the original values of the example here it is a multiplier of seven 39 times total of 133 frames to render the whole animation now the only thing we have to do is put our cap to zero so that we process all the images and we go for it here you can see it I think it looks pretty good all the 133 masks corresponding to each of the frames the reference image has been taken and applied with a face morphing first reference image and then to the robot and finally to the woman with emeralds for the video we are not going to upscale it but we have run it before so you can see how the original image is compared to the original video the animation that we have created with our face morphing and well this is Al I hope you liked it and enjoyed it if you want to know more about stable diffusion com for UI and how to make animations look at my other tutorials [Music] thanks [Music] oh

Channel: Koala Nation

Views: 1,067

Keywords: stable diffusion animation, ComfyUI, Comfyui animation, Animatediff, comfyui Animatediff, comfyui ip adapter, IP adapter, controlnet, OpenPose, DWPose, Zoe depth maps, comfyui animation, comfyui video, comfyui vid2vid, Animatediff comfyui, Animatediff controlnet, Animatediff IP adapter, lora, instant Lora, Animatediff evolved, controlnet animation, comfyui Animatediff controlnet, comfyui controlnet preprocessor

Id: pDR6iie2eLo

Length: 11min 39sec (699 seconds)

Published: Wed Dec 13 2023