NEW UPSCALING method ComfyUI with 2 samplers AnimateDiff v3 + AnimateLCM

using different models for background and foreground learn how to integrate two different Samplers in one workflow it is simpler than you think we made an animated background using Juggernaut and animate diff version 3 the foreground is a woman made with dream shaper and animate LCM with masks and the two Samplers node we combine them seamlessly into one animation first I will quickly explain how to download the workflow how to install cust some nodes than the models used then we will do a walkthrough of the workflow so you know how to use it later you can adapt and change to your taste the workflow and the assets used are available in my page in open art. there you also have a summary with the instructions and some examples made with the workflow in open art. aai you can test the workflow for free using the runnable workflow green button at the end of the video I I will show you how to do it but for the tutorial let's see how to run it in your own comfy UI installation on the right top click on the download button to get the workflow in the resources section you have the three images and the video that are used for this tutorial download them and copy them in the input folder inside your comfy UI installation now in comfy UI drag and drop the workflow inside the canvas if some of the nodes are missing this error message will appear to install the nodes as usual go to the manager and install missing custom nodes the nodes that are needed you can install from there in my case they are installed but some of the custom nodes need to be enabled when installed click on restart to make effective the changes in case that's some fail go to the manager and click on install custom nodes filter by import fail and try again in case that there are still issues you may need to install them manually the list of custom nodes is here described the nodes are all available via the manager next is to download the different models used in the workflow most of the models can be downloaded from the manager go to install models and look for the ones in the list the rest you will need to download yourself we will use the Juggernaut checkpoint for the background you can download from cit. it will need to be copied in the checkpoints folder and comy UI for the foreground we will use dream shaper it is also available in civic. copy it in the checkpoint folder too we will use a couple of loras sketchy RGB and fantasy world download from civit Ai and copy in the luras folder we need two control models not available in the manager the first is control GIF it is a control net very useful to get low flickering animations with animate diff check the link in the description to download loose control allows making creative Transformations from depth maps it is available in Civic AI download and copy it in the control net models folder finally you will need the models for animate LCM they can be found in hug and face link in the description you need to first get the motion module and copy it in the motion modules folder of animate diff you also need the LCM Laura that need to be in the luras folder when everything is installed remember to update comfy UI now that you are all set restart comfy UI and let's start exploring the workflow this workflow can look a bit complex but it is easier to use than you think I recommend to run it in the steps defined in the groups as I will show you in this video Start selecting the background picture that you want to animate in this example we will start using a beautiful picture of a Paradise Island in the load video node we select first the video we are going to use we choose to use every two frames to reduce the number of frames to process to test the workflow we limit the frame cap to 32 this is a vertical video so we use 512 as width in case is horizontal you can use 384 or 768 pixels we need to set the picture size the same as the frames of the reference video for that we use the get image size node because the image can be large or small we have an upscale image by node with the image crop node we can adjust the background to the size of the frames we can also adjust the position depending on what part of the image we want to focus at when we run this group we see that the background image is not what we want adjust and play with the different settings until you have your desired reference image now it looks better a beautiful island on the top right we have the empty latent which we are going to use for the Samplers look that the width height and batch size have been turned into inputs so we can get the dimensions of the frames and the total count of frames from the original video observe that in this workflow we use the set and get nodes with these nodes we can connect remote areas and groups so we do not have lines crossing all over the canvas we can start step two which is the creation of the background animation each of the steps can be started by activating the group using the fast groups mutter for the background we are going to use the Juggernaut model connect the model and the clip output of the checkpoint loader to the Laura stacker the vae is going going to be loaded independently choose in this case the vftm model this V will be used in all encoding and decoding steps for the workflow as a Laura we are going to use the fantasy world Laura strength is 0.2 for the background we are going to use anime diff version 3 for this reason we need to add the version 3 adapter in the Laura stack we keep the strength at one we write down now our prompt for this first test we are going to use something like beach calm clouds moving we use also the trigger word of the Laura in the negative prompt we use keywords to avoid bad quality and not suitable for work images our first control net is loose control using depth maps it leaves some imagination from the prompt to make more creative images the second is the tile control net with this control net we want that the animation still has quite some elements from our reference image of course you can change and use other control net models in the loose control control net we are going to set a strength of 0.5 and the % to 0.5 for the tile control net we use 0.3 and 0.3 we need some images for the control net which is the background image we connect from a get node the loose control needs of depth maps in this case we are going to use the AI Au pre-processor node this node allows us to choose between all the available pre-processors we can choose any of the available depth maps but we will use depth anything in this case for the tile control net we will directly use the background image the case sampler is connected to the conditioning and the empty latent we fix a seed and set the different parameters the final number of steps we want will be 20 but for this first sampler we will use only 12 the CFG is 8 the sampler we use is D mpp2 MSD and the schedular Caris we leave the Deno at one animate diff has a new generation of nodes which we will use in this workflow however you can still use the Generation 1 animate diff loader generation 2 uses the use evolve sampling as starting point from this node you connect to the motion modes via the apply animate diff node and from this to the animate diff model loader we will use the version 3 model for the background finally you connect also the context options we will not change anything this this may look too complex compared to Generation 1 however it allows more flexibility and the use of other Advanced Techniques like using free init for sampling this is not the case for this workflow now that we are ready let's activate the decode and video combined nodes run the workflow to see the results we have animated our image background with animate diff version 3 the workflow works as expected let's be a little more creative and make a new background animation use green fire storm and magic in the prompt let's change the reference image and run the prompt again the traditional Palace is now burning and surrounded with a magical green fire use different images prompts and control net models to create fantastic animations for the background activate group three with the f mutter in this group we take the video frames and create a mask for the person and the inverse for the background the video frames go through the segment detector node we use ultr litic detector provider to Define which detection model to use make sure you are using one of the person YOLO models adjust the threshold limit of the detector for this case we set it at 0.2 but the right levels will need to be adjusted depending of your video with the grow mask with blur node we are gone to smooth the edges of the Mask to do that the expand parameter is 10 use also a blur radius of eight the rest of the settings do not need to be changed we use set nodes for the foreground and background these are later used with get nodes test the workflow for the masks and check that the detection of The Masks is the right one if everything is all right move to step four activate the group so you can start creating the animation for the main character the workflow for the foreground is very similar than for the background but with some important key differences we are going to use the dream shaper model which is a different model than for the background in the stacker we are now to select the RGB Laura with a strength of 0.2 we are going to use animate LCM so we need to select the LCM Laura model choose it from the drop bound menu and use a strength of one different than in the background workflow we use now an IP adapter to stylize the main character for this workflow we are going to connect the IP adapter between the freu node and the Laura stacker use the IP adapter plus model for SD 1.5 select also the clip vision for IP adapter version SD 1.5 our reference image for the IP adapter will be a lady dressing a white shirt and jeans the image might be too big or small so the prepare image for clip Vision node is used we keep the interpolation and sharpening settings but use the pad option for the crop position we use attention masking so only the foreground gets the style of the IP adapter we connect the foreground masks with a get node to the IP adapter in the IP adapter we set the noise to zero .2 and keep the rest of the other settings in the positive prompt we use woman dancing long trousers RGB we use the same negative prompt as in the background animation for the foreground we are using first the control gift control net set the strength to 0.3 the end percent to 0.6 in the second control net use depth use a strength of 0.7 and the end percent at one for the last control net use open pose keep the strength at one and use an n% of 0.8 remember to use the same SD versions for control net and check pints the video frames are obtained from a get node and need to be pre-processed for each of the control nets for control GIF we can directly connect the images for the depth control net we generate the Maps using the Deep anything node and for open pose we are going to use the DW pose pre-processor we also use the foreground mass for the control net apply nodes for animate diff use the animate LCM motion model no need to change the context options but feel free to experiment for a preview of the foreground animation activate the case sampler decoder and video combined nodes fix the seed and use any seed number you want for animate LCM the number of steps should be between 4 and 10 we use eight in our case the CFG should be between 1 and 2 so we use 1.5 choose the LCM sampler select the diim uniform as a scheduler and keep the denoise at one now run the workflow to see the foreground animation results excellent the workflow works and we get an animation of the foreground you see no details of the background because of the use of The Masks it is not really important in this case but shows the potential of using masks within your workflows activate now the two Samplers Group which is step five we also need to select one of the render methods of Step six we will start with no upscaling in the two Samplers group as the name says we have two case sampler providers the models for background and foreground are different so the Samplers need also to be different because we are using distinct anime diff models we also need to have different settings in each case sampler on the top we have the case sampler provider for the background we need to use a to basic pipe node for the case sampler provider this pipe is connected to the anime diff version 3 model from the background animation group the clip also comes from the background group similarly we also get the positive and negative conditionings from the background workflow the same is done for the foreground case sampler at the bottom we use a to basic pipe node and connect the inputs to the foreground model clip and conditionings the settings for the case sampler provider of the background are almost identical as done before we are going to use the latent output of the background animation as you remember we wanted to have a total number of steps of 20 in the previous sampler we used 12 so we set eight steps in this use the same CFG sampler and scheduler as before the latent we use is from the background animation group we set the denoy strength at 0.5 this way the result would be consistent with the background animation the foreground is going to be re-rendered therefore the settings are the same as used in group four use the recommended steps CFG sampler and scheduler for animate LCM keep the Deno at one to render different Samplers we use the two sampler for mask node the base sampler is the background sampler and the foreground is the mask sampler as a latent we will use a latent output from the case sampler in the background animation group our mask sampler is the foreground so we connect the foreground mask to the two Samplers node run the workflow to see the results and here we go seamless combination of different checkpoints lowas and animate diff models let's now run workflow with upscaling activate the upscaling group to use it for upscaling we need first to use the two Samplers for mask upscale provider node as with the regular two Samplers for mask node we connect the background sampler to the base sampler and the foreground sampler to the mask sampler The Mask to use is the foreground mask we need to use upscale model for this node for this example we will use the real ESR gon X2 model we use the iterative upscale node to upscale the animation we set the upscale factor to 1.5x and the number of steps to two the iterative method works by upscaling in steps so in our case the first step is from 1 to 1.25 and the second from 1.25 to 1.5 because we are also using two Samplers it means that we render each sampler times the number of iterations a total of four as you can imagine the results will be very good but the rendering time will be long as before we will use the latent from the background animation with all set you can start with the upscaled rendering the animation is ready the upscale method as you see creates a larger video with similar quality as the no upscale method the results without the upscaling are still pretty good so we will use the no upscale method for the last step face detailer and interpolation Act activ at the group in the fast groups mutter the images from the two Samplers need to be connected we connect the bottom node because we are using the no upscale method if you do upscaling use the bottom reroot node to do the face detailer we need to use first the simple detector for animate diff we need the ultral litic detector provider node select the B boox face YOLO model in this way the face of our animation is detected and can be detailed in the next node for this example we set the Box threshold to 0.5 and the dilation to five for the masking mode we use combined neighboring frames seg pivots is combined mask the images and the segments are connected to the detailer for animate diff we also need to define the abasic pipe because the face belongs to the foreground we use the animated if model and clip of the foreground we also Define The Prompt for the face we do not use any of the prompts used for background or foreground change it according to your animation if for example it is a video of a man change one girl for one man the settings for the detailer need to be aligned with the foreground settings fix the seed and use the steps CFG sampler and scheduler for animate LCM set the noise strength to 0.5 and the feather to 8 for interpolation we use the film model set the multiplier to two and the clear cache to five the multiplier of two is the same setting we use to select every two frames in the load video node at the beginning of the workflow because we increase the total number of frames we also need to increase the frame rate in this case we double it to approximately 25 frames per second with all the settings let's do the final test of our workflow and it seems that we are getting what we want if you are satisfied with the results the last step is to process the complete video come back to the first group with a background and video loading set the frame load cap to zero which will take all the frames of the video and the workflow will process all of them click on Q and let's see how it looks to complete animation and this is the final result you can now combine two different checkpoints and animations with one workflow I invite you to download the workflow test it and change it to what you like as I said at the beginning of the video you can run it for free in open art. just go to the workflow page and click on the launch on cloud icon you will need to connect with your Discord or Google account then you will have access to the T4 machines that are used to run the thousand workflows in open art after signing up and starting the cloud machine wait for 30 to 60 seconds until comfy UI [Music] loads when the workflow loads we are ready to use it the runnable workflow from openart has the same settings as described in the tutorial however the images and video template and some of the models are not available some of the models also have a different name for this reason we need to to- do some changes before running it if you have seen the complete video you will see the changes are very easy to do make sure you have downloaded the files from the resources [Music] section upload the reference video in the load video upload node do the same for the background image and the IP adapter reference [Music] image you will need to change some of the models too for the background the SD 1.5 version of Juggernaut is not available use instead any other SD 1.5 model that you like from the available list fantasy world is also not available use another one or set the first Laura as none the anime diff version 3 adapter is in a subfolder so you need to look for animated if V3 adapter and select it the loose trol model has a different name you can still find it in the list select it in the foreground the RGB Laura is not available therefore you need to change it or select none you also need to change the LCM Laura select the pytorch model under the LCM slsd 1.5 directory the control GIF control net is under the name of animate diff control net animate LCM is still not available so you can choose any of the other models I recommend to use version two change also the clip vision for the IP adapter use the SD 1.5 V Layin safe tensors model with these changes it should be enough to run the workflow results are not identical but very similar than with the models used in the tutorial if you want to save the final video results rightclick on the video combine node and choose to save preview I hope you've liked the tutorial check out my other tutorials of anime diff And subscribe if you enjoy these videos please consider to support this Channel with the Kofi that means a lot and gives me energy to continue making these videos and thanks for [Music] watching [Music]

Channel: Koala Nation

Views: 3,993

Keywords: stable diffusion animation, ComfyUI, Comfyui animation, Animatediff, comfyui Animatediff, comfyui ip adapter, IP adapter, controlnet, OpenPose, DWPose, Zoe depth maps, comfyui animation, comfyui video, comfyui vid2vid, Animatediff comfyui, Animatediff controlnet, Animatediff IP adapter, lora, Animatediff evolved, controlnet animation, comfyui Animatediff controlnet, comfyui controlnet preprocessor, Unsampler, KSampler, KSampler Advanced, Noise diffusion, LooseControl, DepthAnything

Id: XO5eNJ1X2rI

Length: 25min 11sec (1511 seconds)

Published: Mon Mar 18 2024