ComfyUI animation with segmentation: TrackAnything with SAM models

[Music] better control of your AI animations with track anything and comfy UI you can do it stay tuned check out this tutorial for this animation I followed this five steps one pre-processing the original video to prepare for comfy UI I rotoscoped the main character with Runway ml I exported the character with black background and extract the frames with FFM you can use of course Da Vinci resolve Premiere or any other editing software two segment your video using track anything you can segment your video in the parts you want you can do segmentation of characters objects and even parts of the body or clothes three create masks from your segments you will see how to create masks out of the segments and modify them in comfy UI four create the animation similarly as done in my previous comfy UI tutorial I create a video with control net and temperament I will will add a couple of twists to improve consistency five edit your video I will show you a trick in how to polish bad frames using frame interpolation later with a sequence of PNG you create the final composition with your favorite video editor let's get started with track anything you can find the link of my modified track anything git in the description down below with track anything we can segment and track the objects from your video later you can change the segments as you want using comfy UI copy the code and get started use it to install track anything in your system you may need to modify the code depending on the operating system you use the installation will take some minutes when the installation is complete and the application starts a hyperlink to a gradio site will appear click on it to open track anything when the web UI opens upload or drop your video file into the application then click on get video info scroll down to create the [Music] segments click over the sweater this will be mask one if the mask is Okay click on ADD mask now we click on the pants this will be Mask 2 again if the mask is Okay click on ADD mask when ready with all the masks click on tracking this will process the video and create masks one and two for all the frames tracking will take some minutes duration will depend on the number of frames segments and power of your machine when the tracking finishes a video preview of The Masks overlay over the image can be played The Masks created look quite good but they cover hands and the feet in some frames track anything allows to correct these frames we need it for a more accurate frame Creation in comfy UI download the frames to inspect them individually they are located in results image mask folder inside of the track anything folder go go through the frames one by one when you arrive to one frame which is not correct take note of the number go back to track anything in select track start frame slider go to the first incorrect frame the frame number in the slider is the number you took note plus one in this case it was frame 47 so we select frame 48 and track anything click on remove mask to correct the mask start again defining mask one which was the sweater the correction affects the masks after the modified frame and we want to keep the same color for all frames so you need to create the new masks in the same order as before click on the pants for the second mask the mask covers slightly over the shoes we can correct it by selecting the negative Point prompt because image is too small we have to zoom in all the window we select the part we do not want in The Mask if happy with the new mask add mask to then C click again in tracking new frames are being generated we repeat the inspection and correction process until we are happy with all the frames this process is semi-manual and takes some time however is important to do it right to have later a consistent animation in comfy UI when finished download all the results folder in addition to the overlays there is a folder that contains the masks only you will need to use the frames in that folder to modify them in comfy UI before starting in comfy UI we create three input folders the first is test for testing the workflow we will add there one frame with the masks the second folder is frames we will add there all the original rotoscoped frames the third is colored frames where we will copy all the masks frames now in comfy UI we will start creating the workflow the required custom nodes can be found in the description down below we add the image load node which we will use for our test frame you use the correct path for this simple example we are going to change the colors of the sweater and the pants to do that first we need to separate the masks we use the image color mask node in order to know the color code for the sweater we can use any online Color Picker just upload the one of the masked frames and pick the red color of the sweater the code is in heximal so will not yield any result when running the workflow we need to convert it into decimal again this can easily can be done with any online converter this decimal number is copied into the image color mask node When We Run The Prompt we see that a white mask has been created we repeat the process for the green color of the pants now we will change the color of the frames and overlay them over the original frames we start with the sweater we clone our load image node and change the colored frame by the original frame we blend mask and image using the image blend by mask node connect the original frame to image B and the mask image to image a and mask you can see that the new image has cut only the sweater we repeat the same transformation for the pants now we will change the color of the sweater for that we use the image rotate node this node allows us to change the Hue and Vary the color you can use online tools to know how to rotate the Hue to achieve the desired color in comfy UI make sure the rotation value is between 0 and 1 we will use a value of 0.27 which changes blue to Pink you can use other values to get other colors if you want you can also use other nodes to make further changes in the image for the pants because they are black the Hue rotation will not work to change the color we will blend the mask with a blue blank image we use the image size to number to not know the width and the height of the image the measurements are used to generate an image with the image blank node width and height have CT will be changed as input by right clicking on the size node we use again the image blend by mask node connect the mask image to the mask input the Blended image to image a and the blank image to image B play with the colors and use different blend ratios to adjust the color of the pants [Music] when finished you can blend both masks together again use the image blending mode for that use the mode screen and a percentage of one finally blend the masks with the rest of the elements that are not masked to do that create an inverse mask of the original image use the combined masks node to combine both masks make sure op is at now invert the image to have the negative mask the final image is made by blending the negative mask with the original image and then blending this result with the colored masks same way as done for the other masks if everything works okay we can process all frames we use the load image batch node to process the original frames we need to adjust the nodes to make sure the batch is loaded correctly we cannot use a second load image batch node to process the masks to overcome that we will use the text to string trun a node so we take the name of the PNG file and later with the text concatenate node we Define the path of the Mask file we will need the string to text node to create the complete path name including the file extension we can then reuse the image load node by changing the image path widget to input and connecting the path string to the input to save the files use the image save node connect the image to the image's input put change the output path in this case to Overlay masks change the padding to five change overwrite mode to prefix those file name right click and change file name prefix to input connect this input to the string output of the text string trunc a node run the node and correct to make sure the right images are generated sometimes it's difficult to operate with strings so don't get frustrated if first time does not work I will leave a link to the workflow in the description below so you can use it as example if the image is good you are ready to process all frames and obtain the images with a different color of the sweater and the pants to run the batch make sure you click on extra options the batch count field appears fill in the number of frames to be processed and press Q prompt again this will take some time depending on the number of images complexity and your machine with the new frames we have created we will create the final AI animation frames before starting we inactivate or delete the mass workflow you can also start a new one to create the animation I will use a similar workflow as I showed in my previous tutorial I will not show again how to make the workflow but you can find the link to that tutorial in the description down below I saved the workflows as a template so I can load it again by right clicking on the background and choosing the template under no templates we need to adjust the template for the new animation so I first add an image load node for testing we are using the overlay mask for this workflow so I copi the folder created into the input folder we use the path of this folder to create the animation with a test image we will create a first image for this example I Chang the Laura of the template I also change the positive prompt as it is very different to what I had in the template you can use the control and nodes you think suit best for your animation for this video I will not change the line art control net but I will change the open pose pre-processor for DW pose this pre-processor gives better results in my opinion to have better control of the image we add a new control net between open pose and temporal net control Nets we use the tile control net we do not need to have any pre-processor we fixed the seed and increased the number of steps to 25 we also reduced the noise ratio to 0.5 to reduce the artifacts and keep better control of the changes made with The Masks of course you will need to experiment to know what are the best results for your frames and animation as you can see the image keeps the colors of the clothes but the face is not well defined to improve it we use the face detailer node we connect the image generated to the input image we also need to connect the model the clip the VA and the [Music] prompts to recognize and detail the face we need to add the ultral litic detector provider use the face YOLO model and connect it to the batbox detector except for fixing the seed we are not changing the settings of face detailer in this test they are okay but you can experiment with them to get better [Music] results the final image looks much better and we will use for our batch frame generation play with the prompt the settings and test several frames to check the images are [Music] good when ready reconnect and activate the load image batch node and deactivate the image load node to keep the same name order as the original frames we use the text string truncate note [Music] again we connect the file name to the existing image save node and to a new image save node which will save the face detailed [Music] frames if everything is correct you can run the batch for all images remember to click click on extra options and add the total number of frames in batch count to run multiple frames after some minutes all the frames are being generated you can download them and sequence the pngs to make the clip unfortunately some of the frames will not be good if you want to have even better consistency you can manually correct them alternatively in some instances we can use frame interpolation we will need to have some additional nodes installed click on the manager install custom nodes search for the first node and install the compan comfy UI video helper suite and install it after it's installed search for The Comfy UI frame interpolation custom nodes while the installation is running you can create a new input folder for the interpolation find the frame that you want to eliminate and the copy in the frame before and after in our case we want to have a new frame 17 so we copy frames 16 and 18 when the installation is complete you need to restart comfy UI and refresh the browser add in a clean workflow the load images node from the video helper Suite select his directory the interpolation folder and now the film video film interpolation node because we want to add only one frame between the frames we keep the multiplier at two we add an image save node to save the images generated after we run the workflow we see that a new image between the two frames have been created this new frame will replace the frame 7 17 repeat this process for all the frames you want to correct when there are two frames to correct then increase the multiplier in the film node to three if there are three then use four in the multiplier when you have corrected all the frames just overwrite the bad frames with the good ones you now have the final sequence ready for editing the last part is to edit the video create the video sequence from the frames using FFM Peg or your video editor use your favorite video editor to rotoscope add a background and use effects this has been my tutorial about using track anything for better control of your animations in comfy UI check my other videos and tutorials to know more about AI images and animations I hope it has been useful and you have enjoyed the video thanks for [Music] watching

Channel: Koala Nation

Views: 5,604

Keywords: ComfyUI, Stable Diffusion, Animation, AI, Automatic1111, Midjourney, Video editing, RunwayML, Video Animation, Artificial Intelligence, comfyui, stable diffusion, midjourney, tutorial, controlnet, comfyui stable diffusion, runpod, automatic1111, jupyter, jupyter notebook, jupyter lab, google colab, colab stable diffusion, confyui animation, comfyui nodes, comfyui tutorial, SD, SDXL, SD tutorial, SDXL tutorial, segmentation, TrackAnything, Segment Anything, SAM, SAM model, Vast.ai, Vast, Animatediff

Id: HoTnTxlwdEw

Length: 15min 58sec (958 seconds)

Published: Fri Oct 13 2023