Animated ControlNets using the OpenPoseBone tool and ComfyUI

hi welcome to my new tutorial in this video I will show you how to create a variety of animated control Nets in blender which we will then use to create images and videos in comfy UI there's a great blender file available for free on gumroad which helps us to create the control net inputs by simply adjusting the poses we want and we can even retarget mixamo animations to our control Nets and create whole scenes which can be used for specific control Nets like depth segmentation or normals so let's dive right into it first let's go to gumroad and download the blender file link down below just name your price or type in zero if you want to download it for free and add it to your card on the next screen enter your email address to proceed and at the download page be sure to download the latest version which in my case is open POS bones v95 a quick remark please be patient while you follow along the steps of this tutorial it might seem a bit challenging at first but once you've made it you can use this method for any stable diffusion animation with humans in it with great control over the scene and the animations just using text prompts without any real life input footage as you are in 3D space you can look at it from any angle and move the camera CA around freely while separating the scene settings from the actual stable diffusion rendering in a clean way okay let's move on once you've downloaded the file unzip it to any location you like and then let's head over to blender and open the open post bones file in order to get started let's now save it under a new name to keep the original file intact now on the left hand side we have an overview of the different different control net layers for canny depth key pose open pose and so on on the right top you can switch between the different control net layers and when you select one let's say open pose you can see in the left window which elements are visible and which are hidden in this layer now let's switch to rendered mode and you can see how the pose will be rendered for the selected control net when we switch to a different layer the rendered View will adjust accordingly some layers like depth and canny can't be displayed correctly but once you render them they will be okay now let's switch back to the default layer and when you press the play button down in the timeline you will see that the figure is turning around so there's a built-in animation which we don't want at this point so let's click on the rig that's the skeleton of the figure and you will see a lot of controls that enables us to change the poses or also note that blender has automatically switched from object mode to pose mode Let's select the circle with a four arrows at the most bottom of the figure that's the control which turns and moves the whole body and you can see two key frames in the timeline which makes the figure turn around let's remove them by selecting them and pressing the X key all right now let's set the end of the animation to frame 250 in the timeline so the animation will be about 11 seconds at 24 frames per second now select the hip control and select the rotate tool from the toolbar and when you drag it with the mouse you can see that the hip is rotating horizontally while all other bones are following that rotation accordingly then press I to set a key frame choose location and rotation and the key frame will be saved at the current position in the timeline go to a different frame in the timeline roll rotate the hip at the Z axis hit I again and another key frame will be saved when you scrub through the timeline you can see that the animation is interpolated between the two key frames and that works with any of the controls still that's a rather cumbersome way of animating so let's remove the key frames again I will show you a much better solution just a bit later when you click on the little camera icon at the right you get into the camera view That's The View that finally will be rendered then select the camera in the left window and then the output properties on the right window let's change the render resolution to 1920 by 1080 that's the HD video size then hit G and move the mouse you can see that the camera view is moving let's set it to the proper position then open the render menu and select render image to render all control Nets a new folder multicontrol net will will be created within the folder where your blender file is located and within that folder there is a subfolder for each control net let's open open pose full and here you can see an image which we can directly feed into the open pose control net in comfy UI or automatic 1111 now let's animate there's a great site mix amo.com where you can find hundreds of highquality motion captured animations for free so let's go there create a free account or login with Google and search for a good animation I will get the belly dancing to let our figure dance hit download and Export the animation choose without skin as we don't need the Avatar set the frame rate to 24 and Export back to blender select import fbx find the animation we just downloaded and press import fbx you can see a dancing skeleton which we now must retarget to our control net character there are several ways to do it the easiest way which unfortunately isn't for free is autoi pro which you can download from gumroad you can also go with the Roco retargeter which is free or try the rig tool that came with the download of our blender file with a short description on their gumroad page but I haven't tried them as I own Auto rig Pro anyway so I won't dive deep into these tools to install auto rig go to edit preferences addons hit install find the ZIP file you've downloaded from the auto rig site and then hit the select checkbox a new tab named ARP will be created in the tab bar on the right so select it and hit n to open it a quick remark the tool didn't work with the latest blender version 4.0 so I had to switch to a legacy version in my case it's 3.4 4.1 but maybe it'll be already fixed by the time you watch this video okay now select the mixamo Armature and open the auto rig Pro remap drop- down click on the little pipet tool to set it as the source Armature then select our control net rig and select it as our Target Armature and then click on build bones list auto rig Pro will now match all Bones from the mixamo skeleton to our control net skeleton next find the root bone of our control net skeleton called C root master X and select it as root bone one more thing to do hit retarget rest pose to match the standard pose of both skeletons you can see that our skeleton is in a pose while the mix skeleton is in t pose so go to front view and rotate the arms and legs of the mix skeleton so that they match closely to our skeleton you can see that both skeletons are now in sync we now can delete the mixamo Armature as we don't need it anymore and we're almost done with the basics I now want to add some camera movements through our 3D space let's open another window place your mouse between the right side of the main window and the right pane until the cursor turns into a Crosshair then click and drag the mouse to the left a new window will be opened in the left window click on the camera icon to get to the camera view the select the camera in the leftmost window and zoom out in the right window a bit so you can see the camera on the screen now move and rotate the camera until you get an angle you like you can see it in the camera view in the left main window and hit the I key to insert a key frame for the camera go to different positions in the timeline and repeat this process until you are satisfied ified with the camera movements in your scene okay now if you just want to use open pose we're finished in blender and you could just render the animation you can skip this chapter and continue with the comy UI workflows but we can go a step further and build a simple scene in blender that will be very helpful when using other control Nets like depth and segmentation so maybe you also want to follow me with this step let's build our scene a simple room with a door a window some furniture and paintings on the wall all extremely simple create a new collection name it room then hit shift a and create a simple plane which will be our floor move it down a little bit so our figure stands firmly on it then hit s and type 10 to scale it by 10 times name it floor then go to the material tab search for floor and select it this will let the segmentation control net know that it's a floor and when you use the term floor later in you text prompt in comy UI it will be recognized correctly for example a wooden floor or a tiled floor let's continue with our room with the walls a door a sofa a table a lamp and classify them in the same way you can also import some assets from blender kit or sketch Fab link down below but keep it as simple and coarse as possible it's just a guidance for the control Nets I'll speed it up think you get the sense last thing to do in blender is selecting each control net layer and decide which objects should be visible or hidden at rendering in open pose we just need the skeleton to be visible while for depthful and segmentation the whole scene needs to be visible in our case just hide or unhide the room collection once that's finished delete the multicontrol net folder then go to render render animation to create an image sequence for all control Nets then let's start comy UI and build up our workflows for image and video creation for that purpose I will switch over from my Mac to my Windows machine as it has a powerful Nvidia GPU and can render things faster than my Mac still everything I show you in this tutorial can be done either on a Mac or a PC you can download both comy UI workflows for free link down in the description just drag the workflow file onto your comy UI browser interface and install missing nodes using the comfy UI manager install missing nodes if you are not familiar with it I'll leave some links down below how to install comy UI and the comy UI manager in each of the workflow files you can find find a detailed description about Which models you may need to download and which settings you may want to apply but now let's quickly build the two workflows first the one for the image creation and then the one for the animation let's first add a load image node for the first control net and select an image from the open pose full folder with the image sequence that we have rendered before in blender then let's duplicate this node twice and select the same image from the seg and depth full folders we're going to use three control Nets if you want to use more just add as many nodes as you like then we need three load control net model nodes for open pose segmentation and depth to make our control Nets work we also need three apply controlate nodes now connect the load image node with the image input of our apply control knit then the positive and negative prompts with our first control knit for the next step we need to chain the positive and negative prompts to the other two control Nets and connect them from the last control net to the positive and negative inputs of our K sampler next let's edit our positive and negative prompts to describe the scene we want to create and select a checkpoint model I'm going with the Epic realism model which is a stable diffusion 1.5 checkpoint but feel free to use any model you like we also need to set the width and height of the image we want to create which should match the aspect ratio of the control net images from blender we leave the batch size at one because we only want to create one single image at this time we also must connect the control nit models with the apply control net node and also the other two image nodes next I want to change the strength as well as the start and end values for our control Nets the strength describes the influence of the control net on our image generation process while the start and end values determine when the control Nets will start to kick in and stop being relevant play around with these values and see the results I will also add a preview image node and then hit Q prompt to render the image you can see that it matches the pose from our control Nets as well as the contents of the scene we built in blender there is a room a door a table a sofa two paintings everything like we expected I also want to add an upscaler and a face detailer to improve the image even more so let's do it also add another image preview and save image node so we can see the results you can see that the face gets more detailed which is especially important when your you're creating low resolution images there's another trick we will use to keep the face and scene more consistent which is especially important at animations I'm going to add another Square image to our workflow this time a close-up picture of a face alternatively you can also add another lead image node and pick a facial picture of any person you like then let's add an IP adapter and feed the image into it that's just another control net which app lies the style of a specific image to the scene let's connect the model output from the IP adapter to our K sampler to make it work we also need to connect the model and the prompts to the second K sampler let's adjust our prompts a bit further and then render the scene you can see that the face stays stable even if we change our main prompt settings okay that's it for the image creation you can download it for free and play around with the settings next let's extend this workflow to render an animation as we have a whole image sequence of our control Nets created in blender first we need to replace our load image nodes with load images path nodes so we can load the whole image sequences for our control Nets select the folder for each control net right click and select copy as path then paste the path into the path input of the load images node don't forget to remove the quotation marks from the path name or it won't work let's repeat this for the other control Nets and connect the image outputs to the apply control net inputs we can then remove the old load image nodes next we need to replace our load control net model nodes with the load Advanced control net model nodes select the correct models for each control net and connect them then we add an animate diff node which is going to control the animation and make it smooth and flicker free let's use the temporal diff model for that purpose we should also replace our checkpoint loader with a load checkpoint with noise select node and reconnect it okay now we need to connect the model input with our new checkpoint node and the model output with the IP adapter model input now let's remove the preview image nodes as we'll need an animation output and finally I want to use a rather new node collection a detailer for animate diff which makes the animation super smooth and stable I won't get much deeper into it but I'll leave a link down in the description if you want to do some further reading finally let's add a video combined node to view the result connect the image output to our upscaler and face detailer and add another video combine node depending on your Hardware the rendering process can take quite some time so you may want to use another sampler that uses less steps like LCM but I'll stick to a classic sampler as my GPU is powerful enough to handle with it let's further adjust our positive and negative prompts and let's adjust the batch size to 250 which corresponds with the number of frames in our control net image sequence that we've created in blender also consider reducing the width and height of the output animation as it will speed up the creation process considerably now we're good to go so let's render the animation and here is the result as mentioned before you can download the whole workflow for free and I included some detailed notes into it so I hope you will get along with it guess that's all for today I I hope you enjoyed the video even if it might be a bit challenging at some points if it's helpful for you then give it a thumbs up and I'll see you in the next one bye

Channel: Render Realm

Views: 7,464

Keywords:

Id: L3hVYeMWIqA

Length: 19min 9sec (1149 seconds)

Published: Mon Dec 18 2023