Controlnet Open Pose Stable Diffusion Tutorial In 7 Minutes (Automatic1111)

one of the most popular control types within control net is open pose a model which lets you turn a reference image into a control map which can then be used to influence the pose of characters within your image including the fingers and face this video will be exploring what each pre-processor does and some tips for getting the most out of this feature be sure to check out the installation and overview video for control net with links in the description but let me give it to you bite-sized so the first thing you'll want to do is install control net and I've linked a tutorial on the Basics to help keep topics separate once you've installed control net you will want to ensure you have the open POS models installed this can be done by navigate into that hugging face repository that I've Linked In the description as well and draging the models. pth files into the extensions model folder you will also need two additional files for the animal pose and dense models which are Link in the description and note that the dense model file goes into the stable diffusion models control net folder rather than control Net's model folder folder as shown on screen control net will download any missing pre-processors so don't worry if things take a long time to run when processing an image for the first time and any downloads being processed can be seen within the console window which is that black window that appears when you boot up the webui so the first pre-processor we'll look at is open pose which is a standard pre-processor used for capturing the base skeleton of a subject of an image without the face or fingers if I generate a control map you will see the resulting output from the reference image and this is good for capturing the overall pose of a subject while leaving elements like the hand and face for interpretation if I generate an image it should copy the pose while interpreting a facial expression and with a basic prompt our image does have some artifacts but overall is getting the posture correct open pose fall is used for capturing the base skeleton alongside the structure of the hands and facial features in this control map we now have these white dots representing the face with eyes nose and mouth while the finger joints are represented by these blue dots this pre-processor is good for capturing everything including the facial expression and hands to help improve the accuracy of a generated image to the reference images likeness and if I generate an image then it copies everything including the finger placement and face DW open pose is another pose detection model which is far more accurate than open pose full capable of capturing more details of an image with greater consistency and better recognition if I generate the control map something you may notice is that we have a much more accurate image with better hand placements than before when using other models if I generate an image we should get a far better result than when we were just using the open pose model due to the additional details we're capturing with this model open pose face is a model used for capturing only the skeleton and face without the hands and this may be useful if you're getting bad quality fingers and you want to manage those separately the control map reflects this by giving giv us the overall body and face in a way that captures the reference photo accurately generating an image gives us the results we expected as we're copying both the pose and phase for our generated image with stable diffusion interpreting what the hand could look like open pose phase only is used to generate a control map of only the face of the reference image and nothing else this is a super handy feature to have because you may only want to capture a head placement for your image the control map captures this and we can see the head look around in the same position as a reference image although you may want to increase the pre-processor resolution to get a better definition of those individual points if I generate an image we can see that the head is in the place we specified but other details like the body are left for interpretation which can lead to some interesting results open pose hand is used for capturing the body and hands of a subject without the face and is useful for leaving the face to interpretation or trying to get the anatomy accurate to the reference image if I just generate a control map we won't get the same accuracy as the DW model but we will only generate the portions of the image we need which may be useful for adjusting the skeleton as we have less elements to worry about and if I generate an image you can see we have a good amount of accuracy to our reference photo with the face being interpreted but the hands being in the correct Place despite some artifacts animal open pose is a new model which can be used to capture the pose of animals exclusively which wasn't possible before by using any other model which will gear towards human subjects if I generate a control map using this image of a cow as a reference you can see how the overall pose is captured nicely if I generate an image you can see we get a pretty accurate placement to the cow replicating that pose in our reference photo and this can work for a number of animals but results when generating images may vary greatly such as with this chicken was turned into an absolute Abomination it seemed to struggle on more complex subjects like an octopus where if it couldn't distinguish the individual arms and instead and gave me a bunch of lines and triangles but the outputs weren't too bad so worth experimenting to see what works for your workflow while preparing this video I noticed that another pre-processor receiv received Support called dense pose this is a pose estimation model which provides a representation of the surface of a subject similar to other models like depth normal and pose if I generate a few images you may notice that it tries to replicate the pose of the control map as much as it can the benefits of using the dense model is that you get information about the 3D surface of a subject instead of just the 2D body joints while benefiting from using a pose model rather than something like depth or normal now that we've explored what each pre-processor does is worth looking at an extension you can use to modify the control Maps we generate so we can fix issues as these automatically generated Maps AR always perfect and perhaps won't accurately represent the type of image you want to generate my favorite extension when using the reference photo is SD webui open pose editor which allows you to modify the entire skeleton from a control map including adding additional joints multiple skeletons and uploading a reference image which you can use to line up your skeleton perfectly now a use case for pose could be to correct Anatomy issues within images such as hands where fingers are missing to try and get a better result you can do this both in text to image and IM image but I'll be demonstrating this in text IM image this can be done by first generating an image that you like but if you're not a fan of the way the hands turned out then take the image into control net and generate a control map using one of the pose options from there let's open up our pose editor and it's really just a case of adding in finger bones for the control map and making adjustments as we go along so when we go to generate our image we can use control net to drive the anatomical change it may take a bit of tweaking to get right but once you get the fingers into a good place you can regenerate your image with control activated and it should vastly improve the results this will also work in image to image using control net to drive to change and in painting to master hands for modification but hopefully this video has been a useful reference for using open pose in control net and if it has like the video so others can find this resource subscribe and a big thanks to the supporters of the channel this is bite-size genius and I hope you enjoyed

Channel: Bitesized Genius

Views: 3,853

Keywords: stable diffusion extension, stable diffusion extensions, stable diffusion extensions best, Stable diffusion extensions install, a1111 stable diffusion extensions, stable diffusion extentions best, stable diffusion extentions install, a1111 stable diffusion extentions, essential stable diffusion extensions, stable diffusion automatic1111 extensions, controlnet stable diffusion, stable diffusion tutorial, stable diffusion tutorial 2023, controlnet stable diffusion tutorial

Id: kT96mgrtQFU

Length: 7min 27sec (447 seconds)

Published: Tue Jan 16 2024