Photorealistic Characters with IPAdapter and ControlNet in ComfyUI with Stable Diffusion

hello everyone in today's video I'm going to show you how you can use IP adapter to replicate the effect of aaker and instant ID in my last three videos I talked about aaker and instant ID it's a new technology which allows us to use a face as an input and can generate an image using the same face in different OES as well as different styles today I will show show you how you can use IP adapter to have the same effect so we are going to use a face and try to generate a character with the same face in different poses as well as different styles I hope this video helps those who were not able to install photomaker or inent ID in trying to get a similar effect before we get started I would like to say a big thank you to all of you who subscribe to the channel we have reached the the 1,000 subscribers Milestone I truly appreciate all of you support guys big thank you to all of you let's get started so these are the topics I will be covering today if you are already familiar with a particular topic feel free to jump around and skip that particular section I will be starting with a fresh installation of confy UI and I do not have any custom nodes installed I will show you all the steps required in order to install the custom nodes I will be using if you already have a particular custom node install skip ahead to the next section let's start with the first step for that we do not require any custom nde we can simply start confy UI once confy UI start you can simply clear your canvas and click on the load default click on okay then for the checkpoint I will be using the analog Madness checkpoint it gives semi-realistic outputs which is perfect for what we are going to do you can use anying checkpoint including SD 1.5 checkpoints or the poster FR and going to see if face or tree of the woman you can also try to lose up orra middle age or to shoot looking at the viewer you don't want to have an image where the character is facing sideways and only half of the face is showing of course you can add other things but at this stage you want to keep the prompt simple just to get the face of the Cara for the key sample I'm just using a fig seed if you want to copy it if you have the same model you can copy the settings then click on Q promp This is the result we have a character looking at the viewer or the camera and most of the p is clearly visible it's not obstructed by any object so let's move on to the next step and that's going to to be the second category we are going to take the space as an input pass it over to IP adapter we'll then use the face ID model to grab the information from this input image and generate a character with now if you already know the steps feel free to jump to the next step since I am using a fresh installation of comy Y I do not have the custom nodes I will show you how to install the custom nodes if you already have the config y manager you can simply go into the config y manager and search for IP adapter and install IP adapter plus version for those of you who do not have the custom nodes or you do not have the confy manager continue on watching go to your search engine and search for confy UI manager and the first link should be from GitHub rep story and make sure that the GitHub user is this exact name here open the link then click on the code button here and then click on the copy to clipboard button if you have confy UI still running you can close out all that terminal then open your confy UI folder go into custom node at the top type in CMD press enter and then type in get colon pce the link by pressing control V press enter this will clone the confy U manager and from now on we are to use the confy UI manager to install custom nodes now you can close out of this terminal go back to your confy UI or table folder and then run comy UI it will take some time to download the dependencies for comy UI manager to work once it's completed you can go back to your browser and refreshing page now you should have manager on your sidebo click on manager go into install custom nodes at the top here we have the search input box click on it and type in IP adapter press enter to search then click on confy UI IB adapter plus this will open the GitHub repository scroll a little bit down where you can see installation keep on going down you will see work those and for today we will be using the face ID work click on example directory then choose the face ID Basit we are going to start with this one now you can click on ra right click on the page once it's loaded then click on save as go into your confy UI fer and save it somewhere there go back into config UI click on load navigate to your config UI installation look for the face ID basic. Json file and click on open but at this point you may get this box telling you that there are a few notes missing click on close click on the manager click on install missing code custom node it will tell you the node that you'll need and then click on install once it's completed it will tell you to restart click on restall and it will install the necessary dependencies for confy UI IP adapter plus once it's completed simply refresh your web UI and make sure that you do not have any red node telling you undefined next we need to get the different checkpoint so the first one is the load checkpoint this one I will be using the analog mless you can use any other checkpoint since this is an SD 1.5 model when I go into the IP adapter model I will have to use an IP adapter face ID SD 1.5 version in order to get the models you can go into manager install models at the top you can type in and that and then you will have the models here now I will install this version IP adapter face ID SD 1.5 simply click on the install button on the right side next I will also need the low R and in this workflow it tells us that if you are using the SD 1.5 IP adapter face ID you will need the SD 1.5 Laura so you can simply go back into manager install model search for IP adapter and look for that exact name which is this one IP adapter face ID SD 1.5 low. save tens click on install next will require a clip Vision model so simply go to the top type in clip click on search and look for the one that says V not the G1 if you using sdxl then you would download the other version the one that says b GX in the load image click on choose file to upload go to your confy UI output folder and select the out but from the previous step now for this test I'm going to leave everything as default I will also keep the seed as one I'm going to change the steps to 20 for this model if you're using a different model and they require 30 steps or less steps you can change the number of steps then I'm I'm going to change the sampler and the schedul up to match the model that I'm using next I will click on Q promp and wait for an output now some of you may get this error that says new modual name inside face when using the IP adapter face ID it requires the inside face modle to be install so simply go to manager install custom node at the top search for IP inductor once again and now click on the name where it says confy Yore IP adapter _ plus this will open the GitHub repos story scroll down until you see face ID click on installation instruction this will scroll you down to the installation section scroll down until you see this line where it says face ID requires inside face and then there is a check this issue for help click on this issue this will open and issue log in GitHub repository go down a little bit have to scroll quite a bit for this one until you see a post like this and it will be by the developer of IP adap plus then look for the line where it says alternatively you can download the precompile version click on here this will open another GitHub repository you need to check your python version open your confy wi Windows portable folder at the top CMD enter Then type in Python window score embedded on Windows we'll have to use back slash but if you're using a different operating system you may have to change the back slash to a forward slash then type in python.exe again this is windows-- version it will tell you your python version I am using python 3.11 go back into the GitHub repository and download the version for you in my case cp311 click on the file and then click on this download button here I will save it in my confy windows portable folder back to the terminals I'm going to do pyone embeded back/ python.exe dmip install and then name of that file press enter it will start downloading additional dependencies and it will install inside face once you see successful go back into the terminal and this time type in again your python embedded python dmip install and then type in this name Onyx runtime now there's also a GPU version available for this I found add that get errors when I use the GPU version so I'm going to install the CPU version instead once it's done simply restart config UI and try to generate the image once again now for this note make sure that you're using the correct provider either CPU or GPU depending on the Onyx runtime that you install Okay so the generation completed and if it take a look at the input image and the output image we can see that it's the same person or at least from far we can see it is the same person now I've done a couple of generation just to make sure that I'm getting a good result so I've tried with a different sampler name and the different schedule this is the result then I've tried changing the empty latent image to a portrait since the photo composition is a portrait and then I've also used the V which is included in the analog Madness model and plug that in directly into the VA tood and this is the result now you may have to do more testing I've only done four and I was able to get a good result but depending on the model that you're using you may have to do more iteration in order to get a good result now in my case this is good enough if it's not for you then you can experiment by changing the weight the noise usually adding the noise bring the composition the overall composition of the image closer together and be more coherent if you decrease the weight you will see that the final output will not look exactly as the input image sometimes you may have to do that in order to get a good output image you can also play around with the start and end at steps this is called time stepping and you can start generating the face and then at the end Maybe the last 20% you let the model do whatever you want it to do now right now I've been using the face ID version one but you can also experiment with base ID in version two which gives better result now for the sake of the video I will be using this image for the third step which is trying to get these results so we have one input image in different styles if you want to have a closer look to this image you can go to my patreon the link will be in the description below and under recent post this is like a free blog I have and as you can see I am not logged in and I can still look at the video or read the post all of that is free we'll also be posting the workflows that I'm using in today's video over here so you can look at the different mes compare it and see which one gives you the best results okay so let's move on to the next step which is to introduce control net control net will give us more control over the output image and we can control the pose face movement the body shot as well right now all we've done is we've taken a base this can be the real person or it can be like we've done generated person and then we've generated an output image in a different setting this one has flowers in the background as well as a different po so you can see a different shop here if this is all that you wanted to do congratulation you have everything you need in order to generate these type of images you can now start experimenting by changing your checkpoint you can change your positive negative pump change the aspect ratio to get different type of shot change the key sampler settings and that also will influence the output image as well as the IP adapter section now for those of you who want to get the extra control over your are puty image we'll go and install with the control net section of today's workflow again let's go into the manager install custom nde this time at the top type in control net and click on search if you do not get anything like in my case you may need to change the filter here from install to all and then you will get the correct list now what we are looking for is this one auxiliary pre-processes now simply click on install once it's completed and you see restart click on restart in your terminal output you will see that it will start downloading some additional dependencies just wait for it to complete now once control net has installed you will see import times and you should see control net loaded as a custom node if you have any errors it will show as import failed here then you will have to check if you have the correct control net installed so go back into manager and try to install it again now verify if control net was installed correctly by right clicking add nude and make sure you have control net preprocessors and that you're able to add any of the N for example the line extractor we have the cany edge detector and this one will take an image so we can take take the image from our previous generation pass it over to cany Edge and from The cany Edge node we can drag the image out into a preview image press on Q and once the control n The cany Edge completes you will see an output similar to this where the cany edge detector it basically detects all the lines depending on the threshold that we've selected here and I will display it here and we can use these lines as a reference in order to generate another image but following these lines now there are different type of control net pre-processes so if you go back into add node right click add node control net pre-processes another popular one is under the normal and depth estimators and the latest model we got is the depth anything model and this one will basically take an image and output a depth map for that image so we'll be able to get the 3d effect from this depth map by the way once you add the node here the canny edge dep map or any of the other control net preprocesses the first time you click on QPR it will take a slightly longer to download the model in this case the depth anything it will download the depth anything model so in this case you can see here it's downloading the model and the depth anything is about 1.34 GB okay and once it's completed you should see something like this and you can see anything that is closer to the camera is white anything that is in the background really fall is going to be black anything in the middle will have a shade of gray and this is how we can tell the model that you want something in the foreground or something in the background so for today's workour I will show you these two preprocesses however once you understand the concept feel free to experiment with the other preprocessors as well there's a lot you can try each one of them at your own pace okay so if we look at our workflow so far we have the model the low checkpoint model and usually this model will go directly into T sound but when we are adding IP adapter this IP adapter section goes in between the model and the key sample so we can see the routing from the model it goes over to IP adapter section and it goes back into the key sampler when working with control net we will go from the positive up here into control net and then from control net we'll go into the positive of the key sound block let's start building the workflow I'm going to make some spaces in between here so in between the positive prop and the case h I will double click on an empty blank space and type in control net and select the apply control net node this node will take a conditioning the control net model as well as an image and it will output a conditioning so the output we can simply take this conditioning and put it in the positive prop of the key sound plot this is fairly easy then the conditioning at the top will be the positive CR the control net model will take it drag out release and select control net lud if you just install control net you may not have the model let's go into manager this time click on install models at the top here you can type in control net press enter and the list will give you all the control net models we can use if you are using an sdxl this model then look for the Bas here that says sdxl and in order to follow along you want to select the cany and DEP model simply go on the right side and click on the install button for me since I am using the analog Madness model which is an SD 1.5 model I've installed the SD 1.5 version of canny and depth model so you can see here install once it's installed you don't have to restall with this one simply click on refresh and it will be available in your drop- down here so let's start with the cany edge model simply select it and you can see that this control net is going into control net for the apply control net node okay so I'm going to make some space here because the name is bit long so we go from low checkpoint to POs promp to the apply control net and in between we we just have a nde that is loading the control net model now we have one more input in the apply control net and that is the image this image it's not this final image that we have if we are using the cany model this image will have to be the output from The cany Edge pre-processor new mod so to clean up the workflow I'm going to delete this section and then I will add the cany edge pre-processor make sure that you are adding The cany Edge preprocessor one and not the normal cany one you can also find it by clicking add n going to control net pre-processor line extractor and selecting cany Edge on there so the output of this canic Edge will go into the control net the input one you can drag out select a load image and then just choose your input image in my case I'm going to use this as the input image I want to keep the same shot so the same perspective but I want to change the style to a different one so this one is from instant ID and we want to have a similar effect so you can click on choose file to upload go into your config y installation config UI output and select the image from your output folder you can also right click on the save image node click on copy clip space and then go into the load image right click and paste from the clip space okay now since we wanted different sty one option is to change the checkpoint let's say you want something anime like you can select a model that is can give you anime like style or you can keep the same checkpoint and you can change the postive prom so for the first test I'm going to change the positive prompt and then I'm going to experiment with different models and show you the result okay so I'm back and uh this is my input image the first generation just using the face ID and this one was adding the control n the depth map this one is decreasing the face ID a little bit and decreasing the control net a little bit and then I changed the model into an anime model and this is the result now for the second route I have the base ID as one and the only thing that I change is the model so right now I am using an animate model and I've also changed the pr by changing the promp I can influence the overall composition now for this one I've decreased the phas ID weight and again more in the last example I also done another example using a different phas and you can see the result here in different Stars let's do a quick recap this section is pretty much standard you will start with a low checkpoint go into your positive negative and you have your empty latent image then since we want to have a similar face we'll go from the model into an IP adapter section and here you can use the face ID model in order to get the correct face and the output of the IP adapter goes into the key sampler as a model the next part is going from the positive prompt into the control net section the control net section is going to give us the overall composition of the image and that conditioning will go in to the key sampler as a positive then the rest of the workflow can be anything that you want so right now I only have one single key sample which is being decoded and saved and you can go into and upscaler web flow a different key sample with a different model as well just to influence the final amput all right so thank you for watching I hope this video was helpful I will see you in the next [Music] one

Channel: Code Crafters Corner

Views: 3,227

Keywords: comfyui, Code Crafters Corner, CodeCraftersCorner, Photomaker, Instant ID, AI face generation, ComfyUI tutorial, IP adapter workflow, Control Net tutorial, AI-generated characters, AI models, Custom nodes, AI tutorial, ComfyUI manager, AI workflows, Photorealistic characters, AI art

Id: v5qeEZKsArs

Length: 26min 2sec (1562 seconds)

Published: Wed Feb 07 2024