A1111: IP Adapter ControlNet Tutorial (Stable Diffusion)

hi Seth here and welcome to the channel you know images can say a thousand words I am talking about the image prompt adapter in control net it is a very powerful tool if you know how to use it take this AI generated face for example by using multiple control net models I can create a person and a background around the face in whatever style I choose or I can take a photo of a young woman and and change your age even the type and color of the hair it even works with digital art where I can change the elements in the art for example making the character wear sunglasses or change the outfit to a cowboy all this and more just using control net and automatic 1111 let me show you the workflow and technique for using the IP adapter model [Music] all the links will be in the description for this tutorial you need basic knowledge of stable diffusion control nit civit Ai and automatic 1111 I have made a video on each of them and you can find them on the channel homepage under the playlist stable diffusion you need to download these three IP adapter models from the hugging face link also download the IP adapter plus face model from the second link this is a dobin file you have to rename the extension to pth remember that always download download docin files from trusted sources only I double checked the IP adapter GitHub page and this is a supported model so it's safe for the tutorial I would use the Bas sdxl checkpoint from hugging face and other checkpoints from civit Ai and based on stable diffusion 1.5 one of the checkpoints is Rev animated version 1.2.2 the other checkpoint is realistic Vision version 5.1 from hugging face I will use this checkpoint for inpainting so download both the standard and inpainting safe tensor files also download the relevant vae safe tensor files for the Bas sdxl and these two checkpoints after everything is downloaded load up automatic 1111 make sure it's updated and control nit is installed basically the IP adapter model is an image prompt model for text to image-based diffusion models like stable diffusion and it can also be used in combination with other control net models in this tutorial I will showcase four examples in the workflow to use this control net model effectively via text to image and image to image including in painting for the first example I will regenerate this reference AI image in s D XL then I will use positive prompts to add elements to the final image let's start by adding a simple positive prompt like wearing sunglasses I won't be adding any negative prompts here negative prompts will be used in third party checkpoints I am enabling the high resolution fix only for fixing artifacts and won't be upscaling to do this set the upscale by value to one also we'll be running the image through the sdxl refiner in control nit select the reference image and the IP adapter for the pre-processor select the sdxl clip and change the model to IP adapter XL now select the same image and enable the second control net I want to use the open pose here this will give us more control over the subject's body and face angle so I have taken a base image and created a new image Art while adding the sunlasses to the character this is more accurate than the reference image option for control net let's change the prompt and make the character wear a hat you can also use the same seed for consistency let's try again by changing the prompt to a cowboy hat with sunglasses see how accurately the AI created the Hat sunglasses and even the Shadows correctly you can use this method to create consistent characters in different scenes I purposely took a bass image with a complicated background to test how the control net model would handle the generation there's one more trick I need to show you it will help you understand how the AI reads the image for the Regeneration for this to work you need to download the sdxl models for the IP adapter from the second link given earlier specifically this one go to image to image and load the same base image if you click interrogate clip it will caption the image and give you the propt for it using blip it may take some time when using it for the first time as it will download the blip model which is around 850 megab check the command prompt to see the download progress I will do the same setting as before but in image to image also I will not use the refiner here as it won't work choose resize by with a value of one make sure that the base image is SD XL resolution in the prompt let's change the subject to a woman I created it in batches because without using open pose the results are inconsistent to fix this activate open pose full and control net and regenerate the image now you get consistent images you can use one of these images as a base and build on it this technique is easy and you can change the subject without imp painting painting for the second example I will use a checkpoint based on sdxl 1.5 this part of the tutorial involves using in painting and is image to image first I will take a base AI image and change the blonde hair to a curly one from another base image second I will show you the workflow to imp paint the face from another base image for all in painting you should download the checkpoint fine tuned for in painting it just works better also if the checkpoint does not have vae inbuilt you need to download and select the appropriate vae if it does then just select automatic for the IP adapter and control net tick upload independent control image in this case I am selecting another AI generated image of a woman with purple curly hair it's essential to select the face plus model otherwise it may not work I am fast forwarding the video for in painting you don't have to be very accurate while in painting I will avoid both earrings and her left ear but I will make sure that the entire hair area is covered including the hair Shadow on her forehead even though prompts are needed for this checkpoint it's important that you leave it blank when I imp paint the face I will input the negative and positive prompts if you input any props here it might alter the hair color from the IP adapter image just fine-tuning it a little more to cover the forehead this turned out pretty good for the IP adapter image I will use a portrait closeup of a 60-year-old woman generated by AI over here I would use the default negative and positive prompts for the realistic Vision checkpoint the face morphing is not done correctly if I don't use any prompts you can see how fantastic and easy this technique is it has not swapped the face but somehow just morphed it using the bass image and the results are spot-on for the third example I am using the Rev animated checkpoint where I would take an AI generated Vector of a house and use it as an IP adapter based image to regenerate an anime style art using this technique you can manipulate the art in various environments and weather conditions I am using the plus model of the IP adapter you do need to have some prompts based on the Rev animated checkpoint so I will use their default negative and change the positive prompts as per requirement let's first try and generate the image without adding any extra elements you can see how the anime style art is accurate to the base image now what I want to do here is change the weather and environment let's add a beach in the ocean okay this is good but too similar to the base image The Prompt did not have that much of a drastic change reducing the control net w is one way to make the prompt more prominent a value of 4 to 5.5 works if you go too low the image would be nothing like the base image this is much better it is the right balance between the prompt and the reference image used in the IP adapter let's try snowy weather with the same settings perfect I tried to change it from day to night it does add the moan but without imp painting this is the best I could get the sky is a darker shade of blue the lights are on and there is a moon let's check out some more variations I like this one with the big moon adding rain rainy weather and overcast so it has not only made the surface wet but also added some extra clouds while maintaining the image composition okay let's try one last change beautiful the last example is fascinating with this model you can take a face and create the whole body and environment I tried it an anime and a realistic photo and both worked great this works best for creating graphic novels Comics or any artwork requiring consistent characters in different scenes for this to work you need the plus face model and a second control net image using open pose The Prompt was a man wearing casual shirt and jeans Time Square New York City let's try a realistic photo and a simple prompt a female in casual clothes in the park I will just change the IP adapter image I hope you learned something new in stable diffusion thank you for your support we now have memberships enabled in the channel where we would share resources related to specific videos and members only Community posts for this video you will get all the base images I used for the IP adapter and a PDF file with the prompts to try the methodology yourself some videos like the comfy ones will have Json files for the workflows besides by joining you would support the channel Beyond watching and liking the videos until next [Music] time [Music]

Channel: Control+Alt+AI

Views: 36,270

Keywords: stable diffusion, controlnet, ip adapter stable diffusion, stable diffusion controlnet, ip adapter automatic1111, ip adapter sdxl, ip adapter, stable diffusion img2img, controlnet tutorial, automatic1111, controlnet inpaint model, automatic1111 controlnet, controlnet 1.1, controlnet automatic1111, controlnet stable diffusion, controlnet inpainting, sdxl controlnet, sdxl controlnet models, automatic1111 stable diffusion, controlnet sdxl

Id: KHm5Q5TfNvE

Length: 24min 36sec (1476 seconds)

Published: Fri Oct 06 2023