IP Adapter Tutorial In 9 Minutes In Stable Diffusion (Automatic1111)

continuing with our control Net series we'll be taking a look at IP adapter which is a model designed to allow images to be generated with a reference image allowing us to extract and transfer elements from one to another such as clothing styles faces and color schemes this can result in the ability to create images like this where we can mix and Blend images and prompts to achieve something new alongside some other tricks which I'll cover in this video and if you want to access video files we safe for work images and your name included in the video then consider supporting over on patreon for $7 per month which helps the channel immensely but enough Shilling let me give it to you bite-sized so I won't label too long on what IP adapter is as the technical explanations are often less helpful than just seeing the tool in action so you can figure out how it may be useful for your own workflow IP adapter stands for image prompt adapter and it's an adapter used to achieve prompt capability for the pre-trained text to image diffusion models list in practice means to add an image alongside a prompt to influence the generated image there are different kinds of Ip adapters you can use and in this video we'll focus on the IP adapter sd15 models and the IP adapter face models in order to install this navigate to the link in the description which will bring you to a repository of various models and navigate down until you see both the IP ad adapter sd15 and sd15 plus models download the save tensor files and save them within your control net extensions model folder you will also want to download the safe tensor IP adapter 4 and plus face models from hugging face and the IP adapter face ID and plus model this face ID model is required for the face ID pre-processor which we'll be using later on and be sure to change the bin file extension to p and Save in your control net models folder then navigate to control net within your web UI and press the blue refresh icon next to the models and this should bring up your IP adapter models your pre-processors should be selectable and will install when running them for the first time as this is a continuation of our control Net series be sure to check out the previous videos for guidance on how to install if you're new now that we've got everything installed let's explore how IP adap can influence image generation I've generated this image of a simple yellow rubber duck using the prompt photo of a yellow rubber duck and some enhancers using the dream shaper checkpoint we get a fantastic result but what if we wanted our duck to be floating on water well we can modify our prompt to say photo of yellow rubber duck floating on water which will give us this new image but we can use IP adapter in combination with an image of some water to achieve the same outcome but with very different results to achieve this I'll drag my image of water into control net and ensure I've selected the IP adapter clip sd15 preprocessor and the IP adapter sd15 model if I hit generate we get this new image that contains a duck and those signature ripples from our reference image but it's more water than duck and we'll need to make some adjustments in order to make sure that the duck is more prominent in our image now if I were to set the ending control step to 0.2 it means that the IP adapter will influence our image for the first 20% of our total sampling steps this can help to reduce the prominence of water from our reference image and allow those ducts to become more prominent throughout the image as seen in this new photo and we can reduce the impact of the water even more by setting our control weight to 0.25 which will also help the duck appear more prominently in the photo than the water while still trans transferring the Ripples and overall style now if I remove the reference to water in our prompt and replace the water image with random images we will get different results such as with these jelly beans which give us a collection of ducks or even this duck standing in the field of flowers using this reference image of a field you can achieve some interesting combinations simply by combining images and prompts now looking at something different we can use IP adapter in image to image to transfer portions of a reference image to our existing image using the de noising strength to control the amount of change for example we can take this duck into image to image and use a reference image of a swan with control net and a noising strength of 0.5 removing any references to our duck from our prompt this results in a duck one hybrid and our background still maintains that brown earthy color scheme and overall composition but brings in water which you would expect a duck to be surrounded by I also tried this with a delicious Greg sausage roll and the results were very interesting as we're getting both elements of our original duck with the lighting direction and overall wood theme but the duck is now attempting to mimic a sausage role you can use the in painting function to transfer a reference image to a portion or a part of an existing image as an example our mask this duck and upload this photo of a baby chicken to control net and hit generate on 0.79 denoising strength we now get something closer to a chicken than a duck but it's not perfect as we're using two completely different angles for our subject the last style transfer I'll show is a change in the background and we won't need to master the background as we've already mastered the duck and we can simply press the impa not masked option under the mask mode to mask everything except our mask area this results in our background changing to a small during landscape with a duck chilling untouched in the foreground of the image now let's move away from the style transfers and focus on face swaps which is something I'll briefly cover as I want to run some more tests in my next consistent Faces video you have two types of pre-processors you can use to do the face swaps IP adapter clip will take the entire reference image and use it to modify your generated image whereas face ID will crop out only the detected face from the reference image and apply it to your newly generated image the IP adapter face pre-processors only work with the IP adapter face ID and plus face ID models while the IP adapter clip pre-processor only works with the IP adapter fullface and plus face models also be sure that your pre-processor corresponds with your model such as using a plus pre-processor with a plus model for the sake of this example I'll be using the IP adapter face pre-processors as they make the process Easier by accepting any reference image for us and finding the face which saves us having to crop it out manually starting with the IP adapter face ID I'm leaving the prompt field empty so we can see the impacts on this random image I've generated I've imp painted the face area so only our car's face is changed rather than the whole image and I've set the den noising strength to 0.75 we do get the face transfer being applied to the face but the effect is weak and does a good job of blending our existing face with our reference image and next I tried the plus model that we got a much stronger effect than if we were to just use a standard model although it didn't look entirely like Brad Pit I managed to improve the results on Terry Cruz by adjusting my negative prompt and setting the denoising strength to one alongside setting control net to resize and fill the final use case we'll briefly explore is transferring cloes from one image to another using IP adapter this is done by uploading a reference image with the clothes you want to transfer and then an image of the subject whose clothes we will modify we will need to use in painting to mask the area of clothing on the subject then play around with the den noising strength until you get something you like I went with 0.75 this can vary so experimentation is needed to find the right value I'd also highly recommend setting the in paint area to only mask and setting control net resize mode to sides and fill also make sure that your mask is covering enough of the subject so that the entire clothing item can be generated and you don't have faded areas especially around the wrists for long sleeve clothing items you should end up with something like this which swaps out the mass portion of the clothing with the reference image using IP adapter but hopefully you found that useful for learning what IP adapter can do and if so be sure to drop a like so others can find the video and subscribe this is byze genius and I hope you enjoyed

Channel: Bitesized Genius

Views: 19,968

Keywords: ip adapter, ip adapter stable diffusion, ip adapter automatic1111, ip adapter controlnet, ip adapter face, ip adapter face id stable diffusion, ip adapter face swap, ip adapter tutorial, ip adapter controlnet tutorial, ip adapter plus face, ip adapter plus, ip adapter face id, Ip adapter sdxl, ip adapter stable diffusion a1111, how to use ip adapter stable diffusion, ip adapter stable diffusion face, ip adapter stable diffusion face swap, ip adapter face id tutorial

Id: _-_vRtEMyUU

Length: 9min 1sec (541 seconds)

Published: Mon Feb 26 2024