IPAdapter V2 Update Your ComfyUI Workflow And New IPA Features

hello everyone as you may be aware the stable diffusion IP adapter plus custom node has undergone a major rewrite and significant updates in version 2 recently for comfy UI due to these changes all previous workflows that utilize the IP adapter will need to be connected to the new IP adapter pack node therefore older versions and any workflows not compatible with the new IP adapter version 2 will require adjust adust Ms the author of Ip adapter plus has Rewritten the entire code of the IP adapter package which means we'll need to reconnect everything with the new IP adapter plus custom nodes this essentially introduces A New Concept of custom nodes despite the challenges that come with transitioning to these new custom nodes and modifying our previous workflows it's great to see such significant updates today we will be exploring some of the new IPA adapter custom nodes it's important to note that the complete code has been Rewritten and as a result these custom nodes will not work with the previous versions in our workflow we will need to reconnect or recreate any existing custom nodes that are related to IPA adapter in this video we'll proceed with that task and test it out with a workflow involving text to image conversion additionally we'll will revisit some of our previous workflows and attempt to reconnect them using the new IPA adapter so let's Dive Right In to begin I have an image ready to be processed by IP adapter I also have a basic text to image setup let's start by trying out the realism style we have a beautiful city landscape and a young girl wearing a t-shirt and blue jeans that's all we need for now for the sake of YouTube's content policies we need to ensure that the image remains safe so let's proceed with this particular composition now we'll test it out and see what we get all right we have our result let's adjust the settings for the city landscape once more there we go now what we need to do is apply the IPA adapter to this image to accomplish this we'll be using the IPA adapter node keep in mind that there are other custom nodes available that also utilize IP adapter for example we have the impact pack from Dr LT data and the original Ip adapter from Mato however for this demonstration we'll be using the IP adapter from the new IP adapter version too so here we have the load IP adapter node from the new IP adapter version too we can connect it to our models just like this here I have my IP adapter unified loader that is the new IP adapter model loader for our workflow what it does is make it easy for us to use different IP adapter models for example we have the light SD 1.5 model which is the basic IP adapter model you can see it in the chat table here labeled as the basic model or the light model then we have the plus models which are the stronger ones you can find them in the drop- down menu when using the IP adapter plus particularly the strong plus model you would choose the high strength plus option from the drop- down menu this will give you the IPA adapter plus SD 1.5 models with strong capabilities additionally there's the basic model with average strength which is the medium strength model personally I usually opt for the high strength model especially for clothing restyling in animations and other purposes now let's try the high strength model here we have the IP adapter connected to this point we'll discuss this connection Point later because it allows you to load one model and have the second unified loader retain the memories without loading another model again it means that after finishing with the first loading you can pass the IPA adapter data to the second unified loader this way if you have other groups of IPA adapters you can use them without loading multiple IPA adapter models in one workflow which would consume a lot of memory by sharing the same models throughout the workflow using these IP adapter connections you don't have to repeat the process of loading another IP adapter plus model this saves memory in a buffer you can run animations or process lar e highquality images without overloading your graphic card or GPU now let's move on to setting up this will be the IPS adapter Advanced which provides more control over the IP adapter in one of Mateo's recent videos he showcased a basic node for IP adapter loading without many options for customization after watching his video I found the IP adapter Advanced version to be more suitable it offers greater control and includes features such as the attention mask with this we can have more control over how the reference image influences the generated images or videos in the new IPA adapter we have the reference image that we usually use for Styles and capturing small detailed elements additionally we have a negative image similar to a text prompt which serves as the reference for what we don't want in our generated images so we have both positive and negative images the IP and IP adapter stands for image prompt it's similar to a text prompt but in this case it refers to the positive and negative images to better understand think of the image prompt as the positive image and the negative image as what we don't want in our output for example I often use an image of a golden dress as my positive reference for various video examples to specify what I don't want in the generated image I add a negative image that features a Sci-Fi character this tells the IP adapter to avoid incorporating similar noise or elements in the output that's essentially how it works you can also explore the IP adapter noise and preview it for yourself now let's take a look at the IP adapter noise you can directly inject the noise as the negative image or preview the noise separately there are several types of noise available such as fade dissolve Grandeur and shuffles each type has a different effect for now let's focus on connecting the image directly to the IP adapter image and test the noise later so let's keep that in mind as we bring up the IP adapter and connect it to the models linking it with the K sampler that's essentially the new structure of the IP adapter now you might wonder why I didn't connect the clip Visions to the new IP adapter well the IP adapter will automatically detect the model you choose and select the corresponding clip Visions for you make sure you have your model files and clip Visions files downloaded in your comy UI once you have them the IP adapter will take care of the rest remember to download the clip Visions as well as it's essential for the process however I'm not certain if the IP adapter models will be downloaded automatically when you run it for the first time you can try deleting or removing the models file to see if it works that way in my case I already have all the models downloaded on my machines so there won't be any issues one great thing about the new IP adapter model is that we don't need to worry about renaming the model files or clip Visions files different people have different naming conventions for their folders but now we can simply use the drop- down menus and let the IP adapter take care of it this makes it easier when sharing workflows with others who may not be familiar with the file names or where to download them the IP adapter handles it for you let's try it out using different image inputs and experiment with various weight types and combined embeddings by default the settings for the IP adapter are linear and concet which work well as you can see here the IP adapter plus has a strong influence on the image and produces results that closely resemble the reference image if you want more flexibility you can try the light strength options now I don't have the specific model downloaded but I do have the medium one let's see oh it seems there's an error because the new IP adapter only reads the safe densers extensions previously I had downloaded the bin file which is an older version so even if I had the lighter weight model it wouldn't be found here it's better to download from the table list directly I'll download the SD 1.5 light safety tensor files for the Neo IP adapter loader let's click refresh and try again there we go now it should work the lightweight IP adapter models have less influence as you can see they only affect the new image to a certain extent it's a bit funny but that's how it is let's see how it goes when we don't use the negative image okay here's something interesting I've noticed if I use an image as the negative reference the IP adapter can detect the opposite attributes compared to the reference image for example if the reference image shows a person wearing long pants and I connect an image of a person wearing short pants as the negative reference the generated image will show the person wearing short pants too it's like the previous image we generated it reflects the opposite characteristics based on the negative reference that's exactly what I discovered the first time I downloaded the IP adapter when I shared this with friends in Discord groups we had a discussion about it I even tested it by using two images of pretty girls as reference images and the generated image turned out to be not so attractive that's just how it works let's try it again I'll copy the same image and generate it so imagine I have two identical images of pretty girls normally the generated image would o be pretty let's see what happens if I don't add anything else as you can see the color with the highest percentage in the image in this case the golden color has the most influence on the generated image so in this example it would create a golden dress or outfit based on the reference image because it contains the most golden areas however if I add the same image as the negative reference the golden color disappears see when you use the same image as the negative reference the golden color is eliminated this concept of a negative image is similar to a negative condition in a text prompt it's the opposite of the reference image so if you use an image of a person with a different body shape or an unattractive face it becomes the negative reference and the generated image will refine accordingly to produce a more beautiful or better-looking result so that's what I discovered with the negative image however if you connect the IP adapter noise using the same image there are different types available you can try options like fade dissolve and more experimenting with different types will yield different effects for example if I choose the fade type it will reintroduce the golden color now when it comes to the face if I use an Asian face as a reference in the image it will be less affected by the negative image issue on the other hand if I choose dissolve it produces a different result with dissolve the noise type changes as you can see fade produces a specific noise pattern while dissolve creates a different kind of noise pattern by using this pattern it instructs the IP adapter to generate an image that incorporates the dissolve effect based on the reference image essentially it brings out the texture of the dress in the generated image for example in this t-shirt the texture matches the reference image but the color is not influenced by the reference because we applied the dissolve effect using the negative image which is the same image this eliminates the golden color and focuses on the texture there are several combinations you can try the new IP adapter offers more settings compared to previous versions with we now have more control over how our generated image looks it's not just about connecting everything to the IP adapter and hoping for the desired style we can set the weights adjust the strength and Achieve consistency in Styles the new IP adapter allows us to have greater influence for example in this Shuffle we can influence the colors and see other styles emerge we can make long pants disappear and have short jeans in instead by influencing it even if I do blue jeans it won't affect the generated image because of the strong IP adapter weights applied there's a lot of underlying logic in the IP adapter and it's challenging to explain everything in one video If you have a programming background you can experiment with it and understand the logic behind it explainer videos can only convey so much without Hands-On experimentation different scenarios they absence of other images and various combinations with the image and negative image all play a role in generating different effects that's the way to go as you can see there are always different styles emerging let's try one thing for example if I have a Sci-Fi cyberpunk style image and I want to enhance my generated image to have less influence from animations or illustration Styles I can aim for a more realistic look the IP adapter advanced settings offer more options including different wage types by default it uses linear and concat adding add it have more influence in style while subtract has not much impact creating different styles tweak input in the wage type is like a formula that introduces other styles strong and medium settings produce noticeable differences in the image as you can see there's a strong contrast between the previously generated image and the weak output which reduces the influence of the IP adapter there are many settings to test including reverse in and out once we use the wage type and the combined embedding since I'm using subtract the IP adapter image has less influence if I use average or non- aage it will result in different styles as well so yeah this one has the linear and Norm average settings which brings us back to the realism Styles the clothing outfits in the generated image are influenced by the reference image leaning less towards cyberpunk or illustration Styles that's essentially how the IP adapter works in the new version V2 there's a lot to cover and I can't explain it all in one video we also have face ID and the tile which is a great feature the IP adapter acts like eyes able to see the contents of your image the tile has a wider angle of view allowing it to process the entire image rather than just a specific area let's disconnect this for a moment and focus on the prep image for clip Visions by connecting it here you can see what's going on previously clip Visions only used a square image to analyze a particular area of the reference image if you set it to the top it would start from the top and exclude the rest of the image you using the pad option adds a black screen around the square image however with the tile it works differently you can see it in action here let me give you an example I have this IP adapter for a normal image and I also have the IP adapter tile both are connected using the same IP adapter models in this workflow diagram but if I only want to use the previous IP adapter and a void reloading the second one with additional models I can set the previous one here which connects to the new IP adapter group this way I can save memory by only loading one IP adapter model let's take a look at the tile as you can see this image compared to the previous one using the prep image for clip Visions has a wider View the prep image only captures a square area of the larger image but the tile can process the entire image both the top and lower parts this custom node analyzes the image Dimension and automatically processes each part whether it's top and lower or left and right screens the output is futuristic and provides a reference for different styles it's really cool that the tile allows us to fully reference refence our image to the IP adapter in the prep image we often miss out on referencing the full body figures in the source image however with the IP adapter tile we can generate the entire image both the top and lower parts we have more control with the IP adapter advance and the IP adapter tile we can manually set the clip Visions using the old clip Visions loader or the unit however for me I don't think I'll need to do that since the new IP adapter automatically handles it for me there's no need for me to do the heavy work if the AI system can handle it on its own as we explore the new features there are many possibilities let's try using the new IP adapter to connect it to an existing workflow when I load the workflow Json file it may show that the IP adapter doesn't exist that's Norm normal and you don't need to worry you've installed the new IP adapter so you just need to reconnect the old one to use the new version here's how you can do it quickly connect the models files and model connections to the animated def fire using the purple dot you no longer need the old IP adapter custom nodes now we have the new unified loader for the IP adapter lastly you can use the reference image if you want to prepare the clip visions you can do that within the workflow if you have a large image over 512 * 512 pixels the clip Visions won't be able to cover the entire image in that case the IP adapter tile is a better option allowing different dimensions to influence the generated animations or AI images you can remove the image for clip Visions node and connect the models and the image directly to the IP adapter user the Green Dot we don't need that and we can just use the IP adapter tile for image referencing this way we can reference the styles of the entire image for the AI generated animations we're working on for example in this case I have characters with a Christmas style atmosphere the generation process involves splitting the image into top and bottom sections then passing them through the model's data let me provide a preview to help you understand what's happening here this approach is particularly useful when you have images with different proportions and dimensions that you want to fully influence the IP adapter for this demonstration I'll be using open poses as the main control net as it has a significant impact on the overall appearance of the generated characters okay that's about it for this part let me double check everything and then we can proceed some people have noticed that the recent youTube shorts videos I uploaded are more detailed that's because I've been using the motion model version 3 of LMA diff which provides the best and most detailed results despite longer processing times so let's move on we don't need the free in it at the moment so I'll bypass that step everything looks good so far if you want to use LCM you can apply the LCM Laura before the loader groups and you'll be good to go addition Ally the V3 sd15 Laura adapter can enhance your Styles so those are the three lauras I mostly use let's give it a try now let's take a look at some examples the first one showcases a red dress in various Styles different from the IP adapter reference image this happens because I chose the Strong Middle setting the Strong Middle widget type differs from the regular IP adapter since I have a text prompt defining in the red dress it heavily influences the text prompt first and then some elements in the generated image align with that prompt such as the hairstyles and the face as you can see the source video is an Asian dancer but but the generated hairstyle is not the expected hair it's completely different however we can try different settings let's go back to the default setting which is linear if we use add or normal the IP adapter reference image will have more influence on the generated result let's quickly generate and see what happens in the second example we use the linear setting combined with embed as ad this way we allow the IP adapter to influence the coloration of the outfit and the background of the main character the generated character's top part like the t-shirt is green referencing the IP adapter's outfit the white and red dress is also influenced by the IP adapter's outfit although with fewer details its style in an animation style since we need more control and additional custom notes to achieve the level of detail seen in the YouTube shorts videos so far the results are decent when compared to the original Source videos this demo dancer style is different and the hairstyles face and background lighting are all influenced by the IP adapter we can try using the lightweight SD 1.5 IP adapter loader or the high strength plus model and see how it performs in my experience with LMA diff or any image workflow that requires detailed IP adapter influence using the IP adapter tile loader is a better approach it allows us to reference an image with different dimensions whether it's a proportionate or landscape style image we can provide the IP adapter with a full view of the image instead of just using a prep image that chops the 512 * 512 reference image into parts for the IP adapter reference so far using the IP adapter tile loader seems to be the way to go look at that the results are even more influenced by the IP adapter image I'm using the strong setting and the IP adapter plus SD 1.5 model it offers a variety of styles similar to the IP P adapter reference image however it's important to note that these images are generated by Ai and not actual photographs of real individuals now that we've clarified that let's proceed so far the results have been quite impressive in the second sample the flickering lighting effect is more enhanced compared to the first sample this Improvement is due to the increased number of samples which enhances the overall quality now let's take a look at the final result where we can see the transformation from the original Source video to this particular style here we have the IP adapter reference image and the source video and the generated image Falls within this style it's quite a remarkable transformation for those who are eager to try out the updated IP adapter workflow I recommend reconnecting your IP adapter groups using the new structure simply switch to the IP adapter unified loader to IP adapter tile and connect the load image accordingly this should work seamlessly for you if you can't wait for my workflow update on patreon feel free to give this structure a try I'll be updating my open POS animated videos to videos workflow in the patreon group so you can find the details there if you prefer you can also update it on your own using this existing workflow in this particular workflow I have the free init connected which allows for more experimentation it's a straight forward an easy way to update the IP adapter notes in your existing workflow depending on your specific situation if you only need a square size of 512 * 512 you can simply use the regular IP adapter Advance model and input the IP adapter image into the clip Visions data for the image workflow that approach works well too however personally I prefer the IP adapter tile as it allows ows the IP adapter AI to have a fuller view of the image in different dimensions that concludes this tutorial I hope you found it helpful have a great day goodbye

Channel: Future Thinker @Benji

Views: 6,083

Keywords: stable diffusion, stable diffusion tutorial, IPAdapter V2, IPAdapter Plus, ComfyUI workflow, updating IPAdapter, custom nodes, text-to-image conversion, IPAdapter Advanced, reference image, negative image, IPAdapter noise, IPAdapter models, IPAdapter Unified Loader, IPAdapter settings, IPAdapter V2 In ComfyUI, IPAdapter Version 2 Features, IPAdapter Plus Rewrite, IPAdapter Advanced Settings, IPAdapter Reference Image, IPAdapter Negative Image, IPAdapter with Clip Visions

Id: sWHgc7QgPtI

Length: 27min 9sec (1629 seconds)

Published: Wed Mar 27 2024