ComfyUI, how to use Pix2Pix ControlNet, and Animate all parameters and prompts for a dynamic result

Hello friends today I want to show you how you can convert your image with simple prompt to Outer scenes and some different effect okay here this is my main image okay and I turned it to fire or Christmas or night or a snow or a spring or changing the brick material okay uh for doing this process I use the control net peaks2px in this video at first I am trying to provide some information for beginners and after that I will provide some information for more professionals that are familiar with confi UI or some extension like was note okay so at first let me show you that what is Peaks to Peaks okay it's a control model and it can get an image as input and after that it can analyze your prompt and make a condition based on your prompt for that image for example on this sample you can see that with this prompt they make it on fire and after that you can see that the house is fired or for example on this prompt you can see that the prompt is make it winter and it changed the image to the winter so Peaks to pick scan understand and can and can analyze and can figure out that what is in the image and for example it can detect the ground building or outer parts Okay so with this way it is a little similar to semantic segmentation but still it's different because it can add some effect to them and also it's not very accurate and it's not very high quality but I think it's very interesting to use and for example here you can see that they try to change this man to Iron Man and with some simple prompt it's not very bad because they didn't use any outer control Nets and working and having some experience with this control net is very interesting I think and you can see that uh the best way for working with control net is writing simple prompt okay and you can write your prompt as action for example make it on fire make it winter or turn it or swap it something for replacing uh is this this kind of prompting can working very very better for Peaks to picks and here is another reference this is the main repo and Main paper and also you can see that here are some other sample for example here swap sunflower with rose or add Firework and it can make some changes in image or for example this one make his jacket out of leather and I think it could be very very improved it's very interesting okay so in comfy UI how we should use that and what are the important rules Okay so let me move this image here okay because they are my samples and this is my reference image and this is my saved image okay and this is my work follow for loading the control Nets okay let me try to explain that and after that you can figure out that what is this at first I load a checkpoint I have positive prompt and negative prompt for example here on my positive prompt you can see that I write make it on fire and a smoke and then I sent them to a control net this time you can see that I used the advanced control net okay this is a new node and if you update your coffee right in you can get this control net it's very more powerful it has two connections one connection from negative problems and one connection from positive prompt and also you can adjust the start percent and also the end percent and I use this control net control net v11-e e e Dash sd15 ip2p Okay and just please pay attention to the to the E okay because all the experimental control net models are defined it by E okay so it's not a final model it's still experimental but it's interesting and I sent the control net to the sampler here I have an empty latent okay and this is my vae that I use that for vae decode and also I can show you that how you can use the image with image to image after that and it's very easy and this is my image that I sent it directly to the controller and the control net ip2p or pix2px it doesn't require any preprocessor and you can connect directly your image to the control net okay for example here you can see that I write this prompt and let me do some adjustment just the CFG and here I have written some sample prompt for example make it winter night Christmas okay and decrease Mass has some more weight and let me cue my prompts okay so you can see that this time it's dry to make a night scene for that building add some more tree elements and also add some lighting let me bring the reference near that and then you can come I think sometimes the result could be interesting and this is the reason that I try to share that with you and for example simple prompt to make it winter without night and Christmas okay so you can see it's the building in the daytime but on the ground or a snow still we have the same trees but we have the snow effect on top of them and yes also it's interesting or for example swap one with dark brown brick tile and this time excuse me I paste them in wrong place okay qpro and this time the dark brown brick tile or with lover with and you I can see you that way I did I can show you that why I did that okay because you can see that with this prompt it completely changed the environment okay and I think still maybe I have to reduce the value okay no problem if you lost the environment also you can play with CFG and if you change the CFG value the result will be more similar to the image and also let me change the seat mm-hmm okay so uh working with this sump with some parameters are very important in this control net and if you don't pay attention to them maybe you cannot get a good result okay uh at first pay attention to the width of your product okay [Music] for example then you should pay attention to the CFG number and if you check the control net website you can see that we can adjust the weight of this effect by tuning to CFG and the CFG is very important for this control Nets for example on this prompt you can completely understand okay let me remove the video of my prompt and change the CFC for example to number three okay it's going to turn it to fire and a smoke okay and now if I increase the CFG value then you can see that the effect of the fire will go also to outer parts of the building okay and for example with some higher CFG you can see all the house is on fire and if I again if I change the CFG value to some higher the amount of the fire will be increase in the generated image Okay so it's very important it's very very important to pay attention to the CFG value and also David of your prompts okay uh the next thing that is very important is writing the true seat okay for example let me change my CFG to five or six and Q a prompt okay for example here with the seat number of 16 you can see that the fire is it started at top and also some middle part of the building is on fire and if I change the seed value [Music] for example to one you can see some different position of the forties fires okay and finding the right value and a good value for your seat is very important for working with this effect okay so here you can see the CFG number is fixed on six and also just I am trying to change my seed okay for example in the seat of number two the result is interesting but it's very high so if I like to get some lower a strange of fire then I have to change my CFG value okay but the position of the fire and other things is defined by seed okay and just changing the CFG value can help you to adjust the fire amount so this time for example with CFG number of 1.5 only the sky is yellow and there is not any fire but after increasing this the CFG slowly you can see the fire Okay so exactly with this technique with finding the right seed value and after that play with the CFG I made this animation foreign for example the fire position is correct let me go to the next frame oh okay and if I go frame by frame you can see after increasing the CFG value the fire is going to grow on the building and it will cover all the building on fire so it's only with the rule is playing with the CFG and also the prompt with okay so let me show you another sample for example with the CFG number four okay I want to change the video of my prompt okay so the weight of the prompt is changed and you can see that the building the color of the building is more similar to our reference okay and this is very also it can be useful and it can be important and here from this image I sent this image to vae in code and also I can use this image as latent image so this time the empty latent image is not working and it's something like image to image technique but the denoising is about fun and in this condition if I run this prompt you can see there is not any effect of the fire or a smoke on this building so if you are using this technique for example image to image you should use very higher value for example if I increase the amount to something like this maybe after that we can see the fires and this house okay what is the benefit of this technique sometimes sometimes it can help you to get more colors similar to your reference it's on nothing more okay and if you decrease the dinos amount after that you should increase the width of your prompt or increase the CFG so you can understand that what is the true balance uh okay now the beginner part is finished and I want to explain some technique for animating the parameters in comfy UI okay uh for animating the parameters for example for animating the CFG or animating any other parameters it is not related only to CFG value and the parameters in comfy UI could get input and also you can make some animation video for the input okay uh let me try and let me explain that to you at first you should go to manager and make sure that you have these three extension okay the first one is efficiency node let me show you the control that is very necessary also number 15 efficiency note and also I will place the links in my YouTube channel and that's not Suite wsa not Suite it's very very important and number 55. python goes adding something like that and also this extension is very useful for animating the parameters okay let me show you how I am doing that okay at first let me connect empty latent keyword prompt on CFG number four okay okay great so the safety like two I think could be better for this condition uh this is my sampler and this is a value at first I will right click on the sampler and convert click on convert CFG to input so instead of defining the CFG from a writing value here the CFG is changed to input and I can connect another input to the CFG and what is my input okay uh if you install the efficiency node you can go to the efficiency node and after that go to simple eval and select the evaluate fluids okay and you can see this node okay this is exactly like the yellow one that I have okay oh let me add that again efficiency node simple evaluate evaluate fluid okay here you can see that we have three parameters a b c these are our variable and here is a python expression and here we can write an expression and play with these parameters okay number a number P and number c for example you can plus A to B after that minus from C and divide them by two any any kind of expression you can write here and if you want to have good information about all the Expressions if you load this node here you can see all kind of the Expressions that you can see that you can write and I think it's very powerful and useful okay here I have the evaluate fluid and for example I want to change the variable number B to input convert B to input so I can Define the B value by a input node if you right click and go to the utility then you can see the Primitive note okay this is a default node from confi UI and once that you connect the Primitive node to any kind of input it can get the value of that input and you can change that value from the Primitive node and in primitive node there are four useful parameters for example here I can start with 0 but I can change this parameter to increment okay and each time that that I am pressing the queue you can see that the value here is increasing okay three two four five six okay so it can work like a counter for us this is a trick that I am using and actually it's something like a loop yes and for example if you press this uh check box okay automatically Q prompt when the queue size hits zero uh if you check this and press Q it will continuously increase the number okay so it will it could be some infinite Loop so for changing the CFG value in animation mode I use this technique here I place the Primitive note for B and here I have the a the starting CFG for me the event I will use the a value as the CFG and I think here 2.5 could be useful and uh what is the c number okay here is my expression okay I wrote a Plus B divides C okay so it's the B and it's the C and for example if I divide the value that is coming from the B with four after that in each step the CFG value will be increased for 0.25 okay and if I change that to number two in each step the value will be here increase about 0.5 and after that I can connect the fluids to the CFG okay and here you see that I used a note the name is show text and it's coming from the extension number 55. okay and also it's very useful so it can show me the output of this node and I can make sure that I am on the right track so I'm going to the string and using this one show text okay so I can delete my old notes because I explained all of them to you and now you understand what is the workflow so let me Cube prompt now The Prompt the CFG value sets to 3. it was started with 2.5 and after that it plus about 0.5 and the final amount is three again if I press the Q it will be changed to 3.5 okay you can change this amount with adjusting the c number it's very easy and if I press this stick and press the Q prompt automatically it will continue the generation and here you can see that each time the value is increasing okay and if I have some higher value some higher CFC value then I have some more effects it's all okay and this technique this technique combining these notes is very very useful for animating many kind of parameters in com vui maybe I think in the future or now there are some other ways but at this moment I am using this workflow with combining the this node from efficiency and yes and this note from the extension from number 55 python goes to show me the value now it's seven point five and now it's eight okay so remember the CFG number eight and I will cancel the process okay this was the image right click copy clip space and paste it here okay I want to disconnect this node and add a primitive note utility primitive and here I want to write the eight so with cfga exactly it should generate same result okay yes this is previous image and this is the New Image and you can see that exactly the CFG value for example was 8. and now it's 8.5 but it is not connected okay this technique is very useful okay but how you can animate a parameter in your prompt for example now the weight of this prompt is 1.25 and how you can animate this number okay let me show you this time I used a node I used two node from vas Suite w a s Suite the first one is text multi-line and the second one is text concatened excuse me for me okay here I split my prompt okay for example here I write make it and prompt with this icon okay and here I have again a note from I forgot that efficiency note yes and again here I have a primitive with the increment okay and here I have another text line and I close that so this text read this number that is coming out from here and merging with this parameter is equal with this structure okay so here I want to add another note this node is very useful show text okay and you can see make it on fire with this number 0.92 exposition okay I write that here and this time I want to add a smoke Q product okay so you can see it is trying to combine all of our text and numbers together so with using this technique every time you can animate the value that is here okay now you can see each time that you are pressing the queue it can increase some amount and how you can Define that amount from here from this expression and if you change the value of the division for example to to 20. each time it will increase more you can see here and if you use this text as as an input to your clip input then you can send that to your conditioning okay this is the this is the name of this node is clip text encode okay let me create that from zero clip text encode instead of writing here you can right click and click convert text to input so if the convert text to input is activated then you can connect the strings from here to here okay and also the clip text in code needs some inputs the input is clip and you have to connect the clip to here and the conditioning should this is the positive conditioning and it should go to the control net positive it's all so now each time that your doing the Q prompt the amount could be increase David of your prompt everything so uh with this work follow easily you can animate prompt and limit the parameters and many other things in config UI again this workfollow is available on my patreon and if you want to support me please join me to my patreon don't forget to like And subscribe and if you have any question please let me know I will try to answer the comments also if you want I can give you the consolation over a video call with our with hourly rate of 60 per hour and I can learn you directly or I can help you to solve your problem okay thank you see you

Channel: Arch_AI_3D

Views: 6,841

Keywords: AI, stable diffusion, AI architecture, ComfyUI, ControlNet, architecture, artificialintelligence, Tutorial, ai robots, artificial intelligence, ai video, pix2pix, pix2pix tutorial, pix2pix gan, pix2pix stable diffusion, ip2p, ai tools, generative ai

Id: tyPt7ge2VO0

Length: 37min 26sec (2246 seconds)

Published: Mon Jul 31 2023