Mastering ControlNet on Stable Diffusion: Your Complete Guide

hello there in this video I want to go over control net and stable diffusion what it is how to install how to use it so let's go ahead and start first to look right here I'm using aftermatic 1111 installations for the control net we're going to use it as extension control net is neutral Network model that controls stable diffusion models you can use control net along with stable diffusion models if you need it any notice right here because we will use a stable diffusion models on the top as as we're going down below we will use the control net models so also speaking about stable diffusion let's go and see how we can install it so we can use it you can approach this two ways one you can go directly to the GitHub and download it there I Stand Alone to the stable diffusion or as the part of the magic 11 and 11 installation this is not hard to do but the problem with this you will need upkeep to every new revisions and go check the easiest and most recommended way to do it is going to your stable diffusion go to the extension tab an extension tab if you do not have it already installed you can go to the tab available click on load from and type in a search control net from these options you want to select as the web control net manipulation installations click on install after installation is completed it should will be available for you to see in your installed extensions at this time I highly recommend for you to click check for updates and after it's done click apply and restart this should be enable control net for you in some cases you will need restart the server but don't rush it because when you click apply and restart some models of control net will be downloading on your machine if you use it installation from the extension tab then it will be located in your stable diffusion installation folder extensions and down below you'll see you have an SD web UI controller if you expand this folder you'll notice another folder called models click on this models and you should have it here models installed at that time of recording I have 14 models installed if for some reason you're missing some or you want to upload it downloaded them or reinstall it then I will recommend for you going to the hoggingface.com and lasers control net with version 1 1 and going down to file and versions all of these links for every resources that I mentioned in video will be available down in descriptions here you'll notice you have it all of these models you need all you need Python and yaml so you need both of these one will be declaration and one is a model to process be sure you have it a latest models and personally if you have already control net installed previously I will recommend you to check be sure that you have a new models the reason it will look on old models you can see the size is quite a bit large they're about five gigabyte each it's quite a bit uncompressed big models new ones it's much more smaller it's only about 1.4 gigabyte the optimize and the much faster loading in the memory of your machine also if you finish downloading models and you install it I will highly recommend again restart your server so it will reread all if you have a control net properly installed you should have it drop down box notice if my expand I have a control 1 2 I have three different types of control net those configurations will be in a setting Tab and on a left side you can see you have a control net option click on the control net this is open our model right here on the model notice I have it version 1.5 or it's v15 this is what we're going to use control net utilize stable division 1.5 in some cases with special settings you can run version 2.0 but in most cases it will be only 1.5 next below you want to specify where you want to direct for maps if you have it it should be fill up by default things what you want to change it is control netmax model and you notice I have a three models and also cache size the cache size for me is matching how many models so I don't need reloaded them again if I use more than one and right here we have it max amount it's a three which is correspond to those three tabs or three models I can use that same time you can go much higher if you need it but mostly I found three is about mostly what you will probably use if you're creating even animations or other down below you will have it checked on the box do not appear on the map to Output you probably want to uncheck this at least at the beginning because when you render and create image it will also showing your map you can see what map was used it's very nice to debugging tool at least at the beginning till you get familiar with control net next below also check a lot other script to control this extension this will help you in the future if you decide to use like the Forum or any other application to access control net and use it for the animations after this completed click apply settings and reload your basic use of stable diffusion it can be when you use a tx2 image we have a prompt where we can type for example Dark Forest and without even any selections we can click generate so this is the basic basic use however what is control net add it is additional condition to what we want to do so here is example we just heard one text all men in a hood sitting on a chair Rembrandt lighting epic dark majestical this is a render without any specific conditions that we can apply I can apply additional condition if I wanted for example using Ctrl net by placing image inside switched the open pause click on preview be sure the preprocessor is working and after by enabling use of the control net I can generate again but in this case the condition that I want to use it it is this pose that I extract from previous photo and now because we apply this condition our image that we created look very close by the pose to what we have on our photos of course the control net you can notice have it many many different type of models in some cases we can use like candy Edge detection of the object and also close and as a result you can see we have a little bit more resemble with our close what we have it so right here that's what we created and this is our image you can see the close very closely resembling what was before control net can create the point of reference or a shape of the object based on the edge depth math and all different options beside the installing control net models we also can use a T2i adapters that stand beside the Great points on outlining they also are neutral Network models that provide extra control to our images by generating with a diffusion model together select t2a this is for the adapters and from the adapters we can go select what type of adapter we want to use it notice in this case we don't use it any prayer processor preprocessors it's allowed us take input image and analyze in specific way for example you can see right here we have an image and we're using the can preprocessor which is detect edges for our image and now when preparator is run we have our adapt map created but because that map it's not yet integrated with our stable diffusion this is where the model step in model will take an image that already passed pre-processing with all the data that created and controlling the stable diffusion how it will going to create our image based on our text or other input because the model need to understand prayer processor what is expecting the model should be match our pre-processor in this case we can go to control add and select depth as your collection is growing it's maybe sometimes hard to find so in this case it's much easy if you just select options for example if I can select depth you can see right here it's select preprocessor and select models I'm not necessarily like to create with this preprocessor in this case if you want modified you can select different preprocessor the reason for example I like to use it later plus plus because it's allowed me to remove near percentage or background so I cannot isolate front or back of the model and work that way General working with control net it's a similar to what you do with a text to image we select our primarily model or checkpoint we want to use it we write our text prompt if we need it we can write any negative prompt next we're creating selecting our sampling method the method is will be based on what model you use it before or your personal preferences next we're selecting sampling steps how many you want to apply it of course options for the width and height which in many cases need be what is a model train on currently because I'm using rpg4 it is 512x768 next we have a CFG scale by default is lift to the 7. in some cases you maybe one keep it similar seed number if you want consistency and that will be mostly if we're using the Forum or some other animation there you can see I have step animations this is different extensions and not related to this video so I won't go in there then next we have control net Dub with our three primarily that was selected Choice when we work with images notice by the single image we also have a batch processing in this case you can point to this specifically directory with a PNG or JPEG files and control net will be sequentially reading those files and using inside and as I said second sequentially it's meaning images also need to be named properly for example zero zero one zero zero two zero zero three and so on below we have an option to select notice we have it enabled when you enable the control net it will display on the top and also allowed us to um control Our Generation by deselecting control net options if you do not select properly preprocessors in models the control net most likely will give you error or will be ignore it also we have it options for the Pixel Perfect and it is optional can be selected and it will use it image height and width that you're specifying text to image to generate pre-processed image so in this case it will match your settings inside the text to image or image to image and if your image is a little bit different from audio specify I recommend to kind of enable and select this check aloud preview so we can see result we have the model and this is actually notice what I said before if you go to preprocessors and you can see you have a very large selection of them and it's very hard sometimes navigate so with this selection it does help you to turn down to specific set for example if I'm going to click open pause selection you'll notice my preprocessors selection uploaded will apply to open pose as well as the models will be selected to open pose as well if not all models or processors are selected you can always click on refresh and this is Will reread directory with the RM models is located and repopulate this list next to the preprocessor we have this kind of spark icon that is allowed us to click and run preprocessor and preview how its will look this is very useful if you work with too many options in one be sure the you receive the item that you wanted you receive this look let's go ahead and look on what processors and models we have it to see what ability provide for us we'll start with the Kenny in Kenny is the general purpose kind of edge detection options so when we select we can go click on the preview and you can see it is definitely showing us the edges this is general purpose kind of work very well in most situations however withdraw with this model because we're not just a passing for example how model look and everything is passing close elements around so it will be creating settings very similar to what our reference photo is and if we select any pair processors we have it also option below Kenny threshold high and low so this is amount of details in a low or high so generally if you want to increase all details in everywhere we can bring them a little bit down in values and run preprocessor again and you can see we have way more details and same things apply to opposite if we want to reduce amount of details it's producing we can apply for both high and low or you can just separate for them and run preprocessor and you can see we have it much lower so next we're going to look on a depth and the depth selection will provide for us less plus plus meets and Zoid by default depth meets it just will generate and this is a classic one depth model so we can preview how it's created notice we don't have it any optional adjustments that we can apply to that depth model in this case and this provides some details the models large and less plus plus will provide even more details in many cases it will even render background and because we can render background we also have additional options below where we can remove percentage of near or background keep in mind because this is does not have it true 3D mapping it is guessing so it's a basic on the AI guessing by analysis of the image overall we have our classic we have a classic as meters we have layers with more details layers plus plus with much more details and the absor is somewhere between layers and minus also when we select it we have our model to use in my drop down you'll notice I have some other elements but those ones it's for the adapters which does not necessarily require preprocessor when you use them depth map can use it for multiple reasons for example if you want to create a depth of field or if you want to remove any specific background or any specific planes inside the image so our next is a normal and it does create it by default bias but let's go to the middle one first one thing when we're creating normal mids it will tend to actually separate other objects it's work very well if we need to separate and isolate our objects from the background and next the buyer it will also will take some background in considerations and will create details for both background and the model itself open pose probably one of the most popular option in control net because it allowed us to analyze our object or objects in image and based on this creating specific poses for them by default first selection will be open pause full which included everything included a facial expression phase position even hands and fingers if you have it but if you want anything specify you can specifically we can expand and select just open pose which will only display the position of head arms legs we have also option just for the face in this case if we want to display facial expression we also have it face only what is on this face only will remove it any position of the hands full that we saw before include everything and specifically if you want just the hands because the hand and fingers will assign to the arms it will create everything but just remove it our face position okay let's select our face and everything and we'll go ahead and generate based on selected model of open pose the preprocessor open pose full and model open pose here we have it our image of the old man in a hood Rembrandt lighting Epping dark majestical and you can see the pose was based on same from our model that we preloaded I said this is one of the most popular problem open pose at least for me when I'm using and many times I may use this pause with conjunction with other ones like Kenny open pose maybe included the depth for the player or maybe even normals to isolate the object okay our next is mlsd which is mobile line segment detections it's a straight single line detection it is very nice to extract some outlines architectural building Street something that have its straight lines work very well and you can see as we run play processors we have a model is almost all removed on the middle but all what we have with its straight line so it's worked very well in combinations and if for example you decide to render background you can use this model to create very nice beautiful backgrounds next we have it a line art and that this is try to create and outline the image something great like almost simple um anime simple drawing line drawing type of render and we expanded preprocessors you can see we have it line art anime specifically that is a little bit more applied we also have it line art the noise coarse realistic standard and as we render with old man the hood we created this kind of Animation style and it made it drying little bit with based their line art anime soft Edge will act somewhat similar to the edge detection only is ADD little bit more flexibility and gradient to the edge allowed more flexible render so if we think about the lines restrictions this is a little bit widening restrictions and may create more interesting result and here as we're creating for example a space girl futuristic lighting epic Majestic and animated style you can see we create very nicely base it on the our image with a soft lighting so if we're going and replace for example two head quality let's go refresh it we'll have a detection a little bit more details included like I and moth if you notice right here we don't have necessary Expressions because in previous model does not provide this amount of details so but without details it provide also more flexibility and here's our result scar Scrabble preprocessor will try to turn our image to those like hand drying type it's self-explanatory almost scrubbling creating um very nice if you try to create again some painting and painting and other things but it is personal I think it's a little bit more limited due to similarity to other preprocessors but only with a less resolution so personally after using all the spray processors I found the canning actually most useful and you can always reduce amount of the details to achieve similar results like the previous preprocessor created this one is again it's a my personal opinion and you maybe find more use than me notice right here has it because it does not recognize it's actually booted eyes and created kind of a little bit uh weird creature for us okay segmentation next it will um label for us what kind of object inside and if you work with any post processing with 3D rendering you many times try to isolate objects and use it so in this case it's very useful when you save the map because in this case you can take it and use inside another applications to isolate objects or to apply other elements and you can see right here it's try to differential some object but that's a little bit harder with transparency so let's try in different image in this case and there is a perform better you can see separation trying on objects they also have a different type of preprocessors and they corresponding to the different type of the data sets it was training with to Creation you can try to readjust them and see which one may work better in your case you can see it's separate object try to separate table flowers and background Just Produce depend on the word preprocessor and would be purchase a train on so we'll use it so let's go Shuffle is we'll try to shuffle all images steer up kind of input on the image and may create very interesting result but not always it may walk work very well as space or some other ones if you want to create some unusual look so you can take your image put it in for our next model we're going to use it in pain this has allowed us to analyze image based on a style color in other elements and change scale change elements inside so we're going to actually use it as old painting increase in size or change the ratio for example you can see right here we have it with an i5 12512 even our image is 16 by 9 Dimension also I'm going to switch this to the settings and set to 35 on the steps we'll leave it default with an height in painting here we also want to switch to resize and fill so it does not crop otherwise the image will be just cropped on the side we don't want to do this and inside here let's go it says watercolor I think this will work fine problem with this method sometimes when you tweak a little bit more with sampling steps and other properties like reducing scale so we achieve better blending on our edges so let's go ahead and try right now and here we have it our image you notice just like a touch on the top bottom actually come up very good but hope does a little bit visible so we need to adjust maybe CFG scale a bit better on this case but generally you can see it's extended our image and created different dimensions so this is actual very nice in pinning tools if you want change something inside image and definitely if you want to upscale or change out painting on your image okay our next model it's really going to use the options ip2p let's work with images you notice as we're selecting our pre-processor is not available because we're going to utilize image itself what is important it's using special control model after this and we said correctly ip2p model how it's working stick image and we can apply specific effect to this image for example let's go ahead and says work here we'll just go set make cover with snow so I won't put it all of this inside the snow let's go ahead enable our control net no preprocessor model and everything seems correct let's go ahead and click generate and here we have our image so it's not as it's our chair and all our stuff around cover with snow so it's gonna work very well now let me switch this to 768 because you notice right here's image we add gray bars because this resolution of the original image is not Square so let me go ahead and generate one more time okay right here and we have it perfect generated all chair in snow so definitely with this model you can create a different type like the house put house on a fire or other things very nice very um interesting type processor so next one it will be references and this you'll notice right here only pre-processor because we I will reference specific image and nice about this you don't need to have it model because it will render specifically to reference of this water so for example let's go ahead and I'm going just to place one here and as our prompt let's go ahead and type maybe um so you can put it wherever you want it I think this one will work very well and we'll just leave it everything as default again we have a Fidelity of sterilization so it's a balance do we want more towards words or image so we can switch and just a little bit more towards our image let's go ahead generate and here you can see we generated different but it did reference our model colors Arch and everything of course we need to apply some negative prompts to remove extra limbs and other things but for the test it's very um useful mostly if you want to create like portrait based on somebody else but you don't want necessarily A precise copy so in this case it will use it as a reference to this image for the color some major big details and positions okay next we have a tdo AI this is our adapters notice when you select adapters you don't really have it preprocessors you have some of the color sketching stabilizations but generally you have it a lot of preprocessors most of them will be duplicate or doing similar with dust control net okay let's go ahead and look at some preprocessors and we have it of course color grid if we look on a preprocessor color grid this will give it as a position of the colors it's useful to reference to um what colors can be using of course we have a sketch as well you'll find some of this very similar and familiar what we have in control net and of course clip Vision this allowed us to create specific noise same with the model selected it is apply similar to model that will look before with a control net previous selections so let's do a couple things one let's go disable our preview and we'll go back this is actual models it's quite a bit large collection of the models and of course as you work alone you can find something your favorite models but beside working with the specific models and work with single image as we did before you have an ability to mask you'll notice we have some properties that this will change some below depending on what model but just under selection of preprocessor and models those selections is consistent during the old time they represent the control weight it's how much control will be con how much control net will be affecting the resolution in stable diffusion how much of the effect will apply on the model in stable diffusion as well usually either one if you want it if you you can increase higher however effect and result won't be as desirable sometimes the fund does not work well and maybe control weight you maybe want to reduce if you're using more than one control net next we have it starting steps again remember we have it steps just above and still when on which Step control net should step in because we have this denoising things happening and at first you have a big flexibility and selection and you can say I want to control net start effecting maybe on the third or fourth step so where is already a major object created and same when you want to stop control net to be effective the ending control step may work better if for example you have too much utilization on canning and you don't want some of this fine details in the end you want allowed something more going from The Prompt then you can cut off ending control steps when you want to do this okay next we have the preprocessor resolution and this majority if you draw by yourself were created some other elements personally I like to set preprocessor to higher size so for example if the image was 512 by 768 I put it here preprocessor resolution 760 so it's the least it's be process everything but the same things you can do if you just without putting there if you just click Pixel Perfect then you'll notice the option is disappear so it will read its same size of your image and will create that way so you have different options Pixel Perfect or you can also just type it in size that you want to use it next okay below we already spoke about this control mode we have it balance it prompt or control net more important in the self-explanatory it's where you want to stable diffusion pay more attention and resize mode just resize image this will affect if your image need be too clipped or use it you'll notice sometimes we'll cut maybe top head in many cases if you want to utilize all control on that image you probably won't go to the size and fill in this case and definitely when you work without painting it's what you want to use it loopback automations it's work if you need to have it on the same image in many cases will work a little bit better with the batch files in most cases I found I don't use this option speak about batch file if we go switch you'll notice the image drag and drop disappear but you still have it all the other properties here inside the input directory you just refer to the path we have the JF or jpeg single image located then next we have a second control net which is identical to another one and you can also enable but in here you can have a different type of the control like models for example if in the first one I won't have this done by the poses so I want to be sure the process is exactly the same and remember when we do the pose it will analyze and put it and sometimes scanning not necessarily pick up properly position of hands and other things and then in my another control net I for example want to have it um let's go download there and I won't have it Kenny so it will have the detailizations for example on our outfit maybe some like smoke and alien so I want to do this way overall control net is have a very big ability to kind of to create properly image the tools a lot of flexibility and everything it's maybe overwhelmed at the beginning but as you start using you find this very intuitive and nice to use one other big plus about control net because it is extension and if you allowed to work with other applications you can go even inside application like the form and use here control net as well for your animations well thank you for watching this video I know it's baby was a little bit long but I won't be sure we go we can touch all details and possibilities if I miss something or you have better information please let me know give me notifications um send me messages and if you find you like this video please share with other people it will be very helpful for me if more people see this video and pop up on YouTube and by the way if you're watching this far away you are my hero because you don't fall asleep and you don't give up on me and I appreciate this thank you and have a great

Channel: Vladimir Chopine [GeekatPlay]

Views: 20,461

Keywords: Geekatplay Studio, Vladimir Chopine, Digital art, How to do compositing, AI art, MidJourney, Stable Diffusion, Dreambooth, Dall-e, Free resources, Free learning, Digital art for begginers, Free tutorials, artificial intelligence, stable diffusion inpainting, stable diffusion install, ai art generator, controlnet tutorial, controlnet ai, controlnet a1111, controlnet install, stable diffusion algorithm, controlnet stable diffusion automatic1111, controlnet stable diffusion mac

Id: M3BQoWA9ke8

Length: 34min 27sec (2067 seconds)

Published: Sun Jun 18 2023