How to Inpaint in Stable Diffusion A1111, A Detailed Guide with Inpainting Techniques to level up!

hello everyone keyboard Alchemist here and welcome back to another stable diffusion tutorial I always try to review and answer any comments left by my viewers and a lot of times people will leave some fun ideas for me to try such as I see that you can inpaint something that already exists in the image but what if I want to inpaint a parrot on the girl's shoulder the answer is yes of course you can do that or can I use luras to inpaint a specific part of an image for example changing the girl's arm into a cyborg arm the answer is yes you can do that as well then you might ask okay but how do you actually do it today instead of explaining in words alone I'm going to show you how to use some inpainting techniques to achieve these fun ideas now there are a lot of great ideas that are left in the comments and I can't possibly cover them all in one video but if you like this type of content let me know in the comments and I will I'll try to cover more of them in due time now let's go to the main course for today first let's make the parrot on shoulder picture we need to download an extension called photopia I know it's spelled like Photo NP but people call it photopia I don't know why in any case this is a great photo editing tool for automatic 1111 it's just like having Photoshop or built directly in your web UI and you can pass an image back and forth from image to image to photop or vice versa thank you to viewer Steve Warner for pointing me to the extension a few months back here are the steps to install photopia go to extensions then click on the install from URL tab copy the photopia GitHub link I'll leave it in the video descriptions and paste it into the first field here then click install and wait for the installation to finish once the installation is finished go back to the installed tab I like to keep my extensions up to date at all times yes I know it's a disease so I'll click check for updates then apply and restart UI once the web UI restarts you will see the photopia tab at the top okay now let's import the original image from PNG info into image to image we are going to get rid of the positive prompt and only keeping the negative prompts here I'm just trying to regenerate the image with original settings setting the denoising strength very low to 0.1 because I just want the image to be regenerated again with almost no change at [Music] all once the image is regenerated click on this red bird icon to send the image to photopia you can do the following steps in Photoshop or if you are more familiar with those tools but I think photopia is just as easy and when you are done you can click this button to transfer the image back to image to image in photopia what I want to do is to just select the parrot and cut out the background in this image there's more than one way to skin this cat but I'm going to use the magnet selection tool it's okay if the edges are a bit rough right now we don't need to get it perfect when you are done with SEL in the parrot go to select then inverse then do a cut or control X this warning message is telling me that the image needs to be rasterized first so then we can go to layer then rasterize now we can do edit cut again to get rid of the background of the bird image then let's clean this up by erasing some of these Jagged edges deselect or press contrl D then select the Eraser tool you don't have to be super careful or neat about it closeing enough would be just fine oh and don't worry about this piece of wood here in painting we'll take care of all of these little imperfections once you are done with the cleanup move the parrot to a location on her shoulder then we can send this image back to image to image after getting back to the web UI click on inpaint to send it to the inpainting interface let's cover up the parrot with a mask notice that I am leaving some amount of padding on the edges of the parrot meaning my mask is larger than the parrot this is done such that we are giving the AI model some context with the surrounding pixels I am using the dream shaper 7 model to generate the parrot and I will add MAA parrot to the positive prompt the inpaint settings are mostly default settings but pay special attention to the following settings masked content should be set to original because we want the inpainting to stick fairly close to what's underneath the mask we are going to use the whole picture setting for inpaint area this doesn't mean that we will inpaint the whole picture it just means that the model will use the entire picture as reference when it is generating the new image I will generate a batch of four images with random seeds be sure to set the denoising strength to something a bit higher I'm using an arbitrary number of 0. 75 here for the first batch but in a little bit I will show you how to pick the appropriate denoising strength value when you are done with all your parameters click generate out of this first batch of images I can tell that we don't have the right seed for our image here because I don't want the parrot to face away from the viewer so let's roll this out of this second batch the first image at least have the parrot facing the right direction so let's use this seed by clicking on the recycle icon and continue on to the next steps I'm going to use the XYZ plot script to test out a range of denoising strength values and pick out the best one for this image oops I forgot to reset the batch count back to one so I interrupted the image generation set batch count to one and then restarted the generation now if you recall from my previous videos a low denoising strength means fewer changes from the original image and the higher the denoising strength the more changes you will see in this case we can see that denoising from 0.15 to 0.35 were not enough to merge the parrot into the image we can still see remnants of the wooden Branch under the parrot's Talons but as the denoising strength increased the wooden branch is gone and the parrot started to look like it is a part of the picture and not just a simple cut and paste job within Photoshop then as we increase the denoising strength past 0.65 we are starting to see distortions in the parrot so this basically means that we can pick a denoising strength value that is somewhere around 0.45 to 0.55 in this case I picked 0.45 turned off the XYZ plot script then generated another batch of four images with random seeds here I would say if you liked the image that you had with the previous seed you don't have to generate again with a random seed you can just reuse the previous seed and generate your image with 0.45 denois strength but here I just wanted to see what another seed would look like and it turned out pretty good as well I'm going to show you what inpaint area only masked setting looks [Music] like you can see that while in the only mask mode the model is only referencing a portion of the area around the mask to generate the new images as opposed to referencing the whole picture sometimes this will have some drawbacks but in this case because we already have a solid image of the parrot underneath the masked area the result will not stray far from it in fact I see that the parrot's tailin are a bit better with the only masked mode so I'm going to use one of these images for my next steps for this next part we are going to use a different method to impt something new into the picture here is our lady with a pirate on her shoulder but let's say that she is a pirate captain and a pirate Captain will need a hat befitting of her status first send this image with the parrot to inpaint sketch then I'll use a brown color to draw the pirate hat or at least a general shape of the Hat because I suck at drawing with a mouse note the area of your sketch will influence what the inpainted Hat will look like also I didn't do it here but you can totally add some additional patterns or colors to this sketch to Define it a bit more for this method here are the important notes and settings put in pirate hat in your positive prompt for the masked content setting choose the fill option and not the original option since we want to add something completely new to the picture when using the fill option your denoising strength need to be set to a higher value otherwise you will just get some noise here is a XY plot of denoising strength on the x-axis and sampling steps on the Y AIS to illustrate my point we can see that is the denoising strength is too low you will just get this blur in the shape of our sketched hat only when the denoising rength is 055 or higher does the Hat start to materialize we can also see that the sampling steps should not be too low at 20 sampling steps we are not getting enough details in the Hat I like the image at 80 steps and 0.75 denoising strength so these were the values that I went with in order to make my picture more cohesive and fix some of the minor imperfections in it I will usually do a latent upscaling in image to image it's fast and easy and it takes care of minor issues in the image and this also answers another question that I got previously someone said after inpainting the skin tone is different doing the latent upscaling would fix any inconsistencies that were introduced by all the inpainting steps and will make sure the skin tone of the body matches the face if you like my videos please click the like And subscribe buttons to help support this channel your likes and subscriptions Are Much appreciated and help me to grow this Channel and continue making quality content thank you okay now let's talk about how to inpaint a specific part of an image like changing the girl's arm into a robotic arm using luras a Laura is short for low rank adaptation it is a Training Method to fine-tune a stable diffusion checkpoint it is a lot smaller in size compared to a checkpoint point and it can be trained relatively quickly ideal for generating consistent Styles objects characters and in our case robot arms to imp paint the robot arm we are going to download this real mechanical Parts Laura from civit AI I will leave a link in the descriptions after downloading the Laura go to your automatic 1111 install folder then the models folder and click on the Laura subfolder if you don't see the Laura subfolder for some reason you can just create a new folder and rename itur Laura then put the real Mech safe tensor file into the Laura folder come back to your web UI and click refresh to see the newly Incorporated Laura we are going to do a few quick things here to make using the Laura a bit easier click on the hammer and wrench icon here and let's add a short description to the Laura properties page usually in order to use a Laura you will need to type a keyword or activation word into your prompt this is not always the case but a lot of lauras do need a keyword to work for this Laura the keyword is real Mech this was noted in the civit AI download page we are adding the keyword in this field so that we don't have to remember to add it to our prompts every time we want to use this Laura there are a lot of lauras out there and this little feature is such a timesaver now click save to use the Laura simply place your cursor in the positive prompt field then click on the icon for the real Mech Laura as we can see the Laura and the keyword we just saved are now both added to our prompt now it is good to go one additional tip before we move on you will likely download a ton of different lauras down the line so in order to organize your collection better I recommend adding a preview image to your Laura here is one method to do it right click on an image that is representative of the Laura then choose save image as in the save image window navigate to your stable diffusion Laura folder the same folder where you have saved this Laura then rename the image using the exact same name as your Laura file here I just copied the name of the original Laura file but changed the file extension to. preview.png note the image has to be saved as a PNG file otherwise it won't work go back to your web UI and click the refresh button to see the preview image okay now back to our main objective I will upload the image of the lady in the blue dress and send it to image to image in paint get rid of the previous positive prompts and add the real Mech Laura and the words robot arm draw a mask at the location where the arm should be as you can see I am just defining the general region so it doesn't have to be very precise to begin with you can always refine it later which is what I did here I didn't like the initial in painting results so I enlarge the mask a bit keep in mind the size and shape of the inpainting Mask will impact how your inpainting results will look the important parameters for this task are as follows set the masked content to fill and set inpaint area to whole picture set resize mode to resize and fill use the width and height of the original image use a random seed and set the denoising strength to a higher value once your parameters are set click generate it's quite interesting if we look at the snapshots of the image generation process here on the right hand side we can see that the AI model is generating a new image that is similar to our original image and then merging the portion of the new image that we have defined with the inpainting mask to our original image therefore if the newly generated image has a different posture or stance than our original image then the inpainting will look a bit off such as in the case of image number two here but if we look at image number three here we can see that the new and the old images matched up pretty well so in this case we want to recycle this seed and try to tweak our other parameters a bit I generated a XY plot of denoising strength versus sampling steps to see which combination provided the best results and in this case I think that 20 steps and a denoising strength of 0.85 or higher was good taking a look at the other images with higher sampling steps even though the robotic arms looked good the robotic hands were a bit wacky so I went with with 20 sampling steps and one denoising strength now that we have a good robot arm how do we get rid of all this random stuff that was in the background well I must say I was a tad too aggressive with the masking initially so maybe a Slimmer mask would have prevented having so much junk in the background but never fear we can get rid of it just as easily with some more inpainting first we will carefully create a mask to cover all the stuff that we don't want to get to the smaller areas in the image we can zoom in by holding down the alt ke key and scrolling with the mouse wheel thank you to quite a few viewers for telling me about this setting the default key is alt but this can be changed in the settings menu under the canvas hotkeys category plus there are some other hotkeys that are useful in there since the background in this image is a white sandy beach I'm going to change the prompt to white sandy beach then change resize mode to resize and fill batch count to four and seed to random and generate we can see that image number four here did a pretty good job with the beach background but while the in painting did change the background it also cut off a part of the palm tree so I'm going to add that back using inpaint sketch let's send our newly inpainted image to inpaint sketch and sketch out the tree trunk then change the positive prompt to Palm Tree make sure we are using a high enough step count I think 60 steps should do fine make sure we are using the same width and height as the original image and a higher denoising strength such as 0.85 then generate a batch of four images with the random [Music] seed we see that image number four is doing a pretty good job of generating the tree trunk so we will take this image and do a final l upscaling to finish the job send this image to image to image and get rid of all positive prompts we can reduce the number of sampling steps to 40 then increase the width and height of the image by about 1.2x we don't need the increase to be very large then Den noising strength to 0.3 which should be good enough but you can always try out different values with an XY plot and that's it the final image looks pretty good I hope the inpainting techniques that we covered today were helpful for you don't forget to leave a like And subscribe thanks for watching and I will see you in the next video

Channel: Keyboard Alchemist

Views: 15,469

Keywords: stable diffusion, automatic 1111, stable diffusion tutorials, a1111, AI Art, AI, Tips and Tricks, Tutorials

Id: akzu3R7lDZ4

Length: 18min 11sec (1091 seconds)

Published: Mon Nov 27 2023