Add characters to ANY environment with Stable Diffusion

if you clicked on the video it's because you want to be able to add a character into any environment making it interact with it and all of that while having it be properly integrated with its surroundings so let's see how to do that you can add a person to your environment super easily in different ways the easiest one is to go to in paint sketch import your background and paint a blob where you would like it to be make sure that the blob is big enough use the noising strength from 0.8 to 1 generate and you are done probably you will want to control the pose of the character for this we will create a control net reference I will go over to the open pose editor extension use my background as a reference and pose this rig until it fits what I wanted now I will save as PNG and import this into control net with open pose you might want to redo your mask to match the pose better and that's it you have the character even after upscaling we will have some work to do which we will see in the most important and last step but let's not get ahead of ourselves because this was a super easy example and should not satisfy you if you want good results in most cases there will be four problems that we will solve and two of them will be taking care in the first step let's say I wanted to add someone in this room for example sitting in the sofa how big should that person be in relation to the scene well that brings us to the first issue scale good news is that this gets solved automatically when you create your control net reference most of the time if you have a scale issue it might as well be a skill issue because it's something you make an educated guess on the best tip I can give you is that do focus on the nearby assets and then try to think about how tall your character would be compared to those this gets a bit harder when you combine scale with our second issue though perspective and lens Distortion if I were to use the same size I used for the sitting character for a person that's standing close to the camera it would feel very out of place quick tip time to much perspective of an image you will extend and the lines naturally created by the objects in it this will get you the banishing points with this I could find the correct proportion for the character like putting it next to the car then placing the pivot on the vanishing point and scaling from there with shift plus alt but this doesn't help much in our room example because of the camera lens you could try to guess the open pose position but this can be pretty hard to get right so I will use POS my art for this I'll try to replicate this room in 3D using pure ref to have the IM much as a reference this will help me match the angle of the camera remember to also play with the field of view to match the camera lens Distortion once you feel like it matches the original good enough and there's no need to be super precise here you can use the character and scale it so it has the correct proportions if pure rev is bothering you contrl T makes it invisible to the mouse now I can move the character wherever I want in the image with the right camera lens and perspective you can save the camera position post the character and Export your depth map in my case I exag ated the camera lens a lot making the character way larger than it should artistic choice I guess sometimes you will need to match the horizon line too like in this example we will use later with this you should have no more issues with the scale or perspective if you really need to mat a scene with absolute Precision you can use FS spy with blender another way would be to remove the lens Distortion using Photoshop for the actual in painting part I'll be using control net in paint as I feel like it does a better job at keeping the original background consistent you can use one weight and one ending step if you have an open Poe model like for the dude in the sofa import it and use a high weight too and make sure that your reference is clear in cases where the character will be small on the inpainting model I usually just mask by eye but you could also create a mask in Photoshop and import it later remember to mask the floor though characters cast a shadow now we prompt and input the aspect ratio I will use highr fix to upscale which is another huge benefit of using control net in painting in text to image it might take a few tries to get a good result you can disable highres until you find a generation that you like pre the seat and activate it again once you do you can see that the lighting is already pretty decent in this case of course the shadow could be better but I will leave that for a later step first I will change the open pose model for my depth map and generate the character close to the camera and I will use this example to show you how to fix the next issue lighting I will go to photo sh and transform the original image to black and white you could also use the recolor lightness reprocessor you could use the steps to create the Shadows too by the way now I will use this as a control net reference and use the brightness control net model I like to use something around 055 weight and 0.8 ending step getting us better lighting results overall we can also try again now with our sagy and it Shadow this control net allows us to control the values a bit better which can be used to control the lighting values used for this will bury a bit though sometimes you will need to lower the weight a lot more to add characters that get cut by the scene like someone sitting inside this restaurant the process is pretty much the same create an open post reference and use in painting control n model but make sure that you don't mask the table parts that aren't absolutely necessary if you will have an arm over the table then mask only for the needed space if a part of your open posst reference won't be visible don't mask it either now just prompt generate and enjoy okay that's cool and all but how do you make the characters interact with the environment liking the example where she cuts the waterfall with her hand or someone breaking the floor this has a bit of a trick to it so I will be working on this image and showing you the full process until it is finished including the last and crucial step and a bit of a funny story around that I will import my control net in paint and since this is a complex posst I will use a control net dep map for it too after playing with the prompt and parameters I landed on this but even if I prompted for him to break the ground it doesn't really matter to do that I will bring the image into in paint sketch this is where most of the magic happens and it is my favorite part of the process with this we are able to make our character cut aside a waterfall add complex shapes and even other stuff in here we painted the shapes of what could be the groundbreaking with some rocks flying out and away I will use the canvas Zoom extension to take col from the background you could also do this in Photoshop and then use regular in painting or uploading a mask once you finish the sketch prompt for what you want I will use a mask blur of one and 0.9 or one as the noising strength it is very high because we will also activate control net tile this is a key part that will allow us to maintain the shapes drawn but adding detail inside them if you want to maintain the shape more use a higher ending step now it's time to generate a bunch of images with different prompts and playing with our values you can also retouch your sketch if you need to of course the goal is not to have a perfect generation but a bunch of images that have cool parts and then in Photoshop we will import all the images we like and combine the best parts of each to create a better overall scene there is no need for this to be perfect because to improve this we will use a very cool trick that I came across in a pretty funny way we will import this mixed image to control it and use the instruct P2 model for those of you who don't know this is basically a model that tries to replicate your initial image but changing it based on an order that you give it like for example make it snowy I was pretty tired of looking for ways to mix everything together so as a joke I just inputed my image into this model and told it make it a masterpiece and it kind of worked it doesn't really make it much better at the moment but that's why aside from this we will also use a tile model apart from that also add a a bit of context to The Prompt about what is happening in the image and the prompt matters a lot too by the way and now your job will be to find the perfect balance where the tile model will try to maintain the original input image and ip2p will try to modify it using lower weights on ip2p will make the image change more so play with the ending steps of the models I also like playing with the CFG scale which I'll use nine in this case for this image I really liked 0.65 ending step on tile with 0.8 weight and for ip2p I will use one weight and 0.85 ending step then I will activate after detailer for a better face and use highr fix to upscale the image giving us this result right here the problem is that we lost a bit of the original background so let's go back to photoshop yes I added the shoulder pads after this and I should have done it before but whatever I will also import the original environment here all you need to do is mask the parts you want to add in my case I want to leave some destruction on the part where the man is but none on the background and with this you would be done unless you want some tricks to make it better the colors on the new image are a bit off so I will use a match color adjustment to fix that very easily much better something else you need to know is that your character will be affected by indirect lighting this is the light that reflects from objects that are near it and bounces into him so I will add a tiny bit of color from his surroundings like so if he had a stronger color palette this would also be appliable in reverse then using some brushes I will add dust and particles I will also add motion blur to the rocks and maybe a bit to the character I could also add a gan blur to the background finally I will add a filter to make everything more homogeneous and here we have our final image if you're interested on how to create consistent environments now that you know how to reuse them you can click on this video I hope this was helpful see you

Channel: Not4Talent

Views: 6,166

Keywords: stable diffusion, sd, stable difusion, promting, prompt, tutorial, guide, stable diffusion prompt guide, stable diffusion tutorial, ai art, ai, crate ai art, beautifull ai art, talk to ai, controlnet, contrlnet, inpanitng, characters, consistent, understand, skere, not4talent, notatalent, notfortalent, inpainting, integrate, environments, add, insertcharacters, colormatching, interaction, characterinteraction, interacting, groundbreaking

Id: BcLjsWChAn8

Length: 10min 0sec (600 seconds)

Published: Wed Mar 06 2024