Next level AI art Control | My workflow

air you describe what you want in a prompt and a few seconds later you will have an amazing image that tries to represent what you ask but it can't really generate complex images that match what you were actually envisioning or can it no by itself it can't how about you using it though that's exactly what we're gonna learn today by taking ideas and turning them into final images with AI and human skills First Step planning in this step we will think about how we tackle the process of bringing this sketch into life what does the image need what can we provide as help and how much time are we willing to spend on it each sketch we'll need different things some might be easier than others with time and experience this planning step will become second nature but for now it is really important let's look at these images and see what challenges might arise when creating them this image presents multiple characters in a beach and to make it even harder I want them to be specific characters from the series konosuba where this tune will be aqua and mega Min and the dude on the right will be Kazuma for this we will need posing even though we have a sketch I'm not sure it will understand the pose that we're looking for so we will need to use the open pose model this background will be easy to create but if we really want to be precise lines we will need to use this segmentation model we can add this sketch as a scribble model just to help a little with the overall image to tackle the multiple characters issue we will need the extension latent couple I will explain how to install it in just a second this will allow us to separate the image in different prompts and mix them together with more accuracy unfortunately it is pretty meh with loras so we will need to do that later with impending the second image shows High complexity in terms of depth as there is an object literally attacking the camera while the main subject shoots at a target practice without touching the guy with a sword obviously we will need a good death map for this to use with the depth model probably we will need an edge detection model like Kani to create The Sword and the gun somewhat correctly and again open pose finally latent couple it is very unlikely that the sword or bullet get generated correctly so I will try to add them with Photoshop later on in the image and then fix it with AI this image presents various challenges the first being the perspective and the multiple characters it will be a short from above with a man facing the enemy volleyball team we will most definitely need open builds for this but it is likely that it doesn't really get the perspective right using the death model could be helpful here but it takes too much time to create a good depth map segmentation will help us Define what is ground and what is people as well as probably the bomb we will use the sketch as a scribble and if it doesn't work we will switch it with something else like the death map we will of course use latent couple to try to get the best possible character results then another problem will be the net it is probable that stable division doesn't know what to do here we will probably need to fix it with photovashing this is a little more complex as there are multiple elements and we don't want them to interfere with each other too much we can approach this by creating multiple images and then combining them but first we will try this a segmentation Model A scribble model with the sketch that we have and we could use open pose to see where the face of the thumbnail would go this image will need latent couple mainly for the robot and background a Next Step 2 creating the references for control net as well as the masks for and couple by the way keep in mind that this is my current workflow so it could change in the future and it is also not a to-do list if you have some other workflow that works for you make sure to share it in the comments below and let's look together for a super workflow I may pin down one of your comments if I think it may help others I will ask you to install some extensions skip this step if you think you won't need them first open pose editor this is an easy to install extension and it makes it really simple to create open pose references for control net you can detect the post from an image and then correct what it doesn't get right you can also input a background image and then post it yourself following it you can just go to the extensions tab click on available and look for open pose there is the open pose editor and the 3D open pose editor you can install both since one of them can build in 3D and has options to post hands and feet I'll be using the 2d1 as it is faster and easier to use now go to installed and click apply and restart UI latent couple there are two different extensions with the same name here so make sure you only install are one of them to install it go to the extensions tab install from URL and paste the text and leave in the description then click on apply and restart once it has restarted close the console and after you reopen it you will have to do the following open the CMD as an administrator clicking here when you open the CMD type CD and a double dot then hit enter you will probably have to do it a couple of times until it doesn't let you do it anymore if your install is in the local disk or the cdisk now just type CD double dot and copy your stable diffusion path which will be up here enter again and now you type this which is also in the description if you don't have it in the C disk like me then the process will be the same but here you will type the disk name that you have it on so for example in my case D now CD and the path here you will paste the other text now just run stable diffusion again and it should be working fine of course you will also need control net so watch this video If you haven't installed it yet and with only extensions installed let's begin generally I just make a quick sketch and then take a photo of it import it into Photoshop and make it to the contrast is high enough to get almost only black and white then I cut the thumbnails to make sure they are 16x9 by creating a rectangle with those Dimensions if you don't have an initial sketch or don't know how to create a base image to act as one watch this video right here I'll explain all methods to create a good starting base no matter your skill set and now based on the planning we've done previously we will make different references to input into controller for the first image we will need open pose sag and scribble so let's open our recently installed extension click on input background image and then we will choose our sketch since it will not detect the process correctly let's just do it manually this part up here is the head and the blue line represents the neck if you want the face keep in mind that the interior circles will go in the eyes and the exterior Ones Will Go On the ears then the joints here are pretty intuitive the shoulders the elbow the wrist not the hand the wrist then the point where the legs will start the knee and the start of the feed I hide the part that won't be visible like all of kazuma's body on the other side of the girl's bodies you can insert new rigs by clicking the add button I usually just download the image just in case I want to edit it later here's a neat little trick to create more accurate backgrounds remember that segmentation had a Google document with what each color means well we can take those colors and paint them in to easily create what we want you can click Ctrl F to search for what you're looking for and to see if it is understood by the segmentation model in this case I'll just take sky right here copy this color code and in Photoshop paste it in right here now we will be painting with the sky color then I'll do the same for the water as I saw it was an option and then the Sun Also I'll make a quick mask for the humans on the scene a segmentation understands that as well later on I found out that it will try to follow the shape more than I was expecting and finally I'll paint the sun umbrella I didn't oh never mind then I forgot I will add it once this is over and now we will also use this to create the latent couple masks and create three sections one for each character painting in bright saturated Corners so it is easier to see keep these masks wide and not too precise and use a hard brush without transparency or fade also add a white background for the second image I'm gonna take a more complex approach 3D modeling this image has some depth complexity so I'll solve that by rendering a real Dev map of this scene I downloaded and duplicated a mail model then using the sketch as a reference I post both models into a somewhat matching stand moving their positions in relation to the camera in a way that would be understood and I did not care about the camera angles that weren't seen in the render view also modeled with basic shapes The Sword and gun the sword was my main issue in terms of depth so I needed to get that right then I added the targets and rendered outputting obviously the depth map and I will use this map for Kani or soft Edge too again I'll make the reference image for open pose and finally late couple see that I didn't leave much space for the background this will make it so the generations don't have a very cohesive background most of the time but we will fix that later and then we have the volleyball image even creating the open pose reference was hard as the perspective is really weird didn't Focus too much on the secondary characters and instead try to get the main character as best as I could also ended up not needing a death map which is nice created a segmentation map this time too for the people the ball and the floor there isn't a color that defines net but I painted it with a pole so it knew it wasn't part of the ground and this time I refined this cat with Photoshop a little so it had cleaner lines on the main character for the last image I tried a fast open bills segmentation latent couple and scribble it didn't end up working as I expected so I moved into image to image in the future by photovashing once we have all our control and models it's time for step 3. iterating this means inputting everything into stable diffusion and then generating over and over again and then prompting or playing with the weights of the control net in order to get what we want for them yes we will drag our latent couple mask in here then adjust the size of your desired image and click I've finished my sketch don't worry if it doesn't let you input the exact size you wanted as long as it is close enough which even on General prompt we will write our desired background for example in the first image I typed animation style illustration of a beach beach umbrella which I didn't realize I hadn't made a sec sketch for it ocean Etc next down here you have the colored masks prompt individually for each part of the image I tried using lures here but it didn't work so if you're looking for a specific character I'd recommend describing it without the use of loras here the result doesn't have to match the real thing but the closer it is the easier it will be to impaint later here I'll leave you a few frames of how the prompt actually looks like when you're done hit the prompt info update and then you will see that your positive prompt is now separated by the word and you know what this does time to input our control net references depending on the accuracy of your references you may want to use a higher or lower control way for example I want the segmentation model to be used as a way to understand where I want what but not the specific shape of it so I will lower its strength then open pose can be fairly high on control weight as it is pretty well done this time I want my scribble model to Mark a little bit of the composition but it is a really rough sketch so I'll put it at a really low strength for starting or ending weight I follow this general rule of thumb if your model is aimed to help the composition but can hurt the details of the image in the rendering phase I will lower the ending weight to what point it will depend on the problems it can give if it is really problematic then take it out as soon as the composition is decided like at 25 of total steps if your model is aimed only to change your odd stuff but you don't want it to affect the composition you could put the starting strength at 10 or 15 even though I usually don't touch the starting strength for much if make a well-posed open pose it is aimed to help on the overall generation you can just keep them both at once finally click generate and see what you get adjust your prompt and weights accordingly we are not looking for a final image here not even close we want good shapes and if possible colors to help us later when we in paint aiming for the real final results for the first image I started with this where the two girls are basically blobs and the Dude Looks Like chat for the second image I used this robot thing with a nonsensical background they heard this one for me was actually the volleyball image because AI made it so the dude spiking was looking at the camera all the time or created different balls I ended up using this one with a Blu-ray protagonist and lastly here you have examples of what the robot creepy thing was creating even though at the end it made this but it was too late this can take more or less depending on the image you're looking for and the control and models you're able to use usually about 4 or 5 Generations should be enough to get your starting base some threads you can use are changing the model to one that has a better chance of generating a good composition you can just test different ones if you don't know making the prompt really basic and focusing more on the in painting and if you aren't getting what you're looking for at all maybe it is time to create new control net references or change strategy for example in this sketch I moved on to painting over a generation that had some parts right to create a better starting image then I looked at the last chance I had given stable division to see that it just generated what I wanted but I still use the photovash to reference and now step 4 in painting and Photoshop it is the hardest and more important part of the process you will probably spend the most amount of time in here mainly depending on the Fidelity you're looking for and a little bit of luck with your AI Generations the process is very similar in all cases but at the same time it is really different what I mean is change stuff you don't like then use AI to improve your changes but how you do those changes or what you want to change will be very different for every image I will go over some of the hardest parts to correct on every image as well as working you through the process of creating the volleyball image as it was the one that got the best feedback thank you for that as I said as a starting base I picked this image from here I see that I don't like a few things the first one is that the ball looks too big this is my bat as I painted it like this in the segmentation mask I'm gonna select it make it smaller and then and fill the Gap with content aware if you have the latest version of Photoshop this will be too easy for you by the way I'm not gonna go in depth on how Photoshop works if you want a more specific tutorial on using it then I'll make it depending on how many people ask for it but for now I will use pretty basic stuff that pretty much all photo editing software has and that is really easy to learn next I'll create the net for this and we'll go and find an image online transform it too much what I need and done you don't have to make it be a perfect Blend or anything just erase the parts that shouldn't be there like the net in front of the main subject I will also take out the net in front of this guy's face so the AI doesn't have much trouble understanding what it is I will add it back later if necessary there are other things that don't really match what I want for this image like the water on the top and I will also paint the lines that Mark the volleyball field another example of this would be Indie gun image where I actually had to use Photoshop to make one a better Target dummy to put the bullet in which by the way if you ever need to post something in a very specific way like I did here you can go to sketchfab and look up for example in this case bullet then picked one that you like and pose it to match your image here I just took a screenshot and then imported it into Photoshop and finally I used online images for the fire effect and the smoke even a more extreme example is this robot where I ended up photovashing a main part of the image okay I'll export this image as PNG and use it as a base to work in the in-paint tab for now the process here is fairly straightforward in paint stuff one by one until you get the result you're looking for I import the open pose reference we made and use it to create better results while embedding people and even more so in the main character with a complex pose I just in paint one by one if I feel like the post is a little meh I will use open pose to see if it helps and if it doesn't I will lower the weight or even take it out completely for the main character I used to load the noising strength and just looked for it to be in better focus once I have that done move on to the next and so forth only masked did a pretty good job in this case even though you're risking getting a character that doesn't really fit the perspective if you need your characters to be a Laura or embedding trained this is the time to use it here in the first image I actually invented both characters to have a better base first and then impainted them one at a time using obviously their corresponding Laura and same thing with this image when I wanted to put a csgo skin as the main character so I needed to use this Soldier 76 Laura and then just imprinted this samuraiian continuing with the volleyball example one thing I would like to keep are the corners of the characters our main character is dressed in blue so the other team should be wearing red to give emphasis to the fact that they are on opposite teams I didn't mind the shirt of this girl not being red as long as the plants were even still if I wanted to I could change the color easily in Photoshop now repeat the process for this one keeping in mind that the open pose might be helping or hurting the last guy I wanted to change was this one as it really stood out and did not match the perspective at all I used the whole picture and latest noise to see if it generated something that matched the perspective better even if it wasn't what I was looking for and then I used that generated image as referenced in the original mode there is no need to have in painting do the job perfect directly you can find a good enough generation and then in paint on top of that or even change it manually to go from there a good example of this is returning to our short image where I actually had to combine two different sword generations to get something I liked and now that I had the guy in a pose I like I just went into Photoshop and make adjustment changing the shirts color with a hue layer also fixing the sound really quickly by painting the parts I didn't like using the colors next to the bad areas I just changed the parts that could look like they were not sent to the AI why well because the next step is changing the Sun for a better one we will create a mask for every character and the volleyball then leave the background including the net as an impainting area you have some ways to create this mask based on the image you have I usually create a quick mask by painting with a brush even though it might not be the best way as it isn't really precise not that it should be super good but sometimes I'm a little too lazy while masking if you don't want to do it manually this trick could help you if you're lucky and your image isn't too overloaded just input it into control net and select the segmentation model and you can use the of8 20K pre-processor click allow preview and then hit the explosion icon this will try to find what everything in your image is making a mask for you now you're just going to photoshop and change the character masks for black and the background for white also you can correct the things that are ported on we will go into in-paint upload put the image with the character changes up here and the new created mask down here it will act as an in-paint mask to change only the background I use whole picture for this also the sound that I already had is not so bad even the net and ground marks are good so I'm going to use a loading noising strength like 40 or 30. also I will use the segmentation model for this with like 75 percent of control weight just play with the mask blur depending on how precise your original mask was and make sure that the background is all properly changed don't worry too much about messing up your characters as you can later overlay them in photo push up like nothing happened these are the before and after of the other images following this same process here you can see that my mask was a little too wide but ended up being easy to correct in this case I have the same problem I should probably start using my own tips how about that clean up time again but this time I'll try to clean up everything more accurately as this part of the image will stay like this until I upscale I want them as clean as possible for this I will use the Clone stamp tool and erase the parts I don't like this weird thing close to the hand or this tail looking thing on the girl here something I usually don't care about are shadows the floor here doesn't have shadows that make sense but I don't really mind that you could create those in Photoshop and then make them better with AI but I'm too lazy for that what I did was just repair the parts that really stood out that way people don't think about the Shadows when they see the image if you have some weird stuff in the image that you don't like now it's the time to change it but be careful because depending on how you want to upscale it or even how many times you want to upscale the image you may want to save some efforts for later for example this is the clean top version of the Kazuma image I did before the first upscale and this is the upscaled version so the hands I bothered correcting are now well done and I will have to make some heavy changes on Kazuma as he started doing some makeup and wears an earring now now we repeat the same process with it for every character but this time changing only the face make sure to use only mask to use the most resolution possible this way you will avoid undefined or distorted faces if you feel like the lighting or poses off increase the only mask padding pixels to give AI a little more context when generating and now we can move into the final step upscaling and post processing upscaling I'm still not super good at this but I have tried some stuff even though I haven't found a way that works perfectly for everything so for now depending on the image you want to create you will use one method of the other we want to use different types of upscaling as these Generations get really complex and stable diffusion starts to misunderstand what's happening sometimes to the point where it is really hard to upscale in normal image to image without a super low denoising strength if you want to upskill more than once and your image is fairly simple to subscribe then you can try up scaling it by two in normal image to image if not then you can try the tile model on control net with the ultimate as the upscale script like we did in the last video If you're looking for a simple not very good upscaling then just use the extras tab with the ultra sharp upscaler when I'm looking for a good image I use all of the above and then mix all the results together with the part I like the most in the volleyball image I upskilled once with the time model and these parameters then the faces got a little weird so I went into in painting and fixed them super important to use only mask if you don't want to downscale the whole image again I also went into Photoshop and did a little bit of post-processing like adding some motion blur to the hair and hands and the opposite team or raising the contrast of the image and some other little adjustments these were done using the camera raw filter on Photoshop but you can also use layers which by the way thanks to storm or storm for reminding me that this existed I will leave their Twitter name in the description so you can go follow them they post some really cool landscape art also helped a lot giving upscaling tips in the comments so if you really care about upscaling I suggest you look up their profile after the whole image was completed I brought it back into stable diffusion and upscaled it again this time with controller but instead of 20 or 25 sampling steps like usual I used 80. found out that in image to image you can afford to use a lot of sampling steps this will add some very nice Minor Details be careful with the artifacts it produces though I had to go into Photoshop again and then left out the artifacts or the parts that I wasn't interested in this was my original final image and this one I think the 80 step details another interesting example here is the konosuba image I copied the same prompt I used at the very beginning by changing the ends for breaks not using latent couple this time and by the way this was only a test I still have to see if adding break here is worth anything I don't think so I don't know how to prompt for the tile up scalarian this was the result I got after a while working on it I ended up with this then re-upskilled it again plus Allure to a detail I'll give you the link to it in the description I haven't tried it much yet though just thought it wasn't interesting experiment then I make the two upscales to get detailed on parts that didn't have any before getting up per 2 upscaling and not a per 4 but with really nice quality for the short image I upscaled it with extras and controller then makes the best part of both to create what I needed here I wanted to do some extra post processing like adjusting the contrast putting in some motion blur or movement effect on the blade and the body it didn't turn out great but that's my own skill issue and the robot thing I didn't even bother upscaling as things were staying until now and while you click on this video down here I'll show you how I quickly added the umbrella onto the Kazuma image I basically painted a sketch to use the scribble model and invented it with a custom mask made in Photoshop then used blatant upscale and a simple prompt and voila hope the video helped and thank you so much for watching t

Channel: Not4Talent

Views: 95,487

Keywords: stable diffusion, sd, stable difusion, promting, prompt, tutorial, guide, stable diffusion prompt guide, stable diffusion tutorial, ai art, ai, crate ai art, beautifull ai art, create good prompts, talk to ai, talk to stable diffusion, controlnet, controllnet, contrlnet, latentcouple, latentcople, composablelora, inpanitng, inpainting

Id: aBiGYIwoN_k

Length: 23min 1sec (1381 seconds)

Published: Fri Jun 02 2023