How to Colorize Photos with AI - Stable Diffusion + ControlNet Tutorial 2023

restoring and colorizing an old historical photo is an art for itself but as in almost every field AI can help us specifically stable diffusion using control net if you want an accurately colorized photo there's almost no way around a ton of manual work and historical research but nonetheless today I'm going to show you how to turn a picture like this Civil War nurse into a colorized image like our handsome man over here you can use AI for this process as much or as little as you like I think it's an excellent jumping off point and you can really get some great results if you spend enough time without further Ado let me show you how this video is going to assume that you have some basic knowledge in image editing and already have the automatic 1111 web UI installed for stable diffusion if you don't no problem just follow this guide that I Linked In the description on how to install it you will need an Nvidia Windows computer and just play around with it a little bit to get the basics of prompting done the rest I'm going to show you how to install and use now we will be using a different checkpoint than the standard V 1.5 or 2.0 namely one called realistic Vision what you can do is you can follow the link in the description go to civitai.com and download realistic Vision version 2.0 just click here it's a safe tensor file and then place this model in your stable diffusion web UI folder under models stable diffusion and just put that in here next you're going to want to install the vae now it doesn't really matter what that is you just need one to use it to make some slightly better images than usual so go down here click this link and you're going to download another file that file you put in the models folder as well under vae just paste that in here and this is the name I already have it now to activate this vae you go to your UI settings scroll down to user interface Quick Settings list and write comma SD underscore v a e apply settings and reload your UI and after you've done that you can switch your checkpoint to realistic vision and you see that you have this new menu up here sdvae that you set to the vae you just downloaded now one more thing of course you're going to need the control net extension to well get the actual image you want out of this so for that you go to extensions install from URL and paste this URL that I have in the description in here and hit install once you've done that go to install Tab and put a check next to sdwebui control net you can also go to the available tab load in the extensions and find it in here and do the same thing it doesn't make a difference and after you fit that check mark apply and restart UI great now you see down here you will have a control net tab but you will need to use multiple control net steps so to do that you go to settings go down here to the control net settings and set your multi-controllenet max models amount to three and now when you go down here click control net you will see you have three different tabs for control model 0 1 and 2. so we're going to use all of those and that way your stable diffusion is set up for the process we're going to do today now it's time to get our image ready right I'm going to be using Affinity photo 2 but these methods are possible in Photoshop in pretty much any sort of image editing software I have this Civil War photograph here this is a union nurse called owu weight now I only want this guy's portrait up here so I'm going to crop it with a custom ratio that works well with stable effusion of course you can use any ratio but I recommend one that's just easy to handle in my case 512 by 768. and now we're just gonna move on up in here and get this guy cropped like this so we have his face and parts of his body great now I'm going to resize the document to the size I want as a final result and because I'm going to resize by four I'm going to do 512 times 2 right and that's going to automatically resize to 1536.4 I can adjust that to just 15 36 right I don't need that extra comma resize and there we go it should look exactly the same now before we go into stable diffusion I'm going to clean this up a little bit roughly already using just standard tools like the Clone brush tool for example to remove some little artifacts like that just some spots because later we're going to ask stable diffusion to run the Kenny algorithm over it you don't need to know exactly what that is but what it basically does is it recognizes outlines in this picture to figure out what's in it right and if there are spots like this that might irritate the algorithm and we don't want that so I'm just going to clean it up roughly now historically if we were doing this really clean I would figure out if that's actually like a birthmark or something he has on his face but because it looks like the other spots here I'm going to remove it but that's sort of attention to detail could be important depending on who you're doing this for family member for example you might know that kind of thing there isn't much more I think that's just part of his beard here and now I'm going to run some denoising over it in Affinity that's available here in filters noise denoise and I'm going to just adjust the voluminous settings to kind of make it a little smoother because it doesn't matter if the details get lost here we're going to get those in with candy again right so this is basically just removing the noise now that's nice I don't need this in this sepia tone so I'm also going to run like an hsl adjustment over it remove all the saturation and finally add curves and just bring a little bit of contrast back in here just a Teensy bit to kind of bring back some of that shading that got lost over time and in the original photographic process now you'll see we will get some noise returning so we can just kind of Clone tool some new spots that may have appeared there this could be a rough process that's okay and now we're going to save it export as a PNG and funnily enough even though our canvas is really big we can bring this back down to 512 by 768 it doesn't matter really because this is going to go into control net and what that does exactly we'll see in just a second I'm going to iterate my files I'm just going to call the first one I export one that's a great way to kind of save these and now we're in stable diffusion so the first thing we're going to do is before we forget import our images into control net so that's this new tab down here that we set up earlier control model 0 we're going to bring in our one.png and this will be turned into a depth map so for that we hit enable preprocessor depth model depth and that's already it for the second control model we're going to bring the same image in enable and set the preprocessor to canny and the model to canny as well now down here we are going to change the thresholds so low threshold I found a good value is 30 and a high threshold can be 130 that usually ignores some of the extra noise right and gets the the outline of the person well of course this might be different for the photo you're editing so make sure you play around with these settings if you're not getting the exact results you want and for the third control model we're going to get a pose to really make sure that our guy remains pointing the same direction we set the preprocessor to open pose and the model to open pose same we don't change any settings and now up here scrolling up we're going to change our sampling method to DPM plus plus 2m Keras that in my experience is just a really good sampling method for this process our sampling steps we can set up to around 30 is good I'm going to set my batch count to four but you set that to however many options you want to get see if G scale is great width and height we change the height to 768 of course so we get the same aspect ratio as our input this is why I wanted the input to have these settings already pretty close to what we want or exactly to what we're going to enter here and last but not least our prompt now before we type anything in here there's a pretty cool trick we're going to go to the image to image Tab and load our image in here and here we have a button that says interrogate clip and this is going to try to recognize what's in the image and give us a head start for our prompt so let's click that now funnily enough this actually goes pretty hard on my GPU so I do have to stop recording when I do some of these things but as you can see it worked we now have a man in a uniform with a beard and a mustache in a black and white photo with a white background good now obviously we don't want to take the black and white photo thing over from this because we want it in color but everything else is pretty good so a man in a uniform with a beard and a mustache we're going to copy that bring it into text to image and start our prompt like this now I'm not going to get into all the historic research but knowing that he is on the union side in the Civil War we can figure out some colors we want right so if I look up Civil War uniform union you can see most likely this man's uniform is a dark blue with golden buttons so we can add that to our prompt um dark blue uniform with a beard and mustache golden buttons and now some other things that um on Reddit among others has worked well for people is starting the prompt with raw so raw photo with a weight of 1.5 another way you could weight this is actually selecting these words and Ctrl Arrow key up to weight this to 1.5 those are two ways and what I also like doing is adding healthy skin to really get a good skin color in there high detail high quality and Studio lighting now negative prompt is almost more important than the positive prompt it's pretty long though so I won't read it out loud I'm just going to copy it and you can get it in the video description and copy paste that as well it just basically makes sure that it looks realistic rather than being a cartoon zombie is great to have in your negative prompt when making people because you know people look healthy zombies look pretty much the opposite of what a person should look like so that kind of negative prompts it that way and last but not least black and white in there ugly colors and we don't want them looking sick or pallid so I kind of add that in there as well good now we're set up for our first round you may have noticed that the only input we're giving is in control so image to image comes later if you know what that is that we don't need that in this step and now it's time to generate before we continue I want to share this video's sponsor with you brilliant.org as creatives I think it's very important to understand the tools that we use so I was curious about the technology behind stable diffusion I found the perfect interactive course about artificial neural networks on brilliant and was able to learn the fundamentals in just one short interactive lesson per day you can pick from thousands of professionally made lessons for whatever level you're on brilliant has content from basic to advanced math AI data science neural networks and more with new lessons coming every month brilliant is built for people like me who have a busy day-to-day job and want to learn real problem solving that we can actually apply so it's a great low pressure fun and Hands-On way to learn I sincerely recommend it if you want to understand AI like I'm learning to or check out anything else brilliant has to offer for 30 days free click the link in the description or visit brilliant.org Albert BOCES on the first 200 of you who sign up via MyLink will get 20 off your annual premium subscription should you choose to subscribe now let's get back to storing that photo great and here are our first results before we look at the guys close up I want to show you these three Maps it generated so this is the so-called depth map where you can see it got the outlines of our guy pretty much perfectly even if some depth you know is missing in his face or details this is more than enough this is our canny map I would probably adjust the threshold like raise it like bring them a little closer together to get more detail out of here but it worked pretty well already and here is our open pose data so this one I'm going to just save and we can bring it in here and just call it open pose because then we don't have to pre-process again same thing with the depth information we can just reuse that one and the canny I think I'm going to check out the guy results first so here we go you can see it really colored very well I mean the jacket is almost perfect our guy looks too different from the original I think and again it's like giving him a side beard we may have to fix that manually I'm not entirely sure yet and here it did something weird to the sleeve but in general the results are quite excellent I'm just going to run it again with a better canny map and adjusting my prompt so to do that first I'm going to replace the results we like with open pose for example we can remove the preprocessor here same thing with depth remove the preprocessor keep everything else now for the candy thresholds like I said you have to kind of figure out your own numbers here but for this Union nurse I'm going to go for 40 70. your aim is to find a good balance between detail in the canny and limiting your model too much for example my guy has some lines here that stable diffusion seems to mistake for Beard growth and I don't want any beard there so that's what we're trying to balance and let's check out the background here so what is our background actually I that looks like some kind of wood I suppose and wallpaper I don't know if we're going to be able to reconstruct that we might have to do some in painting here later I'm not sure let's just uh put background is wallpaper for now and instead of the beard I'm gonna do with a goatee because that's what it is and try clean shaven cheeks and see where that takes us great so here's our new canny image and you can see it did add some more detail we do have some noise here but that should be fine most importantly we have the rest of the uniform so she we shouldn't have any more sleeve issues then the results it made me are pretty good too they look kind of painterly but that's our next step right so they all look pretty cool and I think this is my favorite now this is not close to final right so what we can do is right click copy this and bring it into Affinity or our editing program and paste it it's going to be too small but that's okay because we are only using this for color information after all it's still not perfectly close to our original guy so for that we're going to change our blending mode to color it's going to look worse of course just at first right but what we basically got is we've colored in this original image and now we can go to our original image and lower the opacity just a little bit to create kind of a nice little mix between our dude dudes here and if we want we can already adjust the face which is going to go further and further off from our original if we want to and for that in Affinity I'm going to go to the liquify Persona in Photoshop it's also called liquify it's just an effect so we're going to make sure our layer is rasterized and then we're going to click liquify Persona now in Affinity we have this little tool here where we can just kind of push around elements of the face so I'm going to bring my eyes over a little bit basically what I saw as a difference between our new guy and the old guy kind of Click between them and see you know his eyelids are a little lower and so forth that's what I'm trying to fix here but don't be too detailed here right and once you have something you like you can also adjust the curves a little bit so this is already closer to what you want as an end result so make sure your lighting is here how you want it you know your contrast Etc for example I also kind of want to desaturate this a little bit just a Teensy bit excellent now we can save this as number two and again make the size small and now the one we liked we can click the send to image to image button and it's going to bring all of our settings over here now we don't want to use this image of course we want to remove that and bring our number two in here but the button still made sense to bring all the features over all the settings now make sure you set your sampling method back to DPM plus 2m Karis sampling steps everything else should be just fine and our denoising strength we're going to bring down a little bit of course because we don't want to change the colors and stuff here too much this is just going to be um basically optimizing this result over here removing some of the noise Etc now with control net we can bring in our depth again no preprocessor depth we can go back to text and save our new canny image here bring it in and because we're changing so little we don't really need to use the open pose anymore I think it understood what we want and now we're just going to produce another version and as you can see it kind of just cleaned up our image a little bit it removed some of the weird grains and stuff we just have some new dudes here now of course you may notice the face has changed again and that's what we're changing we're fixing in our next step we're going to send this image the one we like to extras and here we're going to resize this by two with the upscaler our ESR again 4X Plus and now we can just click here copy image and paste it in here and immediately we'll have our new guy in here now how do we get the face back though right well we're going to mask quite simply we are going to create a mask on this layer and slowly but surely paint our original guy's face back in now that's why I like Affinity because it has this nice little masking preview that photoshop doesn't so we're gonna especially take care of the eyes here because those are really easily um recognizable our mouth survived pretty well the nose is a bit different so we're gonna get the eyes and the nose back now he looks a little sick because it's a little redder but that's easy to fix we just need to put another curves adjustment onto our color and just go through the different channels and try to adjust the color down right so there's a little less red good and our guy has his face back it's that easy so now we export this version and this time we keep it at the full resolution so export three we go back to image to image bring our three in here and hit in paint now after a couple seconds you will get this brush here and with that brush you are going to select the facial area that you want to adjust and give it a little bit of a margin right just make sure it has enough room around the features you want to change like that and now down here you will see these settings you need to set the inpaint area to only masked so it uses the full generating resolution on just the area you want to edit that's very important and width and height it's kind of square shaped so I'm going to go with 512 by 512 for this area and keep all the other settings denoising strength is okay and everything else down here too and now we're going to hit generate the results are done and here we go we have a couple different options all of them very handsome this model makes people very handsome so they all kind of have more of a smolder than he does this one looks closest to him I think the beard is very similar he has kind of the same nose shape I think this is actually the closest so I'm going to bring him over here copy him and paste him on top of all the other images here it's kind of lost a little bit of color but now we're entering the final stages right so we're very close to the original we can duplicate our original and bring it up on top just for you know comparison purposes and also copy any curves adjustments we may have made we can go back to liquify and just bring them together a little more so here's where we kind of enter our manual editing phase now you can also mask it a little bit right so if we do have some areas that we want more fine-tuned like for example where his eyes meeting and what's also helpful if you're editing really individual channels RGB you can also just remove like un show the other channels and adjust right if you go to composite green for example and change this to Green you can easily see any differences in shading and more easily adjust the colors and blue great now this is as far as I'm willing to take it but obviously you can edit as much manually as you want after this go through more in-painting um iterations so there's a lot that can be done to bring these two closer together I mean obviously there's still a little bit of difference between their faces but you saw the tools that you could use to bring these closer together if it's for example a loved one that you actually know what they looked like you have more reference pictures you can spend as much time as you like on it but I think this is nonetheless an excellent method to really accelerate the process of colorizing or restoring images and is great in any workflow whether you're a professional restorer or colorizer or just doing it for fun as a cool gift for somebody so definitely worth a shot and a lot of fun to find these old pictures and make them pop like this so I hope you enjoyed it as much as I did and learned something if you did feel free to leave a subscription and a like to help me continue this channel if you have any questions leave a comment down below and I try to answer as many of them as I pass possibly can and again make sure to check out today's sponsor brilliant.org for thousands of courses for math computer science and much more and as always I hope to see you next time I'm Albert and I hope you have fun with stable diffusion and control net

Channel: Albert Bozesan

Views: 45,272

Keywords: ai art, concept art, stable diffusion, midjourney, dalle, open source, artifical intelligence, photo, photo restoration, history

Id: dHQ4rkAGghs

Length: 23min 56sec (1436 seconds)

Published: Wed Apr 05 2023