IC-Light: Stunning Photo Relighting with ComfyUI and Stable Diffusion

hello everyone this is Amon here from flow scale and in this video I'm going to be talking about uh IC light something that has recently just come up in uh uh and sort of is taking a lot of people by uh surprise because something now is possible which was not really possible with um with a a stable diffusion which is to be able to adjust light in a fashion that is very realistic now uh we let's deep Let's uh dig deep into that in this video but before that uh uh if you want to reach me out you can find me on Twitter with @ nerdycap 007 and uh I usually post about uh comfy and uh uh how to work around comfy optimizations on comfy and a lot of interesting stuff so let's get into it so comfy you u i has this new even before we lap into comfy UI let's talk about IC light first so what is icite uh icite is a project to manipulate the illuminations of image you can uh illuminate an image in different fashion and uh not only that uh it is um it allows you to you know sort of add backgrounds and uh do stuff with uh of by fixing a foreground it it allows you to you know add background and do a lot of stuff over there so to just give you an example like uh let's say this is an input image uh you uh it automatically takes the foreground out and uh uh uh based on the prompt that you have provided and the directional uh light that you have provided you can generate this sort of image now we'll experiment more with it um right now it has been launched on uh both on hugging phas and also on GitHub there are three models over here uh this one the FC IC light uncore sd15 uncore fc. safe tensors is the one which is the best performing model so we will uh use that across but uh let's go through their um uh hugging face space let's let's try this out before we uh move on to comfy UI so let me let me pick an image whom shall we try let's try Jason mumua I have an image of Jason mumua here let's try Right light okay so they also have a few um uh options to select out of here where which you can use as an example uh but let's let's see I I want to try something on my own though um man standing on a beach uh soft sunlight I have put Right light let's let's try to relight this very interesting now if you take a look uh the light is so natural as if it's a it's an image which was uh you know shot on a beach but uh there are a few like drawbacks to this process uh if you see like the distortions are happening a little bit on the face and it's changing the details to some level although I I mean real big kudos to the way they are uh still maintaining the details probably I guessing there's a good um sort of effort gone into just retaining the details now this uh look at this the hair and every everything is so natural here the now right now these details maybe are changing a little bit but I'm pretty much sure in the in the uh in the uh later versions with of IC light we are going to be seeing like really amazing progress there so let's let's try a few few more things um and standing in a living room um soft sunlight Through the Windows coming through the windows let's let's put uh left light this time let's try o nice but you can see uh the details of the clothing are changing like uh more than the details the coloring of the clothing are changing so as as I already said like it's retaining things to a good level but still there is um there is a lot of improvement that needs to happen in terms of uh retaining the identity to uh its fullest um they also like have a bunch of examples you can try out here where uh the they also have the seed and everything here but uh so these I generally when I try out uh um any new uh you know uh space or any new model that gets launched I don't don't try there uh generic test data because it's it's their best performing data you already know that right so it's better to try out on your own uh inputs and uh see the uh how how exactly is the model performing so uh I think that's pretty much it there are a few more options Advanced options here you can move around the steps more steps let's maybe try with um 30 35 maybe let's let's try with 35 I don't usually go above 30 because uh 30 is like enough uh for the number of steps usually for me but let's see I'm just trying to see if uh we can retain the these details which I'm talking about if we increase the steps maybe it retains it more who knows but see working with stable diffusion or any uh thing like that is like uh shooting an arrow in the dark but as you can see like it's again not retaining the uh um the clothing but clothing not being retained can be fixed a little bit to a good certain uh level so if I um if I go here and let me let me put pull this image out man standing in a living room um okay let's let's go through the same one actually let me relight this dude oh my God okay this is another uh issue here as well like it's darkening the uh the person the foreground a lot I don't know how to fix that here okay but but uh now if you see the image the the the the t-shirt is getting a little whitish over here let's let's try adding that actually yellow shirt um and black pants let's try adding that there you go uh so as you see right uh you can do a lot of stuff like you can do uh um uh if there's a major use case of this in fashion uh industry uh I mean the output right now which you're seeing I'm pretty much sure they are using an SD 1.5 model at this point I guess I can just tell from seeing the outputs but uh and also we are also using SD 1.5 in the comi workflows that I'm going to be showing you right after this but uh the the output uh here is good it is retaining the details the the texture and the structure of uh the clothing and the person uh as if it would happen like if you are using canny Edge detector to a very good level this something like this you would see but uh it's still changing the details a little bit I'm pretty much sure that will also improve as I already said but uh you see right to a good level you can solve these issues by specifying the color of the shirt pant and you know specific details but um yeah there's a there's a chance you will hit a roadblock uh when when it's like any any sort of complex clothing uh that's where you will hit a roadblock here okay so let's see how we can use this in our comfy workflows what can we do here so to move on to the comfy side I want to show you the comfy node that we're going to be using it's comfy UI IC light and it is like uh pretty new 3 4 days ago or probably last week people have been working on this very very new so and they also specified some of the example workflows require the very latest featur features in KJ nodes so if if you try out their example workflows probably just have KJ nodes I also have installed KJ nodes it is mostly used for this uh lighting setup I'll show you that um yeah so let's let's uh dig deep into this I've already set up a a workflow which I'll attach in the description of this video you can go through that and let's uh let's try to see what what we can achieve here so uh just to explain the workflow uh this is the input image which I am taking in and uh first of all I'm using uh image remg uh for removing the background and just isolating the foreground and uh after that that image is being resized while also keeping proportions which is very important if you lose the proportion it'll be well you'll lose the proportion so um uh after that this aspect resize mask is coming from the mask area of this I'll show you how exactly am I using this and then uh the output from this the mask that is being created is going into grow mask with blur these two nodes are from KJ nodes and then we are converting it uh converting The Mask to image just to show case here and uh uh to Showcase here and also we are putting that in vaen code here which is eventually going into a k sampler now apart from that uh we are also using as a base model we are using epic realism SD 1.5 model which is this model the uh one uh which has been there for a while now and that I'm connecting to the load and apply IC light uh node in which we are loading uh the sd15 FC you can download this uh uh uh this and also there is IC light conditioning you can download these models in uh um custom nodes right out of comfy so if you open comfy and uh let me check out see IC light uh if you just search IC hyphen light you can find out this node comi IC light and you can directly install this and if you want to install the models you can go and search I see light uh and you'll find all the three models all three models have different purposes but uh I generally I'm using uh the one which has FC because uh it's written on their GitHub page that that's the one which is the most uh best performing uh model out of their uh uh different uh options so uh the multiplier uh I'm not very sure what exactly does it do but uh what I found is like around. 2 183 or whatever at that uh uh number it's working decently fine um there are some negative uh negative embeddings and negative uh um uh prompts that I've given here and also some positive prompts which I have connected here uh here and eventually the K sampler throws an output VA decodes VA decode happens and you see the output here okay now let's try uh running with this also one more uh thing which you won't find in the uh in the uh hugging face space demo is that uh I have also introduced a IP adapter here so that is a a good thing uh that we can do like that's that's the good thing about comfi you have a lot of freedom in comi you can do a lot of stuff so right now what I'm doing here is I have introduced a uh IP adapter for uh giving a reference image for the background right so uh I'm doing something like that now this is not the traditional way to uh do the background based uh relighting on uh IC light there is a separate way to uh properly uh you know uh overlay something on a background but the problem in this is it just you know puts the person the foreground on this back it takes this as a background and this takes this as a foreground and just puts it on top of this right so I mean that is doable but eventually it will look quite fake to be very honest so uh but but the good thing is this is if you if you really look at it this is like in painting on steroids because in painting so far has been only um uh this is not particularly in painting this is more like cut uh cutting and pasting by mask or so in the previous previous uh you know uh uh um iterations if you if you did something you would be doing cutting and pasting by mask but right now um you can do it with the lighting and everything in place this is so awesome otherwise what usually used to happen is you cut by mask and paste this dude here and then you run a k sampler on top of the final output uh final image to with a very less denoising strength to make it as if you know it's blending in now what makes things blend in only lighting right so that got fixed so you don't have to run a separate uh sampling step right so there is a separate way to do this uh background conditioning and everything but the way we are doing is quite different we are just introducing uh IP adapter um because I just want to give a reference image I don't want to uh do a proper background uh sort of setting okay now let's run this uh enough talking let's run this let's see how we can uh add some lighting on this person so let's say I want uh right side left side lighting okay let me put this and yep and I'll save this okay uh photo of a man in a living room let me add [Music] um see uh without yellow there there was no yellow shirt so you see right it was changing the uh the color yellow shirt black pants black pants let's see so you can see the mask is generated through the KJ nodes uh this uh this node grow mask by with blur resize mask through this and voila there you go so uh now you can see right this uh background image is coming from sort of this let you can also try with you know without uh this of course let me try to put this person in a let me let me bypass this node for now let me try to put this person in a garden or something for man in a garden yeah yep beautiful it's looking very good although the color again is changing but it's it's coming out really well actually uh if I were to just look at it from uh a beauty perspective it's coming out really well okay so that is IC light for you guys and uh you can also uh I'll share this workflow in the chat in the in the description of this video you can go ahead run this workflow on your end the best way to run any workflow is to just load that workflow use manager to install uh comfy manager to install your uh Missing nodes and models and you know just experiment further so thanks again guys uh do not forget to subscribe to flow scale uh we will we are posting uh different uh you know new innovations that are happening in this space and how it can benefit different people uh so just tune in and uh more interesting stuff coming uh along the lines thanks guys

Channel: FlowScale AI

Views: 378

Keywords:

Id: tCoknRU4c-U

Length: 20min 11sec (1211 seconds)

Published: Wed May 15 2024