Faster Video Generation in A1111 using LCM LoRA | IP Adapter or Tile + Temporal Net Controlnet

hello in this video we will be using LCM in automatic 1111 order to generate videos 3 to five times faster using video to video method in image to image image to image video generation is simple doesn't require extra extensions to use and gives us the ultimate control over the generation as I have explained in a previous video we will see LCM without control net and some nice control net combinations that may produce some interesting results as you can see here while LCM sampler is not yet implemented in automatic 1111 LCM Laura still produces good results using a a sampler for example as we will see in this video so we use the V resolve which is a free software in order to generate the BNG sequence now from the settings we would use vertical resolution for example for a sample video I want a video with 720 by 1,280 I want the timeline frame to be for example 16 now in Photoshop we can have more control over the number of frames for example we can generate 10 frames per second more control and it's also very fast in Photoshop we save now we bring a video for example if we use this video this video from free pick I'll include the links in the description we just drag and drop it to this location for instance change frame rate don't change because this video has a different frame rate for example we check the details we can see a different frame rate all from the information here so I wanted 16 frames per second now uh okay now we go to deliver we make sure that the settings are correct I want it like this so from format which is something setable such as BNG I explained this in a previous video BNG and we reduce the compression ratio because I want don't want the file size to be so large now we choose a file location for example if we choose this location add to render cu and render all this will produce or convert the video into the NG sequence we see that it has been completed very quickly now if we come here we can see that in the face we have all the images so we bring a frame into image to image in the image to image section we can use this button to get the dimensions of this image now initially we will be using we can use interrogate Pro for example this will allow us to detect what this image is exactly we can adjust this by adding Aura such as the LCM Laura now LCM Laura will allow us to generate an image to image okay I can adjust this prompt a little bit such as like this now LCM Laura is very important in this case because we want to generate something very quickly so we only need to use five samping steps for example four to eight is is perfect now regarding the CFG scale for LCM Laura because I'm using LCM Laura we need something between one to two maximum so I'll be using 1.5 now I'm going to use control net so we will have a large noising strength it's possible to use maximi maximum ding level which is one which will produce an entirely New Image but it will be controlled with help of control nits now the control nits I'm going to apply now uh this is a face image this is why we don't need to use after detailer if the face was small in the image such as full body then we need to apply after detailer just click here and just add a Lura we don't need to use a prompt for the face and or we can use it you can we can use a specific prompt for the face but in general we don't need to do that uh for simp we just keep it like this I'm not going to use after detailer because the face is very large in the image now I'll be using here three control Nets this is experimental the tile control net will reduce the details the level of details of the image in general we don't need to use a pre processor we can use bu silf with contr is more important and all three control Nets we will use control net is more important okay this is just experimental it's not really a must to do now since we are using only five sing steps we should start the generation from the first step till the last step now if we are using more steps then we can increase this value as I explained in previous video the second control net is temporal net the temporal net is used to increase consistency between frame generated frames okay I'll be using no pre there will be no preprocessor the thir control net is soft Edge it's possible to use another such as line out or or something like this but soft software soft Edge is good enough all of them will use control net now regarding the control net the temporal contet because it's not an uh one of the official control net 1.1 models we can download them from this page for example by going to I will provide the link below we just download this control net and base it in the models folder of control net this is the model folders of control net we just based the temporal net in this location now regarding the IML file this is the IML file we can download it along with tempol net and give it the same name in the location of control net folder now once we're done we have set up tile temporal net soft Edge we can generate a sample image click generate okay just double check five all looks good generate now the biggest time goes to the control net okay so the image to image only takes approximately 6 seconds and and often it takes half that time but because this is uh it's being it's loading control net then unloading it Etc because I have limited memory so the control net takes the longest time so we are doing image to image alone with CET that will take that will be extremely fast okay regarding noise multiplier I'm using noise multiplier of zero I have explained how we can make this zero in a previous video now once we are satisfied with the results such as this result here we can go to patch and put the folder of the frames okay this is the original frames this is the folder of the frames and this is the output folder this is where I want my generated frames to be once I'm done okay I'm satisfied with this result I can start press generate and generate all the frames of this video now that I have the frames generated we can check them okay we can see this is a great generation even using LCM law it automatically we are getting some very good results now soon when we have LCM s the results will be even better than this currently with a it produces acceptable results now in in com UI the image will look slightly better now if we assess fi with this image we can directly process it and combine the frames however if we think that we need to sharpen the image a little bit we can import all this image for example into toas photo studio for instance Now toas video will allow us to sharpen each frame individually or each image or we can generate the video and later on sharpen it using toas video for instance which is which are useful softwares to do these processes okay now for example we can sharpen US Standard we can sharpen all the image not just the subject we can recover faces if the face is not so good we can recover it as well give a suitable resolution for example 1.5 to get 1080 to to get HD quality for instance then save all the images we can change the image format into gbg and change the quality in order to save some disk space now it's very important that after the generation of the images to make sure that the images are in a certain sequence for example we come here click the first image contr control control a press need rename and give it some name and press enter and it will give us a sequence which is suitable for the V resolve now usually it's also better to check each frame individually because sometimes some of the frames are not so good so we need to remove some of the bad frames I always recommend to remove the badge frames okay so that our videos are of higher quality now once we are done with this we can Brak this image for instance we can bring it into we select the folder like this and drag and drop it into the V resolve okay now in the v resolve now it it treats the pngs as a sequence so so it would be like a video now now once we have the video down here we can change its speed for example this is 16 frames per second now while this video is 24 for example we can use something like time recol and choose a suitable speed for example if I want the speed to be 75 or I want it to be 50% 75 looks acceptable for example okay that's suitable I wanted something different I can also change the speed from this location for example if I want the speed to be through 70 I choose change and just come here and drag it so the frames would be 70% in terms of speed so now regarding the rtime and scaling now we should make this slightly smoother using retime and scaling settings by retime process which choose Optical flow now Optical flow would make that image slightly smoother and we can change the motion estimation into something like enhance better so this would allow us to produce a slightly more smoother image now this imag is of high quality but sometimes when you generate a video it's not always as good as this one so sometimes you might have more flickering for example if we try to change the style or not use tile uh tile control net for example we might get some flickering so in order to remove the flickering we come to the effects and open effects which just search for deflicker now deflicker unfortunately is only part of the studio version which is the commercial version so once we have this we can drag and drop the defa into this for example like this just drag and drop and we can drop multiple def flickers to improve to reduce the flickering but don't always it doesn't always work in that way so the more deflectors we have the slower the rendering would be now this is deflicker we can change the deflicker settings from timelapse for example to flly and test which one would produce better results so this one does not is not affective because it's already good so adding deflicker is just a waste of time okay so after we are done we just go to to to deliver and produce the video by choosing the MP4 and for example we just sort in this location for instance use the vertical resol okay this is a vertical resolution the project settings are not vertical save use vertical resolution okay this is edit vertical vertical use vertical here this looks acceptable now which choose a suitable path add to render q and render R you can see that the rendering is very quick and result once we're done we can see that our video is in this location we can double check it and we can see that this is our animation which is pretty decent this is using LCM Laura in automatic 1111 the point is that we are just trying to produce very fast videos to produce videos faster using LTM lers in automatic 1111 despite that the LCM s has not yet been implemented now if we want to use the LCM Laura for example without the contet which produces faster videos we need to use suitable sampling steps now the sampling steps as I've said before it should be four and 8 now if we reduce the denoising level by default this value will be would be multiplied by this value so I'll show you what does that mean now if this value was one then we should put something between for example example five this value was then this value should be ideally 14 15 16 something like that so we can generate this one now we can see here that despite having sampling step 14 it was multiplied by 035 and it give us five sampling steps so five sampling steps would produce the results very quickly using the LCM Lura and the results are pretty decent okay this is without control net now with control net we add more control and we would be able to CH to change the picture a lot more that would also take more time now if you are not comfortable with this computation and the multiplication issue you can go to settings and go to image to image and just checkbox this option which would render for example if you would five it would render five even even if the denosing level is slightly less than one okay but for me I find it very comfortable to use it this way the noise multiplier we can just right Mouse click here and inspect if you do not have it zero you just change it to in this location minimum we change the minimum here then you go once again to this slide bar here and inspect and change the minimum value to zero as well so that you can set the the noising level the image the noise multiplier sorry the noise multiplier to set it to zero this would allow to produce smoother images and reduce the changes produced B based on a certain prompt now assume that I want to change this image more dramatically more dramatically and I want to example add something firey and magic or whatever and change the dress more then the best way to do this my opinion is to use IP adapter so for example if I generate an image and text to image like this one okay I take this image and go to image to image then in the control net I would go to control net and use something called IB adapter now in the IB adapter if you want just to change the face you can use the face module if you want to change the style of the image then upload an independent control image like here press enable exect and use IP adapter then I can use more control nets for example I can use another control net such as temporal net I can use a third control net such as soft Edge I can also also use open PA for example so with IB adapter the IB adapter allows us to transfer more of the style of the image into the target image more more like the face and the hair and the dress and slightly of the background into this location now if we use IP adapter instead of the tile as the first control net this would allow us to ruce uh more changes into the target image for the deising level because I want more changes then I should increase the deising level instead of 0.3 for example now lower D nois level allows us to maintain more of the style of the image the higher we would reduce the style of the image so we can use 7.5 0.75 or even up to one and press generate now for example if I used IB adapter at a SP pH control net the second control control net could be something else sometimes I use uh temporal net other times I use open pose for example now in this example it's just an experimental process now open pose normal maps and the point of open pose is just to maintain the pose of the face the lips Etc because this is what open POS DW does the facial expression the nor of the point of normal Maps is just to separate the background from the foreground and fill the background with something different okay we can also use temporal net and see if which produces better results so this allows us to generate higher level of changes in the output okay even using the LCM Laura now like I've said this is uh rather experimental but in general if I want to change to to the results to have the style of this image then an IP adapter could be the way to go now reference could also work sometimes but I see that IP adapter is producing some amazing results recently now unfortunately LCM laas did not work well with anim diff they produce really really bad results on automatic 111 possibly we need a new sampler for this process to work properly now anime def also now for allows us to have a video to drag and drop it here in order to guide the generation the problem is that with animate diff and using control Nets actually it requires stronger GBU in order to run in my case it did not work both with control net and animate diff so I did not like the results that I was getting so I did not display any of them in this video this is why I strongly recommend that if you are planning to work with anime diff to use comi at the time being so in general so LCM low is allows us to generate videos and images faster in stable diffusion which is which is worth trying and using now I hope this video was useful and enjoyable at the same time and have a good day

Channel: How to

Views: 27,901

Keywords:

Id: k4ZWJD6W8d0

Length: 18min 11sec (1091 seconds)

Published: Sat Nov 18 2023