Unlock The AI Video Potential in Stable Diffusion With Deforum Hybrid Video

in this video I'm going to review a free game changing technique that lets you create warp fusion quality videos inside of stable diffusion after following this tutorial you'll discover that it is actually quite simple once you know how to do it I want to show off some really cool art just so that you can see the kind of quality that people are getting out of this already this Creator unreal unit used to create videos with warp fusion but recently he has been only creating videos with this new technique so let's look at a snippet of one of his dancing videos I think that looks really good this one is in the unim style and I am very impressed with the consistency it just looks amazing and the last one we're going to look at is something which is very trending at the moment Dance videos in Lego style that looks really amazing look at that awesome the next Creator we're going to look at is stable swirls he creates different style videos but they look absolutely amazing like this animation of a guy in a room he sort of morphs into a zombie and a robot and all kind of stuff is just amazing and the last creative we're going to look at is really big name this creator creates many different styles and we are going to look at the transformation from somebody we all know and didn't expect that he could do all this kind of stuff look at him going that is amazing and here's another one to show off what you can do with this technique and the amazing thing is that without these three creators I wouldn't have been able to make this video they all deserve credits in their own way and you will find out more about this as the video unfolds so the first Creator to give credit to is Andre from unreal unit he reached out to me and showed me how to use a technique in the Forum to achieve these amazing results and although he's not the Creator who came up with this technique I can't thank this Creator enough for his exceptional support and if you would want a video on order then you could write him a message here directly on Instagram before we start the tutorial let's have a look at the clips that I want to transform but the first one is a dance video which is commonly created with warp fusion this is a video from monolith yet and I want to transform this into a superhero dancing in the second video I'll Transform the same dancing video into a Lego Style just like the video created by Andre the third clip features a cute cat from pixels.com and I want to turn it into a steampunk Style video and later in this video I have a great tip for using the AI from DaVinci Resolve to frame the subject properly in the center of your 9x60 video if your base video is 16x9 so let's begin by installing the Forum if you haven't installed stable diffusion and controller yet there is a link in the description to a very helpful installation video by Sebastian comp so now we go to the extensions tab then go to available click on load from and type The Forum and if you haven't installed it yet it comes up here and you can click on the install button mine is already installed and this is the one you are looking for install this one narrow is something very important that you need to do you have to uncheck all of the big extensions because these extensions could create problems when you use control net in the Forum you can leave the built-in ones as they are now click apply and restart UI I learned this from Stables rules he created this video where Android from unreal unit gained his base Knowledge from on his channel there's another video explaining how to fix control net and deform conflict issues so now you know why this Creator deserves many credits and although he's also not the creator of the technique I highly recommend checking out his channel as I'm sure he will create many more amazing videos in the future one last thing to do before we really start is going to the settings tab then go to user interfaces scroll down click on the triangle here and select initial noise multiplier now we're going to apply the settings and reload the UI and close stable diffusion this initial noise multiplier parameter increases the amount of noise resulting in more detailed images but in the case of the Forum we don't want a lot of detail to avoid flickering and by default it's on 0.5 so we're going to change it so we are able to set this to zero so in the stable diffusion models folder we're going to open shared Pi with notepad in notepad we're gonna go to edit then to find and then type initial noise then find text in here we're going to set minimum to 0.0 then go to file and save it now we can move the noise multiplier to 0 to 1.5 now let's begin with starting the prompts and the settings and the best thing to do is to start with this in image to image because it's easier to test your results and then transform this information into the forum and add additional settings there and the cool thing is once you have done that in the Forum then you can save these settings as a text file and use them every time you create a new hybrid video in the Forum okay now let's begin with the prompting for the superhero video okay in the image to image tab I'm gonna type in my prompt and the negative prompt and I will drag my image in here and the best way to export an image in the free version of DaVinci Resolve is to go to file then to export and then curved frame as still and then choose export for the superhero video I'm using the ref animated model so download this I am also using the bad hand V4 as a negative embedding and an easy negative so download these two now put the ref animated model in here and the two embeddings in here make sure that you use embeddings that are trained on the 1.5 model all the models I use in this tutorial are all 1.5 model I will leave all the links that I use in this video down in the description below so now let's start with the rest of the settings I will leave Euler a as the sampling method and put the sampling steps to 15. now I'm going to set the width to 576 and the height to 10 24. I would recommend to stick to this resolution if you want to make 9x16 videos I've tried different resolutions and well it just doesn't work that well I will set the CFG scale to 3 and the denoising strength to 1. then make sure the noise multiplier is on zero so now we're gonna go to the game changing technique that I mentioned in the beginning of the video which is using the dial control net model in the control net section of the Forum this technique was discovered by really big name he is the developer of the hybrid video compositing section in the Forum and she added in the deformed Discord crew he deserves many credits for this so check out his YouTube channel for more beautiful videos to get comparable prompting results we're going to apply this method also in the image to image Tab and later we will transfer this information into the forum tab so now we're going to go back to the image to image tab again so we're going to scroll down to the controller tab so we're gonna enable this enable Pixel Perfect then go to tile and now make sure in the preprocessor set it to none and then this is the right one I'm going to set my control weight to 1.75 and control net is more important now I'm going to add a second control net enable it not Pixel Perfect because this will slow your computer down then I'm going to go to open pose and select open pose here I'm going to leave the control y to 1 and set the control mode to control net is more important now let's hit the generate button okay this is quite a nice result and the reason why my character is not completely changing into a superhero is because of the tile controller model this ensures consistency later in the process in the Forum to get a more intense stronger results we need to use loras and I will show you how to do that in the transformation of the same video into a Lego Style video so I will leave my superhero as is and I will save all the settings to apply them in the Forum later okay now let's start the Lego figure I am very excited about this for this video I used the protein V 2.2 model and I'm going to use the Lego Laura as a Laura so I'll download this and place the Laura in this folder now I put the Lego prompt in here and the negative prompt and here change the model to protodine 422nim and I'm going to leave Euler a as a sampling method sampling steps to 15. the only thing I'm going to change is something in the control net unit here I change the control weight to 1.5 and in the open pose control net I'm gonna leave everything as it is as well and hit the generate button okay this doesn't look so bad and when I ran this through the Forum it wasn't consistent at all so that's why I asked Android from unreal unit if he could help me so he shared his settings file for his Lego video with me so before I'm going to apply these adjustments I'm going to show you how my settings look if you run off through the Forum well it looks sort of okay but it is not consistent at all and this is something that will change if we change the prompt and settings I'm showing you this to help you understand that the key to success lies in the prompting and the control net settings so here is the new prompt now and in the Laura Lego I set it to zero point three and a nice little tip is if you click and hear an ALT Ctrl and then go up and down you can scale it which is pretty and nice and this is the new negative prompt now we're going to change the steps to 13 and in the control net tile model we're going to set the control weight to 1.2 and in the open post control net we're going to set this to 0.8 and my prompt is more important for the control model and hit that generate button again wow this looks so much better and especially if you run this through the Forum then the video will become more consistent stay tuned because I will reveal it soon but first let's make this awesome steampunk cat so let's drag this cat image in here and for that I use the mechanical cat Laura for the model I used ref animated and this is the prompt that I'm going to use and this is the negative form that I'm going to use and if you're new to loras then put your cursor here add a comma and the space and then go to this button go to to loras and I'm also using the Laura Laura and we're gonna add it just by clicking on it click out of here again and now you see it's there and now I'm going to change it to 0.4 now I'm going to change the steps to 15 and leave the CFG scale and the denoising strings as they are in the control net tile unit I'm going to set the control weight to 1.5 and control net is more important and the second control net I'm going to change to soft Edge the preprocessor goes to soft Edge head the model here stays the same and I'm going to put the control weight back to 1 and the control mode to control net is more important and hit the generate button again okay that looks pretty cool to me now we are going to transfer the steampunket settings into the forum and this deformed app looks a bit overwhelming in the beginning but we are going to run through it in the most logical way starting from the left with the Run tab and then moving upwards to the right after you've done these steps once then you can save the settings and load them the next time and then it won't take much time to set this up at all and that is truly amazing in the runs app I leave the sampler to Euler a and I'm going to put the steps to 15. then I change the width to 576 and the height to 10 24. I'm going to leave the seat at -1 and give my file a name now the Forum will create a folder in your outputs folder in the image to images folder in the keyframes tab we set animation mode to 3D we don't need to set the video input here because that is something what we will be doing later in the hybrid video tab now I'm going to leave the Cadence to 2 Cadence refers to the number of frames generated at once if you set Cadence to 2 it will generate 2 frames but only the first one will be diffused the second one will be interpolated for videos with a lot of movement you set this to 1 and for videos with a bit of less motion you set this to 2 and I set border mode to wrap now we are going to move to these steps which are part of the keyframes tab the keyframes tab can be a bit overwhelming and I was a bit confused in the beginning too to make it easier to understand remember that these steps below are separate tabs and they don't belong to these tabs now in the strength tab set the strength schedule to zero in the CFG tab set CFG schedule 2 3 but this is also something which you could play around with to see what different CFG scales do for you in the C tab set seed Behavior to fixed because we want to have consistency in our video and we can leave subseat step sampler and checkpoint alone we don't have to change anything in here now we go down to the motion Tab and set translation Z to zero then go to the noise Tab and set the noise schedule to zero you could also try 0.02 but I would leave it at zero then in the coherence tab set coherence mode to none if you want to have your own color reproduction from The Prompt if you want the color reproduction of the original video then set this to video output in the anti-blur set amount schedule to 0.05 you can leave depth warping and fov as it is now we go up again to the main tabs and select the prompts tab now we are going to move our prompts and settings from the image to image tab into here make sure that the double quotes stay behind the colon sign and also at the back now paste it in and these are the quotes that I mean you have to leave them in because otherwise it won't work now I paste my negative prompts in here now we are going to load our original video and we do this in the initi tab but before in the image image unit set the strength to zero now go to the video in it select your video and copy path then paste that in here now make sure that these colons sorry colons know double quotes go away and then enable override extracted frames now we are going to go to the control net Tab and now we're going to move all the controlled info from the image to image tab into here so we're enabling it and also enable Pixel Perfect then in the preprocessor leave this to none and in the model set this to control tile and the weight schedule is the same as the control weight so we're going to set this to 1.5 and set control mode to control net is more important now we have to set the video path in here so to do that I go to the inner tab and I copy this then go to the control on it and in here control net input video image pass you paste it in then I will enable the second controller model enable it and I'm going to paste the path into here as well then I'm going to select Pixel Perfect now in the preprocessor you won't find a soft Edge head but you have to select head and then here in the model you select soft Edge my control weight was one so I'll leave the weight scheduled to 1. then for the control model this control net is more important now we're going to move to the hybrid video settings tab in the hybrid video tab you set hybrid settings to before motion make sure you select generate input frames for hybrid motion select Optical flow and set flow method to this medium then open the hybrid schedules by clicking on this triangle and set the comp Alpha schedule to 1. now we go to the output Tab and here I set the frames per second to the frames per second from my original video now you can click the save settings buttons and hit generate so the next time you want to create another video you can load the settings files and the form is prepared to load these settings select your file copy path and then paste it in here and then click load all settings and then the next time if you want to create a new video you just go to the image to image tab you do all your prompts and your settings and the control Network and you just copy them over into the deforum tab so before we're going to look at the video I want to let you know that I optimized the video I obscaled it I applied frame into position and I add the dirt removal and the deflector node in resolve I will demonstrate how to perform these steps and after that we will look at the two other videos that I've created okay let's look at the video wow I really like the video look at the consistency in the head of the cat and the background is just amazing I am really impressed about the video possibilities with the Forum and I really want to thank all the three creators for making this possible so to upscale the video I use topaz lab and I applied the frame interpolation here as well so that saves me quite some time with for the people that don't have toppa's lab or the studio version from Da Vinci resolve I'll show you a free tool called flow frames that can also do interpolation you can download here at this link and consider giving a small donation for the great work done on the program in flow frames drag your video in here then you have to either change it or it does it for you make sure it is on x2 and then hit interpolate so now we want to frame our 1024 video into a 1280 timeline and the best way to do this in resolve is go to file go to new timeline and then uncheck use project settings then go to the format Tab and change it into the one you need this one make sure your timeline frame rate is correct and then in a mismatch resolution choose scale entire image to fit now if I drag files in here with different resolutions then it will automatically scale the image to fit now change the speed to 200 to get the video back to its normal speed now I want to share the tip I mentioned earlier in this video about using the AI from resolve to frame and track a subject perfectly in the center of your 9x16 video so if you are in a 16x9 project make a new timeline uncheck use project settings go to format choose use filter resolution and then take the one you need this one I need and then in the mismatched resolution set scale full frame with crop and then create now if you drag your 16x9 video in here it scales it automatically to the subject so now you drag it from in here or from in here and ta-da now select the clip and go to the smart reframe and click reframe now I will analyze and do is AI work and here is the result not bad huh now we're going gonna go back to the cat clip now right click on your clip select new Fusion clip go to the fusion tab hit shift spacebar type in dirt and click enter then hit shift spacebar again and type in the Flicker and enter that as well you can leave Automatic dirt removal to default and the flicker node you're going to set to Floral light and don't worry if you don't have the studio version from resolve because I don't always use these options sometimes it works and sometimes it doesn't I think the frame interpolation does already a really good job and the other things are just a bit like you know tiny bit extras and as it worked fine for the Cat video it totally didn't work for the Lego video if I hit Ctrl P I can put the note on and off and if you put it on you can see that it's just like I don't know not so good so I deleted that and also for the deflector note I had to adjust some settings so I changed this to Advanced controls then I set frames either side to one motion raised to large threshold to 42.5 and the motion treasure to 43.3 now my background flicker is less but I've got tiny artifacts on the Lego person it's just you gotta choose now you can select the note hit Ctrl C and Ctrl V two times to have 40 flicker nodes now export the video and you're done we're almost ready to watch the two other videos but before that I want to mention that I'm very excited to see how this technique can work on photorealistic videos you know can it compare with a Tokyo depth temporal consistency method okay let's start with the superhero video I am pretty satisfied with the consistency of the background the subject is not completely consistent especially his head but I think that comes because he's wearing a hat I think if it would have only have his hair then they think it would be better so now let's have a look how the same video looks in Lego style wow this video looks so much better and consistent than the first attempt that I did I really love it so I wanna thanks Andre from unreal unit one more time for his impressive help and if you're interested in Tokyo depth temporal consistency method then you can click here to watch the tutorial and meanwhile I'm gonna work on this and if I've got something ready then I will put that here

Channel: Digital Magic

Views: 54,340

Keywords: digital magic

Id: vzssE1iW4bI

Length: 19min 46sec (1186 seconds)

Published: Fri Jul 28 2023