Transform Video to Animation in Stable Diffusion | How to Install + BEST Consistency Settings

in this video I'm going to show you how to use AI to generate animations out of a real video and yes you can use the same workflow to create various animation Styles we've previously learned how to use disco diffusion to create more complex animations but today I'm going to focus on the best tips that you can use to maximize consistency in your outputs using stable diffusion and a couple of other tools to begin install the stable diffusion interface on your computer by following the link provided in the description be sure to download the latest version once you have downloaded the setup file go ahead and launch it the file is safe to run so click on run anyway and choose where you would like to install the UI launcher I will go with desktop here if this is your first time installing stable diffusion be sure to enable clean install and click on install this process should only take a few seconds and once the installation is complete open the new web UI launcher file you will start updating some settings and loading a bunch of files it will ask you if you want to download the stable diffusion base model go ahead and click yes now you might be wondering what this means and what we need it for to put this simply models also known as checkpoints are pre-trained files used for generating a variety of images the version 1.5 model is a general purpose model that is suitable for most use cases however if you're looking to produce specific Styles and Aesthetics there are many specialized models available and we will shortly get to which one we'll be using in this video upon installation the base model will begin downloading this process may take a few minutes to complete once the download is finished you will be presented with another window where you can customize additional settings enable the auto update settings you can enable this if you have a graphics card with less than 8 GB of vram to check how much vram your GPU has go to start and open the run app type in DX diag and click OK look for your GPU under one of the display tabs and search for display memory and if your GPU has 8GB or more vram you can leave this unchecked enable xformers and click on launch web UI so once again the installer will start downloading a bunch of stuff you'll see a new folder appear in your directory called stable diffusion Dash web UI just be aware that this might take a bit of time depending on your internet speed once everything is downloaded the stable diffusion interface should automatically launch on your default browser to run it manually open the web UI user batch file and type this address in your browser in this video we'll be using the image to image feature so switch over to that on top you'll find all the available diffusion models currently we only have the base model available but I want to use something different that can give us more of an animation or cartoon style so the model we'll be using is called Arcane diffusion which is inspired by the popular arcade series I'll leave a link to the model in the description and to download it head over to files and versions and look for Arcane diffusion V3 checkpoint right click on this arrow and choose save link as then go to the stable diffusion web UI folder find the models folder then stable diffusion and save the checkpoint file head back to the stable diffusion UI click here to refresh the checkpoint list and now you're able to switch over to the Arcane model before we start working on the style head over to settings look for face restoration enable code former and bring the weight down to zero this will allow us to keep the stylized face as close as possible to the original one click on apply one more thing you need to do here is to go to the user interface look for the quick settings list and comma typing in painting underscore mask underscore way click on update settings and this time click on reload UI you'll notice a new slider has been added on top we're gonna keep it set to zero for this example now to stylize the video using stable diffusion you'll first need to export individual frames you can use pretty much any editing software for this I personally use Adobe Media encoder to do this import the video you want to stylize open up the settings and change the format to PNG you can also lower the frame rate to save time and give the video a cartoon or animation feel enable render at maximum depth and use maximum render quality then select an output destination I like to create a separate folder specifically for the frames once you've done that just hit the render button and the frames will start exporting to the folder you chose now that you've exported the frames from your video it's time to head back to stable diffusion under the image to image tab click here to upload an image select one of the exported frames that you'd like to test the styles on first it's best to choose a frame that is sharp not blurry and has most of the elements from the scene visible especially if you're seeing shows hands or teeth at some point once you've selected the frame click on interrogate clip this will give you a pretty accurate description of what's going on in the image if you feel like the description is missing an important detail you can add it in to apply the Arcane style to the image make sure to add it to the beginning or end of your prompt scroll down to the settings and make sure the width and height ratio matches that of your image keep in mind that the higher Dimension you use the longer it will take to process alright let's keep the rest of the settings as default for now and apply the Arcane style to our image by clicking on generate you see the image change and it might look really cool but keep in mind that if we apply this effect to the entire clip it won't look consistent let's see what happens if we reduce the denoising strength you'll notice that the output gets closer to the original image and this is because the denoising strength dictates how much the AI can be creative at low values the AI will try to stick very close to the original image let's talk about the other importing settings here the CFG scale the setting dictates how strictly the AI must stick to the text prompt at this stage it's important to experiment with different values for both settings until you get the desired look A good rule of thumb to follow is that if you don't change the settings but keep getting a completely different look every time you click on generate it means you will have trouble achieving consistency in your video later on to improve the overall look let's reduce the denoising strength and in label the restore faces option as you can see the stylized face now looks a bit more like the original now I'm going to show you the settings that worked best for me but keep in mind that you don't have to use the exact same settings try to approach this with an experimental mindset and be patient because every input will require different settings for example I found that changing the sampling method to LMS cameras gives me more consistency I also bring the sampling steps down to 10 set the CFG scale to 1 and the denoising strength to 0.7 in addition to that I enable the image to image alternative test script and uncheck all the settings then I bring the steps decoder down to 10 and disable this as well and finally I enable the sigma adjustments let's see what it looks like now if we click on generate I think it looks a bit too similar to the original image so let's try reducing the sampling steps and see how happens I think that looks much better now it has a more stylized look that could pass as a computer animation keep in mind that changing settings gradually doesn't always result in a gradual change to the output you need to find the sweet spot for the effect you're trying to achieve and that's what makes this process a bit more challenging and fun at the same time once you're satisfied with the look you can click here to reuse the same seed of this particular output this step may not be necessary for this context but it will usually help ensure consistency across the other frames next head over to the batch tab paste the path of your original frames folder into the first field and the path to where you want the stylized frames to be exported into the second field and finally click on generate to start processing do not panic if the preview suddenly looks a little weird you can check the output folder to ensure that the frames are being exported the next step is to take the exported frames and Stitch them together into an actual video I like to use After Effects because it has a specific plugin that helps me achieve even more consistency right click here and select import file find the exported frames and select the first one make sure PNG sequence is enabled and import the file right click on the imported sequence go to interpret footage and change the frame rate to 15 which should match the frame rate of your original sequence now drag the sequence down and drop it over here to create a new composition and here you can preview the animation although I think it looks pretty good here there's still some inconsistency on the subject and in the background to reduce that I use a plugin called the Flickr it works on both Windows and Mac and it's compatible with several editing programs including Premiere Pro and Final Cut Pro you can find the link in the description once installed you might need to restart after effects before you can access the Flickr from the effects tab there are a few options to choose from but we need to go with the Flickr high speed apply it to the clip switch on the GPU usage change the time window to 1 and set the refinement mode to refine and you can see here how much reducing Flickr has improved the overall look of our sequence and to push for even more consistency you can duplicate the effect I encourage you to try out different settings here as the result will depend a lot on your sequence another great way to improve the overall look and feel of this video is color grading I like to play around with contrast shadows and Hues to polish the animation further now again just like the other steps the final look will highly depend on the input but if you're interested in replicating this look for your own sequence you can find the project files on my patreon page we're almost finished with our animation but there's one more step we can take to really make it shine first let's export the video click on composition and add it to Adobe Media encoder and change this to a video from format then click on the settings button make sure to select maximum render quality and enable render at maximum depth next choose where you want to save the output and hit render now because we chose a relatively low resolution when processing the frames through stable diffusion we may have lost some details and sharpness to fix this I'm going to use an AI software called video topaz AI to upscale the animation to a higher resolution import the video and let's go with a four times upscale ratio enable frame interpolation and you can change the output settings as needed then click on export as and choose where you want to save the output the process will take a few minutes depending on the duration and complexity of your video once it's finished you'll be able to see the difference topaz video AI has clearly increased the sharpness and details of our video on this channel you'll find plenty of educational videos about filmmaking visual effects and digital art recently I started including more AI tools into my creative workflow and I can't wait to share more of my new findings and techniques with you I invite you to subscribe for more videos and if you guys have any questions about this entire process or would like to see me use stable diffusion differently please feel free to reach out in the comments below if you learned something new today give this video a like other than that have fun experimenting and see you guys in the next video peace [Music]

Channel: MDMZ

Views: 400,896

Keywords: ai animation, stable diffusion, img2img, video2video, ebsynth, after effects, stable diffusion tutorial, stable diffusion for dummies, artificial intelligence tutorial, artificial intelligence art, video to animation, video to anime, cartoonize video, how to install, automatic1111

Id: sVmi2Yp43c0

Length: 12min 50sec (770 seconds)

Published: Sat Jan 28 2023