Easily Turn Videos into AI ANIMATION ! | After Effects & Stable Diffusion Control Net

if you're looking for any plugins presets extensions to save you time and create awesome videos my digital asset store is at the top of the description today I'm going to show you how to easily create AI video animations it feels like behind this is advancing at lightning speed a couple of months ago I made a video talking about how to use stable diffusion AI to generate amazing looking images the only issue with that when you try and create a video with that method there's a ton of jitteriness going on because each frame in that video is being processed with slightly different results so today we're going to try and fix that by using the new extension for stable diffusion control net also disclaimer everything in this video is free to download so let's dive in I'm going to show you how to set this up and then we'll start pumping out some crazy AI animations today's video is sponsored by surfsharkvpn as content creators you guys know we're always on the Move working on various different projects whether you're editing in a cafe or the airport surfsharkvpn can help keep your data safe and secure when using public Wi-Fi's surfshark lets you send or receive files securely it keeps your device virus free and it lets you change your virtual location on the fly so for example if I'm bored at night I can't find a movie to watch I can fire up the VPN and set my location into Canada and it gives me access to all those Canadian exclusive Netflix titles so that's really cool but more importantly keep in mind that everything these days has some sort of digital footprint I highly recommend you guys keep your identity and your data secure by using something like a VPN if you guys are interested in surfsharkvpn click the link at the top of my description to learn more alright guys so first things first to get started you're going to need to download the stable diffusion web UI this is great because you won't need to do any coding yourself you have this nice interface here and you're really just messing around with the sliders and tweaking the results to your liking so click the link down in the description I'm using automatic 111s stable to Fusion web UI here now the first thing you'll want to do is scroll down you want to come down to the readme section for installation and running and this will depend on the computer you are using so I'm using a Windows PC with an Nvidia GPU which you can see is recommended they have information here for AMD gpus or if you're using Linux or Mac so if you're running into any issues make sure you check here first so in my case what I need to do is install Python and you can do that simply here I'll also leave a link to this below where you can download python onto your computer once I've done that I need the actual code here if you guys are into coding you can clone the repository with Git I'm going to keep it simple and just click this button here and just download the zip folder so once you guys have downloaded and unzipped the folder this is what it's going to look like here you don't have to worry about all this craziness all you need to go down to is web UI user batch file so you want to right click and go to edit on this and where it says command line you guys want to put in what I have here dash dash xformers this will make things run a lot faster and dash dash Auto launch whenever you double click on this it's automatically going to launch the user interface anytime you want to use stable diffusion always open up that web UI user batch file so we'll fire it up and the first time you do this it's going to install all the dependencies so be patient let it run through now while stable diffusion is installing on your computer let's go ahead and prepare our video that we're going to turn into an AI animation so I'm just going to go on pixels.com and grab some royalty free shots of course you guys can use your own custom footage so in After Effects I'll click this button to create a new composition let's go 1920 by 1080. I'll drag in my footage I can right click transform and fit to comp so that we're all ready to go and then I can go up to file export and add to render queue so in the render queue just click on this lossless button and you want to change the format from Avi to PNG sequence next we can click here this is going to be the output location so let's create a new folder because again it's going to export every frame as an image so you want to have that all neat and tidy in its own folder and then render that out and now our video is being chopped up into each frame and we can bring that into stable diffusion once it's installed and there you go as you see we now have our web UI opened up so from here what we need to do is add in control net so a brief explanation of control net before there was no efficient way to tell an AI model which parts of an input image to keep but now with control net you get extra conditions for controlling diffusion models and that in turn creates more consistent results so we'll talk about that more once we start breaking down the different control net models but for now let's get this into our stable diffusion so we're going to go over to this extensions Tab and right here you can go over to available and just click load from and it should give you a whole list of different extensions that you can add into stable diffusion if for any reason that doesn't work you can also go to install from URL I will have the URL to the specific control net repository here but you should be able to find it here under available just search I already have it installed here so it shouldn't be showing up and there you see it right here SD web UI control net so this is already installed for me go ahead and click install on your end once you've done that go over to your command prompt you can actually see the process of the download once that download is complete go over to this install Tab and you should see it right here SD web UI control net now there's one more step for actually getting this to work and that's adding in the control net models which allow this to function properly so I will have a link below here where you can easily download these just click and download each one of these packages they're a little beefy once you have it all up and running it'll be worth it so download all of these once you've downloaded the control net models you just want to go back to your stable diffusion Master folder here find the extensions folder within there go to control net models and then just paste those models that you downloaded into here so once you've done that we're now ready to start piecing together our AI Creations you'll see your control net here you can have multiple of them all in tabs which is really nice one last thing I do want to go over before we start actually building our AI animation I've been using the word models a lot transferring models from one folder to the next what actually are these models so these model sets are images that are trained through the AI meaning the AI is taking those images and learning about them so that it can recreate that style from the lighting the lines everything and you can see right here with the base stable diffusion this is the default model set that should already be in there but a really cool thing you guys can do is download some custom model sets so these are custom models so these are custom models that are trained for a specific look so this is on hugging face again link below so you see this model set is trained on different images of Pokemon this model set is trained on classic animation so for example Pixar style things this one was trained on analog images so so for example like those viral 80s Dark Fantasy AI generation you guys could use these custom trained models to really hone in on a specific look so again this is optional you don't have to do this in a second I'll show you how to download these it's extremely easy just go to files and versions I'll show you how to download here this is what you want the ckpt so you can download that and then you just put that in your stable diffusion models folder so I'll show you that in a second get another site which I'll link below here civit AI a bunch of Community Driven trained models here which you can download so super cool to actually add this into stable diffusion you just want to come to your web UI Master folder you want to go over to models you want to go to stable diffusion and you want to put your diffusion checkpoints here and again that's what you're downloading these stable diffusion checkpoints I'm going to download the analog diffusion here and I also found this GTA 5 artwork one which was pretty cool so here's my checkpoints this is where I place the custom checkpoints that I downloaded I'm just going to copy and paste them in here once I've done that back in stable diffusion I can just click here to reload models and now we see them all here I thought that was worth the mention before we get into the animation workflow you guys can even train your own custom models so if you're really trying to hone in the look I know Corridor did that in that anime rock paper scissors video that's a bit more complicated I'm trying to keep it simple with this video so I can switch over to the GTA artwork model here I can type in Drake as a GTA 5 character dramatic lighting generate and there you guys go that easy let's try with the analog diffusion one we'll go Tom Hanks Polaroid picture wearing a yellow hat all right guys let's finally get started working on our animation so we're going to go to the image to image tab because again we want to process our videos that we've already turned into PNG sequences so we'll click here and here is the PNG sequence folder so I'll just select the first frame and now what you guys can do to start you can click interrogate if you'd like to if you want the AI to sort of think about what is in this image so you see here it says a man dress and a suit green hair with a green flower let's go ahead and just click generate and see what it thinks it is so again it's taking man dressed in suit green hair kind of just giving you this rough idea of that so let's start optimizing this to make it closer to our prompt so first things first you want to make sure your width is dialed in here so this is this is like a 1920 by 1080 image so we're going to want to try and keep this closer to 512 because a lot of these AI models are trained by 512 by 512 Square images but we still want to extend this a bit to try and get that wider aspect ratio so let's put this up to 812 by 512 and then let's go ahead and generate again so you can see our composition is more accurate in terms of the background in terms of the width of the shoulders we're just having some weird broccoli people Generations so let's go in now with our aesthetic that we want to apply to this so let's say a man dressed as the Joker comic book character DC Comics dramatic lighting Bloom something like that just to give it some nice filters so try and be specific here you want to say a man dressed just so the AI is kind of pulling from those folders of knowledge that it has on what a man looks like pulling from the folders of knowledge of the Joker comic book character so this is getting a lot better the only issue here is this is far away from our starting point so if we scroll down here let's actually look at the beginning settings you guys will see these two sliders here these are your most important CFG scale you can think of this as your stylization so the more you pump this up the more it's going to get all crazy with the style that's being applied to here denoising strength this is how much noise and blur is being applied on top of this so if you put this all the way up to one it's basically like you're looking at this image and squinting and it's trying to think and create based off that blurry start and it leaves more of the imagination to generate on top of that so if I put it up to one again it's going to be extremely far from our original prompt you see it's just completely different composition but that looks pretty cool now if you put it down to zero it's going to be basically the same as our starting image as you can see so that's what we want to do we want to find a nice in between where we can keep the composition while still getting some stylization so let's put this up to 0.2 and you see we're starting to get something let's check on restore faces here just to clean this area up a bit okay getting a lot better and if you want you can play with the slider these settings are always going to change depending on what you're inputting here so you don't have to follow exactly what I'm doing it's all about tweaking and finding that happy medium so let's put in some negative prompts here just to try and clean up our results so we'll put in blur we don't want any blurry images uh we'll put in noise we don't want any noisy images we'll put in jumbled face so looking a lot better and that's a basic starting point for what you can expect here right now we're set to random seed so you guys can keep clicking generate it's always gonna and it's always going to give you a different result so if you find a seed that you like maybe like this one for example you guys can click this little button here to save that seed so if I like this I can copy it I can go back maybe do a few more randomization just to check around got a little zombie Joker and say I want to go back to my other seed control V and paste that in and you should go back to your original original result with that seed assuming you didn't change around any of the settings so that is your basics for normal stable diffusion trying to get the prompts to match up now we're going to take it one step further here and we're going to use control net so this gives you way more control over the image output this is amazing I think that a tool like this is super Central for having full control over what the AI is spitting out now if you want to run multiple control Nets and multiple tabs just go to your settings go down to control net and bump up the slider that way you can have more than one control net running at the same time you see we have different tabs here because we set our control Nets our multi-control net in the settings so let's start out here with our preprocessor we want to enable this let's go over to preprocessor HED and then let's set the model to HED as well let's take this guidance and put it down to something like 0.5 and then let's just run this through and see what's given us all right so we can look at the HED filter here and it's basically just generating the lines the edges here and trying to match that up more accurately so you can see the jacket lines we're still getting a little bit jumbled you may need to just play with these settings a bit more so I'm just messing around with the prompt here and with these sliders to try and get a more stylized result this looks pretty cool starting to get more of that comic book style color just by adding in these prompts so again just experimentation let's keep going through here let's add in another control model to fully tailor the look and try and make the eyes look a bit more clear here so the second control net model I ran is canny and this is going to give you a lot more fine detail you guys can see my settings here and if we look at the generation result you can see what Kenny's doing really getting those fine detail lines so if you're going for something really stylized or cartoony something that looks hand-drawn I highly recommend men you guys put candy in there and of course there are a ton of other models out there that can help for example depth which is great for larger scenes with a bunch of things in the background you can create a depth map you can create normal Maps there's the open pose model which is great for really capturing the pose that your initial person is in those were the main ones that stuck out to me of course you can look through the documentation yeah here's where we're at right now I put the denoising strength up a little bit more just to get it a tiny bit more stylized and if I really wanted to go crazy with it I could even put a negative prompt like realism so that it goes fully to the realm of cartoony stylization which is pretty cool so I'll try and find that sweet spot in between the two and then let's just test this on some of the other frames in our composition here so whenever they're moving around because again the name of the game is keeping things consistent and there's not too much going on here so it shouldn't be that hard to do that let's try so there you guys go pretty consistent style which should help with the jittering so now we're ready to actually export this out into an animation so let's go over to the batch tab here and it's going to ask for input directory output directory so here is our input directory that we already have set up right click and copy this path and then let's create an output directory right here copy this path let's go ahead and generate this and check out what we get so I'm still in the process of rendering this batch but I quickly wanted to show you the difference between old results and the new results with control net so this is some older Generations that I did and we're gonna go frame by frame pay attention to the hairstyle here um or even just any particular detail like the coat the hair you see every single frame is so different with the old method compared to with control net here and if you pay attention to the hair obviously there is still some small deviation but it's so much closer to the last image the real deviation is coming from the lighting and we're going to fix that by using some D flickering once this is done but I wanted to show you up and close just the difference here and why I thought this deserved a full video because the improvements are great here in terms of our consistency check out our results we're going to hop back over into After Effects here and to bring it into After Effects we're going to right click import multiple files and let's find our output folder here select import as PNG sequence import as footage that's good click here and then click done alright guys so whenever you import that in here is your Raw results now as you can see still very flickery and that's the key here for cleaning everything up I'm going to show you two different quick methods so again if we pay attention here's just raw image sequence nothing going on there now here is going to be our first method this is the flow frames with d flickering I also put in some posterized thyme so if you guys don't want the slowed down frame rate here's what that looks like and then you have your EB synth method where you're basically pasting a frame over and in terms of movement that looks the most accurate but you do have some Distortion going on here I think this still looks pretty cool let me show you how we can do this for the flow frames shout out to enigmatic e he made a video on this I'll link him down below you guys should subscribe if you haven't already just go to this link I'll put it down below you can download this for free give it an install you've done that you'll have this actually run this again go to the interpolation tab you need to do is your input video so what I did was I just took my raw output here going to export add to render queue I exported this out as an mov so go down to QuickTime okay and then render so once you've done that you can go to your input video browse and then it should be right here jokervid.mov should analyze the FPS here make an output directory click here and it's as easy as that so that ran through you just need to go to your output directory so flow frames outputs next to tackle the flickering you guys are going to need some plugins here now a lot of people have been using DaVinci Resolve have a d flicker plug-in I know in the chord or crew video that's the method they used for that anime rock paper scissors video I'm not sure if that's paid or not it might be in the paid version I've been using this flicker free plugin from digital Anarchy and paid plugin for Final Cut and Adobe or flicker free if you want you can place a few of those there you can play around with your presets again it's going to depend on your clip it's going to depend on your lighting oh I think that looks pretty cool again the last method which is EB synth to do that what you need to do is make sure that your raw output here make sure that your raw output here and your initial frames are the same aspect ratio so if we go to Project we'll reveal this reveal this in the project right here you can see that this is 936 by 512 so all I did was make a new composition 936 by 512 drag in your original footage and fit to comp and then you can just export this out as a PNG sequence next you want to download EB synth very easy to download link will be down below you want to set your keyframes here let me make a new folder here just so you don't get confused with everything going on so for EB synth you want to make a keyframes folder and in here is going to be your stable diffusion image to image processing so let's go Joker output again these are all the frames from stable diffusion you can take one of these here Ctrl C paste it in and if you want multiple keyframes you can so for example if you want different parts if there's something messing up you can grab multiple keyframes from this folder but I'm just going to go with one so there's our keyframes folder this is the original video that you're pasting onto so so again this is Joker plate resize this is that PNG sequence that's 936 by 512 and that's really about it say your output here so I'll put folder and then just run if you get an error here it's probably because the aspect ratios aren't matching up so again make sure that the dimensions are the exact same so let's check out the output again here's all my frames that are being processed from EB synth that's good to go so there's your EB synth method and yeah here's the flow frames method after I applied this tutorial here just a little graphic novel comic look uh After Effects tutorial so you guys can have fun play into the flicker if you want put some cool filters or whatever I think this fits perfectly gives it that kind of comic book dark Sinister Vibe have fun with it I think this could be an amazing music video a unique type of look this looks so much cooler with the flickering and with all the processing going on then if then if I was to do like the exact same thing here with you know just our you know just the normal testing footage as you can see that is the power of AI you guys can really make some cool things and that's why I'm trying to work these type of tools into your After Effects workflow combine these different tutorials together make your own unique stuff that's the beauty of it so that's about it guys I hope you guys enjoyed today's video there's a ton of debate going on right now because AI is such new technology AI should always be a tool that's helping people create something so I hope as time goes on as things are figured out all the controversy Behind These AI tools is removed whether that's forcing them to Source their images from royalty-free databases or whatever it is it's not up for me to decide but I do know that I think that these things can help us a lot in the future I know that AI is going to power a lot of the systems that we're running through our creative workflows it already is it's embedded in things like after effects Photoshop and as time goes on I'm going to be covering different ways people like you and I can utilize that so if you guys did enjoy it slap like on the video for more content like this subscribe comment down below if there's any anything you'd like to see from me next and as always guys thank you so much for watching thank you so much for supporting and I'll see you guys [Music]

Channel: Max Novak

Views: 255,771

Keywords: after, effects, ai, stable, diffusion, dreambooth, img2img, corridor crew, vfx, artists, react, max novak, how to, animation, ben marriot, adobe, premiere, photoshop, control net, corridor, anime rock paper scissors

Id: zDvpJIp0rl0

Length: 22min 21sec (1341 seconds)

Published: Fri Mar 31 2023