Ai Animation With ComfyUI

hey everyone this is Bastian tus and I'm going to show you how to turn these 3D renders into lifelike images using comy UI and the LCM luras stick around to the end and I'll show you how you two can get a hold of this workflow for comy UI a couple of weeks ago I showed you how to use LCM luras to create animations with automatic 1111 and for the longest time ever I've been saying that I was not going to get into comy UI that has changed because comy UI has changed the game completely when it comes to animating using St diffusion so what I'm going to show you now is how I take this character which I've animated in blender and feed it through a workflow in comfy UI that's going to completely change it into a more realistic sort of vision now what are the benefits of this the benefits are that I can export this out through blender without having to use let's say 15 minute render times per frame I can render this out in let's say 25 seconds at most maybe 30 seconds using the Cycles Render engine because you can't really use the eeve render engine it doesn't really have the sort of lighting that you need to make it look realistic so let's have a look at this I've created this character using the human generator if you haven't used a human generator it's amazing I'm not sponsored by it but I really swear by it it will hide the hair so the view portal will move a little bit faster but we've got all these options here that we can change on the character so the head shape the muscles everything now with the standard clothing it's very limited it doesn't really have a whole lot to choose from so I've taken this flight suit and actually change the colors around and one of the shoes and change color around so it'll look like it's part of the suit now the good thing is because we are feeding this through stable diffusion it won't know the difference between this and the shoe so it'll think that it's all one piece which is exactly what I want and the animation that I've created I'll show you the camera angle let's play through that and it cut to a closeup of the face to do a little bit of a slow motion on the face and she flies away as you can see I've added this explosion this kind of like Shockwave effect and I've actually used embergen to create that but if I show you what it'll actually look like once it's rendered you'll see that the shock wave actually looks like smoke and we've got 15 seconds render time now this is crucial because we don't want it to take too long because what's the point otherwise so now the other thing to consider is the background what have I done to use to get this background use the hdri maker both of these are paid add-ons for blender but I really swear by them because they they save you so much time you can do this on your own but really to get the sort of effects that you can use using the HD maker it's going to take you quite a while to to get used to it because you can actually put the character on the ground as you can see we do have an actual ground plane that we can put a character onto and the the good thing is that you also get the Shadows which is fantastic now obviously I don't have that here because I've got her floating in the air so she's kind of like a uh superhero Supergirl sort of character pink hair okay and so what I'm going to do is actually export it out using the cycles and you'll notice that I've got 512 as our Max samples and I've actually put this down to 05 noise threshold if I was to raise this to 0.1 it'll be a little bit less quality uh but I believe it comes in like stock it'll come with 01 and the problem with 01 and I think 4K Max samples is that it will take you roughly about 5 minutes to render each frame of these scene we don't want that we want to be nice and quick nice and clean because it's going to go straight into stable diffusion it doesn't need to be 100% perfect and I'm also rendering at 1440 by 8815 pixels 25 frames a second and it's 118 frames just make sure you tell it where you want it to save make sure you save it before you start rendering just in case it crashes uh blend is notorious for that but it's still an amazing product now this is going to take a while so we're going to skip ahead and jump into comy UI now now this is the workflow that we're going to be using comy UI and I have to actually give a big thanks to AI warper who got me onto this workflow you've probably seen AI warpers work on X that Pixar styled Matrix scene looks unbelievable now that we're in here you'll see I've loaded the image sequence that we've got out of blender and I've changed the width to 1440 by 815 and over here we need the Pixel Perfect to be a similar size I'll be using the realistic Vision 5.1 this system actually works with anime diff I know that I've harped on about anime diff not being ready for animation but for some reason in comy UI it's more than ready I believe I'm going to run it through the 1.5 Laura here and let's keep the clip skip at minus one I've noticed that clip skip Works quite a bit differently to the way it does in automatic 1111 so just keep that in mind we got quite a few different things here so the our control net we're going to use the control gift which AI warper told me is actually the Holy Grail of animating in confy UI we'll also be using open pose using the DW pose estimation we've got this anime diff loader with the improved 3D motion model from civid Ai and the good thing about using the LCM again as I said in the last video we can keep our steps quite low so I'm actually going to go with 12 steps and our start step needs to be at six if we have it any higher than half of what our steps are it'll crash the workflow as a prompt we've got Asian girl with pink hair wearing a black leather superhero costume motion blur now I'm not too sure if this is like automatic 1111 where I should have this here or not but but I'll leave it in there for now and obviously i' got a negative prompt here so then I'll make sure to actually put where it is I want it to save so it'll save it within a folder called superhero 4 and it will actually name it number four and this is our video combin so it's going to take all our frames and turn it into a 24 frames no we want it to be 25 frames a second 25 frames a second MP4 video now the quality of the video isn't going to be the best but it's not bad once we have all that ready to go we can cue the PRP let's zoom out so we can see the process as it goes along and now we'll and now we'll click prompt and now we'll cue The Prompt and you'll see it actually runs through the entire workflow in this case this part of control net will take a little bit of time because we do have about 118 frames so we go here using open pose it's detecting one person it's going through each frame create an open pose image for each one now you can see that in these last couple frames it couldn't detect a person and that's because as you saw in the original video she flies away from the camera so at that point in that angle there's no way that open pose will be able to see what's going on now it's moved on to the vae encode it actually takes in the original image plus the vae so now it's over to the case sampler sampler and you can see that it's going to take a while here now it won't show you every single frame as it's going like we did with automatic 1111 it's actually going to go it'll it'll show you that it's done 25% 50% sometimes it does odd numbers but usually it's about 20 and it won't actually show you the frames the reason why it won't show you the frames is because it's actually saving it to the ram keep that in mind if you have uh low Ram you're going to struggle doing this especially at this size I would actually go a much smaller resolution and then upscale from there and now I'm going to skip ahead to when this is done because it will take quite a while and now it's going to save the images and it will send it over to the video combiner where it'll combine all the frames into our actual video now the good thing is that it's actually going to save the frames in one folder and then the video as well that way if you want the frames and not the video because it's a lesser quality you've got the original frames that'll work with as well if you zoom in here you'll see got all your frames I love that it's I love that it's actually given it regrow so it looks a little bit more realistic than what you get out of blender okay so that's finished now and if we go here and we do Show preview 's our video and we'll do resume preview now that looks pretty awesome obviously this animation is quite rudimentary but if you take the time you you can actually get some amazing results using this system so if you're interested in doing something like this using comy UI I'll be releasing a longer video for the people on my newsletter so that way it explains exactly what it is that I'm doing how it is that I'm doing it which models you require uh if you know how to use comy UI you probably already know how to get this up and going pretty quick and so yeah for the rest of you you be able to use this workflow to get started pretty quick let me know in the comments down below what do you think of this system are you going to be able to use it in your pipeline click like And subscribe and check out this next [Music] video

Channel: Sebastian Torres

Views: 13,621

Keywords: stable diffusion, ai animation, stable diffusion tutorial, stable diffusion img2img, stable diffusion controlnet, stable diffusion animation, ai animation generator, how to animate with ai, ai video editing, blender animation, stable video diffusion, stable diffusion anime to realistic, stable diffusion realistic vision, stable diffusion action scenes, ai filmmaking, ai film industry, ai filmmaker, comfyui tutorial, animatediff, blender tutorial

Id: YulLJ7ixBvg

Length: 9min 42sec (582 seconds)

Published: Mon Dec 11 2023