Stable Diffusion Consistent Character Animation Technique - Tutorial

hello everyone I have recently been working on a technique for creating animations using stable diffusion you can see right here two different results the character animation which was my pathfinding attempt I spent a few hours working on this developing the techniques different ways to work through it and then the actual orc that's attacking me was what I made in this tutorial so that's about a come up of how to make how I made that I spent about two hours working through the whole thing you could definitely spend more time cleaning it up making more frames cleaning different things up creating different directions so it really depends uh this was without a character Laura I also created a whole bunch of scripts to help me with creating them and I tried making them as non-programmer friendly as possible so if you have no program experience you should still be able to utilize them to help create whatever animations you want to so if you want to figure out how I did it and how you could do this yourself then stay tuned and I will go over everything so the original idea for this came from XYZ disk on the stable diffusion subreddit who showed off a technique for using a turntable Laura and open pose to get consistent characters without a specific character embedding and I sort of modify this technique and extended it Beyond and created some helper scripts to really allow for these larger scale Animation Sprite sheets to be generated so the GitHub that I mentioned uh it's linked down below here are all of these scripts and they read me to how to run it but it's a little more difficult to explain this kind of thing in text so I will just be running through going down the list and showing exactly how you would go about running this so the first thing that you want to do is you want to download the code so you can either download the zip or you can click git clone it so I run into here get clone and I've got the files here completely raw like before the next thing you want to do is install all the pit packages that you need if you don't have them it's pip install opencv Python numpy and REM BG I believe that's all of them if it complains about more just Google it and install what it tells you there's also an interpolation that I'm running locally this is not required it's optional uh things were a little more difficult with this one I had to make a anacond environment with python 311 and install tensorflow and tensorflow hub in there along with these three but I'll explain that in more detail once we actually get to that part of the tutorial so to initialize the workspace there's a few things so the first thing that we have to do is we need to move this over we need to create a folder in here called workspaces and then we need to make a folder zero zero zero it's currently the default and if we open up in here there's also this file called settings.pi this is our main place this is where it expects to find certain things so this is setting our workspace to workspace zero zero zero if you want to pass it a a different folder then and actually mistype this works faces zero zero zero and if it doesn't find it'll like credit hard create folders and look in there and it won't find things so you need to make sure that you name things properly and you tell it where it is then when we move on we need to create a folder opposes folder in our workspace so in the zero zero zero we need to create a folder called poses and I actually have one already created so I'm just going to copy paste this into here now this is going to have your open poses open pose poses for the control net in the automatic 1111 UI so I have in here I have what's called this turntable so this is going back to this Reddit post this is what this user was using but I also have specific ones like a stand it's just like my idle animation and then we have a walk one two three four five and six so this is where my actual animation is going to come from so all of these are just thrown into a poses folder within here so we also need to create this turntable.png pose picture this is a sort of specific one that it looks for the name of this is in settings.pi if you want to change it something else and then we need to create a reference image so this is where we're going to first come in and start doing some things so this is where begin the stable diffusion so I have my prompt here and I am using this this Laura embedding I will have that linked Below in the description uh it's the same one from this Reddit post and these prompts are also in this workspace so you can see some example prompts so this is my positive and my negative but the positive I have this percent prompt percent percent that you are expected to replace so here are just some examples of different things that you could use for that and you know feel free to make your own modify them I'm using this new orc one because I'm trying to make an orc enemy for my game so at the beginning I start with just a gray background and I have in painting filled to 100 I'm going to turn up my CFG scale make my batch you know to eight to see different variants and I'm also going to have my sampling steps fairly low because we're just going to want to churn through a whole bunch of iterations right now I have my control Net Zero control net one it's disabled I have this one enabled no preprocessor because we already have the poses and I have this open pose fp16 enabled on low vram and my weight set to 1.0 so I can go through and I can generate a few now this one will not actually be in our final Sprite sheet or animation this is just deciding what they look like and this will inform all of our other Generations what kind of image or what kind of orc what we want them to look like and the kind of thing to make so I'm going to go through and I'm going to I'm going to make what I want and when we get some interesting sections you know we'll catch back up and I'll explain them all for you so I ended up reaching something that was in the direction that I wanted but I don't like how the pants look I wanted something more more leathery so what I'm doing here is I'm just coming into into paint here just normal old Ms paint and I'm just drawing over these sections and very broad Strokes the kind of colors that I want and I'm just going to keep running this through through stable diffusion to just get it to clean up some of the details and really get it more to what I want so I've reached something here where I'm fairly happy now with the results but the one thing that I wanted is I wanted this guy on the right to face backwards now it's difficult to get the the two guys to face forward and backwards at the same time so the technique that I always use is I just get a good looking forward pass and then I mask this guy on the right I give it a fairly high like a 0.7 denoising threshold um set this back to about 20 and then I just start on the positive facing backwards back towards camera and I just start adding these into the positive and to the negative we have this facing forwards and then we also do things like face and chest this really turns it around it might take a few iterations but generally speakings will turn your guy around so I have something now that I'm pretty happy with and it is important to take your time and making sure it's exactly what you want because this will inform the rest of the process what kind of character to make how to make it so if you spend your time to make this really nice and what you want then it will help you down the line but there's one last thing to do and that's the faces I'm not happy with how these are now one of the things little tricks to do is to fix the faces you just click on one of the faces and you do this only mask button and it will really zoom in and try to fix up that face and create you I right now I have four variants at point one with 50 sampling steps and we'll really try to polish up that face and really give it lots and lots of details so I'm going to go through and I'm going to get these these faces all fixed up okay so I've landed on something that I'm fairly happy with here so The Next Step as we go back to here is we create this turntable pose picture uh we've already done that we're creating a reference image and following the turntable pose and we're going to put it in the root of our workspace named source.png so I've got the file right here and if we head back to our workspace we're going to copy it and paste into here we're going to rename it to source.png and this is of course in here we see where the name here is the source name and what this should mean now is that we can run python generation.pi to initialize our first interaction our iteration so come back to here we're going to CD into that's the animutils and we're gonna in the root of the repo not in the workspace we're going to do python generation.5 hopefully everything works properly and if you look here it created our stripped version so let's strip the background and then also made us this clean one where it puts on a gray background and we also have this error zero zero folder which is what the readme tells me should happen we should see a zero zero zero folder if we look inside the side by side if you look inside it should house a suite of folders which we see in here and they all correspond to different pose names so we had the stand and walk one through six pose PNG files so now we see a folder in there and we don't have the turntable folder because this is like a root for all of our images if we take a look inside we'll see a data.json file that's what the program uses we'll see the input which currently has our turntable on the left and a blank on the right but the pose actually has a combined one so we see on the left our turntable and this is for our stand which is in this stand folder so this is what we will be using now to generate it so we're going to go through the step so we're going to run folder by folder and the images and poses we're going to run them through stable diffusion to generate an initial version of each of the character in their poses and once we found an image that we would like for for instance stand we're going to throw it into this folder here so I'm going to run through one of these iterations we're going to come over to image to image and we're going to Chuck this into here we're going to Chuck the pose into the bottom here similar settings 1.0 low vram and don't forget to change your width now we're at 10 24. and crank up the CFG scale denoising strength all the way to 1.0 this is roughly where our character is so I'm gonna create an in paint in that General vicinity turn the sampling step spec down I'm going to generate eight and remember just checked us back the whole picture because remembering the faces and let's generate and we'll see what it makes but I've ended up something here that I'm reasonably happy with this doesn't have to be perfect you just want the general shape the general features to be roughly aligned I wouldn't spend too much time on your first iteration with this but we're going to come down to which one is this this one and we're just going to drag this into the stand folder you want just four files in here these original Three and then your new one you don't want any extra like five or six files in here just this new one so it's going to grab that one when we run through the future stuff but you just drop it into here now when we go to the second one one thing that I highly recommend is instead of starting from scratch since you already have something that you reasonably like I would just pull this into here you pull your pose for your next walk animation into here potentially even crank up this weight and paint over it maybe even just starting with the legs here at the bottom and give it a decent amount of denoising strength something like a 0.7 and you just start running through and you start morphing it into the next frame so it can go ahead and keep some of your your features oops I had only masked in make sure to do the whole picture but really start to bring over features and just start to move some of the limbs and I'll see you on the other side okay so I've ended up something here with roughly you know more towards the pose but the issue is that these legs are not in the correct order that I want so what I'm going to be doing is I'm just going to be pulling them into paint and I'm going to do very broad sort of what I want so I'm going to grabbing this color and we're going to be sort of just drawing in the front line here the front leg here and we're going to then pull in this almost like background green a little bit too dark and we're going to start painting in roughly what we want and then of course this background it's just a middle gray so you can end up putting that in there but we're just going to of course we're going to be lining it up with our pose so you can look at that pose image and you can see roughly where it comes down and this does not have to be perfect at all this can be just super crude you're just giving the AI that General hump that it needs to get to where it needs to go we're going to grab this we're going to throw this back in a stable diffusion we're just going to look at this leg section be something like a 0.40 noising strength that we're going to run it through and we're just going to have this thing clean up our rough drawing we have it with this open pose model so it knows where it needs to make things and we're just looking for roughly the right shape and orientation of which leg is in front of which leg just so that our animation is smooth when we play it back later okay so I've now managed to clean this up get the rough feet and the shape and everything the face is not perfect but roughly what I want you know good enough for our first iteration here so once again let's take that image and we drag this into this walk 1 folder and we continue on with the next one so once again we drag in this pose and I drag in this one that I enjoyed this one and there's a few things changed here so I'm actually going to be grabbing this image and about just be slightly editing this in paint and I'm just going to go through and I'm going to do all of these if I have anything interesting to do I'll come back in the video but I'm just going to go slowly work on these so I've got everything filled out now so each one of these folders has the image for the pose that we want so we have this step finished now so we're going to take a look back at here and we've completed this initializing a workspace we've got the first iteration through and now we're ready to iterate a workspace so to clean up the images and increase of course coherency we have this iterate.pi script that will pull out the images that we just did and with a few parameters in settings.pi we can actually control what we want if we take a look here on the settings.pi these are the four we have keep turntable images left images right and randomize order so the keep turntable will ask you if you want that turntable pose at the very left of your image it will pull some images to the left some images to the right and you can choose to randomize the order so we're going to try to do a coherency pass so we're actually going to set this keep turntable to false we're going to put one image to the left and one image to the right and we're going to keep a non-randomized order and so the order of these will actually be this order so walk will have stand to the left and walk two to the right so we're actually coming here and we're going to python iterates dot pi and you can see the key ordering here if it was randomized it would tell you exactly what it's running on if we come out here we see this iter 0 was created if we take a look at stand we actually see that it has stand in the middle here oh oh we never saved it that's why so this is the thing that you should know is that if you ever don't like what your settings are actually delete this and we'll actually come in here to 80 and we'll actually delete that entire folder and now if we rerun this oh not generation iterate.pi we have this inner one here and there we go so this walk one see how it has stand to the left and walk two to the right so we can come back to stable diffusion and we can plonk this in and we can plunk this down here make sure to increase your width now this is a little bit bugged uh the web UI from automatic 1111 so this ends up cutting to the right but we have just this section here that we want to run over and so it's not a huge deal now what I end up doing is run with a little bit more sampling steps like 30 we're on a Passover at like 0.3 and I simply run over these images we try to increase the coherency between the images on the left and the right though I actually had to stop recording because when we increase the image size it became too much to record and run stable diffusion at the same time but I'm back and the process was fairly simple you just drag an input you drag in the pose you mask over and you run through a fairly High sampling step low denoising strength and you just polish it up you got them to look a little more like the one before and after and you really start to get things to look a little better so we're going to run through and we're going to do one last iteration we're actually going to turn this turntable back on zero images to left and we're gonna have one image to the right with no randomized order and we will run iterate and we should see that we have an enter 2 here and if we look here we get turntable and we get the the one after it to run so let's head over to stable diffusion we will drag in the image we will drag in the pose and I will actually turn off this control net completely remove it out of here simply because we don't actually need it when we're running this low we're just polishing up some details so the first thing is we set this to 0.2 I'm actually going to set this up to 40 batch count down to one if I don't like what I have I'll just rerun it see if G scale can come a little bit down and we run over the image one time turns out my control net model was still loaded in memory so I had to restart my stable diffusion web UI but I got it back now I can run it while recording so I ran through one time washed it up a bit but then we're going to mask the face in we're going to say only masked and we're going to run through the face and we're really just gonna let stable diffusion polish up this face get in lots of details and it's a little bit better not quite exactly lower to the noisy strength a little bit and just run this one through and that seems pretty good so we're going to save it we're going to come to the next one and drag this in reset select everything 0.2 make sure whole picture and run so I'm just going to do this whole process with the whole body with point two and then do only the face and I'll get back with you once I'm done with everything and here we have it I have gone through and I'm pretty happy with the result so now we can move on to extracting and cleaning so the first thing we can do is we can do python extract dot pi and what this will do is come into our workspace we now have this extracted folder it has all of our animations extracted so we can actually take a look at how he walks around which I think it's a decent result uh given how much time I spent on this of course you can spend more time and you can polish this up go through a few more iterations make them look more similar really have them move around with the control net but for what I want this is pretty good and then oops stupid windows we can run python clean dot pi what this does is makes this clean folder and it grabs them all and it will remove the background and then put them onto a clean background the clean background is later used for interpolation if you wanted to so we can see it goes through and it does each one of them Now we move on to interpolation this is an optional thing that you can do I have a local version of one working you can use different things like Runway ml has one I haven't used it but it seems decent from what I've looked at however when I was just in my base installation in 3.9 python 3.9 things were a little finicky so I ended up with the because I use the tensorflow and the tensorflow and it doesn't like things so activate in terp so I ended up making an anaconda environment so I had a python 311 I installed the tensorflow and tensorflow hub along with these three to have it in my installation now to actually use the interpolation I tried making everything as programmer friendly as possible but this one you know was a little bit weird and difficult to get in so again to interpolate come down to where it starts here are where your your different options are so for iterations this will run two iterations so it'll find a middle frame between each frame pairing per iteration and then it'll run it again including your interpolated frames so if we open up paint here let's say you have these frames here and these are frames that we just made so this is you know run one run two run three so we have one two and three if we run for one iteration it'll interpolate a frame between those if we run it for two iterations it'll run the first iteration will create these red and then it'll actually create two extra interpolations in between and so you'll end up with four times this menu with two so two equals four x the frames if we run it for three iterations you'll get 8X the frames for four interpolations you'll get 16x I recommend sticking between one and three two is normally The Sweet Spot for me uh where it allows it to really spread it out so the other setting is Loop interpolation and this is if you have a cyclical animation so for instance we have with our run animation this is actually what we want if you had something like an attack where you had an a start and an ending it'll interpolate between the start and the end but if you have this Loop interpolation such true it'll take your last and your first frame it'll interpolate between those and put them at the end so that you have a nice smooth Loop the last thing so then we have interpolation folder and we have to interpolate so in my case I actually want to have these custom things because if you notice we have a stand here but then we have this walk we don't actually want the stand to be in our animation that we interpolate so I have return walk-in file name if you wanted to you in theory could make a walk subfolder and you could create a folder here and put walk and you put the frames into there uh I haven't actually tested that but in theory you should be able to but I have this just return walk and file name so it only grabs the files with walk in their name so then if we run this python interpolate.pi we'll actually go through and it'll interpolate all of them and this does take a little while to run so I'm just going to let this run and I'll see you on the other side so it's not finished the interpolation some of them ended up a little bit scuffed but we're just going to try this for now we're going to see how it turns out and we're just going to make the Sprite sheet so if we head over to the Sprite sheet there's a few settings up here we have this skip existing uh this will just not remove the background for those that already have a background removed if you want to rerun the interpolation you might need to set this to false to then have it override and recreate you these blank background ones we have this Max width and resize so you can resize the images so if you set it to 0.5 they'll have half the width and half the height uh when it puts into the Sprite sheet and if you have Max width it'll start creating extra rows to not exceed this width this width right here is because Godot has a max width on Sprite images so these are the settings that I want I'm actually going to turn this to 0.5 and then we can run python Sprite sheet dot pi and I'll go through and it'll create us the clear backgrounds and once it's done all that it'll create this little sprite sheet picture and there we have it we have sheet.png and so now this is our full Sprite sheet that we generated from our images so I'm gonna pop over to Godot and Implement a few details so we can get the Sprite sheet implemented the way I want and I'll show you the results and I finally got it implemented so you can take a look at how it looks in game it's a little bit scuffed but I think for the work that we put in it's a decent result and you can of course in your own project spend some more time working on it refining it making it exactly what you want getting it so that the interpolate looks better or using a better interpolator and you can make some really clean animations with this so if you enjoyed the video definitely leave a like subscribe or leave a comment and I'll catch you in the next one thank you

Channel: Tobias Fischer

Views: 135,868

Keywords:

Id: Ffl8b_GfJ-M

Length: 34min 11sec (2051 seconds)

Published: Tue Apr 04 2023