Amazing video clips with animateDiff for automatic1111

hi in this video I want to show you how I did two of the clips of a um stable diffusion um animation clip uh a couple days ago and yeah let me show you guys first the clip in question so let's watch it together real quick [Music] [Music] [Music] die so yes this was the video that I've posted a few days ago and some of you were wondering how I did it and how I got it to look as consistent as it does I mean obviously it still has some issues um but I guess it is a nice uh step forward to what we we have seen um a couple months ago and yeah so I just want to um guide you guys through the whole process of creating One Clip basically inside of automatic 111 using a custom trained model um in this case it's a still unreleased beta version of newsphere V4 but you should be able to create something very similar with the already released new sphere V3 I'm going to link um the um address where you can download it um in the description so the first thing that you will need I mean obviously you have to have automatic 111 already set up and everything uh is an extension that is called called animate diff um so if we go to extensions here it might already be available here let's have a look load from let's not filter for already installed and then look for animate diff and yes there it is so just look for animate diff and then install it and then you will also have to download the motion modules and the model itself um so if you guys do a quick Google search you will um find tutorials how to do that um there are like lots out there on YouTube um otherwise I will also link another video to um yeah give you an example of one of the tutorials that um that I've used to get into it so this is basically going to be text to image only so in this case none of the clips have been created by rendering the AI image on top of a real recorded video so that is not the case here it is purely synthetic purely prompt based and yeah no extra step needed really it's just it just works if anime diff likes your prompt it's going to already produce some nice results um so I'm not going to write the prompt from scratch since that could take quite a while to find something working and what I've done basically is really just grab a prompt from one of my still image um generations and in a lot of cases that's already enough and then you just in enable animate div here and it mostly does all the magic already on its own one thing to bear in mind is right now 16 frames which is the default really works well to produce good results anything higher I haven't experimented with too much so far but I didn't get good results I tried for example 32 frames which is the maximum that you can use and yeah it didn't didn't really we um it didn't produce really good results so I would um keep it at 16 for now and I'm hoping in the future they will have a model that will be able to handle higher frame numbers um so yeah let's get started um so the clips that I'm am going to show you that I've done is going to be first off um this one here the opening shot this so for that let me grab the prompt first so how we can do that is really just go to PNG info here which is installed by default and then just drag and drop one of those images in there and then here we have all the parameters and then just click to sent to text the image and then it populates everything there and I am using a 49t RTX and I'm only able to upscale by a 1.4 until I'm running out of vram so depending on your configuration you might not even be able to I guess with some of the uh command line arguments like m v it could work but I haven't tested out so so yeah this is basically what works for a 4090 with 24 GB of vram um so I don't know why I have used uler sampler in this case but for some reason animate D seems to really like it and it's producing really nice animations with it uh you can obviously try the other samples as well but for some reason this works worked really well for me and yeah otherwise really nothing special here samping steps right now at 25 but I found out if the sampling steps are too low there could can be like jumps in between um you know like poses so I would recommend if that's the case for you to try to increase a little bit to have like more frames in between movements and yeah I have then here the upscaler which for the first test I'm going to disable to just have a faster um preview and also I'm using stable diffusion 1.5 right now because that's the only um stable diffusion version that animate div works with right now [Music] and yeah so let's first generate an image without enemy diff to just get an idea how a still image would look like for that purpose I am going to enable um after detailer um let's try this and yeah let's just I guess with the highest fix click generate and see what we get so this is the still image that I'm getting from this prompt and let's see if that is going to look I mean let's see if it's going to look anything like it with um animate diff enabled so let's quickly enable the check here and I'm really leaving everything else at default right now um yes so let's just click generate and see what comes out this is also not going to take a lot of time maybe up to a minute or so um actually in this case no I forgot to disable the upscaler the higher s fixed so this is definitely going to take a little bit longer um so let's wait and see and yeah I will also provide you guys those sample pngs that will have the generation data and just to get started you can then just grab it and use my prompt and then just you know modify the um the keywords and the main part of the prompt just to your needs and then you should get something similarly uh similarly in the quality go um yeah in that um Vault park for sure and then here we are so this is the gift preview that um it is giving us and you can see there's like some Oddities with the legs going on and I have noticed upping the sampling steps can help with that uh let's just try that real quick like slowly let's try with 30 31st yeah I think now it already looks like less jumpy maybe let's try 35 the thing is if you go way too high with the sampling steps the animation is actually going to change actually in this case I can see it already is um so maybe let's leave it at whatever it was 26 25 I guess because yeah we don't want to actually the motion to change too much so this is my point if you increase the sampling steps too high the overall animation might change and if you were already overall happy with the previous results then yeah that can be an issue so this is something to be mindful about um yeah so we have have the GIF and then also the generated images themselves and as you guys can see the face looks not as we would like it to be so what I'm doing next is I am grabbing the images and then inside image to image I'm actually using um after detailer to um fix the face so let's do that next real quick so let me just first open the generated images um so I'm guessing it was these ones the previous ones that I wanted to use so now here are the images inside the outputs folder text to image images today's date and then what I usually do is I create a folder and call it like clip one for example and then I'm just grabbing all the images contrl X to cut and then in the folder contrl V so this is kind of important that you put all the images into one folder um because otherwise if I would do the image to image batch process in here it would just do all the images and I don't want that so now we got them in here and one other problem is the naming is an issue not right now but later so before I even actually yeah let's not worry about that right now because it's going to produce a new folder and new images and then we can rename those um so let's copy now this path and go back to Alterna 1111 and then we go to image to image and before that let's also send all this data to image to image which is here this button nice and then we don't want that image but let's just put one of the actual Generations in there now what I do is in the settings here um so so what we want to do first is click this button to get the resolution if it's not um passed on already um actually one second maybe this is not the right resolution [Music] 760 oh it is actually okay good so this is the right resolution um just to make sure and then what you want to do is you want to set the denoising strength to zero because we don't want to change anything else because once you start to do that it's going to start to lose cohesiveness in the frames and it's going to start to morph and flicker a lot so we really just want to focus on the face so that's why the den noising strength is zero and the seat is not important at all so let's just set to minus one and then we want to enable after detailer and what I figured is in the first I like to use the second one that seems to be a little bit more consistent and then in the second I like to use hand and then let's set the Imp painting settings Den noising I like to lower a little bit for the hands otherwise you might get some very pronounced hand details and for a lady it could be a a little bit too much and then in the first one we can start with 0.4 for the D noising strength just to see and I also like to up the in paint musk l a little bit on both and make sure animate diff is disabled and yeah let's give this a quick gen to see what comes out so this is the result um which looks kind of similar to what I had in my um V video so I guess we can go with that one second the hands are not improving somehow it can be if the hands are looking too abstract that after detailer won't be able to detect it um second hand let's maybe try to increase this a little bit okay it's skipping the hands but that's okay for now um and this looks pretty good actually um at least the face now we can go to the Second Step since we like the result we can then now switch here to batch and then we just paste in the folder with the images and then in the output directory I just paste the same but then add another folder here and now yeah we can now just click generate and then it's going to go through all the images it could be that we have to do this process um a few times sometimes it happens that one phase is like kind of jumping a little bit um and in that case you can then just just run the batch again just make sure that you have C2 minus one so it use a different seed so that will then you know otherwise if you have exactly the same seat it's just going to produce the exact same issues that you had in the previous um generation and we can see here it actually did detect the hands sometimes now we can go here into the folder already and we can see the images being generated and while it's still added we can just already have a look and it's on some of the images to see if they're fine and this actually doesn't look too bad you see on the forehead is a jump therefore I'm just going to run it again so we can actually cancel this but it already finished it so I'm just going to delete these and then really just hit generate again and if we're still seeing jumps in there we can start to try reduce the D noising of the after detailer for the face a little bit further let already check here yeah we can see there's like some stuff generating on her for had so maybe let's actually try to decrease the den noising a little bit so here oops I think it's bugging out a little bit oh um yeah it's bugging out right now so we can just go here to settings or in this case we can't or just reload the page okay now it's working again but somehow the generation data got lost so we can just just um use one of our previous generation images to um in PNG info to get the prompt back there we go and now we have to double check if um one [Music] second the resolution is not really correct here oh sorry I send it here of course [Music] oh yeah maybe it was not this one one moment but I think it should be fine yeah so let's just do that set send this to image to image and then um make sure the after detailer is set properly again so in painting yeah so we want to reduce it so let's do that um and the same here for the hands make sure it's set to a little bit lower 25 I I think I've used last time enemy diff is off let's just quickly do another test gen mhm yeah that looks similar I'm just one wondering if that is really the last resolution we use but it's fine we can just go with that it looks a little bit more blurry but that's perfectly fine we're going to upscale it later and now let's try the batch process again so let's put here that and generate yeah this seems to look a bit more consistent now no jumping so far on the forehead yeah I think that's pretty good if some of those things here bother you on the forehead you can try to with the prompting with the negative prompt to tell it to not put anything on the skin but or in Photoshop just use the healing brushes um to paint them out so that's your choice okay so we have all these images now with the face restored looking pretty good and the next step now is to First rename the images because we want to load it as an image sequence and this naming won't work so for that I'm using a tool that you can just download for free bulk rename utility so then we just grab this folder and then paste it in here and now I just select them all and then what we want to do is here remove all characters basically and then you can here under add put in the prefix like whatever I'm just going to call it image Dot and then here under numbering suffix padding let's set it to five and this this is what you want so it's like image Dot and then the frame number and then dot the extension this is going to work and then just hit rename done and what we want to do now is we need to basically uh what I did is um because it's only 16 frames it's like super choppy right so what I want is to convert it to 60 frames for seconds so we have a nice and butter smooth animation of it so for that I'm going to use topas video AI highly recommend it there's a feature in D DAV Vinci resolve that does it too but it is not as good it's like kind of creating jumpy motion and so on I haven't played around with too much but this works pretty well right out the box so what I do now is I would ra normally use a preset but I'm going to show you guys how to set things up um from scratch here so first what we want to do is set um set the frame range uh the um um frame rate to 60 and then resolution I want already two times upscaled um maybe we can try four okay that's maybe we too high but that's fine let's try four um and then the next thing you want to do is um go under frame interpolation and then set it to slow motion uh 3x I used um in this case and then I left all the other settings by default enhancement also left all of this by default if you want to see the results first before you want to save the video you can click on preview here and then it's going to create a 2C preview of the result um yeah let's just quickly do that and check how it's going to look like before we continue for and here we are now we can just press play to see and we can actually see the slowmo in this case is actually too slow so we actually want it then maybe two times slower let's just do a quick [Music] um two times oh actually yeah the upscale let's leave it original because I just want to quickly see the preview um now let's hit it so this is going to process just way faster um yeah that's going to be better yeah I think this was the kind of the correct speed that I had previously and we can even rtime it later inside of D resolve which I'm going to use after this um so yeah I think these settings are great now let's just set the upscaler back to 4X let's just leave it at yeah 4X is good and now all we have to do is change some settings here I like to not use press in this case I mean the quality is not that high that you would need it and it's just way bigger so I like to use h264 and quality level high and then export as and then it will already like kind of select the clip one folder that you had there and I guess it makes more sense to just dump it all into the parent folder so you can just select them easier and put them into um ding resolve more conveniently rather than you know going through folders trying to find your clip um just to keep the steps down and this is almost done now let's have a quick look how it turned out looking pretty neat so now we can go to the next step in Davin she resolve let me quickly launch that and what we want to do first is we want to change the settings of the project uh to reflect the resolution that we have here so let's go there project set right click project settings and then here uh what do we have 2176 by 25 no to 688 and then frame rate 60 FPS and then just save double click and then now we can straight go here to the edit menu and grab our clip here and then drag it in yes and now we play it we can see it's nicely moving and then what I've also done is um I've added like an extra kind of slowmo at the end we can do that real quick here too right click and then um retime controls and then we um how did that work I think I had to like yeah retime curve that's right so we enable the retime curve and then with middle Mouse you can like pan here or you can just increase this I guess yes and then here we have this and then what you need to do is you go here on this straight curve and then you press control uh alt click that's going to add another Point here and then now you have like two of these um um speed controls here percentage and then what you want to do is change this one here to the speed that you want for it for the last portion so we want it to be slower so that's why you have to then um lower the speed here in percentage so let's try like 50 50 and um try to like oh yeah we changed already and now let's just have a look how it looks so that's how it looks let's um set this here to where was it to oh yeah here to Loop and then and let me quickly expand this here but it's not creating any extra frames right now also if you find it like too harsh because right now the interpolation is stepped it is literally like immediately switching from one speed to the other from one frame to the other which could be harsh in some instances what you can do is um select this point and then you can select click here and then it's going to um produce a smoother curve with um um yeah with like interpolation and now we can have another look maybe that's a bit too smooth then we can just select here one of those handles and then click and move them to the right to make it you know um smaller the interpolation range and I think that works here pretty well and then what you want to do next is also select your clip and then on the right side here under um rtime and scaling I would set the rtime process to Optical Flow In Motion estimation I usually set to enhance better I'm not sure if that does much I'm not a um thein resolve Pro by any means and now when it's going to slowmo it's then also going to generate extra frames so that it doesn't um hold two frames uh because we only have 60 and then if it's half it's going to be 30 right so now let's play it back again I think it's not very noticeable because it's kind of has to in real time generate but it is actually doing its job there so once we actually render out the um the video it is going to look super smooth so that's what I done here and yeah what I want to do now is I want to show you guys how to generate like an extra image so because of um I mean an extra clip because of my prompting style every generation is going to produce a very different result because if you take a quick look at my prompt you will see the main part here says a beautiful Loring Asian woman and then I have like a Magic Mix um Laura in here to get that Asian um face look and then I'm describing her with long flowing hair and brown eyes ruled by the Divine powers of the underworld wearing beautiful modest fantasy attire and then Flawless skin melancholic woman wearing garment and then eye color Death Stare extremely long flowy hair cloth swirling in the wind atmospheric storming turbulence Wind full body shot disturbance cool temperature colors because it was um getting a little bit too red it still is and then I have that detail aura that is like kind of boosting the detail in the images but not like the um classic detail Ora that we had which is like kind of just making the image a bit more contrasty this is actually like adding in extra details anyway but you might have noticed from my prompting style I'm like extremely wague with the descriptions and that is actually very intentional I'm basically just implying certain things basically just kind of like uh um you know creating kind of like a recipe but not being extremely specific with things because I want to actually leave a lot of uh room for imagination for stable defusion itself to come up with um ideas how to kind of you know turn um the idea around and I really like that because when you do that it's then producing different results every generation and that is perfectly fine for me because I really just trying to be creative creating like beautiful images beautiful motion and the fact that it's creating you know very different um variations from it is exactly what I want basically and what I'm doing then is I'm just like generating a bunch of clips and then looking through all of them and a lot of them are actually not good not usable either the animation is not very good or the hands are extremely broken and I just really generating or if I feel like it's not um you know yielding enough um good results then I'm looking into modifying my prompt a little bit to have have more consistently good results and in this case as I said you all you have to do is really just set the seat to random and then it's going to generate very different results in U each generation and this is basically how all my um all the video the Clips in the video have been produced it was like a single clip that was producing then all those different results and also I'm get guessing the waiting of the keywords is kind of like on the line of um is it going going to be um you know taken into consideration in the generation or not kind of on the edge and depending on the seat it seems to then like grab one keyword and leave out other keywords and I'm guessing that's what's causing what's also causing or adding to the very varied results so let's really just um generate a bunch of Clips so we're basically going to add a brand new clip into it that hasn't been seen in the video yet so that you guys can kind of get an idea how I create the new clips um let's just click generate now and see what we get actually we have to do it here and it's hanging again let's reload the interface again not sure why this is happening could be because of OBS um which I'm using for recording um it was never happening before that so it could be that has something to do with that so just to be sure let's go to PNG info again and send it here to make sure everything is the same seed as I said to minus one to be random each generation animate diff is turn on highrisk fixed let's leave it off for the time being and then when we see oh yeah this is good then we can regenerate it with um with the highr fix so let's have a look here surprisingly it's quite a similar result um let's just play around a little bit with the weights of the luras here the detail slide R is going to set to 1.5 from previously 1 Point uh from Two and maybe here the Magic Mix Laura let's just set it to 0.1 less and let's generate again it this is often just what it really takes to you know change things up a little bit and as we can see yes and I can also see that this is not going to be a good one it's going to be doing something odd yes so let's just click generate again and as you guys can see there's like a lot of motion in the hair and I'm guessing that anime div is also taking into [Music] consideration um keywords that would encourage motion so as you guys saw I have like for example hair and cloth swirling in the w wind which is kind of implying that it's a motion and yeah surprisingly this is also pretty similar let's reduce the CFG scale a little bit and one other thing that I like to do a lot is um if I want really varied results I really often like to work with clip skip 4 I do that a lot um yes so just as a side note okay so this is different let's just generate another one okay let's cancel that that's already going to look pretty similar now let's just be cheeky and raise clip skip to three nice animation here though this also looks pretty similar let's go to clip skip 4 yeah so this looks quite a bit different now and then if you guys feel like a lot of similar results you can also try a different sampling method so this produced that let's see what for example alltime favorite DPM Plus+ 2m Caris does so yeah I do that a lot I you know play with the sampling method I play around with a CFG scale clip skip yeah clip skip is a big one so I'm really utilizing the higher you go with the clip skip the more kind of random the results can get like not based on your prompt as much anymore but I really like to play with those things so here we have a problem where she has like a double arm but it's already already nice it's like very different than what we had previously so let's just generate another one to see if we're more lucky next time okay her hands are not great so let's just try another one it's just fixing hands is always kind of a hassle it's just really breaking the flow of this whole workflow even if the rest is amazing I just often feel like create just generating more results kind of is more straightforward that doesn't look too bad let's just generate another one maybe these are all a bit too close up so I want to use um Aura a zoom slider Aura now with this we could potentially just you know zoom out a little bit um let's see if we're able to I guess minus is going to zoom out further let's see oh yes zoomed out a lot um maybe 0.5 so I really like these slid deloras um on Civ you have like tons of those that can like control certain aspects of your image like bigger eyes or age body weight and stuff like that really helpful then you don't have to you know kind of try to enforce it with a prompt but yeah it seems to zoom out quite a lot now let's try minus 0.1 for the next one nice extra people here in the background love that still zoomed out too much let's try this now if this is still zooming out too much I'm just going to remove that Lura okay maybe it's just too sensitive we don't know I will just remove it and see what we get could have also been just bad luck that those seats that have been picked at that point in time were all just very close up and I think that's the case because now even without the Laura it's like zoomed out further than the previous generations so yeah this is good I guess this is not too bad and I think this is generally a lovely generation something that I might use as well so let's go with this one a brand new one so now we go back into the generation folder um but before we do that let's oh it yeah let's do another version with highest so if you find a clip that you like you just then click on here this button to reuse this seed and then we're going to regenerate the same with um just the high res enabled to get a little bit more um better quality and as you guys can see now it takes way longer um yeah because you have to imagine this increase in resolution it's going to not be applied to just one image but like 16 images and and I'm guessing it's processing all these 16 images at the same time loading them all into the vram let's check how much vram is currently being used and as you can see um just before it was already a 25 21 gigabyte of vram and then here we are with a bit better quality very nice now we just do the same procedure again as previously um we grab those 16 images then crl X create a new folder clip to paste them in there copy and then go to image 2 image make sure to send this so we have the prompt in there put one of the images in there and then seat minus one enable add detailer set it the in painting value 0.4 and then for the and um 0.25 and the resolution here actually you can change to 7 uh 68 if you want this is like still something that stable diffusion supports actually with the lower D noising strength you can even go pretty high like whatever 1,600 as long as the D noising strength is not too big big it's going to be all right but I think 768 is fine it's like a tiny face here inside the whole image so um do know a strength at zero this should all work already yeah let's just do a quick test gen to see what we will get but in my case actually what I did is I used Magic Mix um don't forget to press this button to get the the right resolution otherwise you will get this weird stretching on the sides so as I said for the faces I actually used custom model you can actually Define it down here in the after detailer um use separate checkpoint and then here we can just oh second no we wanted for the [Music] first oh we have to switch to first here mous blur let's not forget and then you separate checkpoint and then Magic Mix whatever let's try this maybe and enabled and generate looking great now we can just go to batch here again um update [Music] folder and then just click generate now again while it's running I'm going to already take a peek here because while it's generating if it's doing something bad in the first images you can then already just cancel it and then um adjust the settings and try again to get better results so I usually wait for three frames and then go through them so you can already see there's a little bit of wobbling going around going on here so I definitely want to lower the D noising in the after detailer let's try three but before that let's just do it on a single image to make sure it's going to look good but unfortunately it's hanging now again so I'm just going to click generate still looking good yeah I can already see there's like less of this wobbling going on because after detailer at the end of the day is not really aware of that we're doing an animation it's just doing its single frame in painting kind of thing so that can very quickly introduce some wobbliness we have some wobbliness here but there are some tools that we can use to fix the jittering a little bit which we're going to utilize after so let's just let this finish I know there's like a little issue here but I'm just going to ignore it in this case just for demonstration purposes odd stuff going on in their hands and I also actually sometimes if it's this severe I will go in and in Photoshop just paint this stuff out by hand I figured it's just way faster than I don't know trying to use control net and then the depth thing and trying to pose the hand to match there it's just kind of a hassle right now almost done all right all frames there there's one extra frame at the end let's delete it okay this looks pretty decent uh um now we do the bulk rename thing again the bulk rename tool has a little bit of a quirk like it doesn't update the folders that are being created new so you have to like um as you can see clip one is two is not there so you just close it and reopen it and then there it is and then all the settings are still working rename now then again we go to to plus videoi contrl v to close the current clip and then just drag in the new one as you can see it all works and I already saved a preset for this so I'm going to select that directly [Music] here [Music] and though I still have to change this by hand export s and The Parent Directory save oh I guess we actually upscaled it to four times didn't we Let's uh actually I don't think so all good we can just it should just scale correctly inside of Dinger resolve regardless all right now we go back down here and we can see or we can't see um there should be a new clip um can't find the clip let's do it again I guess oh already there see h curious let's just call this X and save it again seems to be there though um so much now I saved it again there it is that's the old one this is now the new one sweet now let's just drop it into D in resolve so the first clip and then the second clip and if it's even like too fast we can just enable the retime controls here and then reduce the speed a little bit and then again do the thing where we have the retime and then Optical flow enhance better for the motion estimation it's going to kind of look laggy because it's doing it in real time so going to see the good results afterwards and then this is really basically it and then all the other stuff is like adding a vignette and what not this is stuff that everybody should I guess figure out for themselves it's not really part of this tutorial but yes so now we have recreated the clip that we had previously already seen in the original video and we edit a new video in here new clip so yeah this is basically the whole process and then obviously you can then also add some extra music to your liking so in my case it was this one over here yeah so not very difficult at all it's really just knowing how to use these new extensions and the rest is really just playing around and um just really following your own creativity um as I said I'm going to link all the necessary links in the video description so you guys can easily follow all of the steps that I've shown here and get all the resources like the luras and um how to install animate div or how to get started with installing automatic L level and all that kind of stuff um yeah this is it and yeah see you guys guys on the next video

Channel: Pronto CG Tutor

Views: 15,045

Keywords: automatic1111, stablediffusion, animatediff, aiart, generativeai, animation

Id: Gz9pT2CwdoI

Length: 69min 0sec (4140 seconds)

Published: Sun Oct 22 2023