Next level of AI Animation. ControlNet and DeForum in Stable Diffusion.

hello in this video we're going to create animations inside the stable diffusion local installations with using the form before we start let me show we have a deform installed you can go to your extension Tab and if you don't have a deforum located and Shrink right in your extensions you can go click on available click load from and after it's finished load just type the form right now it doesn't show because I need uncheck install it and mine is installed in your case it's your action will say install so you just click on this and install it after installation completed you want go to install it and click apply and restart but because in this video we're going to take step further from what we did before and I'll put link for you in a previous video we was using this we're going to use the actual control net with the deformer the former right now can support control net from the inside which is make it much more smoother and nicer animation to be sure we have it as well the control net installed we can same go to available and just type control net you'll notice okay let's see if we can find right here we will should have a control net down there right here it says SD web UI control net so just be sure you click and install it same way you want to go after apply and restart and if you already have them installed I highly recommend for you to go and click check for update I do this almost every time when I restart my server or on daily basis and this is go over and verified if all plugins or extensions we're going to use they're up to date if they're not up to date you will have come up like menu and you'll see right here it says new commit for the form and dream Booth so we don't go use a dream Booth right now but the Forum I want it and we won't go to click apply and restart when you restart sometimes you may have it something says air come up and it's a reason because the server on backend is still kind of relaunching when you'll see it says running on local it's meaning that is done and all what we need to do just reload it refresh the page and that should be fully working after reinstall the extension for the control net we actually need to adjust or set specific configurations for this so we'll go inside the settings in the settings on your left side we'll scroll down and we'll find control net let's go ahead and click on that notice right here I set the control net Max 3 it minutes allowed me three tabs but this is option these settings will affect the X2 image image to image and therefore they have pre-default 5 tabs for the control net we can use it but just in case still we here be sure you have it your max model set to 3 and model cache size 3. I usually match them next I want to be sure my version says to 1.5 on us deep okay and as long we're looking on some settings let's click on the phone and in the Formby show you FF MPEG set on proper location otherwise you will get it some errors when we start creating I think we're ready right now go inside the Forum I have it in a previous video when we gone over some settings just in case I will refresh them so in our first Run tab it's where we have our sampler we're using we also have a width height how many step we're producing and also place for the seed below we have some batch processing which we want used in this video in a keyframe it's where we set our motions for the keyframes where we also said that how many guidance or how many repetition frames we can have it so in this value what is meaning it is between each frame can create one kind of like copy of the frame or blending frame so it does not necessarily process as the render AI but it's more will take blending between two frames usually by default it says two if you want to render every frame you can put it one I do not recommend go over five but just for fun I was experimenting with some settings and you'll see example right here every two frames whereas before any can notice where the head is rotating and this is was done with using control net that's what we're going to use the reference on the video and and this one was using 10 frames and you can see when I start progressing the motion does not apply because it is too many frames need blend it does change you have a little bit better non-changing background versus this one but the problem is then you don't have it motions so you have a plus in minuses with this you have a little bit better standing together but again you will don't have motions we'll look on some of the settings shortly so let's go ahead um go over them a little bit fast noise it's how many will apply notes at the beginning and you can have control to some of this point if you want a little bit more consistency same with concurrency anti-blur when it's a frame start kind of blending together you will notice you will have a blur effect so in this case you can increase and a value going here from 0 to one by default it's lower normally I will increase little bit slightly higher so maybe about three or four okay next we also have it warping this is apply more to 3D which um we will use just 2D for these videos we also have our guide ncfg the recommendations going from six to nine for using with the form we have our seed subseed amount of the steps simpler and if you need it customize depend what sampler you wanted very different frames same like you can switch to different checkpoints if you need it for example you can go from photorealistic to anime on specific steps and change style all the way if you need to do this we also have our prompts and prompts you can see straight it is what frame and what prompt should be applied in this Frame this is what loaded it's a default with a deform but we're going to modify as well you have your negative prompts if you need it there are um image initialization that we needed and of course we have our control net so for this one let's go just start using and create it as we in control net what I want to do I want to enable notice right here we have the five control net models even we set up three before in the settings but settings will apply to text to image image to image here we have preset five so when we click enable it will take a little bit time to load it so let's go ahead and click down select one I will go with a canny preprocessor and we also want to set Canyon model for this also we'll look on the bottom we have a control mode balance it if you want priority prompt I want to see switch to control net more important so it's more based on control net in this case and resize we'll just keep it in our feeds scale or you can just resize but we're going to with the same size as we have it before so let's go just for now we'll go enable also our second model and then second model we're going to set this to let's go open face open pause face same we're going to select for this open pose for the model same things we'll just set control net is more important on this case so if you notice in any of these um areas like canning we don't have an input on the image however if you're going with open face you'll have a control net input image and path and I'm just going to put it path to this test image that you can preview here okay so let's go ahead and start creating animations with the form when you're going to The Forum tab you'll notice you have it all these different tabs we went over them in another video but just in case if you missed that video let me go very fast over them we have to run keyframes prompt initialization control net hybrid video and output we're going through all of them to set some settings and as we're going alone I will try to explain keep in mind this is quite a bit sophisticated the extensions so some of them will require little bit more time to learn how they work don't get frustrated and time will go over them so first the Run tab we have the very basic settings we have our sampler steps with hide and Seed seed it's in case we want to reuse similar look example if we go to image to image and just type portraits of Cyberman and we'll go ahead and click generate you'll notice it will generate for us portrait if we need it we can go ahead and reuse it seed copy of the seed if you like it how it's look otherwise we can create more and more but usually it's what happened it's very nice to go and preventer something and copied not exactly the same but very similar settings inside our work notice in this case you probably won't keep it sampling steps CFG height and width similar settings copy inside of the form but we'll just go use it straight from here without copying this is just option so it's what you can do same if you have it single frames you can do same with image to image just to be sure to see what the result you will get it before you actually start animating and this is very helpful okay but this we used Euler we have a 25 steps we have it as default with I'm going with 768 so I want a little bit wider screen a little bit more cinematic look in this case even we won't go to use any seeds so it's a random seed it's meaning every image will have a basic purlin noise applied a different on everyone so this is does not necessarily provide for us consistent will have some changing details but it's a good way to do in the next step to see how we can increase did we do some flickering by using here and create a little bit more consistency okay let's go ahead in our keyframes and keyframes when it's open notice we have additional types and this is kind of keep it in mind that we have it also motion tab here we'll look in a second we also have it our addition string safety CD no additional options usually keyframe it's where you specify what you want change settings based on the each different frame through the timeline of your animation we're going to use a 2d 3D just add additional Transformers the Z Vector for us with 2D we just only have X and Y okay we'll just keep it on this next uh Cadence it is how many blending happened per frame so for example if it's two it's meaning we render one frame we're under frame three and we'll create special blend between them that will reduce time how a long cuts take it and also will create somewhat little bit more smoothness between those frames however be careful when you um adjusting because if you have a too much of that value you will start skipping some motion so let me show you example right here you can see this is animations we're going to use as a test and here example we're just using two frames notice it does still kind of Applied um flickering not flickering but changing and that's because not consistency in this case and of course if we have it does increase blending between frames we'll have this problem when it's motion does not translate it you still can see some changes how it's applied it does keep a little bit more consistency on the screen versus this one but we don't track motion so this is kind of need good balance between those values and of course between two or three will work very well in this case down below okay we have our string the guidance where we said with using different models will require different same like with the form and the form size between six to nine it's very good level so I will just leave it as a seven the smaller number the more closer will be to the text string the bigger number the more far away flexibility notice right here we have a zero over and uh column and seven it's mean we can specify on a frame so for example you can say I want on the beginning be very close to my text so I'll even put a two or three then we can have it comma and on a frame 60 we can say let's go with a seven or more so you can play around with the different settings and this represents frame column and value so but for our effect we'll go up to the 7 as default okay we have it also you can do with a seed subseed amount of the steps let's go back and string this is the frame influence on the string so if you want a little bit heavier influence you can increase I will say by default start with default and see if you have too much changes you can increase seven eight and just play around but remember if you put it too much it will be almost no changes even just motion when I'm applied so this is kind of um adjustment in small increments on each that also when you do change I will highly recommend changing van one variable at a time don't change 10 20 because something happened you don't know which variable what was affecting the next motion is kind of very important because if we're going to create some animation based on a control net reference or a reference of the face and we don't modify like a zoom or other things this is what happened you can see it's countless Zoom constantly jumping and the reason is why because it does perform zoom in between those remember two frames that we rendered before in blending but next time when control net kick in on another frame it says hey this is the size so let's go back to this and this is what will happen it's made good effect if you create some special music video or Mark have some other visual effect creating it could be very interesting but if you want consistency on a motion that probably not a good way to do so be sure these motions for the if you're using with control net specifically there are set to like zero or nothing and here's a different that Zoom the sum of this properties work from one till zero with a um same like a zoom for example and you can increase 1.1 and it's going on different directions so one is no Zoom increasing to two will zoom out and other ways it's a zoom in so Moana does not move here however it will going from zero to one so they're a little bit different and we're just going to set everything to zero Zoom to one so we don't need it any motions or other things in this case uh noise we'll just leave it as default in a noise it's what purlin noise will apply to make animations you can have consistency you can increase to create more details because I said before we'll just leave it as default right now at this area kind of currency with same will won't touch any on the contrast or any other things anti-blur this is ways between frames sometimes you notice kind of very blurry happen in animation and here's the example you see how the sharp and nice this one first frame watch what's happening you see how it's become very blurry as we're zooming in and this has mostly happen when you start zooming in it's become more blurry and blurry because how works on this case you probably won't go inside an anti-blur increase the amount here for this video we don't need to do this because we don't use it the zoom itself but if you do some zoom in animations I would recommend you to see where's the blur kind of occurred and you can set for example on frame 10 increased 0.4 and so on so you can increase um actually right here should be column and you can increase sharpness to avoid some of this blurrenness what happened with your animation okay so we have an anti-blower and we don't worry about this at the moment this is for 3D and we're running in 2D mode as I'm repeating what said before 3D will add z-axis so we can move our cameras but in this current station our camera is stationary we just have motions of the person rotating head and that is in will be in 2D positioning for the camera okay then one last things I want to mention in a keyframes it is maximum frames notice right here it says 120 frames if we do 12 frames per second it will be equal to 10 seconds of animations the also it's meaning how many frames will be extracted from original video you remember I showed before original video where rotated head and this is much longer video so if your video is longer but you put it on so many frames it will extract those first frames from the video like right here and you'll notice it will extract 120 frames and use them later as a reference for different control Nets options that we set up so for this test we'll leave it as 120 but you can increase more if you need to create um I would recommend small trick if you do zoom in animations create little bit smaller frames because you will start having worse and worse blur effect but after take last frame and based on this Frame start render again that will increase the quality of the sharpness okay let's go next we'll go to the prompts and prompt you'll notice we have again our keyframes our key frames where we want to do and it's what we want to add because I'm not going to change stuff I'm going to erase all other frames portrait of Cyberman and we'll just put it the word that we created otherwise you also can have it your positive prompts and a negative prompts for example I can put it LED lights so it will be sure they are right there but we also don't wanna run NSF W or nude in the pictures well nobody wants to see me in that way trust me okay so next what we want to do this is the decisions we look on this a little bit later what it does and in a control net this is where all magic will start happening if you remember in extensions we got in the settings and set up how many we want to do and we set up three control net right here we have it five because it does not related to this those three where we set up it will happening right here we have a control net and you can see one two and three options when we're going with the form the predefause set to five one thing I'll notice when you use this model sometimes you may need restart let me show you what may happen we'll see if it's happening here or not if we're going and click enable one mode you'll see how it's pre-loaded and right here we already have display our path for video or image we want to render a few times when I did before sometimes that path does not shrink there so in this case it doesn't it just UI does not refresh properly or something goes wrong just reloaded UI and try it again notice when we when you reloaded UI if you made any other settings you will use those settings so what you want to do after you create your own settings you want to probably go and save this deformed setting so next time when you open that open them as a default or you can pre-load it this way when you're reloaded your interface you still save some of these elements so be sure if you've done all this work create your prompts create navigation Zoom animations before reload it go ahead and save the file okay so we are in a first model and what I want to do I want to go select the canny model for this there are first and we're going to select also sorry it's preprocessor and we'll select any for the processing we are schedule one we leave it as default you can change the weights depend depend what you're doing and the weights can be changed between different control Nets so for example you can switch from one control net to another let me give you example if you have a figure with a face and you're going zoom out from portrait to the figure you probably won't have it more weight original set on the open pause face and after this on specific frames you want just pull out and have it maybe open face just itself okay so right here we have our preprocessor resolution and usually I want to have it the largest we're using and while using remember I want to create a wide screen and this is where we're going 768 we'll leave it as a default and right here we need to put it what we reference in in this case I will reference to um oops okay I will reference to the video that we're going to create use as a regional I don't need any mask and on the bottom this is control mode you want to balance it between two of them more to the prompt or more to control net for me I want more important control net because I think this is I want consistency to support you can use it more than one so for example if we're going to Second and let's go ahead enable this as well and notice what's happening right here we don't have uh come up our stuff so sometimes if you open enable and we don't have this path to our videos I don't know why it's happening I think it is still kind of erroring sometimes because in this case right here we have an arrow video path so if it does not showing sometimes UI just go berserk or crazy let's try see if it's sharp nope and I don't want to reload it yet so we'll just use the top three with the displaying so we'll go ahead same so we'll save same things control net only right here preprocessor I want to actually use it open pose face and for the model we want to also use it open okay so we using two models model one as a Kenny and three the reason is Kenny will give me consistency on the background what we're going to do and open face give me consistency or emotion on a face that we're going to process we leave it all schedules everything same from frame one so I don't want change in this case and this is we set up control net so let's go now we'll skip hybrid we don't need to do anything there we'll click on the output and the output I want to change this to 2L frames per second and we leave it everything default but before let's verify it we don't have any soundtrack that we want edit we want to create videos so I don't click skip video creation okay we'll leave it all the stuff we don't need it uh create gif upscale it's fine interpolation so we'll just leave it everything as default we don't need to worry about this we'll just have a 12 frames before let's go to run I just want to verify it we have our 25 frames we could reduce to a step sir we couldn't reduce to 20 make a little bit faster but I think with default here won't be sure my width is set properly don't care about seed okay next go to keyframe and keyframe as I said before be sure the Motion Set properly so we don't have any problem in this case we have our schedule set so I think we all ready in this time and all what we need to do just go ahead and click generate when we do generate at first time it's maybe take longer and the reason is why it takes longer it is let's go over here to server and I'll just go look on the server you'll notice it will pre-loaded some models mine is already pre-loaded so it starts jumping directly into creating the image itself but in a case if you first time doing after you start your server it's need to take all models and pre-loaded them as well it's need to go inside and take all these frames extract from your video your video will be only extract 120 frames its Max frames it does not will extract amount of frames you have it how long your video so be sure again you know how long is your video or how many frames of this video you want to use it okay till it's still creating we can see the frames created and also it is created our frames not as control net one it's first control net what we is using and control net too the it's separating them because you technically can use it two different videos for that so you can actually have it like one architectural video where you're putting architectural effect and applied and second you can use it for a grant for the actual or open pose of the person we'll try to do this little bit later to show you how you can combine a two different um effects to different build background in one and foreground or actor in another ones it probably will be another video not in this video due to the size um and Link of the video itself and because we're going to do in another videos I'll just let you know and I will also creating a music video how to create and synchronize and this is good time for you to be sure you actually subscribe to the channel click on that notification button to know when those videos is released if you're interested in this technology okay so here few examples that I tried and animated after this and you can preview this so this is again using control net notice um some flickering we can rework a little bit better on this it does affect what prompt you put the seat number right here with the very high flickering so it was adjusting to quite a bit noise adding so we just want to see how it's happening um try to create a female look by my beard come out and it's probably on this case could be overdone if I remove canning because I'm still using two I used canning and I used open face if I use it just open face then you will remove the beard you can replace fully character but canning was adding the beard effect also notice right here you see those lines happening it's when you increase your sharpening quite a bit so that is a kind of be careful with this as well but I do like how it's create like line art almost you know preview and here I create myself try as a Simpsons Comer Simpson of course it's very low consistency but it was fun and for some reason I was creating the Marge hair on my head um this one with a problem with the too many steps was skipping the reason I brought just showing how clean it could be like background and other elements again if you're using um open face or other effect so overall here's the example how we can use the control net I'm going to work more with this to align a little bit more concurrency more um consistency between images and we'll look how we can take also the background image separately and Implement on top of the character itself as well and be sure you check that another video that upcoming about how to you can integrate music or how you can synchronize motion inside the video with the music if you want to create some video musics or some other fact thank you for watching this video let me know what you think if you like it if you have additional tips and have a great time to create

Channel: Vladimir Chopine [GeekatPlay]

Views: 37,291

Keywords: Geekatplay Studio, Vladimir Chopine, Digital art, How to do compositing, AI art, MidJourney, Stable Diffusion, Dreambooth, Dall-e, Digital art for begginers, artificial intelligence, ai animation, stable diffusion tutorial, ai video generator, deforum tutorial, stable diffusion video, stable diffusion deforum, stable diffusion animation, deforum stable diffusion tutorial, ai video, stable diffusion ai, ai art generator

Id: d2hf4cohg_4

Length: 32min 8sec (1928 seconds)

Published: Fri May 19 2023