AI Animations with ComfyUI and Stable Diffusion

hello there so in this video we're going to create this spider web with a lot of stuff we take one video animations and we're going to process with the uh control net animation diff and creating some animations of course this is was a little bit flickery because I played around with different prompts but you can see it does create all those different frames most important for me was it is how accurate with a pose and pause was very good how it's done with this of course AR rest will be playing around more with settings and we have a few different videos was creating let's go ahead and jump into the compi and start creating realing our nodes see how is everything working to creating video inside com fui I'm going to launch to the stability matrix it's a just nice interface that allowed me to control all my instances as well checkpoints and other however some elements we need install manually and most important in our case it we need have it the manager manager help us to control what uh packages or what custom nodes is installed example if we open you notice right here we have it install missing custom nodes and we're going to use this even I'm going to create from the scratch from beginning from all nodes but I will provide for you special Json file that you can drag and drop on your screen and that will help to to create all nodes as well will'll show you missing nodes noce after even you click and install missing nodes after installation all of this you will still need go and restart your confy UI after you restart confy UI you'll notice it's well a lot of packages will downloaded missing packages from those custom nodes so this is reason why you want to use it as we open you no us right here we have it our default just create image it's a default preset you can also reset and access if you just going and click load default that's what does it well preloaded all of the nodes and we're going to keep some of this but not all of them right here this save image will keep it we'll keep it the code Via de code and case sampler we need to change some load checkpoint empty image we'll keep it one overload check en code and another ones will replace with something else so let's go ahead first and we're going to delete this checkpoint and we'll delete empty image as well let's remove one of the clip check and code this is what we only left so first let's go ahead and create some input for our video first if you try to create with scratch maybe you will missing some of those nodes it's the reason why I recommend for you to drag and drop Json at the beginning to preload those missing nodes but afterwards you can create by yourself I also like create by myself so I can follow all of this path and and see how is everything working what we're going to do it's using video help for suite and from here we need go to click load video upload so this is our video uploader next we want to connect the image to upscale image so we're going to node image upscaling and we're going to select upscale image you can select upscale image by but in this case we'll just use it One upscale and we have it here width and height as well available for us let's go ahead and connect right now our image to image next we need load special checkpoints and this is with animation diff animation diff it's what we're going to use so we're going to add nodes going to our animation KN going to reload it and we want to load it okay let's go ahead find what we have it right here we want load checkpoint with a noise select CU we want to apply noise on every other frame and this will give it us this motion okay let's go also select our schuer to the sqrt ler animation div next we need add our loer and we're going load me so add this one right here next we need to create our our clip or processing we already have it one isn't use it so we'll just take one we have it we'll put it down below here next we need put badge prompt scheduler here and by the way sometimes you need just to reload youru your UI if for some reason some of the nodes is missing even and there after install it so remember to do this and we're going here to fast notes going to batchit scheduler and right here we have a badge prom scheduler just use it normal not for the sdxl we'll just open this okay let's make it just little bit bigger we also want to have it like pretext to input so originate and for this we'll just need go right click going down and we'll go to find where it says convert pretty text to input and you notice right there it's created for us one nodes that we can create a two um check node and those basic nodes you can also apply if you want like size remove it but I like size in here we're going to nodes dtn it's a basic node going to the text and going to select text node so this is our basic text node and we'll connecting to pre text also as we connecting let's connect our clip node clip and we also have right here our clip text node we'll just connect this to that one as well so this time you can see we have it our text going to the pretext we have it our checkpoint clip connection to our clip text and batch schedule Pro loaded so so far we actually already start connecting okay next we're going to put it our the animations one one so for this let's go take image we'll just move them a little bit away from here and we'll just go insert around here and because we're using animated diff we're going to node going to our animate diff and right there we're going animate diff ler so this one and because we're going to use um with a context options let's go ahead and install our our uniform context and this will be same in animated div and right here we got uniform context options and we'll just connect this to our options also notice we have our model so we can go select drag and connect our model at this time and also as we work around for test purposes we'll just click down and we'll need select SDM 15 so let's go ahead with version too by way I'll provide link down below for the models so you can download it and install them okay right here we have our image but we need to convert this image if you remember from our previous video to Laten space something that can work by the weights because it's Laten we go to nodes Laten and right here we have it our video um encode and after we'll use it remember we have video decod right here so we'll take our image connecting and we'll also have a connect our be from here to right there however because for animations we have our special one we actually need to take it not from there we need to take one from our special input there you go that is correct so don't use it from via from um our checkpoint load just load separate next we actually need Connect Now control net and it will help us analyze our video missions and work with that okay so let's go ahead node also we go to advance control net and we'll need it to we need it the apply Advance control and we also needed loaded our load model okay let's go to this way and the model will go out and here's which model will go to use it um we can open open pae of course it's what we're going to use it and let's go to control canning and we just need it just use it open pose and I'm going to control net control net there you go next let's go ahead connect ours so we have our positive to positive connecting here and just to the global so we can also just take from a sampler from here we don't need it and we'll just need to connect to our negative okay we'll just select from here and drag oh it's a little bit too far okay so let me go take bring this a little bit below I don't want minimize screen too much and we'll just connect right there to negative okay now we can bring this back up next what we want to do do we actually want use it our image that input to feed back to our control n for the pause but we cannot just connect directly we need to have the pause inator so for this we'll go to the nodes and inside control net processors run here we're going to face and pause estimator and we'll just use a DW POS estimator for that okay let's connect our image here and and as output will take from image and connecting to our control net I know it's getting little bit spaghetti and maybe too many noes but hopefully you're folling along with this and we also just change this to 3D works onx okay I think we're kind of ready to fed this to our sampler now so right here we have a sampler and we have it our positive to positive negative to negative we C our um model and our model going from right there let me Zoom slightly out so we'll take our model and connecting to our model here and we have our latent image and this one we going from our encoding we also want to control our seed and for this we can just go and says fix it if we needed one seed and then you can modify the seed on whatever you want it of course if you want it you can just select and create the convert seed to input and then just connect other ways like we done before with our text but we'll just leave it like this another things I want connect it's right here we have it our via connections and we'll just need to take from one we use it and connect here as well I think it's almost all safe except we need to change couple and outputs because we don't want just image we actually want our video so for this we'll go to the node and video helper we're going to select video combine next we'll take image to our image and we don't have it any audio output as a format you can specify we'll just go select down and let's select probably MP4 that should work just fine and of course we have it Harmon with frame rate usually I have it half of 24 24 is what cinematic frame per second 12 is half of this it's a little bit jiggery but it's least look closer enough with this we also have an image as we want to save and preview our image and speak about preview we actually can put it can add some in areas if you want to review the image just to see as the process going on and for this we'll just go to our node going to our images and we'll just go select preview image this is our one preview image and we can do if you want it have it another image going with another preview image right here and if you want it we can put it after so just to see what's happening before and after pose okay um we almost kind of ready except we now need to put it text negative text and also we need to put it elements as well we don't have it even our video so the video I'm using I just downloaded from invanta test dancing so it's what we're going to do okay so right here we downloaded um let's look what we have it we have are disable the frame loot cup let's go set maybe to because we have it seconds we have 12 frames per second so we'll have it 1,200 we don't need skip first frame every other yeah let's go go with a two on this okay next we'll go down here and in our text so we have all of this selected this one if you remember it's our pretext so pretext let's go with um go green neon lights so we just have this as the beginning okay right here we have our frame so it depend how many Max frames you want to do it um and usually it's a frame zero I think what are we going to do we're going just select conrete 20 maybe 60 yeah let's go to H 120 we'll delete those frames that we don't going to use and this is very important you see how the commas okay let's go see if we can zoom even closer there you go you see the commas that is very important for us to have it so and I'm going just to delete all of this okay and here we can put um the area we wanted like disco bar maybe cyberpunk Street uh Forest I know it's will be jumping all over places but kind of it's just for the test and disco club so we'll just have it those ones I think that will work okay hung maximum 20 and what what we have it I don't know if it's it will be on a end but I think that should be just fine and let's go down to our negative if and I just won't put it just protection I don't want YouTube ban me to do this okay let's check what we have here okay and we did change an animator loaders so we using mmsd version 1.2 version two sqrt laner for the beta casers upscale 5 by 512 so it's we size it's what we wanted uh will be a little bit Square I think we have it wide so we can put it 768 I think right there the size and we also need to look because if we have it in other areas I don't need delete this one let me delete it I don't need that one okay uh let me zoom out yeah I think that is is about right okay let's go now back here so we have it wide format we have preview images load advance so we have it open pose and I will leave it link down below where you can download it other one models if you need it okay we have it our estimat here resolution that should be fine for now I know maybe 768 will work better here yeah let's go put it 768 okay I think this is all about right and when we ready let's go see 12 frames yeah we're good let's go put it in and we'll just view in history see what's going on and right there you can see it start processing we look on back so it's a downloading elements that was missing okay and it should start the POS so it's assuming pose and we're ready to look it's happening here and you can see it did preview images created for us the in a pause emulator it's right now work actually I do like how um confy UI showing me what step it is and right now right here POS so we done from upscaling it did take Arrow frames from there and when it's processed we should have a preview image when it's done pause El uh estimator and we should go next control net and after to sampler okay so right here our other image is done and you can see it is analyzing our model okay and it's already should done in our animation and right here our image is completed couple problems May occurred and it's happened to me a few times when I use the wrong check checkpoint be sure if you have an erors like output cannot create or other things you change the checkpoint because sometimes model can be different also I'm going to experiment more with different animations but you see how the background is changing too much it's because what I put it right here it is okay let's look disco bar it's very different environment so we probably want to have it similar environments that it does not jump all over places but if you look on the Motions it is actually similar motions exactly the same by control net so let's go ahead and we'll modify just for fun create different ones and we're going to create this maybe not disco bar let's go create um Street or maybe like a brick wall okay and right here we'll have it you know we actually don't need even that many we can just probably cut like this as well um the video file I did play little bit around because if you put a little bit bigger it's run out of memory it does not it's require very big even I have a 24 GB Ram um vam on my video card it's RTX 390 it's still running out of memory on this so kind of be careful watch that out and right here let's go put it slimy Len of course if you want check all of this um the effect what is kind of rander you can try put it this inside the normal prompt and I think this will do good okay yeah it's took a little bit time to render and let's create look right there so what do we have we have 20 steps 1 eight now we could probably just decrease this maybe to 7 point or even 6.5 kind of I think that one work a little bit better but again steps is very low and it's this work okay you can see the PA same of course yeah have background is just jumpy too much jumping okay and I don't need this preview we can kill it this one let's go ahead cue again and let's take a little bit longer time now and right here we can also have it 512 so let we'll read this value a little bit later okay one nice things about also run with compi because I did not change image it does not need processor those again so jump all the way through those if no changes was applied and specifically with case sampler so it will save quite a bit time for you to work with this workflow here our simpler is done with generating now we decoding and there's our images all selected we'll see how it's come up and after we should combine in a second and here's our animations very flickery but it is kind of interesting to play around and see what what else so we can probably work more with the prompt definitely to make it much smoother and everything I think our prompt what I put it right here it's uh not as good it's very um jumping create a lot of noncurrent so we need to remove as well this one and same Define little bit better in the prompt so it's all happening because of that well thank you for watching if you like it give us thumbs up um subscribe share I think this is a very fun and interesting way to explore and create your animations just to see how the all nodes work well thank you and have a great day

Channel: Vladimir Chopine [GeekatPlay]

Views: 756

Keywords: Geekatplay Studio, Vladimir Chopine, Digital art, How to, Learning Photoshop, 3d tutorials, Digital compositing, How to do in Photoshop, How to do compositing, How to do in 3d, AI art, MidJourney, Stable Diffusion, Dreambooth, Dall-e, Free resources, Free learning, Digital art for begginers, Free tutorials, artificial intelligence, Digital Photography, Photography

Id: OGui8L6-scw

Length: 24min 47sec (1487 seconds)

Published: Sat Jan 20 2024