NEW Stable Video Diffusion XT 1.1: Image2Video

stability AI the same folks that brought us stable diffusion XL have now brought us stable video diffusion and they've released the 1.1 version on hugging face let's check it out and here it is on hugging face you have to actually log in this is a gated model so it asks you a couple questions about how you're going to use it once you fill that in though you can see over here there's a couple of examples on the model card and the general idea here as it says is stable diffusion video 1.1 is is an image to video diffusion model that takes a still image as a conditioning frame and it generates video from it some more details the model was trained to generate 25 frames of video at a resolution of 1,24 by 576 you can expect to get 6 frames per second using a motion bucket ID of 127 to improve the consistency of outputs this is adjustable but these are the defaults that you want to use to get the model jump on over to the files scroll down and you're going to see this SVD XT 1.1 safe tensor file it's almost 5 GB that's the file you need to download simply click the button here we're also going to use a comfy UI workflow if you haven't used comfy UI before I've got an installation video right here otherwise just follow along once comfy UI loads all you have to do is load that Json file so we'll go to load and we'll find that SVD comparison grid Json file and open that up now you're going to have this grid that shows up and you might have some red nodes you might get an error message at this point if you do go and click on your manager and install missing custom nodes again if you don't know how to do this I outlined all of this in my previous video on how to install comfy UI install custom nodes and if there are any missing it's going to show up in this list you'll click an install button you might have to restart comfy UI but it's going to walk you through the whole process from there you should be good and it should look something like this let's go ahead and zoom in on a couple sections here the first thing we're going to do is go to this image only checkpoint loader in the SVD 1.1 section you'll notice in this drop down there's the SVD safe tensor file but it's a different name than the one that we downloaded so you're going to want to go in and find yours mine's SVD XT 1.1 safe tensors with that loaded we've got our model checkpoint in here that we need in order to run this the other thing you want to do is go over to this SVD image to conditioning and you want to make sure all the parameters are the same as what hugging face and stability AI suggested so for width we've got 1,24 height 576 Total video frames 25 motion bucket ID 127 and frames per second six that matches all the settings so we're good to go there you roll over to the left here you'll see this load image box this is where you're going to load the image that you actually want to have it animate in my case let's grab this what I Ed for a thumbnail in another video it says robot from Nvidia it has four wheels for sort of legs and arms might be kind of cool to see what this animates with that once that's loaded and everything else looks good just click on the Q prompt button you'll notice it's loading the checkpoint that's what that green around it means and now it's actually generating the video it's over here now I'm running an RTX 3090 GPU this is going to take probably about 2 minutes to run the full 25 frames at the default settings you can see here there's 20 steps CFG scale of two it's using the oiler and the scheduler is set to normal let's let that run see what comes back and here's the resulting video it's actually been upsampled to 24 frames a second you'll notice that the motion's really smooth this is actually really awesome probably better than I thought it would come out in fact it looks like it's rolling across the ground smoothly look at the shadows and the details there it's almost as if it's been sort of Ray traced which is really cool you can see some artifacting right like if you look at the spokes and the wheels it doesn't quite know how to spin those properly but the overall motion and everything else looks pretty good we got to try a few more here for our second attempt here let's use this image that imagen 2 created in one of my earlier videos when I was testing it this was supposed to be depicting sadness it had these really wild looking weird tears I'm really curious to see how this animates that so let's cue The Prompt and kick it off that's both terrific and horrifying at this same time look at those weird tears they're almost like tree trunks crawling down her face I'd have to crop this image it was a taller image so the wider aspect ratio didn't quite work out here I think you could make something really cool though with this you check out the motion of her teeth and her mouth the nose is kind of wobbly at the end really bizarre one let's try another this one's sort of a light bulb in a forest with a whole bunch of plants and leaves around it all right this one turned out a little weirder than the rest you can see that it looks like it's trying trying to almost shake the leaves like it's in wind and maybe it thinks the light bulb is actually a flower or something on top of the plant but at the end there you see some tearing and it doesn't quite know what to do I'd call this one a fail really weird this one is a robot I generated using mid Journey it's got that nice wide aspect ratio I could see a lot of really cool movement with this depending on how it ends up working out let's check it out while that's rendering it's worth pointing out that most of you watching this are not subscribed to my channel yet so go ahead and H that like And subscribe button while you're waiting so you don't miss out on any of my new content really cool result I was hoping for those fingers to be moving sort of typing away on the keyboard but still cool Parallax effect with the background and kind of rotating around the image you get these cool lighting effects on the top of the head and everything else not bad how about my bacon and eggplants that I created using stable diffusion XL and here's what we get not bad it's kind of just scrolling up and down vertically you see a little bit of change a little bit of inconsistency in the way it renders as the frames get on but not too bad also not too exciting there's no real movement of the objects in there it's just panning over an image here's another mid-journey image I used for a recent thumbnail see how this one does all right here we go I like the panning motion of the image it's sort of panning to the right and downward slightly but the eyes are kind of wonky it's doing some really weird things to the face other than those abnormalities pretty decent and no test is complete without some sort of futuristic looking car and this is what it did to the car looks like it likes to do a lot of these panning shots rather than adding motion to the actual objects inside of it I'm curious to try one more I have an interior shot with some fire and a fireplace I want to see if it'll actually animate that maybe you can see here this is sort of an interior of a cave house in the snowy mountains and it's got a couple of different fireplaces curious to see if it does anything with either of those all right here we go it did actually looks like it animated the flames in the fireplace did some really bizarre stuff with the table looks like it's kind of wobbly and then some of the furniture looks like it's waves it's kind of moving across the scene a little weird not quite what I'd hoped for but some cool stuff in there overall super cool that stability AI is putting models like this out for us to test in an open source way obviously it's not on par with something like pabs and their motion brush technology but cool nonetheless let me know what you end up creating down in the comments below we can see what works well and what doesn't work so well as always I'm Brian love it and remember all your Tech are belong to [Music] us the crown from Basics to complex never let you down all your tax a i earning the renown

Channel: All Your Tech AI

Views: 3,021

Keywords: stable video diffusion, stable diffusion, ai video generator, stable diffusion video, ai video, stable diffusion tutorial, text to video, stable diffusion ai, stable diffusion ai art, stable diffusion video install, stable diffusion video consistency, stable video diffusion free, stable diffusion video animation

Id: 3dH6Q6N-RT8

Length: 7min 53sec (473 seconds)

Published: Wed Feb 07 2024