ComfyUI - Vid2Vid | AnimateLCM - AnimateDiff Gen 2 v3 - IPAdapters - Loose Control

and hopefully you are here too I feel like I'm here I don't know if you're here but I would like it if you were let's find out if I'm online still getting used to this restream nonsense feels like I'm online I feel alive comfy UI all right welcome back to comfyi welcome back to YouTube Welcome Back to all the things here we are creative Exploration with pers hey I see people Al live all right cool I see chat and everything hello Frank hello food all right I think we're live I am happy to start okay what am I playing with today I am playing with animate diff version 3 uh Gen 2 uh animate LCM uh control net masks which we won't get into till the end and IP adapters so with this workflow um you can do animations really fast Within diff uh they uh don't even require prompting you can just use images to prompt and uh you can actually mix images to make prompts so um yeah uh I'm going to actually just rebuild this entire workflow from scratch so you can see the process um and try to understand more like what we're doing um and why uh but yeah let's let's start over and uh and clear it out and start fresh a couple things you're going to need you're going to need the IP adapters node IP adapter Plus in comfyi so add that you're going to need the uh animate diff evolved that's this one right here install that animate diff evolved uh you will need the video helper Suite video helper Suite uh you will also need uh KJ nodes I believe we're using for a few things KJ nodes and I think that's it off the top of my head oh oh uh we will need impact pack for the upscaler we will do after and you will need frame interpolation comi frame interpolation and I think that's actually it and face restore if we want to do face restoration uh face restore CF code former um yep that's B it's all the boring stuff out of the way and we'll also need the advanced uh Advanced control net come VII Advanced control net so I'll throw this workflow on my Discord tonight so if you don't feel like installing the stuff right now you just want to hang out and follow along that's cool you can go and download the workflow and then and then when you open it you can just hit install missing custom nodes and it'll find all the stuff that's missing and and you can go install them one by one and you'll just need the models that we're using for all this stuff and they need to go into the correct folders but uh the new model downloader will do that for you so I'll I'll mention what models we're using and uh how to get them all right so let's start with the heart of all of our uh renders and that's the case sampler and this is the um diffusion step that uh lets us know or does all the work basically so this is our Middle Point uh we're going to go with a vae decoder on the end turn it from a latent to an image then we're going to use the video helper Suite VHS we're going to use a video combine and that's going to make a video for us so 24 frames per second MP4 15 CRF and and yeah that's fine for now and I'm going to put it in um a if testing animate diff and that'll be the uh it'll go into the ad testing folder under and the file name will be animate di uh this will become apparent why I'm doing this after I do this we make a new film node this is going to be our film interpolator and then we're going to make a second video combine plug that into here and just add film 2X to the file name so this will be our slowmo but but smooth version and this will be our full speed version all right uh what's next uh so that's our output so this will give us outputs and now we need to put stuff on the front so what are we doing we're doing animate LCM which we will need the the model for which I don't actually believe is in the thing yet so if we go to animate LCM just drop the link in the chat here uh how do I drop a link in a chat when I don't have a chat there we go that'll do it okay that link there uh is for the uh animate LCM model and if you go there uh it is right here sd15 uh Laura beta you want to put that in your Lura folder and then you use it just like the other LCM node with one uh one little caveat which we'll we'll get to in a second okay so let's just start by just doing a simple animate LCM with no uh controls so that we have a sort of uh you know a baseline for what we're doing here so our latent image is going to be an empty latent image okay and we're going to set that to 512 by 512 and just for now the batch size will be 16 so it'll be a 16 frame animation our model we're going to go with the animate diff uh which one is it animate diff I'm just going to get it from here ad node animate diff we want Gen 2 apply animate diff model Drago from here uh sorry uh we need uh animate diff context options Loop uniform and this goes into I feel like I'm missing something they just changed all this stuff so bear with me here animate diff Gen 2 use evolve sampling okay so here's the three nodes we sort of need to make this work um context options goes in here uh model goes into uh or m models from here goes into models we need to load a motion model so load animate diff model we're going to use version three so yeah animate diff got a little bit more complicated but this is basically all you need to make it work um we're going to be doing LCM so we're going to change it to LCM and we're going to go 1613 for our settings here um yeah yeah these settings here will be fine okay so we need to send a model through this so model goes out into the case sampler from anime diff so here's our anime diff tree and then our model has to come from a checkpoint loader and we're going to load the Epic photog GM uh it's a new model I just found out about not new to me anyway and we're going to add a clip texting code we're going to add two of those and that's going to be our positive and negative prompt so they come out of the clip clip goes through conditioning into the case sampler just drag you guys over here just kind of move this stuff around try to make it make sense make it make sense all right and then no okay uh so our vae goes into our vae decoder we're pretty much ready to go but we need to add our LCM model so let's load Laura Lo Laura loader sorry and we want the Laura loader model only I think I'll double check this in a minute but basically model goes through Laura loader into animate diff and we want to load the animate LCM model that we just downloaded from here this guy right here I renamed it to animate LCM so I don't go crazy and I put it in my Laura's LCM folder all right strength that one this LCM is making sure that the animate diff is using LCM um sampling and then in our K sampler we need to also set it to LCM CFG of two and four steps that's right four steps okay so let's try andf W nude in the negative prompt because we're on YouTube and for our positive prompt let's try swirling ocean uh fire tornado no no let's just try swirling ocean swirling ocean let's just try swirling ocean okay this stuff's all fine okay we're going to add the extra conditioning after this um but yeah let's try this hey Tron yeah yeah LCM animated LCM is fantastic I'm I'm Blown Away uh let's uh let's see if this works for halfhazard plugging it in I'll be surprised if it works I'm just going to go in and code all these um nodes so that we have an idea of what they do and what they're for now you see how fast that was it was damn fast uh it doesn't Loop because I made it a 16 frame animation uh the context window for animate diff is 16 frames so if we make it a 32 frame animation it's going to jump two context windows and we'll be fine hey Corey yes it looks better Tron but please do your own tests so there's our 24 frame per second video and here's our um it's just interpolating now this is our uh 24 frame per second video but interpolated two times so it's you know slower and smoother all right yeah that's cool uh we got prompts we got cool stuff going on uh it doesn't really look great though uh let's go landscape uh uh beautiful sunset [Music] Um mountains mountains uh oh I think we're getting that CFG problem again where the naked ladies are coming through the back of the CFG now what did I do to how did I do that oh you know what I think we need to add clip let's try this with clip so let's use a Laur loader just a regular one this has a clip pass through as well so we'll go model and clip model to the uh animate diff this model out to the K sampler see I think our negative prompt is coming through here um which is strange I've had this before and I'm not actually totally certain what causes it but let's try passing the clip also through the LCM uh make sure sure our positives and negatives are doing what they are supposed to they are yeah see I feel like I'm getting the silhouette of a naked lady in the background here it's happening again what have I done wrong there's definitely a figure in the background what is causing that let's try removing everything from the negative prom okay well let's keep going uh fuse method oh you know what context options looped uniform yeah fuse method let's try pyramid close loop true 1613 B schedule motion model yeah everything's right that's real strange all right let's keep going okay let's get really let's get out all the powerful stuff now comfy models I just have to go to my inputs and move that stuff right okay I think we're not getting naked people in the background now should be good okay let's add IP adapters to get the um to get the uh guidance going so uh colors [Music] blue I yeah it's possible I haven't tried with this let's try with um um Photon I was using Photon earlier uman make you guys [Music] yellow green and green okay cool um we want to slip um IP adapters in after the Laura yeah after the Laura okay let's drag you over so we're going to add one more step here called IP adapters so apply IP adapter from encoded okay let's build the IP adapter rig here so we need uh out of the IP adapter we need an IP adapter model loader and we're going to load the IP adapter plus uh model um for um yeah the plus model which is available on um right here install models IPA you want to get the appropriate stuff you want the IP adapter face mod or IP adopter plus model for uh sd15 not face ID face ID I will cover just not today but it's this one right here IP adapter plus sd15 and you'll also need a IP adapter incoder and that's it's going to let us encode images into the IP adapter and we are also going to need a load clip Vision node clip Vision loader and we're going to load the sd15 clip Vision which is also available if you go to install models type clip it's the clip Vision model clip Vision G that's one right here there's other ones you don't have to use this one but it's just the one I'm using okay uh so IP adapter can take up to four Images it can take batches but just for this it can take four Images mix them together and you can make a a new animation out of that uh using it as guidance so let's do a load image uh and we also need to add one more node here called prepare prep for clip Vision prepare image for clip Vision okay and we're going to go Center on that and plug it here so clip Vision takes very small files uh uh you like need stuff at like 256 by 256 or something so that's what this node does is it crops the center out of whatever photo you feed it and uh sends it into IP adapter and then the only other thing you need to do is just pass that model through uh into the next thing so it goes from there into the um animate diff all right easy peasy I'll just make these little groups so it all makes a little bit of sense group meat meat meat let's put you in here with [Music] y'all group group and group uh group for selected nodes and we'll just leave you guys out for now okay so IP adapter this one is animate diff this is prompt oh what did I do whoa whoa this UI doesn't like me today but that's okay edit group title and this will go prompt settings and this is our [Music] diffusion and output edit group title output okay output no let's let's throw that t in there output all right cool uh okay I've noticed when I do abstract OB like file just throw a random girl in there yeah maybe include the negative tags oh yeah yeah get rid of the people uh pers I've been doing sensive testing with Photon and epic realism and Photon has better motion Clarity yeah I I think I agree with that yeah photon's been doing really good yeah it looks better already okay let's stick with Photon for now then uh and let's make this stuff C I guess and we'll make you blue because you're an image loader okay this workflow is cool okay everything's set that's all good let's go find an image from pixels uh [Music] tropical let's try that bloop so with IP adapters it doesn't matter what your crop is or anything because this prepare image for clip thing is is going to sort it out so uh it's just going to grab the center of the photo and uh you can actually remove the prompt and uh if I'm right it should just work oh I forgot to turn IP adapter Plus on see it it complained because I'm using the plus model but I didn't turn Plus on plus is just kind of better at reading the images sometimes it depends you can try non plus it's usually better with faces but I just I've been having a lot of luck with it lately so we're just going to use it so it's going to take our IP adapter's image that we gave it and it's going to essentially use what it sees in that image as the prompt and then do an animation and you're going to get really good results from this which is awesome because it's you know basically like free prompting right out the box from you can use images you make you can use images you found you can use whatever you like for the streams and stuff I'm just using cc0 images but we can generate some stuff and use that too so that was only four steps I'm going to crank it up to six steps and we'll try that see if we can get a slightly better slightly better look there we go now that's a lot better than what I used to get with LCM that's for sure and it matches the cool input image so um what's another thing you can do uh is you can uh let's make this a little bigger and we can use two images let's grab another one and it should try to mix them together so let's [Music] try yeah let's try that copy image you can just paste like contrl c contrl v right into this node it's great right load image node okay so I should try to combine these two should being the keyword oh I see there are messages is from twitch and Twitter here too Hello friends hey Scott hello zata what's up yeah we're just making stuff there we go we got tropical orange fruits I guess pretty amazing so yeah um that's how small of a workflow it is to just do like these really cool animations uh you can just increase a length of the animation here if you want uh uh you could do all kinds of stuff but what I'm going to do is start adding some uh control net to the mix so that we can uh try to control the um uh what's the word uh the composition of what we're making uh so yeah let's move this over to the right a bit move this over so in between the prompt and diffusion we're going to add some control net so control net apply Advanced uh we're going to load Lo the uh my current favorite control net which is the loose control control net Lo control code uh wait blop there we go and I'll just drop that in the link there you go uh so yeah that's um that's the link to the loose control uh control net uh it's really cool uh it uses depth maps to like kind of assign things in the in the generation so like you can I I'm misusing it today but yeah you can use it like this to do these kind of animations which is you know fantastic gives us all kinds of fun control um I'm just going to use it with the masks that I made uh cuz YOLO all right so we want to load an image sequence so I want to load image uh a VHS node I want to load images half uh my animations are in PNG sequences so I'm using this if your animations are already videos you can just use VHS load video uh and then you can um just upload the video into the app and then set all your settings and use that but mine are um mine are just uh like PNG sequences in the in here so let's try uh getting good results with this one let's try this one this is an animation I made earlier today with blender um yeah so we're going to use this as the mask to drive what's happening in the animation as you can see it's a 90 frame animation so I'm just going to grab the UR or the URL just going to grab the path to that animation drop it in here Boop and then image load cap uh I'm going to leave that at zero and I'm just going to set it to select every n two so that means it's going to play it at Double speed 90 frames becomes 45 frames so how do I tell this that I want to render 45 frames because like that's how long the video is I don't want to calculate that number every time right that's what this int is for INT is integer integer is how many frames are in the video so we're going to add a reroute and this reroute I'm making a rote because I'm going to end up using a different node later to load a video if it's a video so I want to be able to just plug it in once here and Route it to all the things that need it in this case the only thing that needs it is the empty latent image it has to go into batch size but as you can see batch size is not an input so we need to change batch size into an input by right clicking on the Node and hitting convert batch size to input now batch size is a like an input instead of a they call it a widget or a dialogue box so so let's grab the integer which is our frame count and that's going to go into our batch size okay so there we go oops unplug the bottom okay so I know this is confusing because it goes out and then back but it comes out of here into the green and out of the blue into the batch size so this thing is now determining the length of our animation based on the length of our mask our mask was 90 frames we're doing it at Double speeds like every inth to we're doing it at Double speeds it's going to be a 45 frame animation we can see what the control net is going to do if we take this image out and we make another video VHS video combin known and for that we'll set it to 24 frames per second MP4 but this one we're not going to save the output because it's we just want to see what's going into control net so let's um try and clean this up a little bit here all right all right this isn't pretty but it works all right let's get it in place y'all actually you guys can go down here yeah that's fine and then right here okay so our control net strength is at one it starts at zero and ends at one I'll explain what that means in a minute um but yeah so uh oh I didn't run it through okay control net runs on the clip so you need to take your conditioning out from your positive and negative prompt and then run it through control net into our K sampler okay so every time you do control net it's on the clip so if you want to Stack control Nets you just put one after the other after the other after the other on the clip but remember the more control Nets you use the more vram it takes so if you don't have um so if you have more you know you don't have a vram you're going to run out yeah all right um the other thing I want to do is um I'm just going to change the width 912 and the height to 512 it's going to give me essentially 16 by9 that's what I mask is it doesn't actually have to be that exact resolution because you know rendering smaller makes more sense in AI than trying to render it full res so you render an upscale so we're going to render this and then I'll show you how to upscale it as well so let's render that so this is going to go through here it's going to show us what the uh control net is is looking at when it tries to do its thing the strength is at 100% so this should work hey Fearless uh yes it will be available I'm going to put it on the on my Discord after uh the link to my Discord is on pers doxyz um yeah and uh you can just go there and join join the Discord yeah this is already working so yeah these masks you don't have to make them you can uh you can uh pre-process footage um with uh that already exists with the tools in comfy to uh redream footage that already exists so uh after we make this mask I'll show you how to do it with like a you know a video of a person or something there we go that's cool unfortunately the orange part looks a bit like a butthole but that's okay it's an orange yeah it's crazy how I took the this mask and these two images and totally combined it all into an animation for us so let's try a person I'm actually going to add NSFW nude back here just in case and let's go to pexels and see if we can find footage of a person drinking coffee [Music] maybe nope uh maybe dancing dancing those look like pictures I need videos thank you uh I've used a lot of these already to find one that we haven't used [Music] before uh let's try that might be cool it's slow motion but we can speed it up too slow motion I don't want to do too much bare skin stuff just because stable diffusion doesn't always play [Music] nice maybe that's the move it'd be hard to get a depth map from that just need somebody dancing you know like a normal human person oh let's try that I bet it's not going to work because she's spinning and it'll be funny so let's do that all right let's so I have the video I want to use as a mask um I need to pre-process that uh so let's drag it into no we can't do that we have to do what I said earlier you know how we made made this because I want to be able to use a different node here now we've already got to the point where I need to use different nodes so VHS load video upload and then we can use this node here um basically in the same way so let's add our frame count to this and uh the image uh we'll go into a pre-processor which we're going to use depth anything okay and then that's going to go into both of our video combined so we can see what it's doing and the apply control net and the other thing we need to do is switch this because it's a um it's a vertical video so we need a taller height and width I'm going to load it at Double speed because it's you know just somebody spinning around uh I'm gonna make sure I uploaded the video which I don't think I actually did oh good is it broken oh no here we go spin spin spin spin spin spin okay uh you're going to do your thing that's connected to the right stuff to skip so set the frame rate to 24 frames per second frame load cap is zero actually let's just do one yeah I think speed one is fast enough and then you're out to here so we're gonna change this to depth the depth control net and we're going to use depth anything so uh oh and we need to use images that um you know will do something cool for a person uh so let's go to photos uh person dancing let's see if we can put a different person on there I don't know if it'll work but let's try it let's try this guy and I'm just going to remove this one for now we're just going to try to get that one image going all right okay I think I think that's all the settings uh elderly man dancing so we're doing a depth pass on this it'll come up on here we'll be able to see what we're doing and if depth works we'll start stacking some more control nets see what we get good night when Ving videos to IP adapter is there a max length uh for someone running a 3070 these 15c clips so doesn't always work yeah I know uh it's tough um 8 gigabyt of vram sometimes isn't enough to uh do some of the stuff but you can always check how much you're using uh if you install nvi top it'll tell you exactly how much vram you're currently using for a workflow and if it's going over the amount that you have on your card then it's going into your system memory and it'll take forever to install that you just go to command line and it's pip install nvi top ah I think he's doing a pretty good job actually this depth anything is really good at at subjects like people and uh and backgrounds Cu uh actually Tik Tok trained this uh the company Tik Tok yeah so this is pretty fast a minute and a half to do uh 61 frames uh of you know 9 by 16 that's not too bad it would take probably three or four times as long to do regular anime diff um so yeah it's pretty cool oh love that head spin but I would say that did a pretty fantastic job uh with just the depth map depth map in the IP adapter of the of the of the G elderly gentleman here getting down got his suit got his face even got his ponytail nailed it nailed it I love how the head stays perfectly straight the whole time it's beautiful so as you can imagine the possibilities here are pretty insane so let's add another control net so we can see how to stack these out so this is our our control net stack right so we just we just want another control net stack so what do we do copy and paste whoa look at that second control net stack uh positive to positive negative to negative same deal here positive to positive negative to negative and we we are running through our double control net stack so now let's add uh loose control with the depth map as well and see if that helps we don't actually need to process it twice we've already processed it and we don't need to see it again because we're using the same one so here go here and we'll use loose control and we'll put them both at one and see if it makes a difference so now we're using depth and loose control both at 100% and we'll see if it looks good and if it doesn't we'll we'll turn this one off do another one and see you know this is how you can test what the control Nets do and what the control net weights do with only a minute and a half you know between renders to find out what you're doing it's really fast uh Den pose might work too [Music] [Music] make sure all right huh I think it's somehow better I love how the ponytail turns into a hand real quick oh that's trippy so let's try without the depth control net and just lose [Music] control yeah this workflow will be on my Discord as soon as the stream's over actually if I get it to a point where it works good I'll just dump it on soon I just want to make sure it's pretty and stuff first so I don't have to support it like crazy later hey Titi I'll show you guys how to do an upscale too um yeah so I don't know if I don't know if you remember the thing in automatic 1111 called highres Fix but it uh it's still uh it still works in comfy it's just a different node set so we can we can set that up oh the hand that's so creepy that must be where the hand came from the loose control huh so it thinks the ponytail is a hand what happens if we give it something abstract okay while this while this one renders let's build the upscaler okay just got to look at my old one just so I don't go crazy here okay we got our current one we need to uh we need to upscale it and make a new version okay so sorry about the ey for this one we want to use impact pack uh that's in um manager install custom nodes impact impact pack okay and remember after you install these you got to reboot restart comfy or it won't finish installing them okay we need a few special nodes we need the pixel K sample pixel K sample upscaler provider okay that's set to LCM six steps CFG of two all this stuff the same as the other one basically so uh one thing you can do to help uh maintain sanity is set these to fixed seeds and then uh uh that way it won't try to run the same run uh twice if nothing has changed so uh that way you can unplug this K sampler when you or this this upscaler when you don't want to use it and then um you know plug it in when you want upscale see how this turned out real quick looks cool sweet that's really cool def that anything is really good all right so let's upskill we got our uh thingy and we need uh we need an it upscaler uh iterative latent upscale okay two steps I'm going to explain what this stuff does in a minute but I'm just going to show you what else you need to plug in you basically need to plug all this stuff in okay so the model comes from animate diff that's right same model that goes in here vae that comes from our our checkpoint loader which is way over [Music] here positive a negative prompt guess where they come from that's right out of our final control net so we're going to run control net again on our thing um down the road we can add another control net for the upscale and back it off a bit and that tends to work better for like getting a little more dream into your upscale but for now we'll just use the same ones so let's make reroutes so we don't go crazy okay there's our positive and negative Pro uh clip let's drag this over here positive negative hey oh okay so we need to plug in these two as well upscale model options upscale model uh load up skill model load upscale model yeah upscale model loader we are going to load the uh 2x plus real ESR Gan you can get that in install models it's this one here real ESR Gan 2x it's faster than the 4X and we're not uploading we're not upscaling 4X so you don't need to go 4X you only need to go 2x for this so use the 2x Plus in the PK hook we want to add uh two different nodes I believe yeah pixel K sample hook combine and then we want to combine two hooks okay want to combine CFG schedule and uh what's the other one denoise schedule yep and just leave those Target Doo at 20 simple three that's fine leave those alone on default settings this is essentially our setup the samples the samples come from the uh the last one so they come out of our our initial run out of there into our iterative up scale so this is our upscale down here beep and what do we do we add a Laten out to a vae decoder lecoder same as the other one and then another one of these you guessed it okay and I'm just going to add the word up to the front of these so I know that they're up skills up all right so now we got our up skill version and our regular version so uh after it's done doing this it'll do an upscale that upscale will be 1.5 times bigger it will go uh two steps which means it'll run two times and do the this stuff it's going to try to increase the schedule uh the target CFG from two to three towards the end and it's going to try to bring the D noiser down to 02 towards the end um what that does is it only adds detail where it needs to add detail as opposed to upscaling the entire thing um results in a better just a straight up better upscale than just doing a regular upscale Laten by 1.5 okay uh because our seeds are fixed now I fixed them after this run so it's probably going to try to do this again and then it'll do this one but what I'm actually going to do just for the sake of time is just do a square 512 x 512 and that'll go you know smaller so yeah it'll run this one and then when it's done it'll spit all this stuff into the upscaler and and then we'll do the upscale so let's make this uh convert no not convert to group node we want to add group that's the upscaler that's the output add group yeah upscaler upscaled outputs you are purple you are purple you are red you guys are green you're green and you're I don't know yellow I gu I'll make you green too why not I tend to put options and things you can change in green so you can see them at a at a glance anything that's specific like on on the clip like this stuff control net I usually make yellow animate def I make yellow just because it reminds me of control net a real reason for it uh and then uh I think yeah now it should do the upscale you can see it's doing bunch of stuff here ever tried the Igman image segmentation node in the allore plugin pack no not yet uh I haven't got much into segmentation I need to dig in more do we get to go fishing after we combine our hooks I hope so I hope so hey consumption what's up dude I get the Twitter chats real slow I think that should make sense hey exad as you can see the upscale takes a lot longer but it can be worth it I wish I could play music on my stream but it ends up getting copy stricken so you can just pretend you can hear what I'm listening to or if you want you can go on Discord click on my name and hit listen along on Spotify then you can Bob your head along with me I think this is going to be weird as hell well this got out of hand quickly eh started as a simple workflow nice I was trying to figure out why I turned it into an elderly person but it's because it's in the prompt howly man dancing it's my own fault wow crazy they got like the scan lines and stuff in the image very cool so yeah I mean that's basically the whole workflow uh you can swap out different control Nets you can uh do all kinds of fun stuff so let's try a different video oh yeah we're trying that okay let's try uh see if we can make toad break dance no nude toad please how long do we think this video is it's kind of long actually I'm going to do a frame load cap of 64 so it's going to grab the first 64 frames and I'm going to disconnect the upscaler for now because it's cool but we don't need it but uh yeah if you want to connect that so I'm going to leave this here what I'm going to do is I'm going to make a reroute from the model and this reroute here this model reroute this is your way of turning the upscaler on and off disconnect model connect the model if the model's disconnected it won't do this step when it's done uh when you want to run it you just plug this back in okay that way we can iterate on our little uh our small version until we like it and then we can plug the model in bang the upscale when it's done yeah I will check out the the node maybe not on stream but yeah I'll check it out for sure robust video matting is really cool too for this doesn't look like toad no doesn't look like toad at all it's like it made like Mario in a thong not a fan what happens if there's no we just use toad with no [Music] prompt that is disturbing it did a great job with the the actual body though that's just terrible oh now he's just fully nuded okay uh screw [Music] it [Music] seen a person in the middle who looks like a human person oh I don't suppose that would work Yoshi you came in at the worst possible time as you all know there is potential to make NSFW stuff with this even even when you have NSFW in the negative PR this guy looks nude to too okay I am over that not lovely all right maybe it's the clip let's go back to masks masks are safe let's try this input with one of my masks I just made this one earlier this try it so I'm going to switch over to an image so I want to drag this into here actually yeah sure let's make this depth sorry this one up here this int goes here and then we paste the New Path into here yeah and then also want to pass the image to here and to depth to you okay so then we want video combin fine here also with nothing int image out depth anything okay so it's going to do a depth pass and we're going to pass that through the depth then it's going to pass the actual thing just through to lose control and we'll see what we get uh with uh these guys as the image see what it do it did look like it was putting clothes on that person maybe it was just the first step but you know what I'm not taking any [Music] chances you know what that anything isn't amazing at abstract stuff so I'm actually going to not use the pre-processor and just plug it directly in and just try that again so it's going to run the original image which is already technically basically a dep a depth map uh in it for both I don't know what's this going to make I don't know if it's going to make a waterfall of Nintendo characters or you know what [Music] we've lost the IP adapter completely let's try something tropical think you just can't understand what it's looking at actually that might be good yeah we'll try that one one true yep you're good you're good okay and I'm actually going to make it a 912 by 512 so that it's uh wide we're not upscaling so yeah I wanted to look I wanted to get the whole mask or as much of it as I [Music] can I think we've got maybe too much control net going on so I'm going to remove the uh depth control net and I'm going to set this one at like7 and then after this I'm going to show you guys what St n percent does with control net because it's actually really important to understand uh how control net works and what it does uh and how you can use it to control the composition of your animations I feel like this one's going to be texturally cool but it's going to be very smooth yeah I think the depth the depth control Nets just taking over completely so we're going to queue up this run here so that's the depth is turned off now on the control net and we're only using loose control at 70% % so this is both control Nets at 100% or I'll be wrong and it'll look exactly the same and as with all this stuff I recommend you go to the [Music] your results that looks really cool but we've lost a lot of the Mask in it it's just using the mask for a bit of motion you know which works looks cool but we want to increase this a little bit try to get try to get that mask back in there a little let's try and find find the happy medium there it is cool though yeah Jox is here I'm going to do a deep dive on his node pack pretty soon once I figure it out if you don't have them they're Joy metrics they're very [Music] good thanks yeah I worked hard on the flower nipple pillows okay now we can see more of the Mask here let me grab the mask and put it under here so we can compare it or at least put it here yeah let's wait for this one to finish and then I'll sync the impr previews come on baby there we go [Music] uh sync preview so you can see it's got an influence but then the animation itself is actually kind of taken over bit too so that's at 85% so let's try 90% these numbers here this this waiting is finicky and it changes from uh from animation to animation so there's no magic settings for this stuff it just you know you got to try stuff like I'm not really sure what I even want but I feel like this is probably close to it what I want is I want this mask to be to sort of come through the IP adapter Source in an organic way like like I'm I'm trying to make it look like the mask is just pushing the input around that's better we had a nice separation but we're still getting the the input image I like it I like it a lot so let's try do I have anything else in here that would be cool what uh just absolute nightmare my downloads folder has become we could yeah let's try uh let's try our man Nicholas Cage screaming a pigeon I kind of want to just run this at the same settings but let's use a different mask yeah let's try this one nice thing about this is because because we're getting the in the frame count from the the loading node like we don't have to worry about the batch size it's always going to be correct uh what you do have to worry about is how long your input vot footage is though if your input footage is really really really long that's a really really really long batch so you want to use this image load cap um to decide how many frames you want to load so you can load as many frames as you want from any point in the video so if you have a really long video you don't need to chop it up you just need to say I want to load 64 frames a th000 frames into my video you know what I mean and then that'll just grab that 64 frame chunk it'll skip th000 frames then it'll grab 64 frames so you can control what you get out of your video my videos are short they're like all less than 200 pretty much so um if I do them at double time they're you know they're whatever so like the one that I'm working on right now 90 frames with double speed that's that's only 45 frames so that's fine and this is a square so we'll go 512 x 512 and hit it and uh also if you want to just take all the stuff that you ever edit like this this this uh the the checkpoints you know this like you can make a a thing where you just have all the stuff that you're going to edit you know in one spot so you can move all that into one little area for this all your settings and like put that right beside the outputs CU like you never have to look at this stuff if you don't if you don't end up adjusting the the settings ever so you know that's the nice part about comfy is these interfaces are completely modular so just take the stuff that I make and make it your own because uh you know the way my brain works is probably not the same way your brain works so feel free to move the stuff around replace things you know this one's going to be a nightmare I can feel it already that almost makes sense okay yeah I think we nailed it we might have peaked it's early to peek we're only an hour in hour and a half it's pretty sweet let's upscale it quantum cage hey busy guy I'm going to put it on my Discord uh the link is on p. XYZ I keep realizing I'm pasting that into thing but you can't see it on Twitter I'm not logged in on Twitter so it's just pers doxyz instead of do you join the Discord there it'll be in the resources Channel as soon as I upload it I'm just going to you know finish what we're doing tonight and get all the settings dialed in and then uh yeah I'll upload it with some examples and whatnot what is happening here though wait why oh wait where's this coming from did I do a silly I did I took the model for the upscale straight out of the checkpoint so the upscale is actually bypassing literally all of animate diff wait so that's just going to do a whole bunch of images in a row yeah okay this was my bad when I did the upscale I took the uh I took the reroute from the initial checkpoint that's not where we want to get it from cuz it's running through the Laura it's running through IP adapter it's running through animate diff So that's its final output before the thing so I actually want to get this and feed that to the upscaler so let's try that again I bet it's going to hook a look a whole lot more like this it's pretty insane though that that monkey person there that is just a blank prompt straight out of yeah straight out just nothing just that's just a blank prompt strange uh the upscale take takes 12 gigs of vram but I'm also streaming so maybe more like 10 yeah there we go another thing to always check is that your positive is plugged into your positive and your negative is plugged into your negative because if you swap them you can get bad stuff I don't mean bad like nude I mean bad like when you use a negative prompt about you know it's like describing what you don't want and then yeah it can get really gross really fast I've seen some stuff man I'm just going to endlessly adjust the margins on these nodes till they're close but still not correct as is as is tradition all right there we go [Music] m 50 subscribers away from 12,000 subscribers on YouTube that is insane unbelievable who are all you people who are you so many lurkers all right up scills done wow oh that's a nightmare [Music] [Music] crazy all right so let's unplug the upscaler for [Music] now hopefully that makes sense uh you can bypass it it's just doesn't always work and leave stuff in memory so you just don't load the model it's usually better in my experience anyway I don't have any concrete science to back that up let's try the skull where are you skull [Music] skull actually this one's cool too let's do this one Mr [Music] Bean it's got a bunch of random stuff in my folder think I just want that might be interesting hit it yeah [Music] ooh that's really cool I want to see that one at regular speed hey mad uh JX can answer that one actually as far as I understand try update uh will like say your state and if it the update doesn't work it'll revert but I might be wrong about that I haven't tried yet I just update like YOLO Styles like a madman like an absolute Menace to Society awesome yeah I'm not sure i' I've honestly never hit the button it scared me super cool super super super super super cool uh okay I think this workflow is like good enough uh I'm going to add a depth anything preprocessor under here but like as you guys know you can just kind of drop whatever pre-processors you want on the control net and I'm just going to dump this on my Discord right now so anybody wants to play with it can just grab it and get working uh save uh we'll call this uh animate LCM uh animate diff uh gen 3 uh or Gen 2 V3 uh what else is in this IPA um control net multi contr control net he control net I think that's all the stuff oh yeah upscaler so essentially is a videoo video workflow but with uh with some fun with some fun all right so let's add the uh workflow image xort PNG all right let's drop it on the Discord arino resources new post title all that stuff animate LCM okay so vid to vid uh fast animate LCM plus animate diff version three Gen 2 uh upscaler FPA well see control net oh scaler all right cool enter message downloads you and you and let's get one of the videos that we [Music] made where the hell did they go I made a folder for this animate diff tests I [Music] believe 8 if testing all right and I think that'll be the one no where's the one that's not as [Music] silly no that's not it either that's it right there let's take this file and go to downloads process I'm going to do some nerd stuff here all right let's do I'm going to make a folder called uh thingy uh called workflow paste that in there we're going to go to command line we're going to go downloads process I know there's faster ways of doing this but I'm just an old dos nerd at the end of the day all right ffmpeg d i film the output is output underscore percentage 04 d.png it's going to create a PNG SE sequence from our MP4 and then we're going to go GIF ski FPS 24 output uh this is our workflow worklow workflow. G start off PNG there we go and now if I go to that folder Groove uh downloads process uh workflow there we go uh and then we'll paste the stream into here I know this stuff is boring but if I don't do it I'll forget and you know it's important okay and the other thing I need to do is refresh this and then uh come on BR I need the link to tonight's YouTube that's not it this browser multitasking all right and then let's pop that on the Discord com youi animate diff worklow 202 24206 okay friends There's the link all right so it's alive all right and I will send you the link right now uh copy post and as yall know these are fantastic to use with my masks you can get them on the Discord as well all right there's a link to the channel on the Discord with the stuff okay it's live so yeah if you want to play with it plug it in and just remember to reconnect the model if you want to use the upscaler and you make some cool stuff tag me I want to see it you saw how we passed the stuff through so just pass your input through the pre-processor if you want to use the pre-processors and replace the pre-processors with whatever you uh whatever you want and remember that's your resolution there so if you want to go wide you just have to go you like 900s and whatever by 512 if you want to go tall it's 512 by 900 and whatever all right well I mean that's basically the workflow um you can go forth and play and have fun and enjoy it and yeah tag me and stuff you make uh come by the Discord hang out with us uh yeah I really uh really appreciate everybody who comes and hangs out on these streams and thank you thank you so much for hanging out and if you want to come by the patreon and support us uh we really appreciate it it helps keep the channel going and we will be doing more and more stuff with anime diff version 3 as I learn it uh the next thing I'm also going to tackle will be instant ID face ID that kind of stuff um I haven't played with it much yet uh but uh once I get it working I'll play with it and then uh I'll be able to show you how to use it uh so look for that like maybe Thursday night uh I'm going to play with that tomorrow night uh try to get that working and then uh yeah so yeah uh really appreciate everybody chilling and being here I know I said that 60 times but I really mean it um so yeah if you want to rebuild this yourself to understand the workflow and stuff just go back to the beginning of this video and uh you know follow along with me uh if you don't just go on the Discord and grab the workflow I'm also going to throw it on my GitHub uh I'll throw the link to my uh GitHub here one sec uh what's my repo yeah PR com UI workflows uh if you're allergic to Discord that's totally cool uh this is my uh workflows uh GitHub I'm going to keep adding more and more stuff from the Discord on there as as well so that you don't have to I don't want you to have to join the Discord just to get uh access to the files because uh the information should be free right um so yeah uh we're going to get some more uh some more stuff on here and uh yeah and there's also a uh if you're interested in SVD I made a SVD comparison grids uh workflow um that goes through and uh shows you the difference between the different versions of SVD so it'll do a run and then it'll do another run and then combine them into these cool grids so that's on here as well so I'll drop the link to that as well and yeah uh thanks everybody have a great night uh come by the Discord and we'll go hang out there too uh I want to start doing um Hangouts in the Discord where we just generate stuff and hang out so uh I'll figure out times for that they won't be uh live streamed or anything it'll just be more for you know the community so uh come and hang out and make some weird stuff and uh you know we'll have a great time thanks again everybody and we'll see you real soon byebye

Channel: Purz

Views: 2,919

Keywords:

Id: DKfz0skc2oM

Length: 107min 29sec (6449 seconds)

Published: Wed Feb 07 2024