This Week in ComfyUI / Generative AI (SVD1.1, MotionDirector, AnimateLCM, ControLLLite, and more!)

I've been coding since I was seven I've never seen anything move so fast can you imagine that we will soon have spaceships traveling at the speed of light or even faster generative AI is a space where the app of the day is obsolete the day after and we're commenting on a month old thread is essentially considered necromancy at this point so I thought I'd do a little weekly series where we explore what's new in KY UI the latest advancement in the technology and how to keep up with the Kardashians so to speak or the Joneses or whatever works for you the first exciting release this week is motion director for anime diff is out and what that means in simple English is that you can take a bunch of training videos pass it into the trainer and then prompt against that motion Laura like you would do in traditional anime diff to obtain an output video by the way Batman running in the woods is the new meta so go with that now so the training video is uh in this case a man running in the street you can see that if you prompt it with a highly realistic video of Batman running in a Mythic mystic forest you get well exactly that and what's fantastic is the accuracy the the consistency is excellent really excellent and you can of course go completely bananas and change it with a rabbit if you want it to and it's going to track that motion and this is a GameChanger in my opinion you no longer have to use some dodgy stock video website that you may or may not have the licenses to to obtain your motion fo AG you can go grab your cell phone film yourself or film your friend in your garden and obtain that motion you want to then reproduce in each scene so are we're going to see some extra short films made entirely in kyui yes it's possible it's finally here so some people and this is how fast it moves two hours later kiji from the banad doo server was already doing some test and you can see how well it tracks the face even the eye movement is tracked now pay no mind to the fact that the image is morphed and so on because we can apply of course segmentation to this in post and so on within confi of course but the postprocess is now also happening in kfy which is also exciting and you can apply to different anime characters this was released literally two hours I just can't get over this after uh the code was pushed up that's how fast people move and to use it you simply need to use confy UI anime diff evolved by the amazing developer kin kadink yeah pronounces name right which is obviously the chap who give us anime diff in confy UI you install the set of nodes and then you simply apply the motion Lura that you just train using motion director to your prompt or to your whatever else you can mix and match and you're going to get that output at very high speed it's it's pretty fast even on a 208 the training process is going to take around 12 G Ram so more RAM than the usual eight that I'm working with so unfortunately I wasn't able to train my own luras just yet but hey the computer is in the mail as they say yeah I'm pretty sure they say that oh okay next up we have a tutorial by Susie W Susie one you probably know her she's uh in civit AI she's doing some amazing work including uh the creation of the fantastic and completely slept on in my opinion confir roll Suite of nodes uh which is extraordinarily powerful I highly recommend you go check out or work on CVI and on GitHub so Shia has created a tutorial uh that's literally hello world in config UI and it's going to help you create those custom nodes yourself if you wanted to do that personally I would love to have a single case sampler node that does the step proportion between the first case sampler and the refiner and I'm very much thinking about actually giving a shot because it would help a lot of people and it would help me too because I'm tired of having to redo the same thing over and over again uh so go and check it out it's a great way to learn and I really encourage you to also contribute if you can the big problem we have with kyui right now in this environment in general is that we don't have enough documentation Susie is doing an amazing job at building a little wikii for a project but as you can see there's hundreds of no in CR foll and not all the documentation is completely finished so I think it would be fantastic If people could pull resources together and go help her create this collection of resource so we can all learn and benefit from it moving on we have another exciting release and it's control net light now control net light of course as you know has been out for five six months now and it's anime counterpart four month but what's new is that you now have support for for it within Advanced control net and this is important because it's the apply control net node specifically the advanc apply control net node in config UI Advanced control net that works with animate diff today so essentially what that means is you can now use control light as part of your animated diff workflows that's really really exciting and so I've created a little workflow demo for you uh and it's a static image of course cuz I wanted to keep it as simple as possible where I load a completely legal image and then I use a Excel model right because remember before the control Nets were un limited to your 1.5 models and pass it through the standard clip encoder so on so forth and my preview blah blah blah and now I get my output image which is my little anime dude now this is exciting for more than one reason control light is first of all as I said working with this DXL which is a big Improvement and second it's also a lot faster so go check it out and use it play with it in this case of course I could have segmented the background to make it look better but that's not the point the point was to create a node that you could use immediately and the big difference here is simply you have your model being passed as model optional to the apply Advance control net that wasn't here before this node this node connector I should say wasn't here before and you pass it back to the K sampler and off you go to the races very straightforward stuff if you want this workflow I've uploaded it on KYW workflow. comom so it should appear if you have the extension in your confy workflow Gallery within your KY interface itself oh and while I have you here's a little tip for you you see that if I drag this image node over there I get a series of options and sometimes you're limited to those fives by default unfortunately sometimes the one you want is later on so in this case preview image over here let's say I wanted to change that to get more option you can actually change this in confy and change the number of note suggestion I feel that 10 is reasonable after that it gets a little bit crazy but it works very well and that allows you to work a lot faster by being able to pick up what uh confy thinks you might need next in this case a preview image or an upscale Etc so a little tip here I thought I'd insert because I've seen a lot of people struggle with this the other thing you can do is uh use the search button and type whatever node you're looking for in this case I want beauy beautify by the way is by trong is the inventor of the 0246 not set and it's absolutely fantastic just don't forget to set it to full and it can take any input and convert it back to whatever format you expected in this case uh it's going to show that it's a tensor so this is really useful for debugging it works with text it works with int it works with well it works with anything that's that's the whole point of it right so go and use that really useful beyy node by trun 0246 another huge release is SVD xt11 image to vid just came out uh you can download it from hugging face uh it's it works exactly like the previous one so you can integrate it as part of your standard SVD workflow if you already have one if you do not uh there are examples on various websites I particularly like the work done by a member of the banoo community called pers who's made a comparison between the original uh photo that you use as a base the results from SVD 111 and the previous results from svd1 and he's done a lot of those trials and some of them uh will show differences uh and for the better sometimes for the worst unfortunately but there's definitely been an important update here that you need to pay attention to if you want to use this workflow and you don't have an existing one you can simply go on pers bits GitHub repo and and download the workflow from there he's actually uploaded his entire grid workflow uh it's super easy to use you can make your own comparison and see how you get on with that SVD is what was used by mamimi on a 4K AI perfume commercial released a couple months ago I can't wait to see the updated version on 1.1 if there's going to be one but this is the type of content that really demonstrate that locally on our machines today we can create content that's just as good if not better than some of the best software as a service platform and we do this entirely for free using free and open- Source tools evidently here you can see the video has been interpolated a lot it's in slow motion Etc but are we close to being able to create perfume commercials uh using only our local computers yes we're getting there and my Mimi has made this workflow available on open art so you can go and download it and test it for yourself just change from SVD 1.0 to 1.1 and you can give that a shot yourself it's exciting and you should definitely reward the people who contribute to this space another huge release this week yes another one is animate LCM is out yeah so what does that mean well if you like LCM because it's fast it allows you to render at what seems like almost real-time speeds on certain video cards but you couldn't use it for videos all your prayers have been answered you can download the model on hugging face as usual don't forget to give it a like and someone has already done a little test video on credit where you can see the output I think it looks really good of course it's LCM of course you're not going to get the same results as other uh techniques but it's really good for something like trying out new techniques and of course speed so if you want to try it out this person has uploaded the workflow on a site called floaty which I wasn't aware of but I'll give it a shot it looks kind of fun evidently you also have access to the official uh animate LCM civit AI page which also has workflows examples and a lot of tips on how to configure this properly and make it work on your machine so go give it a shot you will need conf UI animate diff evolved to use uh animate LCM it's now compatible again uh coin kadink doing amazing work here and you should definitely support him on his patreon I think is one of the best creators out there and even though he hasn't posted you know what the work he does in his GitHub repo more than makes up for it so last week I showed you mream mream is a vision model that can be run locally and it's absolutely fantastic you upload an image and it tells you everything that's in the image you can imagine that you could use this to create some exciting stuff with Dynamic prompts and KY and that's exactly what some creators have done with KY image KY UI image to prompt which will use uh moondream output to prom the image and recreate something else you could say only pick up the background elements or only the foreground elements and then fit this to a prompt and generate another image I'm personally thinking of doing some Dynamic uh video background fill with this but you can use it for anything you want so what I wanted to show you today was this Audio model not a vision model but an audio model and it's called parakeet and it transcribed Well audio and it's absolutely fantastic I won't play this because this is extraordinary embarrassing to test these things I sing into the microphone and I'm a terrible singer so but it does it does pick up your words and my terrible French accent at very high speed no problem whatsoever once again how you use this is up to you but I think it's exciting will it be implemented in KY I'm sure someone's working on it in some capacity and remember we have the tutorial that we saw earlier so if you want to go and build something cool this could be a good way to do some multimedia installation things like that and then deploy it as part of um an art gallery or something of that nature or maybe just want to do it for fun in any case moving on the next release this is huge uh is segmine Sego everything is Mo these days uh so MO is this Mo mixture of expert not that Mo yeah no not not that one but if you're familiar with what mixture of experts can do for you with llms which is essentially having multiple well experts llm answering your question simultaneously one might be very good at code for example the other one might be better at the generation of Pros or poetry and so on well it does that but it does it for Imaging so it has a lot of advantages specifically obviously these several fine tune expert uh and also it can be upgraded over time I think that if you look at yes the release has four stable diffusion experts model as part of Sego X42 and is also Sego 2x1 which has two stable diffusion expert models uh and it's going to allow for uh the this better adaptability to the data but it has limitation it's not for Speed and it will have some Ram disagreements people that have used it are saying good things about it as you can see it's growing parabolically but it's still very much a niche use case at this point in time but give it a shot if you can everything is available on hugging face and if you want contact me and I'll make sure that you're featured in the next video and I didn't have a good ending for this video so I'll leave you on [Music] this say what you want about America land of the free home of the brave we got some dumbass mother

Channel: Stephan Tual

Views: 2,868

Keywords: AI, comfyUI, generative ai, motiondirector, controlnet, controlllite, svd, svd11, animateLCM

Id: ZPTAs-XGKTQ

Length: 14min 41sec (881 seconds)

Published: Tue Feb 06 2024