Stable Diffusion Video To Anime AI Video (No Limitation By Discord App)

hello everyone in this video we are going to experiment how to transform your video to anime style using stable diffusion and comfy UI you might have seen some Discord app with similar features or some Rd animator YouTube videos have mentioned how to use those apps but now you can run this in your own comfy UI setup you can generate your animation privately without other people watching your result in Discord and create a workflow in comfy UI private enable you to customize specifically for difference video result some video might need more depth layer control and some require character pose control in other words a general AI animation Discord app well might not able to fulfill that let's check it out here's the light version of the video to anime workflow it should be a very easy workflow for most stable diffusion Ki users I'll go through some steps on what you need to get start what kind of input are required and some parameter values that you need to set for example width and height there are also some segmentation objects that you need to set before your click Q prompt pay attention on this video I know someone like to just rush to the link download files and try to run it without knowing how to use it then come back ask how to do it blah blah blah for running LCM enable workflow you have to be aware of the checkpoint mod models and the animated motion models that you are choosing to use for video generation because you have to know what you are doing if you are using an LCM checkpoint you have to choose the right option for the loader group as well I've described it briefly in this note box and you guys can check it out first of all I've tried to make this workflow a little bit simpler than some Advanced one that I did before I noticed a lot of people were confused about having trouble to understand what values to set and for many custom nodes that looks like rocket science to them so I have made this workflow simpler an input output way for you guys to use easier I've also changed the habit of using the load video by path instead of load video by upload so you don't have to duplicate many video or image files from your storage for example once you're using a load image this node will be upload your file into your comfy UI python server side even running in the local machine it is is duplicating one more file in your comfy UI input folder which causes your hard disk storage space to be wasted a lot instead of using load image or load video by upload I'm using load video by path and load image by path so in this way all you need to do is type the file path of your video files in this case I'm using the MP4 video files that I just downloaded the dance showcase we just saw in that Discord app it is going to be try out for this tutorial animation let's confirm one more time this is the one that we are going to use in this tutorial for animation example secondly we can check out the segmentation mask group here I have preset most of the setting in this group except you can fine-tune your mask expand area and blur radius mostly you need to configure these two values and also the segmentation prompts before your run for example here I use human for segment mask prompt or sometimes I use animal like this motion video the Indonesian giant lizard in this animation I am segmenting the animal and using a cyberpunk style on the giant lizard skin therefore it becomes a kind of purplish color on the giant lizard skin so in here you have to segment whatever object you want to change or configure to mask then this group it will output the segment object mask images and the background mask of your reference video image frames after we set up the video and the segmentation we'll go down to the loader group The loader is going to do basically load the checkpoint configuration and text prompting control net setting here as I mentioned in this yellow note box these are the brief instructions so here is the brief instructions if you are using LCM checkpoint models then you can bypass this load Laura model node for LCM Laura because this Laura model node is using the stable diffusion LCM M Laura Therefore your checkpoint model for example my checkpoint model selected in here is using the real dream tune editions turbo LCM for anime or cartoon Styles then the checkpoint is LCM ready by default I don't need to use the LCM Laura model I can bypass this and let it just run by the checkpoint model as LCM mode itself the following Laura node mostly I am using add detail Laura you can use this Laura model or other Laura as you prefer you don't have to follow me exactly next is going to be the text prompt I have set up the negative prompt by default here if you want to fine-tune it yourself it's okay do whatever you want here for the positive prompt mostly I am going to aim for an anime 3D and CGI Style by default in this preset text prompt and on the following line we can do some customization so for example I want the character to have blonda hair the character to look a little different from the original look so type blond hair maybe add blue shirt and black miniskirt something like that I can just try typing that in here hopefully the AI can follow the prompt instruction I've tried to make this workflow as simple as you can and it's not going to have a lot of things to configure most of the custom nodes here I have pined that lock as they are already set for some beginners you won't be accidentally delete them or accidentally move something or remove connections while your mouth Mouse is moving and you drag and drop by mistake causing something to go wrong this light version I'm aiming for allows a lot of people to play around with and try out comy UI and stable diffusion and it will be easily generate anime style video like the Discord app we have saw most of the features in here are very beginner entrylevel friendly and right here continue on our workflow has the control net models I'm using the line art to strengthen the object outlines in each image frame frame as well as soft Edge and depth anything now the depth anything control net model you can check out my previous videos on how to install and update your comfy UI to be compatible to run depth anything control net which also make the pre-processor nodes for depth anything so you're able to run this workflow here's the tricky part that I did by using line art and soft Edge then we're able to do most of the video animations that can detect the character and make the character turn around like this the head isn't sticking to the front even when the body is turning if the control net have not fine-tune the setting the character will present so creepy like a [Music] ghost you have noticed many stable diffusion users are trying to avoid character turning back motion or they will cut that part in video editor you'll see the face appear on the front while the body turn around it's like watching a scary movie but it won't be happened with these settings next we'll check out the animate diff prepared group I have pret everything to work fine for stable diffusion 1.5 with version 3 motion models and the LCM checkpoint models or LCM Laura model most of these custom nodes you don't have to set anything anymore just leave them alone you don't have to touch most of the things all you have to do is if you want to change motion models you can do that as well for example I have one fine-tune motion model using K-pop dancers and you can change the beta scheduler for animate diff in this case if you want to use the default animate diff scheduler you can use that but in my case this time I'm using the LCM linear scheduler I'll leave that alone just stay as it is right now and it will pass all the data here we use the free You2 that is going to make the model data more refined before we pass it to the sampler again the sampler is very clean right now as you can see there's nothing you can accidentally touch or delete you won't have to always come back and ask me what to do and how to fix your workflow that won't be happening right now you basically if you're using LCM don't need to touch any settings in the sampler right now so there you go at the end we'll see the output of the video combined here and by default this light version is using a frame rate of 30 if you want to set it higher you can do that as well mostly a frame rate of 30 is good enough for most anime style video If realism style I will set it to 60 frame rate so let's try this one I have the video loaded in here again and let's try maybe 100 frames this time to make it a bit longer for people that don't know if you're running with limited memory in your Hardware you can run a shorter number of image frames here in the frame cap for example in this case I'm using 100 frames and if you start from zero in here skip first frames start from zero means you're starting from the very beginning of this video so if you're doing in this case I'm skipping 100 frames then that means I'm going to start right in the middle the beginning middle of here or somewhere in here you have to do the math on that so that is the case for using skip frames and we use frame cap so you don't have to overload your computer hardware to load all the image frames in one generation because imagine if this is 16 seconds mostly even I use Nvidia RTX 490 most of the time I set about 400 to 500 frames for each generation to safely run without overload and maybe I'm going to do two or three Q in a batch to complete the whole video generation so this time let's try it right in the middle we'll skip the first 100 frames and we'll generate the frame cap is going to generate generate let's say 150 frames so you guys understand the concept that these two settings don't use the same numbers and there we go we can start processing everything basically but before that let's check out the text prompt and I would like to add bright brackets in between each text prompt to have a strong reminder for the AI to edit the Styles so the character will be blonde hair blue shirt black miniskirt hopefully that will appear in our generated result for the width and height we're going to set a small size just for demo purposes let's say 512 for the width and 960 for the height and we're good to go so let's wait and see the result Okay so we've created our animation and let's check out this result as you can see the color of the clothes and the hair style have changed I also added a little prompt saying that it is on the beach so as you can see there's some water that looks like the sea behind them and then the stage the dancing stage it looks like some wooden floor on the beach but it's not going to be a very consistent style because we're just using the text prompts here also stable diffusion or most AI models usually don't 100% follow the text instructions so therefore we hit this blue shirt and blonde hair prompt and we hit this on the beach instruction but we missed out on the black mini skirt so it happens you know it won't be 100% following our prompt instructions that's what happens with many AIS and if you want more customizations like you want to totally change the clothing refine the face do a lot more stuff you can check out my full versions of this workflow so this is the full version of the video to animate workflow and basically it can be done with the same result as what I have done for the previous YouTube short video examples if can change the clothes the backgrounds color themes and also enhance more details refine the face and add lots of details on the clothing and the character's outfit we have also refined the movement how it turns and the speed of the Motions we've done that in the full versions of this workflow and I'm going to try generating examples with this one and generate full versions in here as well and we will see at the end of this video I'm also going to generate the light version workflow of this full video animation and right now this video at least is 5 Seconds we're using 15 frames to generate 5 seconds so that at least it is you know doing better than what we saw on Discord by just only 3 seconds each generation and we couldn't even present anything by just seeing the hair moving and the eyes blinking you know there's a lot of limitations when I tried out this Discord bot and yeah I couldn't even see what happened in 3 seconds what can we do with that so for the leite version I think using animate diff with comfy UI provides more customizable more configurable things for us to do rather than just uploading your video and letting it change to an anime style without any ability to customize the color of the hair or any character styles that you want that seems unable to do that that's what happened there and we can do that in our animate diff workflow in the comfy UI setup and I think that's going to be a big Advantage for stable diffusion users let's check out the result of this dance video we'll go through all 16 seconds of this one and we're going to use the light version to run this and also use the full version with more customizable Styles the character is more refined with background customizations and of course with extra features to enhance the video generation result [Music] now I have published the Lead version workflow for everyone you guys can download downlo it through this link anyone can download it and create your anime style videos for my patreon supporters you have the full version with an in-depth explanation in our patreon community so let's check it out and I will see you guys in the next video have a nice day see you

Channel: Future Thinker @Benji

Views: 3,809

Keywords: ai video generator, ai tools, ai, anime video workflow, Stable Diffusion ComfyUI, video generation tutorial, segmentation masks, checkpoint models, ControlNet settings, animation customization, line art, soft edges, Depth Anything, ControlNet, model, motion models, AnimateDiff Prepared, video generation results, Patreon community, anime-style videos

Id: IkanfUReMXw

Length: 15min 21sec (921 seconds)

Published: Tue May 28 2024