ToonCrafter - This is only the beginning!

enigmatic e hey everyone today we're going to be talking about toun crafter it came out a few days ago but the way AI moves you might as well say it came out 6 months ago and or a year ago but uh no it came out a few days ago and basically what it does it grabs key frames like you see right here you could get like two frames a starting frame and an ending frame for example uh when you're doing animation you have what they call Key frames which are the main poses and other people animate often times it's other people they animate the inet and what this does it will interpolate that the two key frames and create new frames out of the two key frames and does the inet pretty much so it adds those additional in between frames for you so how does this work right uh I'm going to read this so it kind of briefly explains what it's doing it says toun crafter can interpolate two cartoon images by leveraging the pre-trained image to video diffusion prior if we come here it shows some more examples of what we just saw showcasing 512x 320 resolution and you can see what it's doing right here gets two frames and then it animates the inet and it's doing a freaking amazing job at it and uh you can see here too how oh look at look at the freaking details on this look it it it retains the details what the heck man that's crazy here it has some examples of sketches as well which is doing a great job job with that too here we have a disclaimer says calm down I love that our framework opens up the era of generative cartoon interpolation but due to the variety of the generative video prior the success rate is not guaranteed this is an open- Source research exploration instead of commercial products and I guess this needs to be emphasized it can't meet all your expectations so don't expect to be creating some kind of full-on anime with this because it's not there yet and that's just a reality and honestly like people who see this and say like critique it and say ah this looks like trash or whatever like I I I would think that by now people understand that wherever the state AI is at the moment it's not going to stay there it's not going to look like this forever like I don't get that some people just like critique things and just say this looks like trash it's doesn't look good do you not remember what AI videos looked like a year ago or a year and a half ago or how AI art looked a year ago do you not like remember that so that's kind of weird for me when people critique stuff like that but to me like there's so many things that just look insanely good already as it is and if this is going to improve it's going to be kind of crazy so here's an article by AR roed I believe it's pronounced and they're talking about the people that are behind this and here it says a group of Chinese researchers from the University of Hong Kong the City University of Hong Kong and and tensent AI lab spoke about the creation of Tom Crafter neuro Network the model can be useful for Studios creating anime style videos toun crafter is based on the dynamic crafter model tailored for generating videos it was additionally trained on data set from anime H it would not be possible to generate a long scene this way but to crafter copes better with short animations than its competitors at least based on the example selected by the creators on the neuron Network in addition to generating color animation two crafter has several other interesting features firstly the neural network can generate black and white animations based on sketches secondly tomb crafter is capable of coloring animated sketches based on a single provided reference so basically if you draw sketches you can get one reference image with color on it and then it will do the rest for you that is wild it is not worthy that in this tomb crafter mode you can give more than one color reference this allows you to more precisely control the coloring of the scene you can experiment with to crafter demo on hug and face all right so I'm going to show you in a bit where you can experiment with that but I do want to highlight something from this toon crafter page if you come here to limitations and I love that they did this and something that we say a lot in the AI world when it comes to video is like this is great but it's not perfect and in this case it's not perfect and that's okay right because things will get better with time and it will get better so right here says limitations our second model may not correctly and semantically understand the image contents example the black part should be the rigid body of the aircraft which cannot Sway With the Wind and as you can see here there's uh two images here and this thing is wobbling all crazy like a flag or something I think it's just because it doesn't have any other context there's nothing to tell it hey this is supposed to be rigid and all it sees is this little curve right here and then it says well maybe it's something that needs to be that's kind of like a like a fabric or something that's blowing in the wind right yeah I can understand why it doesn't do well with that and here we have have another example says our model May struggle to generate convincing transition motions when objects appear or disappear in the frame so in these two frames we see in the starting frame this guy with the top hat he's walking out of the scene and this guy with the mustache is walking into the scene and this is what it's doing it's kind of morphing this guy with the top hat into this other guy yeah it's just not doing a good job with that so another thing that's uh understandable but again hopefully that can improve with time and I'm pretty sure it will and so we have some more examples here we got this one of this girl and it looks decent the hands look okay then the Chopsticks here or the whatever tensil this is starts to become like noodles like right here and uh yeah it kind of warps and obviously the resolution is not very high it does very well in some areas like we see here like this is so good right here like the starting frame is like a side view then a front view and then the mouth moving this is what it does really well when it does stuff like that like something like this is just insane the eyes are closed they open the mouth moves um here we have examples of the sketch animation and the references and then it's doing this it's coloring it in pretty much and that is insane like that means it will make the process so much faster when creating animations or animes and stuff like that yeah it's very very impressed at where it's at obviously it's going to improve as we speak right now there are probably and there I know for a fact actually that there are people working to make this even better to make it work a lot smoother to have it glitch out less yeah and probably a month or two months and who knows uh or weeks we never we don't know uh I'm may be making an update to this video where I'm talking about how it's improved drastically um but yeah all this is very very impressive I got to play around with it there are two ways that I know at least uh how to work with this and you can go to the hug and face demo where you come in here you put two images let's go ahead and try that uh one of Goku with his mouth open and then one of Goku with his mouth closed I'm going to leave the default settings maybe I just put here anime and maybe talking I I'm not sure if that's going to help it a lot but I think I don't even need to put that to be honest I'm just going to go ahead and put it in and then it's going to kind of go into a queue and then I'm going to be waiting till it finishes and you can also try this in comfy UI thanks to kiji this person developed a way for you to do it inside of comfy and uh if you come to this page I'm going to put a link in the description where you can download the workflow and if you're not familiar with comfy UI I have a video about it there should be a link above where you can check out how to install it onto a PC and then that will help you get started and understand how it works if you come here to examples you can download the toun crafter example right here they put one for low vram and uh so you can play around with these uh try these and see which one um works the best for you so here we have the video and when I play it you see that it's doing an amazing job at mouth movements and it's even moving the clouds in the background that is very cool and it doesn't like it almost always works when you do something like this at least with my testing it uh does very well with mouth movements it does well with blinking so I try to hear in comfy UI and it gives me pretty much the same exact results here's another example that I did and this time it has it kind of going up and down as he's moving his mouth and I also know that if you play around with the seed it will give you different results sometimes it will improve the animation so here is the first attempt I did this is the second attempt uh the only changes that I made was I added a I just kind of randomized the seed and then also I played around with this number right here I believe this is frames per second because if I come up here to the settings here it has frames per second and it has 10 um and so I put 10 here and I think that's what it is so when I did that it did give me slightly different results as you can see this one is not moving up and down as much as this one yeah and that's what I heard I heard some people say that when you bring this FS down it will create more motion and you can see that here for sure to me this just looks a little bit smoother when especially the mouth movement right here than this so you can go ahead and play with those numbers like the fs play around with different seeds and also your width and height that will affect your vram so just be where try to keep things low KY does give some recommendations when it comes to that here are some recommendation for specific models and it it tells you the resolution like uh 320 x 512 it tells you how much it uses pretty much GPU memory it uses and how long it takes pretty much you can check these out when you download this workflow just make sure you you go to manager you update all make sure everything's updated and then when you have this node here when you select one it's going to download that specific model and then you don't have to worry about finding it you can also add some more images if you want um it does really well from frame one to two but when you add three it kind of like stops suddenly and then suddenly goes kind of crazy but I'm not sure if it's just my testing or if it that's just a general thing that it does but you can test it out for yourself um so you can always add right here uh inputs so just put the number and update input and just uh yeah just copy these and then put them into here and you should have more frames to go by so you can play around with this and I will put the link like I said to kaji's uh GitHub so you can download this workflow so I want to share some stuff from other creators and here we have MC mumpitz and he is showing how he's using real life images to create this scene that looks kind of like anime and he's saying that he has a bunch of ideas of how to improve this so I'm definitely looking forward to seeing what he comes up with but I think it's really interesting that you don't even need animated images you can literally just use real life images which is going to make things so much easier if you don't actually have to draw it in the first place so that is kind of crazy so shout out to Mick mumpitz he also has a YouTube channel where he does tutorials as well with comfy UI and 3D and a bunch of stuff so go check him out here's another example from AI warper he is using six input frames with this and then you see this full animation of what is it Gogeta going Super Saiyan blue I believe um again it looks kind of funky at times but still from six frames to do all this is kind of crazy right I love this one right here where it does four frames and you see this fullon it's over 9,000 animation of Vegeta and uh this is this is wow it's over 9,000 that is crazy and this one is crazy too from Justine Moore look at this one right here yeah there's a lot of examples out there if if you have some cool ones send them my way my Twitter ad is 8 bitcore e unfortunately enigmatic e was taken so I had to go with this definitely add me if you have some crazy results I would like to see what people are coming up with as this improves what it basically will accomplish it will make the animation process and like Production Way less tedious things will be way faster coloring is going to be faster and I really hope that Studios Embrace this and I'm sure I'm pretty confident that they are going to or at least they're already probably looking into this or are already very very much in the process of implementing it into their production I just feel like how could you not like this will speed up the process for shows so much more and tell the story that they want to tell and I'm sure some people are going to have issues with this technology but you know you can't really do anything about it except trying to learn it and try to create stuff yourself so yeah all right everyone thank you guys so much for watching thank you for the support I know I haven't been uploading a lot lately but I want to make more videos and I plan on uploading more frequently maybe not always tutorials but maybe uh talking about AI topics new things that come out or just anything you know so thank you so much for your support please like comment subscribe if you have not already and I'll see you guys next time take care peace [Music]

Channel: enigmatic_e

Views: 9,087

Keywords: comfyui, animatediff, stable diffusion, ai animation, viggle

Id: wHF8LQ1YX_k

Length: 14min 15sec (855 seconds)

Published: Wed Jun 05 2024