Did We Just Change Animation Forever?

wouldn't it be cool if you could film yourself and easily turn into anything you want like a cartoon character like what is this could just be a means of capturing performance and you can visualize whatever your imagination wants afterwards there's kind of No Limits creativity is only accessible to films or animations with multi-million dollar budgets but it's part of our Humanity to try to visualize things that don't exist like let's talk about traditional 2D animation cartoons the most creatively liberated medium is also the least democratized it takes incredibly skilled people drawing every single frame of your movie to make it happen but I think we came up with a new way to animate a way to turn reality into a cartoon it's one more step towards true creative freedom where we can easily create anything we want so how do we do this how do we turn a video into a cartoon well we've been making a series of videos where we experiment with AI image processing this is similar to AI image generation but in our case we are transforming images rather than generating them from scratch so one of the newest pieces of technology is a machine learning process known as diffusion at its core the diffusion process lets computer generate an image from noise much like how we imagine an image from an inkblot or looking at Clouds so if we take a picture we already have and we put a little bit of noise on top you can have the computer clear up that noise while drawing in new details that weren't there before it's a lot like looking at a picture and squinting your eyes and trying to imagine you're looking at something else and if you squint a lot well what you're looking at becomes really fuzzy and you can imagine that you're looking at something different so current technology can do this amazingly with one image so why haven't we seen it applied to video should we be seeing mind-blowing visuals all over the Internet fortunately the moment people tried applying this to video everything fell apart because the first step of the process requires us to noise up our image every single frame ends up looking different and the video gets super flickery the very nature of this text seems to make it impossible to ever work with video and well I nearly gave up on using it but Dean here at the studio kept experimenting and he showed that with a little bit of VFX problem solving this might be something that we could actually overcome because at the time that Dean and Fenner went out and made that spider verse short and I set off on a personal quest to use this tank to enable me to make my very own anime so it was time to Sherlock Holmes our way through this and engineer a solution this flickery problem in the first clue was Jurassic Park six months ago a user on YouTube named hops uploaded an experiment he had processed Jurassic Park to look like low poly Zelda and he used an interesting new technique to noiseify the image and he had achieved excellent results so I'll explain every time you process an image the noise that you put on top changes therefore the forms in the image get changed every frame if we freeze that noise Things become a lot more solid in the image but now it kind of feels like the image is moving underneath a weird warp layer and the details stay inconsistent so the new trick used in that little Jurassic Park video is simple if we can turn noise into an image let's just reverse an image back into the noise it would have come from literally just do it backwards therefore the noise is no longer randomly changing every frame nor is it locked if the two frames are nearly the same so is their fuzzy noised up versions and therefore they get interpreted in their way so we had one piece to the puzzle solved no more random noise on our image but I tried to make my video look like a cartoon and every frame was being drawn with a different cartoon style so we fixed one problem just to encounter another problem dead end well no because around this time style models were becoming a thing and the stable diffusion space a person named nitrosock started creating some amazing diffusion models built to convert your image into one specific style so then it dawned on us this was the key to eliminating the style flicker like imagine telling a hundred different people to draw a cartoon dog you're going to get a hundred different dogs back now imagine giving everyone a character style sheet saying draw a cartoon dog exactly like this the images are going to look a lot more similar so we had to train our own model specifically on only one style that we wanted to replicate another big step forward but it only uncovered another new problem on tests I performed on myself the features on the face were still changing and jumping all over the place so the idea us we made a video where we trained ourselves into a diffusion model so that we could tell an epic fantasy story why not do that here I train a model to not only replicate a specific style but also specifically know a character me I traded on images of me wearing the same clothes on the same green screen background that is using for my test sequence and suddenly boom everything locked in the consistency between frames was hugely improved well almost it still wasn't quite perfect but we're VFX artists who have worked with crappy video files surely we must have a tool in our Arsenal to deal with light flickering so we applied the D flicker plugin in DaVinci Resolve and we set it to remove flickering light and it was that simple suddenly we were there it was working we had a consistent moving emotive cartoon character and is all just driven with video of us on the green screen I think we've cracked a workflow here for getting something that looks like a cartoon and it's pretty Bonkers it's working you know I think we could do maybe a couple more experiments but it's like ready to rock I think Nico has figured out the key to creating a consistent character so now it was time to make that anime that I've always wanted to make because as you've seen on the channel we have our anime videos anime fidget Spinners anime self-driving cars anime baseball but they're all filmed for real because we don't run an animation studio and it's very hard to do so these days so I've had this idea forever anime rock paper scissors what is the ultimate rock paper scissors game two twin princes born at the same time equal claim to the throne it must be decided that day for the King has died and rock paper scissors must happen so we wrote this short film the next step is to treat it like a cartoon which means recording the dialogue first so Nika who are you playing today I'm playing flip and I have no idea how I'm gonna what voice I'm gonna use I figured that we would start and saying be like yeah just go through a few lines and then like we'll throw a filter or two on it to help nail the vibe my character is Jules my research here I just remember the shot of Clint in here screaming into a microphone I'm just gonna Channel Clint today Nico has said that you go you you whisper and then you scream and there's no in between full Dynamics you go really quiet and you get really loud and you do full volume compensation on there so it just sounds like somehow you have this incredible presence where even your quietest moments are as loud as your screams so dni we did the voice acting first and we just blew out our vocal cords yelling as loud as we could sure we did that we then designed the costumes for each character basically just went on Etsy and bought some goofy clothes well look at this high level um costuming here oh no we lost like half the tape what we're trying to do is you see these all these doodads here is not gonna be great for the anime style because this kind of intricate detail wouldn't be included in the illustrations because it would just be more pencil mileage for the animators so we're covering all that up with a color that's similar to the rest of the costume once this is put into the animation space all this Jank gets covered up with beautiful Illustrated lines what we did on the green screen is we basically imagined that we were puppets right we're puppeteering a cartoon character so we're posing like the cartoon character and we're not doing any audio recording instead the audio is already recorded like a real cartoon and we're out there just like we're basically just puppets because once again are trying to treat it like a cartoon so there's a specific way we have to film this we're basically Gathering assets obviously we're Gathering paper cutouts that we're going to then pose in our comic book A lot of these shots are just us standing in a position and we just need a single frame because a lot of Animation is just single frames of characters standing there against the background while you hear dialogue and there's a couple of rules that we've discovered that we have to follow for this so first and foremost you know the style that we're matching it basically has single Direction lighting it's not like you have characters that are painted with a beautiful Edge light and a key light and a fill light in a different direction for an under light like to actually have to paint that stuff onto a cell would be a lot of work and so a lot of cartoons don't do that they just stick to a basic light tone dark tone shading system at most and oftentimes it's just a single pink color at the end of the day we get to take risks we get to try things that might break it because boy look at this giant crew we have oh wait we have like two people and an actor right it's like in Hollywood you'd have 100 people on set doing a big movie so we don't need costume Department we don't need makeup Department we don't need a crazy camera Department right now we're just being very very simple and very very straightforward just focusing on the concept Sam and I spent a lot of time conceptualizing these shots because at the end of the day anime is just really all about tying your visual language to the story and really getting into there stylizing it thinking about things metaphorically all that kind of stuff once again it's your ideas that matter right it's your direction your story because the AI takes it the rest of the way I love it it's such an awesome way to work hey guys so we are commemorating the launch of this anime rock paper scissors video with an awesome limited edition shirt it's only available for the next seven days available in tea and long sleeve tea it's super cool it's designed by human and once again it's seven days only these are limited edition so snag one right now also if you want to little tip here come over here so subscriptions to our website cost 3.99 a month but subscribers get a 15 discount on all merch so if you really want to get cool you could go and subscribe to our website and then use that discount and then basically it's the same price plus now you have a free one month of subscription to our website just saying don't don't tell anyone I told you that this is between us cool all right so now that I walked you guys through the theory and the the techniques that we're going to try to apply here let me show you the workflow that we came up with that led us to 120 effect shots for this piece so here's the video of me on the green screen in costume doing all the acting with the lip sync right so what we want to do is we want to have the AI basically trace this to trace it in a cartoon style so we need to train a model to both know what I look like as well as what style we want to apply so when we were on the green screen filming once we were done we also took a bunch of pictures of me my face my body different poses some full body shots pictures my back different lighting a picture of Nico paringer hey there I am doing like the weird Fonzie point so then we went and we took a bunch of frames from Vampire Hunter D bloodlust which is an anime that came out in about the year 2000 it's actually free on YouTube you can just go watch it if you want to it's actually pretty cool and we tried to grab frames like different people some face shots some torso shots full body shots Hands hair even some abstract things like flowers because with all these different objects with each picture effectively being a different object a different character when we train the model it's not going to learn any single subject instead of going to learn the style in which all these subjects were drawn one little fun little thing here if you look at the vampire hunterd data set you'll notice there's no characters with full beards there aren't any characters that pull beards in the movie and this was a problem actually when we first made the model and it tried to represent me sometimes out of a mustache sometimes I'd have mutton chops it'd be all over the place so I generated a bunch of pictures of me some of them look good most of them didn't but I took the good ones and I re-added them back into the data set and then trained the model one more time I made a new one Nico perringer in the style of Vampire Hunter D there it is it's me as an anime man and notice it's got the the costume details correct it's got my face features correct my beard it's the same character every single time at least consistent enough every single frame so we got the model done and now we can use this for all the shots of me so I've just received a fantastic plate from Nico what I'm going to do now is take the individual images and we're going to run each one through stable diffusion to get an animated character you have our positive prompts which are vampunt D aesthetic style cell animation of Nico perringer man beard profile fist hand but we also have this negative prompt section which can steer us away from certain things like detailed intricate Lazy Eye Photography render CGI these are all the things that we want to avoid coming out and then basically there's a bunch of freaking sliders and it's really boring and there's all that stuff that Nico goes into way more depth over on Corridor digital.com because we don't want to bog you guys down here with the really silly specific stuff but there's a bunch of processes that are laying on top of that image to image process and the output is this there is a really cool animated frame of Nico there is actually an interpretive element to what the AI is doing there's blue shading on the fur on his shoulders and basically two-tone shading in his face it's a really incredible process that we've gotten way too used to at this point we kind of forget how crazy this technology is so what's great is that we can just now run the entire image sequence with this prompt and these settings it's working really well but you know there's a few janky frames like his head just pops in for a second and then pops back out little inconsistencies but they only exist on a single frame basis you know if only there was a way to kind of remove a little bit of this Flicker and smooth out this sequence into a nice consistent character all right so Dean's giving me this whole image sequence now let's check it out not bad it's a little flickery but like it's consistent right you can see the character you can see some of the mouth moving all right so we take the shot I'm going to pop it open in Fusion here and resolve so the next thing we need to add is the D flicker there's two different modes one for time lapse and one for fluorescent lights and if we turn on the fluorescent light D flicker check out the difference much better and if it's not good enough here's how I fix it copy paste paste paste so now we have all these flickers turned on look how stable my face and my costume are the next step is just to pull a green screen key and remove the background and then finally reduce the frame rate from 24 frames per second to 12 to look a little bit more like animation and remove even more of that flickering feeling now that we have an anime man we need an anime world for him to live in that's where Sam comes in so very early on in the process we're thinking how do we get consistency in these shots we want one environment one location so I'm using a environment in Unreal Engine as the foundation for everything and just like the video how we're taking a video still frame and then applying style to that I'm taking a render from unreal and applying a style to that that allows us to have consistency we can go close up we can do wide angles all the same objects in the scene stay consistent here's our Cathedral here I think it was called the gothic interior mega pack The Moment I Saw the screenshot of this environment on the marketplace I knew was perfect I've done a little tweaking redone some of the lighting little modifications here and there to make it match the vibe a little bit better it is just bleeding with sweet intricate detailed Gothic style and it's just ripe for taking cool pictures of I have a camera placed for every shot in the piece and if we go back out here look at that look at all these cameras every single one of these cameras is a different angle for different background flights so shot 250 it's waste of shot maybe something like here-ish yeah there we go so I'm not even going to render these out with like the normal renderer I'm literally just going to take screenshots of this stuff so screenshot all right it's a whip pan so let's pan over pan over Boop and maybe one more Just for kicks all right so now I've gotten four Images stable diffusion kinda takes it from there let's go to my handy dandy prompt sheet of all the different little prompts I've tried out I believe this one right here expressive oil painting dark beautiful Gothic Cathedral interior hyper detailed brush Strokes expressive Japanese 1990 90s anime movie background oil painting matte painting negative prompts blurry I'm all set let's take a look all right so that's basically what the process looks like for all these background plates now that these background plates are processed the shot has everything it needs to get composited basically an anime there's no 3D moves it's all either painted backgrounds or cell animated characters so we wanted to stay true to that so for these shots we have a script that was created by Nico that has everything we need it's got lens Distortion it's got clothes it's got light rays it's got all this cool stuff basically we just drop our foreground our background and the final output it's pretty close to what you want and you just go in and animate little things and add a little bit of flourish here you can see I have the backgrounds that Sam made and I actually just went into Photoshop and Blended them together so you can see the edges kind of smooth out I took those backgrounds and I just literally animated them scrolling past and so you can see as well we've added some directional blur some lens blur and then we land on these two windows and do a final little push in so once we drop Nico on top of that you can see there's this very Dynamic effect that's achieved just by literally moving 2D images behind them on top of that we have this cool plug-in called light rays which creates these beautiful rays of light and we're using the plate of of Nico to actually obscure those Rays so it really marries Nico to the environment what we're also adding to emphasize this motion we took some of the 3D elements of the candelabras from the unreal scene we isolated them and we also turned them into kind of animated candelabras so you can see here as the camera's moving you just whip a candelabra by as if it's in the foreground to kind of emphasize the idea that the camera is literally orbiting around in classic anime fashion to really emphasize the motion we bring in these speed lines right here at the end we have a couple more anime lines to emphasize the point because that's a very important character moment he's challenging my character so all of that together with a little bit of lens effects and then the final piece are these glows which kind of emulate a film camera because that's how anime used to be shot back in the days animation cells would be filmed through a film camera so we try to emulate those glows and after all that we get a sweet shot that I can send back to Nico and he can drop it right in the edit one thing I want to talk about is a democratization of of this process this is a situation here where we're looking at a piece of software that's free that anyone has access to it process here that we're sharing openly with everyone because everyone's openly shared knowledge with us with everyone and the way this technology is developing I want to be able to create really cool things but the only way I can create really cool things is with the help of other people helping make the technology to create those really cool things and so since we've learned and are utilizing an open source program with a lot of contributions to people I want to give back and I want to contribute our knowledge and put it out there so that you can go and make animations and people can experiment and improve upon the process helping all of us get better so in this video I'm trying to make a point of really going through the process in a way that's interesting but I'm not necessarily getting into a Click by click tutorial for those of you who want that we actually have that on our website Corridor digital.com I'll have a link in the description below and of course with every support that's why this video is even possible that's why we're able to take you know four or five people and spend two months just working on an anime about rock paper scissors it's thanks to all the support and the members at corridorigital.com so with practice because we've got this process dialed in shot by shot we just started getting faster and better and at this point we are blowing through the piece and it is working it's incredibly exciting now most people in the office haven't actually seen much of this yet so we're gonna get wrapped up here with the sound of music and all that good stuff I'm gonna show it to them let's see what they think let's see if we can blow their minds it's done theoretically everything Dropped In haven't washed yet just exported it and everybody's gonna wash it with us I think it's safe to say that most of us here have not seen anything I've seen one completed shot and it absolutely blew my mind and I can't wait to see the rest of it I have not seen it with a sound design and that is what I'm super excited for because Kevin sinzaki is so good at what he does and man well I'm here for Kevin damn right you are foreign [Music] feels like a real anime in real life and that's what it is there's like all sorts of wacky AI Jank that Creeps in but you're so used to it that it doesn't feel like Jank anymore the the song that Sam found for the ending Fenner made this amazing shot where my fist comes down and there's just this anime blast that happens and then you just have this drum roll coming yeah it just all melts and then you just get kind of lost it are we gonna get another one Dean is coming Vengeance at the end of all the anime videos we always do one of those montages of like and now here's all the things that happen afterwards which are like supposed to be like the end credit joke but this one is the ultimate fast forward a whole season in this montage and like you know it subconsciously makes you go oh yeah there's a whole story here but we were just like spitballing random shots we get to do another one it's entirely up to Corridor digital.com subscribers we have a tutorial on how we do this whole thing plus a whole bunch of other cool stuff you want to check it out support us help us make more stuff like this super cool you know this is a fun video and all but I'm genuinely curious oh brother I'm proud of you though yeah I'll make you proud would you dare one in real life it happens at normal speed okay one two three one two three shoot [Music] okay [Music]

Channel: Corridor Crew

Views: 3,069,002

Keywords: anime, animation, machine learning, ai, artificial intelligence, dreambooth, stable diffusion, image 2 image, latent space, model, new, amazing, tutorial, how to, making of, anime rock paper scissors, short film, behind the scenes, shot breakdown, react, reaction, image to image, visual effects, groundbreaking, experiment

Id: _9LX9HSQkWo

Length: 23min 2sec (1382 seconds)

Published: Sun Feb 26 2023