How I generated a full CG Short Film with AI

i'm creating a fully animated cg short film with ai let's see where this goes my name is mick i'm a visual effects artist and i'm always looking for tools that make my life easier because making cg movies is pretty hard i don't have access to a team of talented writers character designers or animators like they do have at a big studio like pixar so i have to do everything by myself until now because in this video i'm becoming the lazy director that i always wanted to be leaning back and let my team of intelligent robots figure out everything from the script to the character designs and even facial animation will my computer be able to create a new cg masterpiece for me or will it just be a janky mess what the devil is going on over there so i found this plot generator online and i don't think it's an ai but i think it's a good starting point the hero is a prince from bratislava who is particularly good at magic the nemesis is a prince who doesn't recycle the hero goes on a long quest and ends up bringing down the nemesis i like it so it's a solid idea a very evil villain okay let's check out the second idea frankie is a politician from kansas city who falls in love with his neighbor the two are separated when the neighbor decides the relationship is inappropriate it turns out they were zombies all along what a twist um i don't know if this is going anywhere but okay cambria is an ogre fueled by the death of her baby who squishes teenagers and preserves their heads in vinegar mrs nattoche a farmer from roseville knows she has to stop her eventually natoche captures the villain and wins a bravery award okay i i think i like this one because we have two humanoid characters but one is an is an ogre so i would be interested like what kind of design the ai will come up with will it just be shrek because it's based on like images it finds of ogres will we just make another shrek so now we have our idea but we need the final script and so the people at openai.com have this playground section on their website where you can play around with gpt3 which is like an ai that will finish text for you based on a prompt so i'll just give it our idea like say write a short screenplay about cambria and ogre who is fueled by blah blah blah blah and we'll just see what it comes up with submit [Music] this is actually pretty amazing it is a good it is a solid story it has like a beginning a middle a strong conflict and an end and let's just say it's a pretty badass ending and it's also pretty short so i think it's doable so the next step on our journey is character design the first ai i want to try for that you've probably seen is called crayon it's pretty cool because it's accessible to anyone who wants to use it in the browser but it's also very limited like the quality is not that great way maybe we can use another ai to upscale these images i will just give it the prompt female ogre in a t-pose so we have the character like this like a character like a concept artist would sketch them for the modeling department and i just hope that helps the next ai along the road create a 3d model out of that okay this is looking pretty good we have our characters very cool that it's a white background they are all like at an angle which is weird but this one i think is pretty cool because it's like right from the front the problem is with all these things that the heads are usually cut off okay so i think i have an idea i will just take this picture and let the ai generate like a female ogre face and then we'll use a little bit of photoshop and just slap that face onto the body and then use another ai to upscale this image because the quality is pretty bad the next step now is to create a 3d model out of that one image and i don't want to do any modeling so i found the ai called pifu hd which will generate a full 3d model from every angle from just one single picture okay we have our result from bifu hd and it didn't work this is our result but i have a theory i think it's because of the green skin color green is not really a skin color that humans usually have so it's probably not in the training data so now my plan is to just take this picture into photoshop and shift it so that the green actually becomes skin colored and then just upload it again and see what happens okay it actually worked we now have a 3d model this is cambria like the ai figured out uh like the back side and it's actually looking surprisingly clean the arms are shaped weird but they are also in a weird shape in the picture so that's not the ai's fault this is actually pretty incredible it feels kind of weird i just projected the ai generated image back onto the ai generated 3d model and now we have our first character i mean it's far from perfect but it's still cool to see like a ai generated 3d model that i didn't have to put like any work in so the question now is how do we make her talk i'm using replica studios which is an ai voice tool that is mostly used i think for generating voices for video games let's find our voices come now i can fulfill you every desire i just finished putting in all the dialogue into replica studios and it was a lot of fun to work with the only problem is that you don't have so many emotional styles to choose from for some characters like i just have defeated plotting in vengeance which is fine but i would like to have more but i also checked out all the other ai voice generators and these limitations apply for most of them so if you have any recommendations for cool ai voice generators that i can check out and maybe use in the next project feel free to leave a comment [Music] okay for the facial animation i will use omniverse audio to face by nvidia it is a tool that quickly and easily generates expressive facial animation from just an audio source with nvidia's deep learning ai technology but we have a few problems usually for rigs especially the facial areas you need pristine clean perfect topology for it to deform in the right way the mesh that pifu hd generated is all over the place it's completely messy if this works this will be very impressive the way this works is actually really interesting first you import your rig then you import the default character that all like the animation will happen with and you select only a few points on the target mesh that you want to have animated and the one that's actually animated and then the algorithm will kind of shape your face your rig into the other face and then back again and then somehow it can add the animation on top of that i really don't understand it but it works surprisingly well switching different styles keep creative on the clown sweaters on my brow cause i'm running on these tracks just to keep them running back you know the drilled according back and i've been practicing my crap that's a memory so now that we've got the facial animation covered let's move to the body and for that we're going to use adobe maximo and this is just a library of hundreds of animations that were motion captured and then cleaned up by animators the way it works is really simple you just drag and drop your obj file of the character you want to animate in there upload it drag these markers to the correct places and then we let the auto rigger algorithm figure out everything and look at that we have a character and it's already moving with some animation on it so we can just make our character do everything that we want for example make our character dance roomba another cool thing is we have sliders where we can like exaggerate certain motions so we can like crank up the enthusiasm yeah character arm space i don't know what that is oh okay yeah now we have two parts that we have to combine we have the facial animation coming out of audio to face and we have the full body animated rig out of miximo so i just parented the head to the armature now we have the problem that we can see the old base mesh underneath it so it's like clipping through so i thought the easiest solution would just be to mask it out in the shader so i created this little gradient here and then i connected that to a transparent shader so now we can see the head is missing and if we stick it on top now we don't have this clipping problem anymore but we still see the border here it's it kind of looks like a mask and i don't want that so the idea was to just create a vertex group of like the neck and then use this vertex group in a shrink wrap modifier so now you can see the border looks way more seamless it doesn't look like a mask anymore okay i have to admit it still looks far from perfect but it's a character that we didn't design that we didn't model that we didn't rig that is now dancing around rapping how long is time allows time to keep on switching different styles keep creative on a so the clown thing we need to do before we can create the final movie is to create the sets and the assets and for that we're going to use mid journey mid journey is an ai image generator that works through discord the cool thing is you can also see all the other images that are generated and it's really inspiring to see what people are able to get out of it to create our sets now i'll just take a quick look into our script and it says roseville farmhouse kitchen day so i'll just copy it right from the screenplay interior roseville farmhouse kitchen day and then we can add like what kind of style we want and because we are competing with the best i'm going to put in pixar cgi and then the aspect ratio of 16x9 so let's see what it gives us we can see here it takes some time to generate these pictures but the cool thing is we can see them as they are being generated and i think it doesn't understand the interior aspect so now it's just generating a few nice pixar farmhouses from the outside which is also cool and i think it would make sense to add like one establishing shot as the first shot of the movie so we know it's set on a nice cozy beautiful farm i really like this one and this one but i think this is like a mansion so this is like too much for our story for each of these ones you can generate like four variations of that image oh that's so cool that is so cool okay but we wanted to generate like our interior like our interior kitchen so let's try that again i hope this time it will generate some interiors it's looking good it's so nice i want to live there it's so beautiful i love the kind of retro look that this kitchen has like these these teal colors with mixed with the orange in nature and the weird window shapes okay we generated some pretty cool pictures but i would like to have them actually integrate with the 3d models i found this 3d photo depth map generator okay we're just going to upload this picture and it will use mono depth and tensorflow whatever this is to generate a depth map and here it is and the way a depth map works it's just like everything that's bright is like more to the front and everything that's darker is more to the back of the picture okay so the way i use this z-depth map is i just applied it to a plane and added a displace modifier with the displacement image added into it so now we can add like lights or for example i added this slide camera move it's hard to say right now where the technology will go how fast it will evolve and how disruptive it will be but working on this project helped me understand that ai is not so much a threat but a tool and an incredibly powerful one for me it is a similar feeling to when i first started making youtube videos more than 10 years ago using a cracked version of after effects and andrew kramer tutorials it felt a bit like cheating yeah we were working on shots that could only be done in hollywood or at least it felt like that to us 16 year olds and i think ai is now doing what access to cheap cameras and editing tools did to the film industry it didn't kill it it democratized it in the end it still depends on the underlying idea and how the human instructs the ai even though this short film was in large parts created by ai it wouldn't have been possible without me i picked out and sorted the ideas like a director put everything together and did all the things that are not yet possible with ai and of course i would have had to manually intervene at each step and improve the results to achieve actual production level quality but this project was all about seeing how far you can take it without doing that so without further ado grab some popcorn and enjoy the newest pixai blockbuster cambria the female ogre [Music] [Applause] [Music] what the devil is going on out there cambria stop at this instant i said stop i warn you cambria i'm not afraid to use this i'm so sorry child [Music] 9-1-1 what's your emergency i just killed an ogre [Music] i hope you enjoyed this video if you did please like subscribe ring the bell whatever you have to do these days and if you're using any of the techniques that i've shown today please send me a link i would love to see what you're doing with it see you next time [Applause]

Channel: Mickmumpitz

Views: 368,870

Keywords: AI, Animation, short film, tutorial, dall-e, generative art, midjourney, craion, ai generated

Id: ajyL9FyN-pw

Length: 15min 20sec (920 seconds)

Published: Thu Aug 25 2022