How to Add Animated Characters Into Your Video | Blender + AE VFX

this is shushu he's a cg character integrated into real footage for this cooking series now one shot of shushu as far as vfx goes is pretty simple but multiply this times 35 shots and it changes the whole approach to this sort of vfx work so today i want to go into the workflow for a project of this scale and highlight the key stages of production that helped bring shushu to life [Music] this is kyle balda hi i'm kyle balda like jacob just said and i've been working animation for the last 30 years mostly in the role as a feature film director kyle has worked at ilm pixar and most recently illumination where he's directed three films in the minions and despicable me franchise last year as kyle finished his last film his wife janet finished training as a pastry chef at le cordon bleu in paris and then came the pandemic and a lot of times spent at home and i thought as a way for my wife and i to pass the time that we could you know try something that incorporated her skills as a pastry chef and my animation background and thus sushi was born cal then contacted me to help with the vfx and we were ready to begin to start our journey through this project let's talk pre-production specifically character design to design shushu kyle started with some rough 2d mockups which helped guide some early 3d tests i did a really rough and crude model and walk cycle he's a knife i mean he looks like he's 50 feet tall and he weighs you know two tons he's like way too heavy for this cute little animated character so based off of that i just tried to refine the overall shape and get like a nice silhouette with him silhouette is extremely important because especially in animation you want to be able to look at a character and instantly read what the character's attitude and personality is and a lot of times you'll reference early charlie chaplin films if he raises his glass to drink everything in the silhouette is going to be perfect so that really quickly you can see what the pose is and you're reading what the character is thinking or feeling after further 3d iterations to develop the eyes hat and other parts of shushu his design was solidified and ready for rigging a lot of what i figured out in the rig was thanks to the level pixel level tutorials uh which are very clear and very methodical in the end shushi was given an fk ik rig with lots of specific controls added on top to move different parts to better understand the basics of this sort of rigging you can check out my previous tutorial next onto production for filming kyle and janet used blackmagic cameras to capture some awesome raw footage for all the effect shots the camera was kept either static or on a motorized slider the slider helped to repeat the same camera motion to film both a clean plate and a version with tracking markers finally they captured some 360 photos for reflection and lighting reference with a reiko theta 360. and with that we moved into post production to start janet edited together a rough cut of the video kyle then took this cut and created a rough storyboard by adding grease pencil drawings of shushu and that storyboarding phase among other things was helpful because then i could start to see the types of facial expressions that i wanted the types of eyes and mouth shapes that i wanted to get and be able to make sure that i could get those in the computer this storyboard was also key to getting organized for the work ahead of us as you'll see each vfx shot with shushu gets a shot number i then created this shotless document with these shot numbers and a rough description of each shot this document helped us track the progress for each shot from tracking to animation and compositing now to get the raw footage from kyle and janet we what yeah i see oh we're just gonna do this now i've been releasing some exclusive tutorials like this one over on my patreon that patreon page is a great place to support these main channel tutorials and learn a bit more about vfx in the meantime if you're interested the page is linked below okay good oh okay good goodbye weird guy all right back to footage transfer to get the raw files from kyle and janet we used davinci resolve's media management tool which allows you to duplicate and trim raw files to only what's in your edit timeline we also added about 10 extra frames to the start and end of each shot which is a good way to add some padding in case a final vfx shot needs to be shifted or made longer in the final edit once i received these raw clips i put them in my own resolved timeline and exported an h.264 version for each clip these h.264 files will play back more efficiently for animating and camera tracking i use the mix of both after effects and blender to track these shots to transfer camera tracks from after effects to blender i use a plug-in called ae to blend i have an in-depth tutorial about this process over on my patreon for shots that were extra hard to track i created what is called a high pass which brings out more detail to create this pass in after effects i duplicate my footage layer invert the colors of the top layer and set the opacity to 50 this should make everything gray then i can add a fast blur effect to this top layer which will start to bring out the details finally i'll add an adjustment layer with a levels effect to bring up the contrast and bring out even more detail i used this weird crunched result to track some shots but you can take this a bit further by duplicating the footage and pre-composing these layers then i'll set this new composition to soft light blend mode which should sharpen the footage once i have a good track set up in blender i make sure that the scene scale is set up correctly to help with this janet sent me the measurements of the different bowls or pans in each shot i then create 3d versions of these with the same measurements then i position this prop roughly so that it aligns with the bowl in the shot keeping the z position at zero since it's sitting on the counter but we don't want to scale our bowl down instead we want to scale our scene up to do this i will first parent my camera to an empty at the origin since i can't scale the camera itself with keyframes on it then with the bowl in edit mode i'll select the bottom face and then hit shift s cursor to selected now with my 3d cursor at the bottom of the bowl i'll select our camera empty and i'll set the pivot point to 3d cursor and now i can scale the camera until the bowl lines up from this view you can see how this scales the camera and not the objects in the scene with the scene scale set correctly whenever shushu is added to the shot his skill will be correct finally i recreated any other objects that shushu interacts with so that kyle could know where to play shushu in 3d space onto animation since we were going to add shooshu to over 30 different shots we wanted to make sure that we could make changes to his materials or rig that would automatically update in all the shots blender provides two ways to do this through proxies and library override we use proxies and here's a simplified look at how it works so you'd have a blender file with your character and armature rig all the bones that make it move we'd save this as the master file then if you create a new file in our case a specific shot we'll go to file link and we can click through the master file to select the collection with our character and armature then with the character selected we can go to object relations make proxy and we'll select our armature this allows us to now animate the character freely for a specific shot but now let's make a change to the master file we'll add another part to the character connected to another new bone and we'll maybe change the material color if we save this and go back to our shop file in the outliner under blender file we'll find the linked master file which we can right click and hit reload as you can see our animation stays the same for this shot but we also have these changes added on top with this proxy system in place kyle could animate all the shots and if i needed to make a change to shushu before rendering i would just change the master file and the change would update in all the other shots and now i'll hand it over to kyle for more on his animation process the way that i like to approach animation uh which is sort of an old-school way to do it is to think about as if you're playing with an action figure so that's all that's happening here in this really early blocking pass but very quickly i can get a strong sense of what his overall attitude is and then from there i'm able to go deeper and like do the overlap of the arms get more of the facial expressions a big part of the contrast that i'm getting between these types of movements is that one is a really legato kind of like smooth uh type of movement and you know you can see this in animation like woody and the toy story movies he's a very legato character everything's very smooth where buzz lightyear is very like pose to pose really sharp staccato movements and you can use that to define characters but you can also do it within one performance so if the character is really mellow and calm he's moving in a legato way see something scary switches to staccato into really sharp movements and you're using now legato and staccato as a way to differentiate between calm and anxious by roughly blocking the shots this way kyle was also able to see which parts of shushu's body actually needed to be animated you know when shushu is like bringing his eyes over the little tray and he's looking at all the choux pastries there's no animation on the lower body because you're you're not going to see that part of the character by only animating what's seen of shushu in each shot kyle was able to save time and focus on the most important parts of the performance after animating shushu into all the shots kyle sent the blend files back to me for lighting and rendering i started by creating a master file with all the different materials and 360 reflection setup for each shot i then appended these assets from the master file as needed for lighting given the large number of shots i kept the lighting setup quite simple i added a large area light above each scene along with this 360 photo of the location this is not an hdri so it won't help with lighting but it will give some nice reflections and that completes the lighting i rendered shushu on his own view layer by putting him in his own collection and having it turned on like normal the environment objects were in their own collection set to indirect sometimes i set a bowl object to hold out if shushi was reaching over the top of it next i created another render layer for the shadows and reflections i started by setting the shushu collection to indirect only then i turn off our bowl collection so we just see the floor from this i just want the shadow of shushu and his reflection in the counter to get the reflections we need to look at the glossy indirect render pass the problem is if our plane is set to a shadow catcher we can't see the glossy indirect reflection i could create another render layer just for the reflections but that's extra setup and render time that i'd really rather avoid given we have over 30 shots so instead i'll turn off the shadow catcher enable the glossy indirect pass for reflections and i use the ambient occlusion pass for the shadows ambient occlusion uses the nearness of two objects to approximate the shadows so it's not impacted by your lighting at all and therefore not physically accurate but ambient occlusion looks close to the normal shadows when the scene is being lit by a large overhead light source like we have in our scene for more in-depth info on render passes check out my earlier tutorials finally since i have multiple render layers i'll use multi-pass exr and render the cg elements next up compositing for compositing i used after effects because i'm most comfortable with it for a project of this scale first i bring in the footage which is an exr sequence i exported from resolve i add this to a new composition and since my footage and cg elements are linear i need to create an adjustment layer which acts as a viewer lut on top of everything i apologize as i'm not going to explain the whole color management for this project but it's similar to my last video i will also release a longer breakdown focused on color management on my patreon next i'll bring in my cg elements after effects has a nice system where i can check create composition on import then this dialog box comes up and after effects can automatically generate some different compositions for me you'll see this contact sheet is a composition where after effects puts all the render passes in a collage which is nice to reference then in this other composition after effects has also taken our multi-layer exr sequence and duplicated it for as many render passes as we have then it's changed the name of each layer to the past name then for each pass it's applied this extractor effect which extracts a certain render pass from here i can copy these three render passes and paste them on top of my footage these are shushu his ambient occlusion and his glossy indirect reflection for shushu i applied a levels effect and usually adjusted his white levels to make him brighter then i can use the after effects info window to see the brightness percent of the darkest parts of my footage and i can then compare these to the darkest parts of shushu i'll then raise the input black levels on shushu to match [Music] i can also check the general saturation values in the footage and match those as well [Music] for the ambient occlusion shadows i set the layer to multiply and add a rough mask which i feathered out and tracked for the course of the shot you may need to deselect unmault for the feathered mask to appear correctly i then added a tint effect which allows me to sample and match the shadow tint in the footage [Music] finally i set the reflection layer to the add blend mode and adjusted the opacity to my liking a quick tip say i needed to re-render a shot i can simply bring in the new exr render select a certain layer in my composite and if i hold alt and drag the new render on top it replaces the old render but keeps the effects applied for a more in-depth look at how i created certain compositing effects like you see in this shot i've released an exclusive tutorial out on my patreon as always i used exrs for exporting the final composite out of after effects and for the final touch janet added some wonderful color grading in resolve ah voila [Music] it's really humbling and exciting to work on projects like this with great people like kyle and janet but even after the projects are over i feel blessed to have the opportunity to take a pause and share what i'm learning with all of you and so i want to thank the current patrons who really support my ability to to pause between projects and make these tutorials so if you'd like to support me similarly my patreon page will be linked below don't forget to check out the final shushu video and with that abianthu see you soon you

Channel: InLightVFX

Views: 149,159

Keywords:

Id: G4Azti4pAqU

Length: 14min 58sec (898 seconds)

Published: Thu May 06 2021