Metahuman Animator Part 3: Combining Mixamo and FBX body animations with Face Performance Capture

welcome to the third part of this tutorial about metahuman animator as a reminder we saw in the two first part how to set up Miller human animator performance and to fix floating head issues with a globally idle body in this third part we'll see how to combine moving body animation specifically coming from external fbx body animation files like Adobe mix ammo with facial performance captures from metahuman animator French version will be available soon if you feel more comfortable with LA version Francis Agri biento in a nutshell a bit of theory body animation fbx files may give move still or gently idle information they embed a different bone structure including a different root bone than unreal and metahuman skeletons that will have to be retargeted to our meta-human skeleton and most important they do also embed Head and Shoulders information that we will have to override or blend in order to get our final animation head performance capture on its side performs operations on head and shoulder mesh has to be connected to original dorsal and has no idea what the body is doing at the animated time we'll have to deal with these two structures in order to mix them and perform the animation the best way to have our acting complete get in mind that this tutorial is oriented towards movie creation and not gaming for it involves animation blueprint bypassing and meta human face blueprint modification that menstrual issues on any gaming project in ue5.2 to get started make sure you have all these assets ready you may refer to bus 1 and 2 from my previous tutorials I download The Meta human with an updated structure if your metal human model is based on the classic metahumans before July 2023 delete it updated to the new version and download it back a proper performance including capture data matching meta human identity and process performance with a companion name to deal with next getting 5 for Maximo this tutorial is based on mixamo animations so there's a couple to points to anticipate as told earlier an example based fbx animations files involve both a different bone erk and a different root motion structure if you don't know miximo it's a web app no property of adobe allowing you to visualize edit and download human actions you can browse on an online library with no charge and write free with the mixamo or adobe account created and once logged in you can choose a character here I got my passive Mark command and sign in animations in the animation tab I've selected a walk cycle I'll expand it a bit its range for the purpose of demonstrations otherwise animation would be too short once you're happy with your selection you can download it separately from your skeleton or gathered with the mesh file clicking with skin keyframe reduction is generally advised to be uniform so here we go with the 30 FPS space and download you can also for file size reasons get the mesh once and download multiple fbx animations without skin but for testing only this will do the job next re-dog it in to have our animation correctly set up on a meta-humans gluten we now have to retarget it to the machine structure there is no unique way of retargeting maximum animations so I'll show you a bench you can decide whether you choose first I found was mixmo converter by Terry Billy Studio which you may find by the YouTube channel or direct download on terribly studio.fr it gives you a folder structures with ins targets and exit file and you go through three steps with the matching Target rather recommended the other method is by instructor Advanced ball you may find it here again through its YouTube channel or at eventsbols.gumroad.com it is name your price based on a custom blender distribution batch converting files from in folder to out meshes form a part I selected a third method which is the unreal plugin Maximo animation retargeting too it's a paid one at 49 this is not sponsored content I choose it because it allows you to stay inside unreal to perform these conversions so I'm going to digress a bit on this plugin so about mixamo animation retargeting 2 plugin for convenience reasons I prefer importing a whole fbx package including mesh Red Skeleton and animation I've created a folder named underscore mix demo animations to keep it up in the Yorkie here I import my work cycle fbx files keeping all two defaults here you can choose your skeleton right click and select retarget Maximo skeleton asset after processing it gives you two blue retarget files double click on the one including your animation name here OTG for retarget I initially created the setup using a meta woman but anyway I kept the walk animation in the retargeter window we may offset a bit in the Target mesh to see it more clearly here at X Plus 120 and below in the asset browser double click our animation to check that the retargeting has been properly done then select just above export selected animation and store it with the good name for example in a processed or underscore RTG animes folder to find it quickly and our body animations is good to go then let's edit our face blueprint the solution coming from video maker eyebrows is the next step to follow to mix head with body information we'll have to tweak the global metahuman phase blueprint you can find in game slash metahuman slash common slash face double click to open and make sure you on the blueprint tab with the ER Key icon what we'll do here is to create a working method to mix face and shoulder information with body using a blend shaped node get the layer blend per bone node and on the right hand panel change the blend mode from Branch filter to blend mask then go in the skeleton window here on upper left side here archibano browse 4 head and select it click on the gear right above then in blend profiles add a new blend mask you'll name head mask make sure it has a value of 1 to give it power to control the upper body the same it would act in blending multiple animation timelines for the same objects back in the blueprint window in the blend mode blend mask options select head mask as your mask don't forget to save and compile it up before closing blueprint window otherwise changes won't be visible if everything is okay you'll see blueprint flow animating again and no exclamation mark annual good with your new face blueprint system then convert phase performance capture before adding body animations to your scene we'll have to export our performance capture in a very specific way I tried many walk arounds but this one at all times does the job very quickly what we are going to do here is to convert the performance animation to a new level sequence and grab the face control reanimation information then retarget it to the meta human generic phase and convert it to a new animation not straightforward and radio back but anyway working maybe here again a downside of the better aspect of meta human and middle open the recorded performance window you want to work with here on right hand side be sure that your head rotation calculations are based on Ctrl Ray and not on transform track this will ensure that all rotation baking infos will be processed to control Ray you may then export your performance but not as an animation as we did in the second part of the tutorial but as a level sequence you can add it in level sequence folder or any easy to find folder by default it will include a level sequence file a video plate of your performance capture and a mesh of your identity playing back the performance animation even the recorded audio playback if you want so double click to open your level sequence in the sequencer window by default your face control rate animation is targeted to you met a human identity so it does not fit with global meta human face to change this and we target to any available metahuman go to your face rig track add face control bold Ctrl Ray and in the right details panel edit the skeletal mesh asset by changing it to a generic meta human face by hovering it you can see its path showing slash game slash metahuman Etc change it and you'll see a new mesh in the level sequence editor matching one of your locally downloaded meta humans no and only now you can export your face control rig animation to a new animation sequence give it once more a proper name to avoid any ambiguity when looking for it here for example in a new underscore process performance animations I create my underscore anime phase process rotation phew now let's set up our scene to get a cinematic filling for what's coming next you may use the meta human lighting levels pack provided by epic and add it to your project it adds a handful of beautifully lighted levels including IBL for image based lighting retraced or standard shaded all three categories already set up around a human scale small environment and comfortable to work with quick results here I've chosen an IBL level down tonight I'll add my updated meta humans to the level and position it to zero zero zero in the global level sequences folder I create a new one named LS for level sequence 22 and M5 next we'll add tracking to our meta human animations in object to track add your metahuman by default it creates two rigs for body and face this is an important moment as you may or not choose to animate your character's body and face using multiple methods animation rig which is enabled by default when adding meta human tracking blueprint method rather programming oriented and more suitable to games or more complex workflows and the third the one we are working on using animation tracks by default with a human body and face both use Blueprints and will have to bypass or work around them in order to assign them or fbx stored animations on the body now you can add your selected retargeting animation click the right plus button here I'll add my body work animations if the movement is too quick or short you can still expand it a bit using right click properties and set its play rate to 0.5 or else now with all the previous face steps complete edited blend shapes in face and MVP and re-expot it and correctly retargeted base performance capture will just have to tweak two things before adding our animation with face selected first on our right hand details panel add a keyframe property to the animation mod then just below disable post process blueprint and keyframe this value to keep it recorded to your animation at this point these playbacks may appear closing sequencer and reopen animation may help or simply quit and restart your Unreal Engine session with these two option keyframes you may now track your animation on your meta human face and keep additional head rotational movements above body once on the face line use the plus button and select your backed animation to add it to be tracked depending on the length of on the two animation lines adjust the length or play rate of each one to get it matching to ensure that everything is okay you can scroll back and forth Your Face animation to a specific moment when rotation is more exaggerated and more visible here on my animation body and face playback rates don't match so the result is a bit strange I changed my walk animation to a capillary dance that is already looped using a faster playback speed here you can still see the head movements are effective and once again you can double check your face movement scrolling Your Face animation editing back face and in Blueprint as a last note you may realize at some point that additional neck and head movements and rotations are at best different from what you expected or Worse totally undesirable notable here just get back to your face and impropriate and switch back your layout blend purple node to the initial Branch filter setup with no need to delete your head blend mask save compile and hear you you can now get back to your sequence play it back with no additional head movements but keeping all expressions and face details final thoughts after getting all those steps done you may find that head rotation and movements added to body captures are not this straightforward and this is certainly because it's not intended for face performance capture added to body motion capture in a production workflow has to be strictly planned in order to get the expected result take for example games or virtual prediction and we can see there are a lot of things to be matching Body Face lighting camera setup and so on at this point adding some external and unplanned movement may give headaches in recalculating camera moves or focus and this is why it's safer to deal with strictly frame body animations as in fact real world directors do so the final result has no magic resign it will probably be a mix of motion animation files blended with some rig refining as much for the head done for the and so we end the third part about metahuman animator I hope that the whole process was clear and that it may fit in your future projects drop me a comment if you liked it and found it useful and share it around you because the more the merrier see you

Channel: SUPER3 Visual Studio

Views: 7,119

Keywords:

Id: h_oOZkI47io

Length: 15min 59sec (959 seconds)

Published: Tue Aug 08 2023