DeepMotion to Metahuman | Body, Finger & Face Animation Retarget | Unreal Engine 5 Tutorial

in this tutorial I'll show you how you can create animations with hand tracking and face tracking from video files through deep motion anime 3D and how to retarget them to metahumans in Unreal Engine 5. I want to thank deepmotion for reaching out to me for the second time and sponsoring the creation of this tutorial video their service anime 3D allows you to create animations from video files you found on the internet or captured with your own camera through their AI powered motion capture technology they have a generous free plan that gives you 30 animation credits each month let's create a new project select the third person template and make it a blueprint project input a name and click on create we'll start out by importing a meta human into the project click on window and then qixel Bridge click on the icon of a human on the left this will open the meta human presets you can also use a custom meta human character for this project but if you don't have much experience with meta humans I recommend using a preset first as a beginner you also shouldn't pick a character with Advanced rooms which can cause issues so I recommend you pick the same character as me to follow along for the first time I'll go with Ada whom I already downloaded the assets for if you haven't downloaded this character already you will most likely be asked to sign in so please go ahead and do that then click on download since these are really high quality assets the download might take a while once your meta human finishes downloading click on the add button your Unreal Engine project should now become the active window again and show you some messages about missing plugins which The Meta human depends upon just click on enable missing for all the prompts that show up you should then get a prompt asking you to restart the project click on restart now you might be asked to save some project files in that case click on Save selected once the project restarted open up the content drawer you should see a folder called metahumans open it up navigate to Ada and then find BP underscore Ada drag this blueprint somewhere interior level your PC might freeze for a few seconds and you might have to wait for shaders to compile our meta human is now ready so let's head over to the Deep motion website to generate some animations you will first need to create a free account to be able to access anime 3D after that head to the dashboard and click on create we want to create a 3D animation so leave this as the default scroll down to check out the capture guidelines whether you found your video clip on the Internet or captured yourself you want to make sure it's in line with these guidelines to get the best result for your animation you can also check out the official demo video for more detailed explanation of these guidelines for the first animation I want to use a clip of a person with a full body shown and moving since we want to use this for our body and finger animations once you've found a good clip simply drag it onto the panel and wait for it to be analyzed there are a few settings here which we want to take a look at fbx output is enabled by default and should stay that way root joint at origin is a setting they specifically added for Unreal Engine compatibility so let's enable it let's also enable hand tracking since we want the fingers to be animated for this animation physics filter can often help you avoid clipping and make the animations look more natural so I'll enable it for this clip and lastly I want to enable face tracking to show you how easy it is to get simple facial expressions working with your meta human however full body Clips only allow for simple face tracking and no eye movement this is something we'll tackle later with another clip click on Create and wait for the animation to be processed you can then check out the result here download the file to your PC right click on the archive and unzip it there should be two files but we only care about the one without typos in the name you might want to rename it to something that's easier to work with back in Unreal Engine create a new folder for your animation drag the animation file into the content drawer make sure that skeletal mesh and import mesh are checked the skeleton should be none since unreal will generate this for us also make sure that import animation is checked and also import bone tracks lastly open up the advanced Tab and check import morph targets you might get a few warnings about bow names clashing or smoothing grooves but these aren't relevant if you're using Android 5.0.3 or higher you might run into some issues here and get this error message in that case you can check out the separate video with an explanation of the bug and a workaround you should now have a few new files in your content drawer look for the animation sequence file in my case it's called walk and wave underscore anim ation is already working correctly with fingers and facial movement on the Deep motion mannequin the curves you can see below are the morph targets or blend spaces for facial animations if you aren't familiar with morph targets I'll leave a few links in description that explain the concept since we'll be working a lot with these during this tutorial we now want to retarget this animation to our meta human the first thing we need to do is create an ik rig acid for anime 3D skeleton an ik rig will allow us to perform post editing on a skeletal mesh right click in the content drawer hover over animation and click on ik rig then select our anime 3D skeletal mesh in my case it's called walk and wave let's name it ik underscore anime 3D double click to open it up the way we retarget animations now is quite different from Android engine 4. we now don't need to match a bone from one skeleton to another but rather match bone chains which we create for each major Loom let's start by setting the retarget route the retarget route generally needs to be in the pelvis area in our case that bone is called hips right click it and set retarget route let's add the first bone chain for our spine we can select multiple Bones by clicking on the first one holding shift and then clicking on the last one we want to select select spine spine one spine two and then right click then click on new retarget chain from selected bones and right spine as the name [Music] create another chain just for the neck by itself foreign just for the head let's move on to the left arm we don't want to include the left shoulder bone so ignore it select the left upper arm forearm and hand right click and create another chain name it arm left then do the same thing for the right arm select the right upper arm forearm and hand and create another chain name it arm right scroll down to the legs select the left up leg leg and foot and create a new chain call it leg left then select the right up leg leg and foot bone and create a new chain called leg right then lastly select the root and create a chain for that bone by itself now we want to double check that we haven't made any mistakes it can very easily happen that you mixed up a left bone with the right bone so make sure to check that your chains exactly match the ones you can see on my screen we then need to repeat the process for our meta human navigate to The Meta humans folder create an ik rig for The Meta human for the skeleton just type in the name of your metahume and you should have a choice between the face and the body in Ada's case the body is called what you can see on the screen right now this is short for female medium normal weight body now we want to do the same thing we did for our anime 3D skeleton and set up the same bone chains right click the pelvis and set it as the retarget route then select all the spine bones and create a new chain in this case there are five instead of the three bones our other character hat but that's not an issue create a new chain for the two neck bones thank you then a chain for the head by itself the arms are a bit more complicated here and have a few bones we don't need to select only select upper arm underscore L lower arm underscore L and hand underscore L you can add single items to your selection by holding Ctrl when clicking on the next bone scroll down until you find upper arm underscore R select it and lower arm underscore R and hand underscore R to create a new chain scroll down all the way to Thai underscore R also select calf underscore and foot underscore art as well to create the chain for leg right then select thigh underscore L calf underscore L and foot underscore L for the leg left bone chain lastly create a chain for the root by itself double check that you didn't make any mistakes and your setup looks the same as mine go back to the folder for your anime 3D animation and right click go to animation and select ik retargeter the asset to copy animations from should be ik underscore animate 3D or whatever you call the ik asset for the anime 3D character the target ik rig asset should be ik meta human which we just created for the skeleton preview search for f underscore matte underscore nrw underscore body there might be two assets with this name and we want the one with the gray body from the common folder we now need to match the poses of the skeletons click on new pose and call it t pose then click on edit pose look for the bone draw size and lower the value until it's easier to recognize the bones we then need to move the arm up but with the current snap settings we can't hit the exact value we want set the rotation snap size to 5 degrees now we can move it up by exactly 55 degrees and then do the same thing for the other arm move the camera to view the character from the top rotate the lower arm bones by 30 degrees to match the Deep motion mannequin double-click on the animation to play it and don't forget to exit the post editing mode let's export this animation to see it on the actual meta human click on export animation go to the metahumans folder and create a new folder for the retargeted animations foreign in the animation file you can see that the morph Target curves have also been saved in this asset properly look for BP underscore Ada in the world and select it in the details panel select the skeletal mesh of the body for the anim to play we can now select our retargeted animation to check it out on the meta human change the play mode to simulate so we can have full control of the camera you can see that the body animations have been successfully retargeted and the simple facial movement is also working however the fingers aren't moving yet so let's get into that go back into the ik asset of the anime 3D character for setting up the finger chains we can have a look at the ik asset for the third person mannequin and use it as a reference for the naming conventions let's do the left thumb first so select thumb 1 thumb 2 and thumb 3 and create the left thumb chain foreign index 2 and index 3 to create the left index chain select middle one middle two and middle three to create the left middle chain select ring 1 Ring 2 and ring 3 to create the left ring chain [Music] select pinky one pinky two and Pinky three to create the left pinky chain now let's do the same thing for the right hand select thumb 1 thumb 2 and thumb 3 to create the right thumb chain select index 1 index 2 and index 3 to create the right index chain select middle one middle two and middle 3 to create the right middle chain select ring 1 Ring 2 and ring 3 to create the right ring chain select pinky one pinky two and Pinky three to create the right pinky chain now we need to go through the same process on the metahume so open up ik underscore meta human you can see the finger bones here are a bit more complicated and also have parts we don't care about we always only want to get the three main bones for The Meta human we'll start out with the left pinky select pinky one pinky two and Pinky three and create the left pinky chain we can then collapse the pinky so we don't have to scroll as much select ring 1 Ring 2 and ring 3 to create the left ring chain select thumb 1 thumb 2 and thumb 3 and create the left thumb chain select middle one middle two and middle three and create the left middle chain select index one index 2 and index 3 and create the left index chain [Music] let's do the same thing for the right hand select pinky one pinky 2 and Pinky 3 and create the right pinky chain select ring 1 Ring 2 and ring 3 and create the right ring chain select thumb 1 thumb 2 and thumb 3 and create the right thumb chain select middle one middle two and middle three and create the right middle chain select index 1 index 2 and index 3 to create the right index chain let's go to our ik retarget asset if the animation is still playing save everything close it down and open it up again play the animation again and then clear the target ikaric acid to reset it and select ik underscore meta human again if you play the animation now you can see that the finger animations are already working to some extent however we can improve this a bit go back into edit mode and make the draw bone size even smaller just like we did for the body we now want to match the pose for the finger bones to the anime 3D mannequin for the fingers there isn't really a scientific way to go about it and you don't have to worry about setting the exact angles I'm using it doesn't have to be perfect and will work out even if it's close enough we mostly want to worry about straightening out the fingers after you're done matching the bones leave edit mode and look at the animation again if it doesn't look quite right you can go back into edit mode and make a few adjustments usually the thumbs are the hardest part to get just right so you might need to pay some extra attention to them now the tracking is looking great we now want to export this animation file with the fingers working sometimes I could complete the step without issues and sometimes I would get a crash with the assertion failed Outpost is normalized error this is an issue that can happen when setting the pose for fingers so if you keep on getting this crash you might want to create new posts and try again set up the rotation for the arms again first but this time for the fingers try to change as little as possible while still getting a good result and take things slowly [Music] you should then be able to export the animation rename it to make it clear that this is the one with fingers working [Music] now select the blueprint for Ada and the level again and set this animation to the body as the animation to play we now have an animation with all the major body parts the fingers and simple facial animations working however to get Advanced facial animations and eye movement working will have to create another animation from a clip that is better suited and use it on the face only go back to the anime 3D dashboard and create a new job you want to select a clip that has very clear facial movement and some eye movement we'll need to tweak some settings to get the best result for facial animations enable root joint at origin again for Android engine compatibility also enable face tracking enable upper body only and lastly adjust the eye tracking sensitivity this is a new setting that has recently been added which allows us to adjust the eye movement to our liking I'll select 1.0 for this clip again wait for the animation to be processed you can now see the result here download the file unpack the archive and open the folder again we only care about the file which doesn't have t pose in the name rename it to something that's easier to work with back in Unreal Engine create a new folder for this animation and drag it into the content drawer the import settings should be the same as last time with one exception for skeleton we want to reuse the skeleton from the walk and wave animation this way we don't have to create a new ik asset and make many things easier for us again you might encounter the fpx and Port Buck again here so follow the video in the description if you're having any issues after everything was imported open the animation file and you can see the facial movement and eye movement is being displayed correctly on the anime 3D mannequin we now want to retarget these to the face of our meta human open up the ik file for the anime 3D character again look for I underscore L and create a bone chain for Left Eye then select I underscore R and create a bone chain for right eye go to the metahuman folder and right click to create a new asset go to animations and ik rig select the face mesh of our meta human we now need to find the I bones of the meta human but it's really hard to locate them on a character with this many facial bones in this case it's easy to just go to the bottom right and add a new chain here call it left eye for start bone we can then start searching for L underscore I and select facial underscore L underscore I and then select the same bone for end bone then create a chain for right eye for Star bones search for facial underscore r underscore I and then again the same for end bone go into the folder of the walk in wave animation and create a new ik retargeter the asset to copy animations from should be ik underscore animate 3D or whatever you call the ik asset for the anime 3D character for the Target I curric acid select the ik rig of the meta human's face even though the facial animations and eye movement doesn't show up in this preview it's not an issue simply select the facial animation asset and click on export save it in the meta humans retargeted animations folder again even if you look at the animation here it's not playing on the meta human's face but you can see that the morph Target curves have been correctly brought over we actually depend on an animation blueprint to get this working go to the metahumans folder common and face open up face underscore nmbp then open up the atom graph this is where the magic happens The Arc and mapping pose is what enables a retargeted blend spaces to work and is the reason the animation won't show up properly by itself but works when we run it through the animation blueprint this node copies the animation from the body and applies it to the face and is the reason it was so easy for us to play The Walking animation with facial features enabled but now we want to play a separate animation on the face and don't need this anymore create a new state machine and connect it here instead add a state and called idle drag off from this node and search replay and the name of your retargeted facial animation when we compile and save we can see that the advanced facial animations are now playing on the meta human on the map we can also confirm this and we are now successfully playing one animation for the body and fingers and a separate animation on the face however the eye movement is still not working and the last thing we'll need to set up the eyes for The Meta human are actually a bit special and it seems like they're controlled by many different factors just to demonstrate this if you try to rotate the eye bone of a meta human it won't work out while you can freely move the eye of any other character I believe one of the reasons for this is the animation post processing going to The Meta humans Ada folder and open up the meta human blueprint select the skeletal mesh for the face when we check this able post process blueprint you can see that the eye movement is now being applied the angle is still a bit off but that is something we'll fix in a later step also the facial movement stops when we check this option this puts us in quite the conundrum because we need to disable post process for the eyes to work but we need to enable it for the facial animations to work now I tried many different ways to get around this issue but the only thing that ended up working is with haki and a better solution might become available as we get to know more about the meta humans over the following months and years on the face remove the check mark from disabled post processing again and make sure the facial animations are playing then duplicate the face skeletal mesh called eyes here you want to check disable post processing so that only the eye movement is playing we want to hide the parts we don't need on these meshes open the content drawer and create a new folder for materials right click to create a new material and call it m underscore hidden set blend mode to masked and shading model to unlit hold the one key and then click anywhere connect this node to opacity mask and Save we've now created an invisible material back in the meta human blueprint select the face skeletal mesh when scrolling down in the details panel you can see all the different materials the face is using element 3 and 4 are the material for the eyes make sure your M underscore hidden is selected in the content drawer then on element 3 and 4 click on the left Arrow icon which will apply the currently selected material from the content drawer then go to the skeleton mesh called eyes and do the exact opposite apply the hidden material to everything except the eyes which are element 3 and 4. now we can see the facial movement and eye movement working in unison when looking at the meta human in the level we can see that it also works together with the body animation there is one thing left we can do to improve the quality of the facial animations a little bit on the blueprint of ADA go to the body and clear the animation so we can get a better look at the face this animation is looking pretty clean but there might be a few artifacts or exaggerated movement depending on what kind of Animation you used open up the retargeted facial animation sequence for The Meta human click on one of the curves and press Ctrl a to select them all right click and edit selected curves here you can see all the keyframes for facial animation which we can clean up a little bit click in this panel and then again press Ctrl a to select all the keys right click and select filter we want to use the simplify filter the higher the value for tolerances the more keyframes will be deleted you can try some different values here but for my animation 0.025 was about right I was able to reduce the amount of keyframes from 15 000 to about 650. we now make the animation easier to work with and a bit more stable without losing any important information if your animation is still a bit jittery you can also select keyframes of unstable areas and apply simplify filter again with a higher value finding these areas might be easier when looking at a single curve instead of all of them if you're still not satisfied with the result you can also manually delete some keyframes that are causing jittering by selecting them and pressing delete check out the result every once in a while in the level and repeat this process until you're happy with the result lastly let's take care of the eye Rotation by adding an offset open up the animation blueprint for The Meta human's face again and go to the animation graph here go all the way to the right where you can see the output pose and the component to local node move both of these to the right to open up some space add another transform modify bow node here for the bone to modify first select facial underscore L underscore I for the left eye we only care about the rotation so scroll down rotation mode should be added to existing since we simply want to add a slight offset on top of our animation that is playing we want to change the rotation on the x-axis by about 12 degrees to bring the eyes down a bit in your case this number might be slightly different then compile and Save back in the map you can see that the angle is now fixed for the left eye only and also works properly together with the animation when it's playing go back into the animation group and for the face one last time duplicate this node and connect the pins with this new node selected go to Bone to modify this time select facial underscore r underscore I for the right eye double check that the rotation value is the same don't forget to compile on Save now you can see that both eyes are fixed apply the walking animation to the body again and then you can see how everything is working together in unison I hope you can now build something awesome with your newly acquired knowledge see you in the next video [Music]

Channel: Cobra Code

Views: 41,785

Keywords: game development, unreal engine, gamedev, ue4, ue5, game dev, Unreal Engine 5 Metahuman, Metahuman Retarget, Unreal Engine Metahuman Retarget, Deepmotion, deepmotion animate 3d, deepmotion unreal engine, animate 3d unreal engine, deepmotion animate 3d unreal engine, unreal engine mocap, unreal engine retarget, unreal engine mocap retarget, metahuman unreal engine 5, metahuman animations, metahuman motion capture, metahuman mocap, how to animate metahuman

Id: hKRcEbcwvX4

Length: 32min 8sec (1928 seconds)

Published: Wed Oct 26 2022