Mixamo + FaceIt: EASY Full Character Rigging & Mocap

I know that many of you want to use blender to create animations and tell stories and you just don't have the time to become a rigging expert if that's the case then this video is for you I'll be showing you the easiest way to rig a full humanoid character including a nice body and facial rig you will have a lot of flexibility with it you'll be able to animate it manually or use body and facial mup we'll be using blender mix and and two add-ons the mixamo add-on for blender and the faceit add-on faceit is a paid product and I understand that can be a deal breaker for some of you but I'm trying to outline a workflow that requires as little technical skill as possible and currently face it is the way to go of course everything can be done for free I even have a tutorial about combining mix and the rigify face rig check it out if that sounds more to your liking you can now get access to all CG dive tutorials including my extensive paid courses unreleased videos and supporter only content for just $5.99 a month check out academy. CG di.com slss subscription for this process you'll need to have a humanoid character here is mine and I split it into a body mesh face mesh and if I hide the head um I have the eyes and teeth and tongue as a same separate meshes this is not absolutely necessary but it will help me a little bit in the face riging process but you can also have the whole character as a single mesh the first step is to rig this guy in miimo so select all of your character meshes and go to file export fbx and then you can just give it a name and Export it with the default options but in case there are other objects in your scene you can enable limit to selected objects and that will only export the character meses that we selected then click export fbx next in your browser go to mixim mo.com and you may have to sign up and log in once you're logged in click on upload character you can simply drag and drop your fbx file into this field give it a second click next set the characters chin wrist elbow knee and growing position and click next you may have to wait for a minute or two but it's worth it if this animation preview is looking good then you can click next and your character will be finalized now without applying any animations click download choose original post fbx and download it back in blender move the original character to the side go to file import fbx select the fpx that you downloaded from mixamo and import it with the default settings so now if I select the amateur and go to pose mode my character is skinned to this miimo amateure however the materials were lost in the process that is not a problem select any of the mhes go to the material Tab and you'll see that all of the materials on the rigged mesh have the 00 01 suffix all you have to do is replace them with the material without a suffix which will be the original material before exporting to mixam and I just have to do the same thing for the remaining meshes now my rigged character has the correct materials so I no longer need these initial meshes I can just select them and delete them the default mixim rig is a little bit too simple and not animation friendly for my taste and that is why I recommend that you use the free miimo it on for blender you can get it from here I'll share the link with you and you just have to click on get the add-on and that will download a zip file then in blender go to edit preferences add-ons click install and find the ZIP file that you downloaded select it and press install add-on and enable the Aron and it will be over here in the mixim tab in the side panel just in case this area is contracted you can expand it like this or with the N key with the mix rig selected press the create control rig button and press okay now if you go to pose mode you'll have a rig that is much more animation friendly you have ik arms and what is more important is that the legs are in ik mode so you can lower the pelvis for example and the character will Squat and the feet will stay firmly on the ground if you select the control you may get some options so for example I can switch this arm to FK mode which allows me to animate it easily by rotating the arms for the arms ik or FK is mostly a matter of taste but the legs should almost always stay in ik mode now really quickly if you want to assign mixim animations to this character go to animations choose an animation click download download it without skin that will make things a little bit simpler in blender go to object mode then go to file import fpx and download the new animation I can move it to the side then use this speaker to select this new amateur in this field and then select my mixim control rig and press apply animation to control rig and now we have our character performing that action and you can apply any Maximo action this way I'm going to save my file at this point and we are going to proceed with the face rigging portion of this video video so you're going to need the face it add-on again I'm going to share a link with you the version that I use in recommend is this one the unlimited Expressions version so download the add-on and then we'll install it just the same as the miimo add-on edit preferences add-ons install select the zip file that you downloaded and press install addon activate it and that's it now I'm going to zero out this rig that will put it in the default position and it will remove the animation I have separate in-depth videos about face it but my goal here is to make this video self-contained so that you don't have to jump around and watch other videos to start the face it process select all of your characters face mes I'm going to deselect the body I don't need it and then Press Register face objects here I realize that my objects also have the 001 suffix so I'm going to select everything go to edit patch rename and here type 001 and leave the second field empty and press okay then delete these entries and register my face objects again then we need to assign these vertex groups the most important one is the main one and the easiest way to assign it is to use the Picker and point to the face of the character and click that's all now I have the left eye selected and I can just press the left eye button and that will assign the vertex group then the right eye and press right eyeball that is why I separated all face meshes in advance now if everything was a single mesh then you could just go to edit mode select all vertices that you want to assign and again press this button that will do the same but only on the selected vertices okay I'm going to hide the head for a second select the upper tee and press the upper teeth button select the lower teeth press the lower teeth button select the tongue and press the tongue button now we also have eyelashes so here I can actually assign some vertices in edit mode I'll go to edit mode hover over the eyelashes and press L and also on the other side oops I don't want to select the face just the eyelashes and then I'll press the eyelashes button if you had facial hair you should Mark it as facial hair rigid objects such as a hat or glasses should be marked as rigid but that is more or less the whole process I can then move to rig and press generate landmarks align it with the chin scale it up and then scale it sideways next I'll be able to move these vertices one by one so I align these vertices to the eyebrow these wordss around the eye these ones between eye and eyebrow these are the nose vertices now the mouth and I'm going to keep these mouth vertices in the middle of the mouth and not at the outline this is a chin next click project landmarks and you may want to align some vertices from the side view this one is very important this will become the pivot point of the jaw you can also go to 3D View and keep aligning once it is looking good you can disable snap and do something special select this side vertex on the mouth corner and with the move tool move it a little bit below the skin of the character in these vertices I'm also going to move a little bit inside the lips and I'll do something similar with the eyes next press generate face rig and then bind and okay now if you go to pose mode and rotate this widget for example you'll see that the character is opening his mouth and all of these other widgets will also deform the character now if you like this you can keep this as your final animation face rig but that is not the goal of this atom next we go to expressions and load fa it expressions and press press okay now if you click on any of these Expressions you'll see the character performing a specific action for example this one is I blink left so the left eye is blinking and you have to go through all of these 52 or so shapes and check if they're looking good now how do you know if they look good here is a reference that you can use for this process it contains all of these shapes the exact same shapes for example here we have I blink left and here we also have I blink left so you have to make your character look more or less like on this reference image and what is cool about fa it is that it does about 90% of the job for you you may find that most of the poses look great already but if they need some tweaking here is what you can do first of all once you select this pose you can move any of these widgets and adjust the pose you can click this button to amplify the pose or decrease its effect you can also go to object mode select any of the mes and press this sculpt icon and that will allow you to go to sculpt mode and make any changes that you like to dispose and that will become your final pose a lot of poses are symmetrical so you can just press the Symmetry button here and that will mirror these changes on the other side so face it will do most of the work for you but you still have to go through this poses and make sure that they look good one that may confuse you and it's always a little bit off is the tongue outp pose make it look something like this the tongue needs to be sticking out through the closed lips it's a bit weird but that's how it has to be another one is mouth close it looks something like this and yes it is weird but that is more or less how it should look so here I'm going to select the object and sculpt it a little bit this looks a little bit extreme but yeah it should look as if the jaw is open but the mouth is closed once you're done with these shapes you can go to bake and click bake shape keys and press okay then in shapes you can just check your shapes and if they don't look good for any reason you can go back and back to rigging and you can tweak your poses and bake them again but I'm happy with my poses so I'll go to control and here I'm going to generate a control rig for the face now this automatically created this face rig for us if I go to pae mode you'll see that this widget here opens the mouth this one moves the eyes this one is smile and frown these are lips controls this is the whole mouth control open and close prow controls sneer the controls are quite self-explanatory now one problem currently is that if I go to my main rig pose mode and move the head around the face is not moving with the head so keep this headbone selected go to control rig setup in utilities experimental and press the set child of button and that will parent the face rig to the selected head rig now the face rig moves with the head so we are basically done with the rigging part I'm going to save my file at this point okay with all of this setup we can move to the mockup section next and and animate the face of this character using facial mockup data of course you can go to your face rig POS mode and animate these controls to create manual facial animation which is a perfectly viable option but this video is meant for people who are not very technically skilled so I believe that facial motion capture can help you a lot in this section first go to setup head setup and here we want to set up the headbone of our rig you can skip this step which means that when you import facial animation only the face will be animated but the head will be completely static and if that is what you want then please skip the head setup but the head animation adds a lot of life to your character first I want to choose my Armature and let's rename it actually to something that we can recognize such as mixamo rig and the other Armature is a mixamo mo up I'm going to delete it right um so now here I can choose my mix rig and then I want to choose the headbone so start typing head and for this rig we want control head head action you can disable here it will be imported later and then we can start importing facial motion capture so close setup and expand import Face it allows us to import facial motion capture in a couple of different ways it can import data from face cap which is an iPhone app Livelink face which is another iPhone app also audio to face which is freely available technology from Nvidia it can be very powerful but not very easy to set up so it definitely deserves its own video that is something that I may cover in the future so make sure you subscribe with the all option if you want to be notified about it and yet another way is life facial motion capture so you can literally look at the camera of your phone and your character will be performing and seeing the words that you say I only have limited experience with life capture so today we'll focus on face cap and live link face in the next minute or so I'll give you a very quick overview of using face capap and live link face and I'll even give you a possible alternative if you don't have an iPhone I recommend using live link face because it is free here is how to use it start the app choose live link air kit and continue tap the settings icon and note the FPS I set mine to 30 FPS next point the camera at yourself press the record button record your performance press record again to finish the recording tap the movie Clapper icon tap on your recording finally use the share icon and choose a way to send your data to your PC Google Drive works well for me face cap is a paid app and it does more or less the same as Livelink face but if you're using face cap here is what to do start the app and point the front camera at yourself press record record your performance press record again and finish the recording tap export which sends the face c cap data to your iPhone files so go to your iPhone files folder find the face c cap folder finally use the iPhone share function to send the data to your computer if you don't have an iPhone you can use an Android app called meow and a PC app called Mo using these two I've been able to capture a facial performance but it is a lot less reliable than the iPhone and the quality is lower in the future I'll try to figure out the best approach for non iPhone users and share it with you so stay subscribed now if you're using face cap you can expand this field and click on load audio here I have my face c cap data I'm going to select the wave file and then load the text file and then simply click import and these settings should be fine and I'll press okay now if I press play I should see and hear the voice and the facial performance that I recorded using face cap hi this is a test recording for CG Dives mixamo and face it tutorial okay I'm going to undo now and let's try live link face as well again load audio go to my live Livelink face files Livelink face by default creates an mov file rather than an audio file so if you want to be on a safe side you should convert this mov file to MP3 or wave and you can just go to Google and type convert mov to MP3 and you'll find free services that will do that for you but we could also just disable this filter here and I'll choose the mov file and I found out that it works next I'll load the CSV file and you may have a bunch of them but the ones you can actually use are the row one or the c one I believe Cal stays for calibrated so that is probably the best one that you can use so load it and then import CSV and in these options you should definitely check if your frame rate is set correctly mine was set to 30 so I'll set it to 30 and press okay and now I can press play again and see and hear my character hi this is a test recording for CG Dives mixamo and face it tutorial in this this case I find the Livelink face result a little bit better than face cap but under the hood it is the same technology so you should get similar results something that may bother you is the tilt of the head of the character that probably means that my uh phone was tilted when I was capturing my face but that can be fixed very easily go to your mix rig go to pose mode switch this window to graph editor make sure that this button here is enabled and then you can go to item and play with your rotation axis and see which one will raise the head and that is the x-axis so I'm going to hide the Y and Zed zoom out select this whole curve and then press G and Y and in this case I have to move it down and that will raise the head of my character and if you need to you can play with the other axis most likely the Zed axis in the same way just press press G and Y and see what looks good but now if I play the animation hi this is a test recording the head of the character will be exactly as I want it something else that is important to understand is where the audio is coming from and if you switch this window to video sequencer you will see this audio strip and this is important because you may want to change the timing of your face animation let's say that I want the Face animation to start exactly where my play head is now I can press G here and move this strip and position it exactly at the play head then you can go to your timeline select your head select all key frames and then press G and also move them exactly at the play head and I also need to go to the face rig go to pose mode select all key frames and move them so I have to undo make sure that everything is selected and then move the key frames now we'll have nothing at first and then exactly from here the animation should start hi this is a test recording for CG and the audio is also in sync okay now I want to undo these changes so I'll press control z a couple of times let's also switch to video sequencer and press contrl Zed until I get the audio at the beginning and now let's try to put everything together we want to have some body animation and facial animation we also want to have control over the timing of the animation and even the ability to tweak the animation or make our own animations so I want to go to mixamo and download some more animations but as you can see while I was recording I switch the character and that is actually okay a lot of the mixamo characters are compatible able with custom characters for example this ninja character is safe to use so I could apply an animation to this character here download it without skin and then bring it into blender and then with the mix suo tools I can apply this animation to my custom character and that is okay but if you want to be completely safe you can select your rig and all of your character meses so we don't want the face rig I'm going to hide it for a second and then select the rig and all meses uh then go to file export fpx enable limit to selected objects and on the Armature untick add Leaf bones and tick only the form bones and save your file then go to mix choose upload character grab the fpx file you exported and just drop it into the interface and because this character has a compatible amateur we don't need to rig it again it will be rigged already click next next and then you can apply animations to your custom character and this will ensure the best compatibility between the MX animations and your MX rig in blender and I'm going to want a walking animation and to make things simpler enable the in place setting press download choose without skin and download it another animation that I want is a wave let's say this one for example and I'll search for talk or we can also choose an idle animation let's go with this one and download it now in blender I can go go to file import fpx and here are the three animations that I wanted talking waving and walking just control click on all of them and press import fpx you can move them to the side next we're going to combine these animations that we downloaded from ex and for that I highly recommend that you use the nonlinear animation editor in blender or nla for short so I can switch this window to to nonlinear animation this editor can be intimidating at first but it is extremely powerful in this video I'll try to show you some Basics but as much as I like to show everything in a single video it's just not possible so I have separate videos specifically about the nla and they will help you use the nla properly I also just released another video called mixamo animation secrets in which I show all sorts of cool practical techniques for combining and tweaking some animations the video is also aimed at non-technical people check it out I think you'll like it in the nla first enable the selection toggle so that we only see information related to the selected object and that cleans up things quite a bit next split another window and change it to dope sheet and this here to action editor so to work with the nla you also have to understand blender's actions each animation in blender is stored as an action and the action is then applied to your Armature now I have an action already applied that is the head animation which face it created for me so let's click here and name this head action that is going to be useful and then I'm going to press the x button and that will unlink this animation meaning that it is no longer active although if we click here you will find the head action still in the blend file let's select it again actually and here's this Shield icon in this case it is enabled and you should always make sure that this Shield is enabled otherwise you may lose your animation next I'm going to apply these mix animations to the rig so let's select the first one and apply it okay this is the talking animation enable The Shield then select the next Source Armature apply the animation that is the walk so name it walk and enable shield and then the wave apply it rename it and click the shield now I don't need these armateurs anymore I can delete them and now we can start working with the nla let's switch to the walking animation and press this push down button and here it seemed like nothing happened that is because I hadn't select the amateur okay now now if I click you'll see the the walk animation here in the nla so let's undo select the iure and then press push down and you'll see how this action drops from the active action into the nla so now we have the same working animation but it is in the nla instead of being an active animation and once in the nla we can do cool stuff with it for example if we scroll down here and go to action clip expanded and increase the repeat value and play the animation hi this is a test recording for C Dives you'll see that the walking animation is looping hi this is a this and let me switch to the video sequencer and disable this track for a second we'll enable it later but I don't want to hear the voice now all right now we have our walking animation and it Loops we can also change the playback scale which will make the animation play faster or slower here I made it slower I can make it faster next I'm going to enable the wave animation and push it down as well and let's play and see what happens it seems that we just have the wave animation so what we could do is grab this strip and move it over to the end of my walk cycle and now you may be surprised that the walking is not visible but if you just select the wave and go to extrapolation and choose nothing instead of hold that will allow the walking animation to show through and then we're going to switch to the wave now the switch is very abrupt so you can select your wave and give it a little bit of blend in and now you'll see how we gradually switch from one animation to the other so you may need more overlap you have to play with it so that is one way to use the nla another one I'm going to remove the blend in and move this wave again so that it overlaps the walk now in the nla with the wave selected I'm going to press Tab and that will allow me to temporarily edit this animation then go to pose mode and let's select all controls except for the arm here so all of this selected and expand the summary and remove the check mark next to these controls and the result is that now we have a walk and the wave is mixed with the walk so that is another powerful nla technique next let's enable the talk animation push it down and move it so that it overlaps with the end of the walk and wave again the talk animation is overpowering what we created previously because it is below it and because we have extrapolation to hold so we can switch it to nothing and we can also give it a blend in so now we have walking and waving and then the character stops and does a talking animation if you want to have a longer animation you can switch one Editor to timeline just for a second and expand it here and a shortcut place the play head at the end of your animation and press control and end okay so now we have the walking and waving and then the character stops and talks now now I want to tweak the facial animation so that it overlaps exactly with the talking animation so I'm going to unhide my face rig and this is my facial animation and because this is a single animation I don't need to use the nla but let's just do it for practice I'm going to push down this animation and then shift select the rig and that will allow me to see both animations so I'm going to overlap this with the talk animation and for the talk I want to either use the scale to make it slower or I may try to repeat and then set my end of the animation to around here now we also had the head animation for the main rig so let's select the head action push it down shift select the face rig so that I can see U the face sh action and then I'll move the head action so that it overlaps exactly with the Face animation and that way my facial animation and the head animation that I imported will be synced and finally go to your video sequencer I'm going to enable the audio channel here and press G and also move it exactly at the play head right because if I go back to the non lar editor I placed the play head exactly at the beginning of my facial animation and head action so now facial animation head action and audio should be in perfect sync okay and let's see what we have now walking waving hi this is a test recording for CG Dives mixamo and face it tutorial and everything seems to have been mixed together nicely and again the nla can be complex it does have a lot of gas so again please watch the nla videos to learn everything properly and then the mix s animation Secrets video for a bunch of cool and practical techniques

Channel: CGDive (Blender Rigging Tuts)

Views: 10,359

Keywords: blender3d, b3d, rigging

Id: GvNmk7RteiU

Length: 37min 12sec (2232 seconds)

Published: Thu Oct 26 2023