Face Mocap using ANY Camera

guys why wen't you listen to me I'm telling you the Pumpkin Man is real he's the one that's been putting candies in our hats and he's the one that put snowball in the microwave this is a short video to let you know that finally there is an easy way to create facial Moab data using any camera you don't need an iPhone and everything can be done for free and it works in blender the process in a nutshell you record your face performance footage you can use any phone or a webcam or whatever you have then you download a little free app and let it process your footage and create the mup data for you in blender you need a character with the 52 air kit shape keys and then you have to apply the mup data onto your character for that I use the paid face it add-on and I highly recommend it because it is an absolute Powerhouse when it comes to facial rigging and facial Mo up in blender but I also found a free addon which also works quite well so let's get into it first you're going to need the face Landmark link app you can get it from this GitHub page I'm going to share a link with you and then you just go to releases and download the zip file the application is Windows only as far as I understand then unzip it somewhere go to the unzip folder and find the exe file start it and the software will ask you a couple of simple questions do you want to track the head of the character so not just the facial expressions but also the head rotation I recommend typing yes to confirm and then do you want symmetric eye tracking so here I also recommend typing yes if you type no then both eyes in your footage will be tracked separately and because the tracking is not perfect you may get weird results symmetric tracking will give you better results then press enter and that will open this window to let you locate your footage find your footage and select it and press open and right away this will start tracking your footage and creating the mup data and the process is quite fast this footage was created by my friend darl he's an actor and Movie Maker and lately we've been working together so subscribe because we may be sharing cool stuff in the near future once the tracking is done go back to the folder where the footage was located and you'll find a CSV file in the same folder that is your facial mup data all right now we'll go to blender and I already have a character ready for use and here I'll be using face it and then in a minute I'm going to show you how to do it with the free add-on I have the character fully set up with face it so I have to go to the end panel face it go to the mockup section and now if you want to make use of the head tracking you go to setup head setup and set up your head rig and object so in my case this is my rig and if I go to pose mode this here is the head bone and I set it up like that in the head setup section then go to import recorded live link face click on load audio and find your audio now ideally your audio should be in wave or MP format or another audio format but if I remove filtering here I can actually just select my mov file the facial footage file and click load audio and that does work at least in my tests next click on load cssv go to the folder where the CSV was created and load Mo up next click on this import CSV file button and here the only thing that you really need to adjust is the frame rate and currently we only have 30 FPS and 60 FPS I think in the future face it will allow custom frame rates but for now I'll set it to 30 and the way I know that I have to set it to 30 is if I go to my footage and right click and choose properties details you'll see frame rate 30 FPS so now I can press okay and that's it the mup is applied so if I play the animation I should hear it and see it guys why wen't you listen to me I'm telling you the Pumpkin Man now this scene is very heavy so it doesn't play in real time so I'll have to render a play blast you can do things to speed it up such as switching to solid viewport shading and hiding unneeded objects so that may make it a little bit snappier but doing a quick eveve render is often the best approach okay now let's see what you can do if you don't have face it you can go to this gumro page again I'll share a link and download the blender Livelink phas add-on it is pay what you want so if you can afford it pay something for it but if you don't you can get it for free I already have it so I can download the atom click the download button give the add-on a five star rating okay so here I have another character uh and you may recognize it from my fa tutorial but that doesn't matter here all that matters is that I have these shape Keys the 52 a kit shape keys so for example I have a blink and and the ey moving in different directions as long as you have a character with these standard shape Keys then you can use this free atom so go to edit preferences addons click install find the addon that you just downloaded just select the zip file and click install addon enable it and it will be here in the end panel the addon is easy to use you have to let it know which objects you want to use because your character May consist of multiple objects that are animated by the same shape keys so in my case I have the main face mesh so I'm going to press the plus button and then I have the beard plus and eyebrows and I'll press the plus button but one more time next we have the Stream area and I'm not going to cover this because that will only work with the live link face app and you need an iPhone for that so it won't work with um face land marker link but what we can do is press the load CSV button and find our CSV data and load it and here is my character animated there are a few things that face it does and this add-on doesn't do for example it won't load your audio for you but what you can do is go to video sequencer find your footage and just drag and drop it into blender like this and align it with the start of the timeline and the bottom strip is the audio and the top one is the video so you can delete the top one guys why wen't you listen to me I'm telling you the Pumpkin Man is real but now you should be able to hear the audio and it should be in sync now unfortunately as you see only the character's face is animated the head rotation wasn't applied and that is just a limitation of this add-on it won't import the head rotation maybe they'll edit in the future but for now that is what we get and the final difference is that face it creates an intuitive face rig for you and the mup data is applied to the rig this add-on on the other hand directly animates the sh shapes here so you can see how when I scrub the timeline the shape values are changing now we have the facial animation on the character but usually you will need to do some cleanup and tweaking to make it look as good as possible often you'll see jittering in your animation and some poses may look off such as the mouth not opening or not closing enough this example by darel is actually a little bit unusual because it looks quite good out of the box it's probably because his performance is very fast and energetic so you do not notice the jittering but in slower animations you will see it so I'll show you some common techniques for tweaking the facial performance first I'll do it on the faet rig and then I'll show you what to do if you're using the free add-on so first if you see a lot of jittering in your facial animation select your face rig go to pose mode press a to select all controls and then switch a window to graph editor and here you'll see all of your curves which animate the bones and the bones in turn animate the shape Keys first press a to make sure that you have selected all curves and then I recommend that you press T and switch interpolation to linear if you keep the default basier interpolation you may notice the curves freak out sometimes when you work with them so switching to linear seems to fix that and since we have key frames on every frame the animation should remain the same okay so now if you notice jittering in your animation you can select all curves with a go to key smooth and the shortcut is alt s and choose smooth Goan instead of this you can just press alt s here in the graph editor and choose gojan and smooth a little bit do not apply too much smoothing because you may end up losing the Fidelity of your Mo up data some parts may be more jittery than others so you could for example select a single control such as the mouth and you can even select only specific parts of the curve and then again press alt s and Goan and move it exactly where you want and the other important technique that you need to know is tweaking a specific pose at a specific time so for example let's say that I want the mouth to be a little bit more open at this point so I'll go where the playhead is Select this key frame and enable proportional editing or you can press o in here so o will disable or enable proportional editing I want it on and then I'll select this key frame press G then Y and move it down because that will make the mouth open further and you see how the surrounding key frames are also affected and I can scroll my mouse wheel and make the effect stronger or weaker so I would want it to be something like this and pull down so that will make the mouth open wider and we'll have a smooth transition between the previous key frames and the one that I'm tweaking and I forgot to mention that I'm tweaking the Y location value in this fa it control rig generally the Y value is up and down and x value is left and right so for example if I select the x value here and tweak it you'll see the mouth control moving left and right if I want to open the I here I have to select the Y value which is in this case the only value that is animated and move it down so that makes tweaking this face rig quite intuitive you just grab a control and use the curves to move it in the direction that you want and here is another way to do your tweaks which I absolutely love that is actually my preferred method so for that I'm going to need dop sheet and switch it to action editor and I'll split another window and make it nonlinear animation enable selected only here and then I'll press the push down button and that makes my current facial animation into a nla strip now I can create a new animation and call it tweaks click in here in the nla on the active tweaks action and right away change blending to combine combine is a special mode in which inserting key frames is a little bit different than the default Behavior instead of key framing the absolute position of a Control Blender will take into consideration the existing animations in the nla so for example let's say that I want to make the character Smile around here so I'll go a little bit back select this key frame and key frame it now this will become a special key frame because it doesn't change anything in the current animation so I'll right click here and choose key frame type and change it to something that I can recognize and then go a little bit further and let's say that in here I want the smile to happen push the smile and key frame it and then grab this key frame and shift d and duplicate it on the other side and also duplicate this key frame so that the smile is kept for a while and so the result of this is that we have our regular animation and then in here we start to blend in a little smile and then we blend out of this edit this may not be extremely obvious so let's make it very obvious with the mouth open pose I'll create a key frame here give it another shape open the mouth record the key frame move forward for and close it a little bit and then I'll copy the first key frame on the other side which will basically blend out of this edit and here's the result the mouth is opening very wide and then going back to normal and here if I select this tweaks layer I can go in here and play with the influence and you'll see the difference so this is one of my favorite techniques for tweaking animations using the nla now let's see how we can tweak the on this character that we animated using the free add-on the techniques are actually quite similar but we won't be able to use the nla unfortunately blender cannot mix shape key animations in the nla so we can go to a graph editor and you'll see all of your shapes in here so you can select them select the first one go all the way down and shift select the last one and then you can press shift H in here to isolate only the only the shape key animations again same thing press T and choose linear and again if your animation is jittery press alt s Goan and smooth the curves a little bit but not too much there is no exact value for this smoothing operation you just have to play with it and you have to find a balance that cleans up the noise in the animation but it doesn't destroy the underlying Mo up data and tweaking pauses can also be done in a similar way except that you don't have controls that you can intuitively select so what you can do is for example if you want to work with the mouth you can type mouth here in the search field and that will show all of the shapes that have to do with the mouth so actually what I wanted is to open the mouth and that is not a mouth shape it is the jaw shape we have Joe open so I'm going to select it and press shift H here and let's say that in in here I would like the mouth to open so it is very similar again enable proportional editing select your key frame press G and Y and proportional editing wasn't working for a second but now it does uh then move the frame up or down to either open or close the jaw and you can scroll the mouse wheel up or down to increase or decrease the effect or I guess you could say the ease in and ease out of this edited pose okay so here we have this edit if you want to add a smile you can grab the smile left let's say actually that is smile smile right so let's press shift H to isolate smile left and again move it in here right and then we have the character kind of smiling so this is how you can edit the shape case using curves it's a little bit less intuitive than the control rig but once you know your shapes it is is simple enough so this new free application should make anyone who doesn't have an iPhone happy but that is not its only benefit the main motivation for the creator of this app was actually that I wanted a lightweight helmet for capturing facial mup mounting any sort of phone in front of your face is not pleasant so they created this super light setup and shared a photo on our Discord did you know that we have Discord this app opens up a lot of interesting possibilities so enjoy and before you go if you would like to take a look at my courses or even get exclusive access to unreleased videos check out academy. cd.com I think you'll like it

Channel: CGDive (Blender Rigging Tuts)

Views: 26,609

Keywords: blender3d, b3d, rigging

Id: 1Ygq9052RSk

Length: 17min 45sec (1065 seconds)

Published: Thu Nov 09 2023