Character Lip Sync Tutorial with Nvidia Omniverse - Audio2Face

hey what's up creators it's mike from production crate and today i'm going to show you guys how to do uh this so what i'm using right now to make this face animation is a really cool program by nvidia called audio to face and what it does is it uses ai to generate automatic face animations based on just sound clips the best part about it is it's really simple really quick let me show you guys how to do it [Music] okay so let me show you guys how to get this set up first you're going to want to go download whatever character you want to work on i'm going to use this guy from our store and you also want to download nvidia's omniverse now when you get omniverse it's kind of a a suite you want to download audio to face which is inside of it and then you want to launch that so this is what you get when you open up omniverse and let's just take a quick tour and then we'll process the superhero character and import him into here so it comes with his default head and it's already set up you can just press play and it plays this sound clip the beige shoe on the waters of the lock impressed all including the french queen and what's awesome about that is as i mentioned it's all ai driven it's just driven by the sound there's no animator who made that motion happen let's export the superhero character and bring him into here so jump back into whichever program you're using and i really only need to export just the body or even if his body and his head are separated then just export the head just the part that's talking in this guy's case his head is connected to his body so i'm going to export everything i'm going to click on this body and i'm going to go file export selection or the equivalent in your program and i'm going to export a usd file i haven't tried this with other types of sort of universal cache files it might work with an obj it might work with an fbx i haven't tried it in the documentation they mentioned the usd so that's what i'm using so i'm going to navigate to my folder here and i'm going to make myself a little audio to face folder and i'll call this hero export all right let's hop back over into audio to face and in here i'm just going to go file open and let's navigate to that folder here it is i'm going to import that and i'm just going to say don't save when you import it it may be hard to see it might be that the material is dark or the lights aren't turned on but if you click in the middle of the scene you can see where your character is i can see that he's just either black or the lights are turned off let's try adding a light to the scene and see what happens so i'll go create light distant light and i can see something popping up he's just uh really dark so let's create a new material and apply it to him just some gray material will be fine i'm gonna go click on his body and then go create material omnipr now if it doesn't apply to his body it might be because this character has multiple materials he has different texture sets so he's got his head material and his body material and we can actually see that right here see these different sg's these are different shader groups so i'm going to just drag and drop the omni pbr onto my superhero and replace all four of these in this case this character has four so let's click on this one and just replace it and i'll keep dragging and dropping omni pvr on the him until i've replaced all of these male superhero bass materials there we go finally we just need the head now the way this program works is it has that pre-animated male head and what we do is we sort of are going to shrink wrap our character onto it that character is going to talk and it's going to drive our character like it's wearing a mask it's pretty cool so let's go ahead and bring that default character back into the scene and we're going to sort of map the different points on their faces so the computer knows which points go together so if i go over to the character transfer tab i'm going to add a mail template right here so in my case they popped up sort of inside of the superhero's head depending on where your original character was in the scene and how big it was these heads could pop up maybe way up here or down below and they could be any size really so what you want to do is go over into your outliner or your stage and you want to open up character transfer and select mark and mark open mouth and just kind of move them out like this now you don't have to resize them it won't change anything if they're a different size but i like to resize it so they're roughly the same size especially if they're way off let's go ahead and just uh make that roughly the same size and let's grab the green one which is mark open mouth and just put that over here on the other side okay so i'm going to jump over to the audio to face tab now so here in the character transfer tab we can see that the driver mesh is mark which is this one here that's the one that's going to be animated we also need a target mesh which is our character so i'm going to say target mesh is male superhero bass right here and then you can see the open mouth mesh it automatically picks the last one mark open mouth next step is to click add mode right here under mesh fitting and now we're going to do is go around and click common points between the two faces so the computer knows what's what because to the computer these are just a cloud of vertices it doesn't know that that's supposed to be a nose and that's an eye socket so we need to tell it start on this green one and i'm going to click just the tip of his nose right there and then click the same point on our model and i like to grab these little nooks and crannies on the corner of the nose right here there's no set points that you need to pick but i'm just kind of going by which parts i think are important so i'm grabbing the tip of his chin because it's sort of you know goes up and down when you talk now i have had trouble where the nostrils get messed up so i'm gonna make sure that i get the deep inside of the nostrils uh let's do right in between the eyebrows and then kind of the middle of each eyebrow is good now i'm gonna do the tear duct for each eye and the outer corner for each eye and let's actually do the center of the lower and upper eyelid for each eye too i'm also going to put one in front of the ear when i was watching the documentation for this program that's what a lot of people were doing so i'm just going to go ahead and trust them the other issue that i ran into is i did a creature it has a long jaw and the corner of the jaw wasn't moving right so i'm going to make sure i just add those points too it's probably unnecessary on a human but if your character is very unhuman you might want to select more points around the jaw so this is pretty good i'm going to hit done adding when i'm done putting points now if you messed up and you realize maybe one of them is a little bit off this is never set in stone you can delete points here or you can go to edit mode and just move them around and then i'll zoom in over here and i can drag these points and just get them a little closer make sure they match on the two different models and then hit done editing so the next step is to click begin mesh fitting now this might take a sec so make sure you like the video and subscribe to make it go faster alright guys so my processing finish i'm noticing a lot of issues here and i realized i think it's because i forgot to place points around the mouth notice a huge issue or something that goes wrong um just go ahead and hit delete setup right here and it will let you start over and i can go to add mode and start adding those points around the mouth that i totally forgot so let's just do that real quick just uh corners of the mouth upper lip lower lip and then the outside corner of the mouth here i did notice something wrong with the ears so i'm just gonna be safe and place some points on the ears too so again press done adding and then hit begin mesh fitting and let it update all right so the mesh fitting just finished it looks okay you might see some weirdness and what this is it's actually his entire body kind of squished up and stuck to his face that's just what happens when you have extra parts that aren't part of the head for now i'm just going to accept it and i'm going to click on begin post wrap this might take a second too so the reason that happens is because it doesn't really know what to do with all the body parts that aren't the face so it just kind of like scrunches them up and sticks them all over the place but you can see once i hit post wrap it looks fine cool so theoretically it's done and we can test it now so i'm going to click on this guy here and i'm going to go to the audio to face tab and i'm going to click on add a2f pipeline and i'm going to say yes attach and we should see them both kind of twitch so what's going to happen is it's animating this default head and because it knows which points on our model are associated and line up with the default head it's going to kind of drive our character at the same time as if he's wearing a mask so we can actually kind of just scrub through this audio and see that our character's face is moving and it's really cool just going to zoom out really quick to make sure nothing else on his body is moving yeah so it's not messing up his fingers or anything else so now we can start testing it and having some fun i'm going to go to the character transfer tab and i'm going to uncheck correspondence visibility just to get rid of those dots and then if we don't want to see the original mark heads we can click this character transfer folder and hit the little eyeball let's go back to the audio to face tab and we can press play and we can hear the audio and watch his face move the beige shoe on the waters of the lock impressed all including the french queen it's kind of a dumb sound clip um but you can import your own as long as it's a wave format i went online and i found a couple samples from video games let's just test out something a little more fun feast on their flash pretty cool right all right so i'm just going to go back to that default audio clip because it's super clean and i don't want to mess with any of like the background noises that are happening in my room so down here we can change a lot about how the face moves let's say you want it to be a lot more exaggerated you can see here we have the upper face strength and the lower face strength and we can exaggerate the face in different sections so i'm going to crank this up and if i press play the beige shoe on the waters of the lock and we're getting some strange results huh let's set that back to default so another thing we can do beyond just the strength of the lower face and the upper face is we can adjust the smoothing of the upper and lower face as well so if i crank up just the strength of this lower face and i press play do on the waters of the lock and you can see it might be a little bit twitchy depending on the audio that you have maybe you recorded your own audio and there's some background noise room noise and that can make the mouth just kind of like do this so you might want to crank up the lower face smoothing if you go too high his mouth won't move at all let's see what happens if i crank it up too high impressed all including the french queen before she heard that symphony again you can see it's just it doesn't have enough movement so it's a it's a balancing act now we can also change the expression and this is really cool so down here under the emotion there's the source shot and the source frame these are kind of just different expressions that the i guess an actor made but if i click on a different one like g1a so he looks a little bit angry and i can see source frame i can kind of roll it through different expressions for a different uh starting position if we start here where his mouth is closed then i press play he's going to say the same line but he'll look angry the beige shoe on the waters of the lock impressed all now if i scrub here to a frame where his mouth is open that's the starting position and his mouth will never close right so if you have a character who has big teeth like an orc or something you might want to do this because if the lips close it doesn't really make sense because he's got these tusks sticking out so it's just a way you can refine your character let's try one more i'm going to pick a random expression like it's kind of a sad one and i'm going to do one where his mouth is really pressed closed like that i know his co's mouth that looks all tight let's see how that changes the animation the beige hue on the waters of the lock and pressed all including the french queen pretty cool i'm gonna set everything back to default how do we get this data out of the program and into our 3d package well i'm going to click on the character's head and i'm going to go to data conversion tab up here and i'm going to say export filter let's switch that from visible meshes to just selected meshes i'm going to choose where i want to export it i'm going to place it back into that same folder where i previously exported that usd so i'll select that folder and where it says file name a2f cache i'm going to make it something more specific where it says playback fbs this isn't actually the speed that it plays at it increases the quality of your animation here it says export as usd cache that's kind of a more universal format but since i'm using maya i'm going to use a maya cache you can choose whichever one works best for you and i'll press export it's going to run through the animation really quick okay it's done let's go ahead and pop back into maya and here we are in the original scene with just the raw fbx how it comes down from production crate and if i look at it i can see that there's some joints inside he's rigged now i'm going to get the animation i just did attached to this rig and i'm going to do it in a way that's probably sort of proprietary to maya each program is going to have a similar concept for how this works but the specifics are going to be up to you guys to figure out so i'm just going to hide this cape because i don't need it and i'm going to go file import i'm going to grab that usd that i exported so here's the usd that i exported i'm going to re-import it back into the scene and here it is it doesn't need to be moved but i'm going to move it just so we can see it let's go ahead and make it just a default material for now now this is crucial if you're going to move it off to the side make sure you don't freeze transformations notice how over here it says translate it says uh minus 110 units i moved it over and i'm not going to zero it out i'm going to leave it so that that number is there because i'm going to use a blend shape or it's also called a pose morph in other programs and it needs to know how to get back to the original character so you don't want to zero out your transformations you want to leave it let's click on our geometry and i'm going to go up to cache geometry cache import let's navigate to our folder here's my geometry cache that i just exported if i zoom in on this character and press play his face should start moving very cool now you could apply it technically directly to the rig but i think that would cause problems because there's lots of other deformers on this model like these bones and everything right so it's best to keep it separated and then use this mesh as a blend shape or a pose morph so to do that in maya i'm going to click on this first and then i'll click on my character and i'll go to the form blend shape now it won't work right away if i press play you can see it's not changing his face so we need to go turn that on so i'm going to go up to windows animation editor shape editor and you'll notice that it's all here it's just not turned on so i'll turn that on if i press play it's still not quite working and this is just kind of a maya specific quirk but if i look i can see his mouth is kind of trying to move it's moving just a little bit and that's to do with something called the deformation order there's multiple deformers on this mesh it's the skeleton and the blend shape i just made and the order that they're stacked on the mesh matters so i'm going to right click on his head and go to inputs all inputs and here i can see blend shape is at the top and then skin cluster which is the skeleton and then tweak i'm going to use the middle mouse button to drag blend shape down below skin cluster and press close so now his mouth should move and you can also test to make sure that his head still moves and rotates make sure that these make sure that these joints still work now we do have a problem this arises because i'm new to this so i'm running into this problem here where his chin is moving but his teeth don't move and the reason for that is because of this point cache it's not moving up and down because of this skeleton right here you can see that this skeleton will also move the chin up and down now i don't want to manually do this and keyframe this every time his chin moves just so his teeth will move so i'm going to do kind of a little work around to animate a joint to make his teeth go up and down automatically again this is specific to maya but the concept could work in other programs so share your knowledge in the comments if you know a better way i don't want to actually move this joint because as you can see if i move it it'll sort of double rotate when his uh when the animation starts playing because his chin is moving downward because of the animation and it's moving downward because of the joints we're getting some double rotation we definitely don't want that so i'm going to create actually a second joint chain for his chin just right next to that original one and that's going to drive just the teeth so let's go into the side view here turn on my x-ray joints and i'm going to create a parallel joint chain just these two joints i'm going to imitate them so let's go up to skeleton create joint and you can place these joints right on top of the other joints but just to illustrate what's going on so you guys can see in the video i'm just going to place the joint like right next to it but it's probably better practice to actually place it right on top so now this is a joint chain that just kind of moves doesn't affect the model at all it's just there let's actually parent that to the original jaw rotate so what i've got here is this little jaw that doesn't do anything but it follows the original jaw like that and that looks like something's left behind but that's actually just the other copy over here you can see that when the original jaw moves the second one moves as well what i need to do is take away the influence of this original joint from the teeth so that they follow the new chain not the old one i'm gonna go in there and i'm gonna try to select his teeth and i'm gonna detach it from the skeleton again this is a maya specific solution but the concept should be the same in every other program that you guys are using so i've got his lower teeth selected and i'm gonna go skin unbind skin so now they're not bound to that original joint now let's bind it to the new one so i'll click on that new joint that one right there and i'm gonna click on my lower teeth and i'm going to go skin bind skin and i'll just use the default settings so let's go ahead and test it out i'm going to go to my x-ray mode and if i move that new joint right there the teeth should be attached to it if i go to x-ray mode you can see the lower teeth move but it's not affecting the skin which is good i was trying to separate the motion of the skin from the teeth so that this face animation we did can actually drive the teeth up and down okay next step is to create an ik handle from this joint to that one those of you familiar with rigs will know what that means but those of you not it's okay you can just follow the steps i'm gonna go skeleton create ik handle let's go to the options right here the little box just make sure that i'm using the single chain solver click on that little guy right there and then click on the end what this does for us is it creates this thing right here called an ik handle that allows us to grab it and move it around and you can see that joint is just going to stay pointed at it and crucially it doesn't stretch so if i pull the ik handle away the joint itself doesn't stretch out it just stays pointed at it you can see that the teeth because they're parented to that joint will follow so now what i'm going to do is just kind of glue that little ik handle to his chin so when his chin moves up and down his teeth will move up and down so i'm going to go to vertex mode i'm going to select a few points on his chin and then i'll shift select that ik handle so i have the verts on his chin selected first and then shift selected the ik handle and then i'm gonna go constrain point on poly let's go to the option box and just make sure we have the settings set right make sure that you have maintain offset check that's very important and i'll hit add so now if he talks and the chin moves up and down that ik handle is going to stay pointed at those verts and that'll make his teeth move up and down too let's test it out you can see it's working now a couple things that sometimes go wrong is sometimes the teeth when they're going up and down they twist this way which obviously is not right so again if you guys are more technical than i am with maya or any other program and you can think of a better solution definitely let us know i would love to see it lastly let's go ahead and get some sound in here i'm going to go back into omniverse and so you can see this is the folder where they where they keep their sounds so i'm gonna go navigate to that on my computer and here's all their sounds we can import that into maya so the way you do that in maya is you right click on the timeline go to audio import audio navigate to that folder here it is there's the sound on the timeline i'm gonna give myself more frames now when i press play we should see his mouth moving the beige shoe on the waters of the lock impressed all including the french queen before she heard that symphony again just as young arthur wanted so what's really cool about this is we can actually combine this technique with mixamo we can use these ai generated face animations and drop them right onto a mocap animation so i've got my guy up here in miximo and let's go ahead and just grab a talking animation so i'm going to click download uh with this one i don't need to download the scan i can just do the skeleton then we'll pop back over into maya and we should just be able to press import and grab that animated fbx and it should just work all right moment of truth the beige shoe on the waters of the lock impressed all including the french queen before she heard that symphony again just as young arthur wanted it's pretty good right he thinks this conversation he's having is so interesting now let me show you guys one more really awesome thing i just want to show you guys that this technique can work on non-human characters too with very different proportions from humans um here's a werewolf that we just uploaded to the store um go check it out on render crate and you can see it looks awesome when i press play their flesh nice let's go check that out fully rendered over in marmoset and you can see i've even paired it with some nice animations from mixamo so you can kind of make it look like he's maybe commanding his little underlings on their flesh so i'm really excited to have discovered this program and hopefully it gets you guys excited too if you make anything cool as always just share it with us put it in the comments go up on our discord whatever you do as always make it awesome [Music] [Applause] [Music] you

Channel: ProductionCrate

Views: 50,737

Keywords: aftereffects, compositor, visualeffects, adobeaftereffects, motiongraphicsartist, motiondesigne, postproduction, videoediting, videoeditor, filmmaking, filmmaker, production crate, productioncrate, footagecrate, footage, crate, Stock footage, free stock footage, adobe, After effects, adobe after effects, special effects, vfx, visual effects, fx, tutorial, tutorials, hitfilm, hit film, hit, film, Nvidia, Nvidia Omniverse, 3D animation, 3D, Audio2Face, Autodesk Maya, Maya, lip sync animation, 3D lip sync

Id: 071RiNWC6Bc

Length: 19min 47sec (1187 seconds)

Published: Sat Oct 30 2021