SRanipal Face Tracking Blender Tutorial

okay first we're going to do is create uh masks and we're going to do is create a new vertex group left head and what we're going to do is we're going to go into edit mode and what we're going to do is we're going to do is create a mask to do left of the head so control z does transparency and i want to be dead center okay and i just click and drag and i'm gonna assign that control z to that again um i'm gonna do ctrl shift m mirrors it over we have some stuff that's not mirrored note this that you can't this can cause problems later but this looks pretty good except for the eyes it's gonna cause problems later but that's okay we can work with this i just need to shift and grab all the extra ones and we're going to have right head and what we're going to use masks i'll just call it masks so we can keep it separate these are going to be used later to separate left and right side of when we're creating the shape keys for face tracking okay so double check how it works and that works okay now what i'm going to do is we're going to do a select i'm going to do control minus minus we're going to create a new group called left head smooth mask and what we're going to do is we're going to assign it a weight of 1 and do control plus disselect so it just selects all the ones one extra i'm going to change this weight to 0.75 now we're going to assign 0.75 on those vertexes i'm going to select go plus again to select so it should be in the center so we're going to do set at 50 and this will help create a perfect uh cut in the middle of the face when we're trying to separate left and right and not any harsh transitions so what we're going to do is then assign that control plus again and disselect and .25 assign so what this is going to look if you actually go into weight paint mode you will have this gradient here and you want it to be green on the center but we're doing it manually so we actually get it perfectly exact on the mirror so that's done now we're going to do right and smooth mask and we're going to do a to deselect everything i'm going to select the right and set the value of 1. control minus minus go to right assign make sure you click this before we assign and then we're going to go plus one to select some five assign plus one just select five that's a sign plus one to select and what we're doing is we're adding the next vertex out and we're gonna add assign that to the mask so ctrl a we go into weight paint mode it's nice and smooth so when we do any face transition stuff we can actually separate out left and right so this is without the gradient this is with the gradient now the next step is to [Music] create a upper lip mask and what this is going to do is we're going to basically select anything from this point up so we're going to do is i'm going to select things that i want to move on the upper lip with face tracking so if we noted earlier it does have it mirrored so we can mirror this over if we do select things this could save us some time it's very nice when you actually have something that's actually mirrored now i can't get in here so what i'm going to do is i'm in edit mode i'm going to turn on the mixing and i'm going to do ah that should open the mouth so i can get into the mouth to select things okay i'm gonna go in here and what we're trying to do is we're trying to select what we want to move the lips when we're in sculpt mode sculpt mode is what we're going to be using to create the actual face shapes and now preferred is to actually have a bone rigging for all these mesh in the like different lips and but usually that's not usually a lot of avatars don't have it in this case we don't so i am having fun trying to get up here so i want to get about uh okay so it looks like the lips go up pretty high so we're gonna try to get in there now these lips are very tight with these gums so this is gonna be a little challenging so i'm gonna try to work with around this good control z that a little bit and and thankfully this is mirrored so if we do do this i can mirror this over now this is a little difficult to actually see what you're doing but what will do [Music] okay i think i select one of these that i didn't want to select ah it was this one don't want that one and that is i believe that's the gum okay so i have that side selected so if i do control shift m and i go down to this bottom here and i want to hit extend so i want to extend the mirror selection and that looks like that should be what we want that that looks pretty good right there okay i'm going to select a little bit more on the upper nose just fold this nose i think i actually want to move the nose a little bit with the face tracking so i'm going to move this i have that selected a little bit so i have for masking so let me go over here okay over here now usually with face tracking you want to go move a lot of things so we want to go a little bit farther out here probably around the lower of the eyes so we're going to move it over here just select okay okay oops i selected that one down there i don't want that one just like that and yeah we're gonna go over here i think right here like the corner of the eye bottom corner eye i think that's going to be a good spot to have it cut off at okay now i think i'm going to do that ctrl shift m to add it to the other side and that's all mirrored now if you don't have a mirrored mesh you're sore out of luck you're just gonna have to visually look at it to see if you're mirrored so that looks good i like that so i'm going to assign that i'm going to do alt a i'm not trying to get rid of this gum i don't really want these gums here so i'm gonna try to get these away i'm gonna just hide these bones for a little bit so i can actually work a little bit so i'm gonna do that l and we're going to call this teeth and gums and it's just nice to have a group so i can i can mask things out so i'm going to call it teeth in okay and we're going to just hit h we're going to hide it so we can actually work a little bit better we have the tongue and we're going to just suck the tongue and i'm going to hide that so actually i should have done that first see after i got those out i have i can actually see the back of the top of the upper lip better and i think that'll work i think that's pretty good if if it doesn't move as far as you would like you can add an i think i can add another one here uh you know what let's add another one so we're gonna add another this is a little tedious at first but it'll be very useful later assign okay now we want the do the lower lip lower lip mask and it doesn't matter how this is named or just name it so we can use it later for the process i'll just do ctrl shift plus and [Music] that good good chunk of it and we want this crease right here so we want this corner i want that corner and we're going to go about there and we would well sort of keep that jaw line so we don't really want to move that jawline too much and you're not always stuck in with these you can always add it later if something doesn't work and what we're doing so we can mask this okay and i could have mirrored that right so let's just add that there now these hairs are in the way probably could hide those but i'm just gonna suck this one down here let's see and there okay so we got the lower jawline so we're gonna mirror that over so we got the bottom lower jawline so we're gonna assign that okay now let's go on to weight paint mode and we should be able to see upper lip lower left tongue gums so what we're going to do is i'm going to just scoot this down so we can actually keep it organized so i want to make this smooth so i'm going to do the exact same thing we did with the head so i'm going to call this upper smooth and what we're going to do is we're going to do alt and select the upper lip select it minus minus so we're creating the transition zone so do is assign and we're going to go plus to select 75 assign plus just select and assign plus to select and 25. now you can this is probably how much transitions you want could mess with it later if some things don't work so that way paint mode and we have a nice transition that we manually put in a now we're going to do the lower lip smooth mask now looks like the lower lip minus minus one a sine plus to select oops yep i didn't i didn't assign it to the right layer so i'm going to go back to edit mode go to full minus minus the sine should be that okay and then 175 plus disselect assign plus to select assign and plus to select and assign so we should [Music] have a nice smooth transition okay now i have i think one more that i need to do and this would be for the eyes so i want to make a mask for the eyes so we're undo do i is mask change the one and what we're gonna do is we're gonna select around the eye so we want this and this i'm gonna assign that to the mask and i'm gonna hide it just so we can get two things we're gonna have a lot of layers here that we're gonna have to select so we want here there and i probably could do a ring select on these see control shift actually that may actually work a little bit better let's do that i'll just control shift this and what we're going to do is actually something because this is not connected to the actual back of the eye that's separated we can do a expand on the vertexes so do control plus okay so we have the eyes and let's go one more okay that looks good i'll do a ctrl shift m and i'm going to [Music] let's just add that there and ctrl shift m okay so we have eye mask okay if i go into weight paint mode we have that mast okay so we can do shaping of the eyes now it looks like i would like to have a little bit more up here because if i'm moving the eyes up and down this is probably gonna move the forehead so we're gonna add more to this go up here and get this one under here probably could have hidden this okay do control oh how high do i want to go okay that looks like it would be work now i want to i want to have the eyebrows separate so when you're masking you can actually just do the eyebrows not the add the whole head and face stuff for cartoony type of aspects of it so we want a brow mask and this one won't be pretty straightforward we just go to edit alt a to just select everything l ll and assign and help if i actually spell it right okay and that okay so we have we have all the masks that we need okay sweet okay this is done now with turn the bones back on there's a different scenes and you can add a scene a general scene called sculpting where is it sculpting i already have one out and what do you do in sculpting mode is that we want to make our face tracking shapes now the face tracking shapes are based from the wiki blend shapes now these are case sensitive and so what we're going to do is we're going to try to work on creating the eye tracking shapes what we want to do is not these are not in order on how you would actually create them so the easiest ones are going to be the eye left and uh the blinking so we have i left blank and there is a i write blank those should already exist on the avatar so we should have um what we're going to do is we're going to recreate them it would help if i would have everything shown again because i hid those alt a so we want to make that our link left so i left blank so we're going to do a value of one and do create new shape from mix and we're going to call this i left blink now this is not necessary but it's a thing i do is this i call this spring pull shape keys now this shape is not going to do anything but it's just a way to organize the shape keys so i'm going to call this this is my eye tracking so let's scoot that up and when we hit the x oops if you noticed i accidentally had the mouth open because i was working on it earlier that is my bad and it looks like something like gum is blue i didn't realize this avatar had blue gums but okay besides the point i have this mouth that's not supposed to be there so we're going to x out we're going to minus that this deletes it so we want blink left with no mouth open so i'm going to do that i left blink all right clear and blink right i now the reason why we're recreating these is we can use templates that follow these shape key standards now you can use a template animation set with these shape keys so it's nice to use these exactly as named so what we're going to do is just like how you do mmd models for the work in uh mmd worlds you have to have exact shape key name in order to make it work and just like that you need to have it for face tracking if you want to use templates not required but makes life easier so next shape i want to do is i need to see if there's a wide is there a surprised there's a surprised yes but it has a mouth movement we don't want to mix mouth movement with the eyes so we can use a mask so we can use eyes mask so it should just do the forehead and i'm hoping that it doesn't move too much but it looks like that works so what we did was we put a vertex group into this so i can actually uh-oh what's going on here we have something moving here that we don't want to move interesting it looks like my mask may have a let's look at this looks like my mask may actually have something that's assigned to that we don't want that assigned there so let's look at our mask let's control a let's look at the eye mask so it looks like i maybe accidentally ah if you see that right there i have a vertex here that i don't want to be there so what i'm going to do is going to deselect everything i'm going to select what i don't want i don't want that i'm going to ctrl shift m hopefully that's mirrored that is mirrored okay and i want to remove it and did you just see that it just because we have this mask right now applied to this shape key it instantly just remove that so now should be just eyes okay so we're going to call this let's go and call this eyes wide now okay now we don't want both of them to be wide i think we also would like to have do we want to move the eyebrows i think the eyebrows moving up would be useful in this obviously in this mask case i if i want to add more to this i can actually do a blend from shape i can do surprised look up the that one we were mixing to and we'll add or remove the ad add adds to if these are already moving it will move that okay so we just added the eyebrows to move with the that now i'm a little i'm let me look at this surprise that looks like that looks right uh oh i may want to actually move the stuff in the middle so i'm gonna add these to my ice mask and what i'm going to do is i'm going to cancel this out do and i guess we can make a eyes and brows mask just using both so just select eyes and brow create new one i plus brow mask okay and a sign so we just create another mask cancel that and what we want to do is surprised i am brow mask and we're gonna go boop and now we have a new mask and we're gonna create a new can't do it in edit mode so we have to get out edit mode oops and we're gonna do that delete eyes wide okay so we have eyes wide but now we want to separate it out so we want to have left and right eye widen so we're going to do is going to do head left smooth so now we have a left mask and we can do eye left wide and then we can do this on the other side head right smooth now we have high right wide now i have most of these memorized but what i'm using it's this so just double check should be i left wide i right wide okay now so we got eyes wide i can't close that so we should have this up wide wide now what i want to do is we're going to start creating ones that don't exist so ones that we don't exist yet is eyes squeeze now we're going to do i'm going to set to the value of 1 go to elastic deform go to edit mode and i want to mask out the eyes so eyes masks and do i want to do eyebrows i think let's do the eyes and together we're going to select them hide so it's going to look a little funny oh i did not notice this okay this may be a note i think i need a note okay so we have blush this is a side point if i'm going to be moving the face around i may want to add this to the upper lip mask and the eye and brow mask so let me add these to those so we can actually assign that and i want to assign that to the upper lip mask smooth and the upper lip because i did not add those okay so i'm gonna h hide okay now i'm not in sculpt mode and do a fill mask everything's going to go a little bit dark because we're all masking this out and then alt h to bring things back all day to get things back and now you have a mask using our vertex groups so a i squeeze is a combination of the blink going a little bit farther than blink so you're going farther than the normal blank you're actually squeezing your eye after you blink so it's an additive function of it so in this states i write blink as a value of 100 and i left blink is 100 so it's going to be on top of the blank so what we're going to do is we're going to have i left blank and i right blank and we're going to turn it into x mode i have it mirrored i'm going to select i squeeze f and we're going to drag this stuff down i'm going to drag this up a little bit because you're squeezing it together it's not just straight down so we're going to do is an elastic deform i'm using tri-stale grab and volume preservation so this allows it to be a little bit more flexible not very aggressive and we can move things to make that work so we want to not deform it too much and in this mode i am doing it in with the textures so we can know that we're not messing up the uvs too much and not messing up the textures so we're gonna scale this down a little bit and we can squeeze the in the outside eye and push it up a little bit so when you squeeze it i want to just move that back a little bit so we'll go as well we have a squeeze eyes as i'll squeeze and i want to make sure that everything looks all right with this a squeeze okay that's our squeeze so we're done with squeeze now we need to separate the squeeze now this is without the eye closing so it looks a little weird but without the eye closing but it will only happen when the eyes close so what we'll do is we need to separate this out so we have left eye left and now this is a weird thing so the squeeze actually came out in uh cyranopol uh version two and that for some odd reason had a lower case either they were lazy or something it was lowercase so i'm this is actually from the manual so it's actually the one that's actually lowercase versus everything else that's uppercase for each of those words so obviously that's slightly annoying but we just make do with what we currently have on it we're not trying to change what this rainfall had on that so now we're going to do the right so we have i write squeeze okay so when you do right squeeze and right blink it should be it just a squeeze on the right side okay and we want to i clear the vertex groups because it does break on the import into unity uh you won't have a slider for some odd reason in unity so just need to make remember to x out these when you're mixing them so we have blink wide and squeeze now the other functions are how the eye moves when you look around so you have right left up down and that's for left and right together so in order to do that you're going to have to go into pose mode we're going to do is select the bones go into pose mode and i'm going to select both of these we're going to do our rx and with them both selected and we're going to roll up a little bit and we'll just see let's see how this works it looks like it's not 100 parented to the eyes because it's not moving 101 to these eyes so if i actually look in i think if i look in weight paint mode oops it would help if i go select okay when you go into weight paint mode you want to actually see what each bone is weighted to you suck that and then shift click and then we paint and control control ah okay because these are both weighted to one so it's like sorry like half of it's on this half that's on the other eye so okay so that will have that will work just fine so we'll do is select these r and let's select oh how far do we want to rotate up now these are not moving as well as i would like it to not very not very good for eye tracking let's see how would i make this better on eye tracking i think i need to get rid of these bones so what we're going to do is we're going to get rid of these bones now this is obviously it's just how the bone actually moves the eye it's not needed but we're gonna do is see that's i dot r and i dot l okay so do i want to delete these bones um let's say no let's not delete the bones because i it would be harder to add back so what we're going to do is we're going to go to body and we'll get out of object mode and we're going to go to i l and what we're going to do is delete this vertex groups these vertex groups are what's actually controlling the weighting of it to the bones so we're gonna do that i'm gonna go back into pose mode rx and now it's a one to one okay now the only problem is it's clipping through that way interesting okay now the pivot point is down here well a normal eye pivot point would be up here so let's move the pivot point so i'm going to go edit i'm going to select both let's just do x mode g y z let's go into local mode g y z and we're gonna try to move it i think i want right there on that pivot where the existing pivot is because we want the we want the these bones to be straight up to the world axis and that will make our lives a little bit easier and i'm going to move it on the inside a little bit and now we could make this exact to what actual that point is but this is probably the quicker way of doing it and we'll basically go pose mode okay now that's what i'm looking for so we have this nice rotation point for the eyes so we're going to reset that and we want to set the up upper limit 25 make it a whole number so he's looking up so we're going to sculpt mode it will be in still in pose mode in here and we're going to do is go into sculpt mode we are going to sculpt him looking up so eyes oops i have caps on eyes up and we are going to create a eyes up shape so he's going to look up oh now we can't go too far because this is not really connected interesting okay so we may need to move this back of this eye white so that's eye up so he's looking up and we can squeeze that and maybe move up the cheeks a little bit so he's looking up you can make it like cute like that okay so i up okay and what we could do is have a corrective shape to make these eye wells deeper let's make these eye wells deeper so we're going to do eye deeper and this is just going to be a corrective shape and what we're going to do is i'm going to change up the one change x i don't know where the center is so what we're gonna do is we're gonna try to pull this out uh you know what let's just select everything and just minus one minus oops uh can we suck less please interesting okay it's not allow me to step out so okay we're just gonna manually select it okay ctrl shift m just like that and we want to make a new i'm going to create a new layer nice and white i'm just going to call eyes white mask and we're going to do is assign it we're going to clear this clear this mask and end mode hide mask okay so what we're going to do is we're going to make these deeper we don't want to go too crazy see and from this to that so it just gives it a little bit more room back there i think we can make it a little bit deeper obviously you have to be careful with this okay now we have a deeper well okay now that's if you have issues with that now what we're going to do is we're going to do a clear mask again alt a it's a good habit to do all a because you may have some other things selected that you don't see eye mask hide fill control alt h now we got eyes up and we want to separate eyes up so we go eyes up eyes up and well that will fix the issue but we don't want to have the eyes deeper when we're doing these other animations so we want to just clear everything so we eyes up head left smooth and we're going to create a new shape i left up right and i okay so we got the eye up now we want to clear the mask now go back into pose mode and we are going to go rx and we'll do the opposite 25 degrees so we're looking down scope mode let's go mode eyes down and what this won't do eyes down so we want to look down [Music] and see if the deeper walls actually fixes that issue and not really but we'll see let's just make sure we're not going too far down so we go from this to this i think that looks good so i'm going to do is we're going to clear that do head left smooth and we're going to create the eye down and [Music] down okay so we have this and this okay now next ones is left and right so i go pose mode reset now we don't want x to be on so we want to rotate on the local axis r y and i want to look let's say it looks like it's 25 degrees is what we're looking at on the angles so we're going to do our y and that's going to be 25. okay so we got a 25 degree angle out of pose mode go in to select the mesh and then go into oops scope mode now we do not want x on for looking left right left so eyes left and i'm going to sculpt this there we go so what we're doing is we're the eyes moving a little bit so we're having this shape so he's looking to the left and i think let's see how the deeper looks okay the deeper did fix that a little bit so i think we're good on that and so we have looking left so that's from it was like this now it's like this so yeah actually it looks like he's looking what we're gonna do next it's gonna clear everything and we're gonna do head left and we're gonna do eye left left okay is going to be this one which is on the right you're going to create a new one i right left clear that so we have right and left now i believe there was something on the eyes that did not it wasn't wasn't able to mirror so you can actually mirror this because technically because we didn't mirror it because it's not asymmetrical on how the shape is we can copy this but we're going to just do example oops we're going to copy this so this is a copy of it if i mirror this ah see the the eyebrows are broken so this that i can't mirror that so that's not gonna work so we're gonna have to manually do that so i'm gonna do is i'm gonna have to create a new shape nice right and we're gonna have to eyeball it on the opposite direction so we need to do pose mode select everything reset ry negative 25 and negative 25. okay go back into sculpt mode and we're going to try to recreate the i write soreness scoot this a little bit i want to go down a little bit pull this up a little bit scoot this to the right a little bit scrape down and scoot because i is not perfectly round it's sort of like a it's not perfectly round so it's gonna the long part is at the front eye so it's going to move around it's going to pull towards the actual eye so we're going to have a little bit of a movement right there so we're going to go like that and versus this so we have a little bit of a shape shift and i think that looks good so we're going to do is we're going to do this again head smooth so i left right and i left right right okay so clear that out so we should have right and right okay now the last thing i need to do is i left dilation and i the eye dilation and constrict so we're going to reset the movement on the bones we're done with those okay reset sculpt mode okay now let's look at the eyes okay now the center point on this is actually in the center which is sort of critical for dilation and construction of the the pupils so what we're going to do is we're going to have eyes constrict and eyes dilation so we're going to do this and mode this is not something you do in the sculpt mode so we're going to do is we're going to do ring selects so i want to ring select things except for that went the wrong direction that's not what i want so we're going to do control control control now i believe i want to go to edge select mode okay that's way better okay go to edge select mode and i want to [Music] have it mirrored and i want to do a edge slide so if i pull this out it slides out slides out okay so we're gonna go that's about yay negative one oh it's a wrong way oops so we want to construct and we'll have 0.5 so if we turn on this in edit mode we should be able to see it so we have a little bit of constrict and obviously this texture is not very helpful for actually doing dilation [Music] let me try this again let's reset these from the basis just reset it okay let's just do a negative one negative one and then we have a positive one oh boy i didn't do it on both sides because it's not mirrored shoot okay oh it did never mind okay so we have that dilation constrict obviously this probably could be messed with a little bit more uh i guess we could uh i think we could do these a little bit more and stretching it too much let's do five [Music] five i think oops so smear but we don't want to extend it so we want to do this oh interesting how that happened very interesting so i didn't like that it does not mirror that okay let's just do it manually 0.5 try off mirror and i am not zoomed in enough period there we go oh oops one more out okay so we should have that's our dilation construction and then there's our dilation and let's do the same thing where we have this this this and this go a little bit farther out and let's do 0.5 i don't know if i like that you know i'll just get rid of it okay there's our dilation and our construction so we're gonna do is we're gonna get out here constrict left this is not required but it's nice to keep them if you want to have them separated you can left dilation oops that was constrict i did all right i write and clear and i could be either either smooth or not it doesn't matter dilation okay and that is the eye tracking and that's with everything you need for eye tracking so you know blink and all our movements okay we those are good now face tracking actually technically it's lip tracking face tracking in itself is everything with eye tracking flip tracking and lip tracking is a little bit more involved and what we want is let's say we have our oops wrong thing ah ah is a very good shape it's usually a mouth open so we have our mouth open with ah now that doesn't go very wide hmm now i did see that we do have a mouth bone here if we're going to pose mode it should have like a mouth shape cool okay now we don't want it too aggressive let's reset it unless rx let's say about oh 23 degrees maybe does that look good uh i think that would look good so one of the cool things with bones if you have a bone face rig is you can actually do a pose library now oppose libraries in the object data properties and what you do is you hit a new you have your object selected and you have a new pose library and we want to add to post libraries so it's whatever you have selected as to oppose so now i have jaw open now so if i reset it and i click this i have a jaw open so whatever the pose is for the bone so other shapes i have to do let me go back to here that are using the jaw is we have a jaw right jaw left and draw forward and jaw open so what we're going to do is reset this and these make it really nice because you just do just tweak it and now we don't want to go too far with the jaw left and right let's do negative seven let's do negative six negative six looks good so that'll be our draw left so add new let's hold that jaw left and now we can do a copy and mirror so ctrl c and ctrl shift b and now it's mirrored and nice thing about with bones is they're very not as picky as mesh because if mesh is gets something gets messed up it can become non-symmetrical really quickly and it's very common for these to be not symmetrical but with bones it doesn't really care so we have draw left and now we want to reset i want to do a jaw forward so we have a jaw forward animation and we'll just make it a an actual heart quoted number do a draw forward okay we got draw forward and we are going to do is clear transforms that's another way you can go is you can get clear transforms here and what i've i have been doing with hotkeys is you can clear rotation by alt g all r or alt s so now you have these poses now the way you can do it is i want to create a jaw open so we have this shape i want to create that shape into a shape key now in order to do that i need to enable a function in blender so currently by default it's lock object modes and this what this means is when you click the armature it switches to pose mode if you click the body it will switch to like actual edit mode so what we're going to do is we're going to dissect this now this is not always i'd recommend to have that on but it's it's nice to work with so we want to have in pose mode with that so we have our poses then we want to go to object mode this is our object mode for the actual mesh and we're going to go to this wrench icon this modifier this is what's linking your mesh data with the vertex groups to the bones so what you want to do and this is how you connect your mesh to the bones so if you have anything different separate that you want to link into it and you can add that here but this also can create shape keys so i want to rename this modifier to jaw open and in the thing is save as shape key so now if we go to our shape keys we should have a jaw open shape key that's from this bones we now i want to do left so we have left and what we'll do is we'll go back to this i'm going to draw left and we're going to save a shape key go back to pose go back to our compose library go to the right go back to the wrench right save the shape key go pose drop forward go back to body and we'll pull this jaw forward and then we're going to save that shape key and now we're done with those shapes with that bone because we had a bone there so i'm going to do alt g the revert the transform that's not rotation that was a transform so we'll i can just change this back to armature arm mature would be helpful and now if i go back to sculpt mode i will have these new shapes so if we go to edit mode we have draw open drop left draw right and draw forward so we got those shapes okay now jaw mouth ape shape is a combination of jaw open but with the lips closed so in order to do that i want to do a jaw open mix and it's not going to be all the way down because that's going to be pretty aggressive on the ape shape so we're going to do about maybe i usually do about 60 of the mouth open and i'm just going to duplicate this we have mouth ape shape and now we're going to do is we're going to go into ape shape now we want to create our masks and i want to get back to vertex mode and we'll do the lower lip mask let's do the upper lip mask first because we'll we it gets a little tricky when you have two lips and you're doing both top and bottom at the same time so we'll just do one of that time so we'll do fill mask edit mode alt h and then we got that now we want to turn back on the mirror and we'll scale this down a little bit so what we're going to do is we're going to try to recreate this shape ape shape basically what you do is you have your mouth open like so and a little bit like that okay so that's our we have the upper portion for the shape now let's do the lower portion all right clear the masks alt a lower lip mask fill mask old age now we have the lower so we're going to go up and now we have our a little bit of our gum sticking out so we're going to pull this out a little bit pull this up and now we're dragging a little bit too much on this corner so it's making it a little sharp let's fix that and we're gonna go up now we're getting into a little bit of issues right here so we're gonna try to get in very detailed so we're going to scoot this up a little bit and it doesn't need to be perfect but we're trying to just get it where we want it i can't see okay now when you have this clipping sometimes this is fixed by the clip start distance i'm gonna cut this in half to there change the clip mask and i'm just remove this gum out a little bit and a little bit of artwork on how to make this work look good and there's some things you probably could get away with you don't need to be super detailed because but let's do that now that's our ape shape that's our ape shape that looks actually pretty good so that's our ape shape now our next shape what we're going to do is we're going to create a mouth left and right movement so what we're going to do is run clear mask do [Music] upper lip lower lip hide that fill okay [Music] i have a mouth left [Music] two so you're moving your mouth left should be when the mouth opens moving mouth around a little bit okay it's mouth left duplicate to mouth right and hopefully this should be mirrored so do this mirror it looks like it worked okay that worked now we're going to do is we're going to split the top and bottom separate so i'm gonna do upper lip smooth mask oops hmm i think that maybe it masked out too much oh more lip or actually let's do a upper lip smooth mask okay it smoothed it a little bit okay i guess that's okay that's fine so let's get the mouth upper left and lower okay and the mouth over look at that oops i cancelled it out okay and we're going to is upper lip mask smooth right and lower lip smooth so what that's going to do is we're going to overlay the bottom lip of the top lip it's going to sort of be like a almost like a pug like pouty look to it and now we don't want to get mess up the shape of the mouth too much with that so there's our pout mouth overlay i think that looks good it's mouth lower overlay or inside and you can get see a lot of these shapes from the saranac sdk or from the face tracking wiki page so we want to [Music] mouth lower inside this is going to be like you're biting your bottom lip so we want to put this in like so so there's a there's our teeth okay so that's our point where we want to grab okay now the problem is these gums are really big so this is going to be a little tricky so we're going to try and get this hmm yeah we can't go too far otherwise we're going to clip in the gums so yeah okay so that's as good as we're gonna get so it's you biting the bottom lip biting along that four skills down i think this looks good so we have our inside okay now let's do another one that's on the lower lip so let's do mouth lower down all right what we're going to do is we're going to scoot your teeth down uh his muzzles make it a little difficult to actually see the teeth so uh hmm what we're going to do is we're going to open the mouth a little bit so we can actually see things because we don't want to take it too far down otherwise it's going to look a little weird so we want to grab this out okay grab and we make sure that we're on this layer not the jaw open so we have this lower lip down and we'll just pull this down just a hair more just let me get a little bit rounded and we can pull it out a little bit so when we do this don't really care for that so we have this lower lip thing so you're showing your bump teeth next step we're going to just clear this mask and we're going to do the hopper lip and we're going to go back to fill mask okay i'm gonna do is we're gonna do mouth upper up up and we'll redo is run now with mouth up or up it's a very common uh type of movement so we want to make this one really good and one of the things you want to do is you actually sort of want to pull this back a little bit the mouth it's like you're showing your upper teeth so we're gonna pull this up now i need to be careful with this now you gotta mess with this muzzle a little bit now here is an issue i'm seeing with this right now if you can notice you can't see the teeth when you move the lips up because it's all being hidden down here so we have up interesting and down okay i need to be able to move this so what we're going to do is i need to get that unmasked so this right here i don't want to be masked so we're going to go here oh god there's so much stuff here okay and we'll do another one polly's out uh i think that i think that'll work i'm gonna do is grab that and we're going to mirror this and extend i'm going to make this a upper hide so now we have this new mask i can actually move this so i want to move this up oh so okay see what's i can't see things let's turn it to material [Music] and get rid of this okay oh i gotta be careful with this so i'm gonna drag that out okay now that one point i don't have in my mask so i'm just gonna do it manually i'm just gonna paint it out and we're gonna go back here and we're gonna pull that around and we're gonna scoot that around and there we go so now we have this shape and let's switch back to texture so we can actually see don't mess up the gums too much keep that pretty close so you don't want to mess up the uvs too much on this so we have this we want to keep it pretty flat because this is not a smile but uh it's good to have a natural smile when you show your teeth this would be helpful to it just looks nicer okay and we have a point right there that's not still masked so we'll just leave that alone okay and we can test this so we got this mouth upper up and then we have lower down now i think we probably should move that up a little bit so what i'm going to do is do this subtractive and that's mirrored so it should be mirroring to the other side i can't see we're gonna just just a little bit of masking paint and we're gonna just go up here and we're gonna just scoot that up a little bit and i'm gonna scoot this up a little bit okay so we have this mouth upper up okay there we go so we've got mouth upper up what else do i need to do [Music] we have mouth upper inside so round two is gonna do mouth upper inside grab that and this is be you biting your upper lip now this is a little bit tricky because you can't actually see the i'll just drag it down a little bit so i got that got that okay so let's separate the ups and downs so we want to separate the left and right so we'll do is we'll go full down go ahead use our smooth mask left smooth mask mouth or down left and mouth or down right now so if we do these both together it should look the same and it does because we have it exactly mirrored on that that mask is a perfect split between these now this is not perfect probably could have a little bit better transition here adding another stage because that would be see that's half 75 full so maybe having an 80 there would all make it a little bit better we'll leave it there for now so mouth up or up and we'll do is we'll do a head left smooth so we have the right side left side mouth over oops it's actually upper up left and it needs to be together and head right mouth upper up right so both of them together should be the same so we've got right and then left and then we've got right okay those are split i'm gonna move that up with up so we can keep it together okay [Music] now undo mouth so we're going to just do is create the upper pout shape and we want to not go too much shape it we're just doing the first shape sort of like a kissing lip a little bit like that i want to clear the mask unselect everything lower that mask hide fill hide back to pout so like this this like kissing face just back a little bit so let's go there's pout and i think we can basically that's our pout and we can make this a little bit different scoop that up and call that good you can tweak it a little bit pout okay now another one we can do is mouth o shape now this is going to be both and this is not gonna be a shape that you actually control in game per se you can use it but my template does not use it it's used to help create the upper overturn and the lower overturn to do this we're going to do is we're going to have upper up left upper upright down left and lower down right okay now this shape is used to make an o shape so it's like when you're saying o i want visemes so we're going to do is we're going to hide that fill right so we have our whole face and what we're going to do is on oh shape set this a one drag this stuff out so we're creating an o it's going to look a little sloppy but we'll clean it up a little bit later we're going to go oh have that go like oh shape and we're gonna clean this up and it is popping out quite a bit uh now when you're doing all these movements they can break things pretty quickly so [Music] what we're gonna do is foreign [Music] i think we messed up this corner right here i'll just scoot it in make it a little bit more smoother on these now what i'm looking for is trying to keep it somewhat uniform on the shape i'm trying to look for and you want like your curve like that so we have oh now the thing you have to be careful with o shape and any other face tracking shapes they can stack and they can do some weird things if you have weird stacking on them so we'll just try to be careful not going too extreme now because i think pout sometimes triggers so you have sometimes have pout trigger when you do o shape so we don't want that too aggressive so we're gonna do is we're gonna grab that and pull this o shape in and we're gonna grab that okay okay so we do pout and you see we have a little bit of weird shape but we can mess with it and see if we can make this work a little bit better with pout and when we turn off pout that's not bad i don't know how i care about this corner here and this is a little tedious and now the problem is i'm too close to this upper lip to really make adjustments properly so i'm gonna probably have to uh mask it so i can do some tweaks on that because it's modifying the upper lip but i can just zoom in a little bit and just i think that looks pretty good so that's our o shape oh oh okay and i like that that's i don't like this oh i don't think that's is that the mask ah my mask my smooth masks have an issue there interesting okay okay we have this okay so with these masks i may have to do some tweaking here now i have to make sure i don't have a mirror on because i want to just modify this a little bit just a little bit so what i'm trying to do is when we have this just right one it pulls it down just like that and then on the left [Music] one that looks good oops that's on the wrong side that looks good okay i'm happy with that and oh shape oops see i think this is not supposed to be coming out like this so we'll scoot this in a little bit just need to really check the shapes of how this is because we didn't move it quite a bit here so we'll just try to make sure that we keep it from not looking too shank see there we go i'm happy with that so we're going to clear it o shape by itself is going to look weird and what we're going to do is we're going to do a upper lip smooth that's going to look a little funky it's going to be funky so we're going to have new shape is going to be mouth return and then we're going to have a lower lip smooth this should be just the lower smith of the o shape mouth lower over turn turn okay so now these together should be the same okay and if i do lower down up or up down and right and yes that is correct they do it does make the same thing good okay so we're going to do smile mouth smile and i think there may be a shape key that already has smile and that's actually a pretty good shape key i like that let's use that one because that should work with upper up and upper right and i think that actually that looks great i like that so we'll use that that's i think that's a good shape we'll reuse that shape so we're going to do is we're going to recreate this shape so we're going to do is go okay let's see a smile head left we're gonna do a smile left mouth smile left and then we're gonna do add it's about right all right and so we got the smile we're gonna do is want to make a custom mouth sad and mirror and i'm just going to drag it down like that and we're going to drag that a little bit down so when you test lower down and we're going to just modify this just a hair so we can okay i'm gonna open up the mouth a little bit so we can actually see it and i want to pull that in a little bit pull that in so i'm going to close the mouth so pull it down just a hair okay so now we have a sad face and we can do the eyebrows separate so clear it left smooth said left mouth sad bright get rid of the mask so we got those now the pouty almost looks like puffing cheeks it is puffing cheeks interesting okay okay i think i can work with that so we'll make that cheek puff so let's do this oh the ears move hmm i don't know if i want it to do that let's just copy it for now let's do that and we'll do is we'll clear anything that's not on the face so we'll do cheek cheek puff and what we're going to do is we're going to select anything that is not the cheeks and then we're gonna revert them back to the bases so this should be just the cheeks that's just the cheeks now i want to separate them to left and right so got cheek puff left cheek puff luck and then puff right so we got cheek puff right left and right okay and then what we're gonna do is gonna have cheek suck this one we're gonna do is we're gonna shape this one pull this forward and we're going to pull this in and pull this out pull in the cheeks a little bit so we have our cheek a suck there's a cheek suck okay okay we want our tongue looking here that's selected okay what we're going to do is hide it select alt h now i'm going to open up the mouth a little bit so we can actually see the tongue okay so that's so that's light now what we're going to do it's called tongue long step one now tongue long step one just goes to the teeth now i do have the jaw open just a hair but we just need to make sure we don't edit on that on the jaw so i'm gonna have it open just a little bit so we can actually work with it now and that can happen where you can actually edit accidentally edit that just make sure you have a long enough undo history so what we're going to do is we're going to grab the tongue and we want to grab it somewhere and big enough that we don't distort it too much so we're gonna grab it there so this is tongue step one tongue step one is gonna be to your teeth and a little bit up so it's making that upward motion so we're going like this i'm going to stretch the back a little bit so i'm going to stretch it back so go like that okay that's tongue step one and then tongue step two is fully out step two and that's an additive to it so this is one and this is one so what i do on time step two adds to tongue step one so i go out and this is just straight out and i just need to make sure i don't so we stretch it out i want to stretch it a little bit like that so we got a tongue and we can widen it a little bit so we have a nice tongue now if i close the jaw should be about about there and when you do stick out your tongue your mouth is going to be slightly open anyways so it will be about right here so if you point to it should be like that so we're going to go back to point five so i can work with it and that's our tongues long step one two and two so now tongue up and tongue down going to be this same thing they're going to be on top of the long steps so it's going to be tongue up and what we're going to do is we're going to grab this tongue pull it up without distorting it too much this is good it's a little tricky but uh there's probably maybe a better way of doing this but this is how i have done it so you pull it up like that and we'll pull out and then you can just do some of the shaping just to pull this in like so so this is and we can make it a little bit more curved and we got a tongue up let's get tongue up and i would say it needs to be a little bit longer they're up a little bit longer so it should theoretically should be about the same length when it goes up so and that looks about right you can change the shape a little bit we got tongue up now let's do tongue down and tongue down will be like so and we'll just maybe grab a little bit bigger and we're gonna pull it down like so and we're gonna shape this up a little bit it pulls out then like that just like that okay so i'll just pull this out and you just do some shaping tongue so we have a tongue down okay got tongue down sweet that's our tongue down now let's do tongue left and i believe the tongue we didn't have any issues with mirroring so this is going to be great because um if you don't have nearing of the tongue it will sort of be a pain in the butt to match with the left side the with the right side so having it mirrored helps a lot so we're gonna do this and now with the tongue like a muscle it's gonna stretch on one side and it should be like this pulls forward this pulls back and the uvs should be more like this you want it to be like an arc so we want to arc it like how your tongue is turning and now this can be a little tricky so we just need to tweak it around and the shape can get messed up a little bit you just have to you just have to massage it to where you want it and let's get the tongue about yay and pull that over okay so now looks about right except for this probably needs to be pulled a little bit forward so was that now with the tongue left i need to see if it's about the same length now that's the problem like we have it not quite the same length so you pull there pull it out just a hair more so you want to be like an arc over something that would look decent so go like that and i swear on a little bit better uvs so i'm going to pull these a little bit forward and this little back could look nice pull this back and warping let's fix that pull that up that shape looks good pull this around a little bit check this one just massage it back and forth sometimes massaging back and forth will help you get to where you want it to be and tweak that down a little bit tweak it there and let's bring that about yay here so there's our looks like a bottle left so we got left perfect now i want to mirror this but i don't want to copy the other things so i'm going to do that duplicate tongue right we're going to stick this back out tongue and we're going to open the mouth a little bit 2.5 and now that's the duplicated right one and you select it and mirror shape key except for i didn't select the left one i suck the left one i didn't want the right one there we go and that mirrored so when you do this go mirror mirror okay now that shape is not the best oh that'll work let's let's say i just noticed that i don't like this shape a little bit so i'll move this up just a hair select that [Music] pull this out and you know maybe take some tweaking you'll see something like oh i don't like that so i made this one now i like the tongue right more than the tongue left so if i go to the tongue left i can do this and i can do a vertex blend from shape tongue right and that will override anything on tongue left and i just mirrored again i got tongue left like so now there's some combination of these so when you do tongue left and tongue up you're going to get this weird shape and a way to fix this shape is called a tongue up left morph and this is going to be a shape key that's not in itself used as by itself i guess it's sort of redundant but it is a way to correct the shape of this shape key now i want to drag and fix this shape to look like a tongue again not be warped around drag that that way drag that way and it can be a little tricky because it sometimes squishes the mesh so i'm going to try to drag it out and these take a little tweaking so what this does it like corrects the shape and you just work at it and try to get to a shape that's um someone resembles a tongue going up to the left try to correct that okay now i want to drag this a little bit like that the muscle should be going to an arc now thinking i'm having some issues with this being too okay now that's grabbing on the other side and i don't really want to do that so i need to figure out how i want to pull this either i have to reduce the size or i need to change how the brush works on this let's yeah we'll just massage it just go back and forth dragging it out go up okay and let's make this more into an arc instead of a so and we got a tongue left up and this shape will correct that when you do a combo like that i'm gonna drag that a little bit more and because the tongue can mirror we'll this will we can be tongue upright morph also okay and i am i'm just just nitpicky but this is just just a little tweaking into what i want to look like okay so we got tongue tongue left up now we're going to do the same thing so it brings it down a little bit but that's okay so be instead of up we want down now this one's not too bad this one i don't have to correct as much so we go tongue down left morph so we're going to do this one and then we'll just need to fix this shape just a hair oh this one's not bad at all this is this is a pretty easy one okay all right you have very much at all to do i guess drag this down a little bit okay okay easy okay that one was the easy one so i just duplicate it up right and we'll just mirror that then down right oops there's an either right that and then mirror that so in order to test it we'll do down right long step long step okay and if we went up to up instead of down and now because i did this when the mouth was open so if we change it to that that will be about there so about two now i could make the morph work with the mouth a little bit more uh you may have some clipping with the tongue and that's pretty normal now what we're going to do is face tracking as a little tricky because it's not the fastest updating it will get you like the general shapes but it's to actually lip track what you're saying in real time it's a good practice to actually have it in combo with visemes so one of the things you can do is i have a script that is a face tracking vising mapper by by default everything's at .5 but what i will look at is the current visemes on the avatar so if i go in the visemes so ah so with ah i don't want the problem with face tracking and visemes together it can overdrive your face making your mouth really wide so one of the things you do with this remapper is i can make it at fifty percent of the value so this is just fifty percent for ah and this will allow it to be more reduced but still have a little bit of a visine but like just more subdued version of the visemes so [Music] now p doesn't usually do much so p mouth doesn't move and we can keep that at a hundred at one one so map it to p uh f f f moves the mouth not too much so that shouldn't be a problem so i think we can do the full amount so do f then th th it's a little bit bigger but we don't i think we can do about point eight of th and i go to th now d now that one makes it a little bit i think we can do that 80 also 0.8 d e i can do 0.8 on e e and when this will do is it will take this visine with the threshold and create a new visine that can be used in unity this will be the face track to advising with the reduction so ih and the mouth goes a little bit wide on that on ih so let's not use that one too much this is 0.7 oh now you have to be careful with o o and both of these o's are very aggressive and it can overdrive things so let's keep those pretty low and then in and it's pretty good we can set that up and silence i just obviously silence doesn't do anything it's just there to map it just so you can actually see it okay k is a little aggressive so let's keep that at 0.5 ch we can keep it at full ch at one and s yes we can do full and our or we can do full and then once we get all the thresholds down and everything mapped we just select the body we want the body to be where we want the face tracking visemes click this and boop and now we should have all the face tracking visemes down here let's go to edit mode here's our reduced set of visemes and this is a face tracked avatar now we have our eye deeper we could make that default but i'm going to leave it there just so that it just doesn't mess up the model too much so people can have it as a default state

Channel: Jerry's Mod

Views: 2,865

Keywords: SRanipal, Face Tracking, VRChat, Neos, Blender, Shape Keys, Tutorial

Id: CR6aeNKKaJw

Length: 143min 43sec (8623 seconds)

Published: Mon Jun 06 2022