Creating Real-time Lip-sync in Unity Using Code

hello and welcome to another Unity tutorial in this tutorial we will see how we can generate a lip syncing using code and we don't need any third-party application like Omniverse audio to face for Generate lip syncs the lip syncs are not as natural as Omniverse but it will generate the lip sync using blend shapes and you need just have a three brilliant shapes for your character it's open mouth closed mouth and his shape the benefit of this way is that for a if your application or game has a lot of dialogue you need just pass your voice files or even voice from microphone to an audio Source component and it will be automatically be generated by this screen so let's jump into unity and see how we can create a lip sync using the scripts foreign [Music] should be something like this then eight years since the Geneva Convention Force protocol which said women shall be especially protected against any attack on their honor in particular against rape enforced prostitution or and the lip sync is generated by a single script and we don't need any third-party application like Omniverse Audi to face or Etc the only thing that we needed is that our model have a three shape of blend shape open mouth closed mouth and something like key shape this model has a lot of different blend shape as you can see and I just selected some of them that is more similar to my expectation like most open this one and yeah I can show you this this is the script that we will use and this creep has some parameter and it's the important one is the shape balance shapes uh name the keyspell and shape name that is author team mouthpunker I've selected this is the name of Kiss shape on my model and I can show you where it is that is something like this as you can see and also open mouth it is this one and close mouth that it's not very important and you can see so we can play with these lip close factor to not close too much and the mouse that should that will be very normal so the other parameter is threshold that will is when the lip sync will be started depends on our audio if it's very small even the noise and Reverb of the file will affect the lip sync the smoothness how smooth the lip sync is if it's very large you can't see any lip syncing because it's too smooth they will call the track factor is almost like threshold you don't need to change it at all and these three Factor will control how powerful these balance shapes you will be used and disable or connective the model so after you import the unity package of the model you go to material and extract the texture of the model and the material if it's needed then import the model to your scene you need to play with the material as I did with my main model you can see yeah I can show you some of the changes for example the important changes if for example for the hair okay this threshold is important and you can see it's zero you can see the Y the alpha map so it should be a bit more and the list three shadows resolved and pre-pass result and you can see so this side this is my settings and for the skin isolate I choose the any isotropy for material type which is similar to human skin and just selected and changed smoothness as I need okay so our focus in this video is lip syncing so let's jump into the lip syncs I disable my model again and we work on the new model that is the default one so you you can just you can just make a post that you like you can do it with the [Music] bones you can just maybe for example let's search for upper arm and this one [Music] and you can do the same for the other hand okay so for the uh lip syncs you need to add this script free LS control to the object which has the blend chip as its child for example my model in this in this model the blend shape is in this model in this game object in this game object and I need to add the 3ls control component to the parent of this object I just added three LS control to this and you need to add a audio Source component as well and all Clips any clips that you add to this audio Source will be used for generating the lipsense and even you can use microphone and at runtime you just generate a voice and at runtime the lip syncing will will work so for the settings I will just copy the settings from my main character you can just with your character you need to play with these settings to get the best results and if you will use this model that I will add the link on the description below you can use the same setting as ID so just right click and paste components values fine and for the audio Source I have a clip that I downloaded from internet and it's Angelina Julie speech and now if I play the game you can see [Music] 68 years since the Geneva Convention Force protocol which said women shall be especially protected against any attacker so I can just change some of the settings that you can see how it works so if I for example for zero threshold nothing will happen and for two anything any noises even will affect the lip sync a smoothness will actually as you can see how it's smooth the lip sync is and the factors for example let's just okay so I checked Loop here to repeat the speech and you can see the mouth is more open when the value is larger is the same for the other Island shapes and as you can see for example this one is you need to be careful with liquids Factor and also case Factor okay so and I just for more information about my model as you can see there is some blinking if your model has some balance shapes a blinking blend shapes like this one you can just easily create a timeline and add a game object that has the blink blend shape to that timeline you can just easily drag and drop here and add animation track and as you can see it's doubling shapes here first you need to lock the timeline that you can select anything without losing the timeline here then just let me I delete my original animation here and as you can see there is no blinking now so just record here select the object and for here for example we can create a new keyframe I just add one that has no effect you can even expect zero through it every lane shape somewhere in between another keyframe and now after two three frames added 75 or more it's depend to your model how close the eye will be and then if you frame later go back to zero it will be something like blinking and then double click um the timeline and open it will open animation window and you can just say copy all these three frame and go a bit from and paste it and randomly you can repeat it and then you can see randomly and then if you want to make it as a loop entire game the blinkings work you can just right click on it s homeware to clip track and then here you can just select Loop or ping pong so in tryer during the entire game the model will continue blinking even after the clip ends as you can see I hope you enjoyed this video this is useful for games which has too much dialogue it's difficult to create or impossible for a many dialogue to create lip syncs using third-party application you need to make the animations for example import them for any cutscene to your game but in this way easily you can generate lip syncs it's not very natural like like you can do that in Omni verse audio to face but it will works for a small game or a small projects thank you for watching and see you on the next video foreign

Channel: Code Designer

Views: 5,720

Keywords: Unity, Code designer, programming, game design, tutorial, lip-sync, lipsync, script, code, auto generate, threels

Id: Cgv8naIVa8I

Length: 16min 10sec (970 seconds)

Published: Thu Apr 06 2023