Lip Sync - React Three Fiber Tutorial

hi I'm Wawa Sensei welcome to my YouTube channel let's learn how to add lip sync to our react 3 fiber projects hi uh the 3D version of me just said we'll learn how to enable lip sync on our 3D model for this tutorial we will use Ready Player me avatars mixamo animations 11 labs for text-to-speech generation and rhubarb library to generate the lip sync instruction from the audio file but you can replace any of those tools with the one of your choice after understanding the concept of morph targets that we will discover together let's start by getting the starter pack go to my GitHub repository code and copy in a terminal run git clone the path and then I will enter a folder name error 3F lip sync tutorial once it's done I open the folder then open the terminal run yarn and yarn Dev to start the development server we can click on the link we now have the starter project we have a cube we can rotate it around which means it's working for the Avatar we'll be using Ready Player me you can hit login or enter Hub if you are connected I already have an avatar but I will show you the process to be able to get it I will do customize look but it will be create a new Avatar then you can customize it I will change the clothes let's try this here okay I'll say it's what I want and I hit enter Hub now I can see my avatar with the new clothes there is here a button download Avatar as a glb file it's what we want and we click now in our project in public create a new folder name it models and drag and drop your glb file here okay now we can stop the development server run npx gltf jsx the path to the glb file public models and the glb file Dash o to set the output it will be Source components Avatar Dot jsx and dash R to say the root folder it's public now we can start the development server and hide the terminal we go into Source components Avatar we have our model loaded here me I prefer to rename the function from model to Avatar now let's go to our experience instead of the cube which is this mesh we can try to load our Avatar if we try to look at it it's completely black and the position of the camera or the Avatar is not good for the lights let's add environment with the preset sunset okay be careful that it's correctly imported then in app we will move our camera we want to see the face let's say zero zero and eight when we reload we can see our Avatar but it's very far let's bring it closer to us go to experience the Avatar is here we will change the position we will set 0 minus three and five and scale it up so it's bigger and we better see the face it's better maybe it's a bit too close let's expand the field of view let's try a 42. okay now we see a bit more of it I used AI to generate this background image we will use it in our scene the tool I use is leonardo.ai like my previous tutorial in public let's create a folder textures drag and drop it so you have it here then in experience let's have texture is equal to use texture of model textures slash YouTube background dot jpeg to make it perfectly visible we'll use the viewport size the viewport is equal to use three statestate.viewports perfect and then we will add a plane a mesh with a plane geometry the arguments for the way we will use viewport width and viewport hate in those parameters we don't need them and a mesh stand basic material should be fine we'll set the texture as the map now if we reload we can see our character if we move we can see it's a background but we don't plan to move it so we have our character that we can animate and we have that nice background before adding The Voice we want to animate our Avatar we can go to mix ammo browse animations you will need to log in but it's free and then we need to upload our character but it handles only fbx file not glb to convert our model we go to blender create a new a to select everything and hit delete then file import and glb then search for the glb file public models new your glb file then you don't do anything you don't touch it file export fbx then you can name it my avatar you don't need to put it in your project just in download or anywhere is good on the right you will need to change one setting it's in the transform you will need to go for forward you need to put y forward and for the up you can keep Z up then the default settings but just look that you have applied you need your space transform and you don't need apply transform export fbx now in mixamo we can do select character file and use my avatar now you should see your avatar without texture but you don't mind and you can see it's animated hit next and now when you will try animations on the left it will preview them on your model I will choose a default one so idle which one can I take idle here he's doing nothing it looks like it's breathing so it's good I will download format fbx we will use fbx animation skin without skin we don't want to download the model we just want the animation information and hit download for the introduction I will use this one standing greeting I hit download again without skin download and another one if anyone I will search for angry angry angry gesture and download without skin again of course it's optional for the lip sync and you can use the one you like now in our project into public we will create a new folder named animations and drag and drop your animation file here now let's apply those animation into our Avatar we go to Avatar here we don't have yet animations because it didn't find some in the model so let's load the animations from the fbx file so const animations equal use fbx of animations slash Idol Dot fbx we copy paste for the two others that we have Angry jester and standing greeting but they have the same name a good optimization for our project will be to blend all the animations into one fbx file but here we are using multiple so we will rename animations with double dot and add another name so here it's idle so idle animation here it's angry gesture let's name it Angry animation and standing greeting let's name it greeting animation okay so let's look at what is an animation if we do console.log idle animation Idol animation okay we can see it's an array of Animation clip the first item the item zero it has a name mixamo.com so all our animations have mixamo.com we want to change it so an easy solution is to do idle animation.0 dot name equal idle then the angry animation will say angry and then the greeting animation greeting okay pilot thank you then we need to store a variable to know what animation we are currently playing so const animation set animation equal use State and it will take default idle now to be able to plug the animations from this to our model we will need to use animations hook which takes an array of Animation so we need to pass idle animation but it's already an array and we want to pass it an array we want the first animation from this array then angry animation the first one and same for greeting animation but if we had everything in one file it would just be idle animation like this okay so it takes another parameter parameter which is the root or this we will need to store a group reference so let's name it group const group equal use reference okay and then on our group we add a reference group and what use animations give us is the actions now we can use in the classical use effect we do in all our tutorial use effects based on the animation we are currently playing we do actions of animation dot reset dot fade in so it Blends smoothly go into that animation the duration and play then we return so when animation changes we call that function it's actions Dot from that animation and fade out now if we look at our Avatar we can see it's in idle state if we change in the default one by Angry we can see our character is doing that angry gesture perfect now let's generate some audio files so go to 11 Labs I will put the link in the description and create an account you will have some tokens that you can use for free to generate some files select one of the voice you can pre pretty simple if you do what you've always done you'll get what you've always gotten you cannot find yourself by going into the past you can find yourself by coming into the present okay maybe it is the mark of an educated mind to be able to entertain a thought without accepting it so okay me I will choose Arnold then you can write your text I already wrote this one and hit generate that you can listen to it here do you like pizzas I mean real pizzas with pepperoni not the ones with pineapple who the hell eat Hawaiian pizza you can hear it's very realistic it speaks better than me and download the audio file here I will prepare another one so we have two audio files we can switch from one another the one for the introduction so hi I'm Wawa Sensei welcome to my YouTube channel let's learn how to add lip sync to our react three fiber projects then hit generates hi I'm Wawa Sensei welcome to my YouTube channel let's learn how to add lip sync to our react 3 fiber projects perfect let's download it now we have our audio files we will use that Library named rhubarb lip sync that will take the audio files and generates a Json file that contains all the data for the lip sync we'll go more into that later but for now just let's download it to be able to use it scroll down to this section how to run it download the latest release then based on your operating system I'm on Macos so I click here I downloaded it into my project we do LS go to rhubarb lip sync Mac OS we have it we can also erase the zip we need to put our files audio files here in audio and drag and drop them okay so I named them welcome.mp3 and pizzas it's what we will use in the project how does it work now you are in the folder you can run dot slash rhubarb it's the name of the binary that will process our audio file then Dash f for the format we want to generate a Json to be able to use it easily in JavaScript then the name of the file so it's public um well public audios welcome dot MP3 and dash o for the output file and we will name it um which one welcome so welcome.json but we have an error because it doesn't support MP3 it supports either wave or OG go to convert IO we will convert it from MP3 to OGG choose files and hit convert okay so now I have the OGG file in audio so instead of running out MP3 we will run it on the good extension you can see it's currently processing the file now it says don't if we go into welcome.json we have that file that has been generated it contained the metadata the sound file that was used the duration and something called mouth cues with a start and end duration and the value of the mouth Q so for the moment you don't know what it is but we will look at what it is later let's run it again for the pizza so output is pizzas and the file is also pizzas okay it's done I can take the file that has been generated and moved them into audio now on the left you can see we have the audio file here and then its corresponding Json data file to know the lip sync we will play okay before doing the lip sync let's play our audio and prepare the project to be able to run it so now yarn Dev and go to the Avatar we'll be using Leva we don't have it yet so yarn add lever to not have to create a UI for what we're going to create as it's a development project okay so const we will have a play audio Boolean and a script which will be either pizza or welcome we'll be using use controls from Level play audio it will be false by default and then Scripts the value by default it will be the welcome and we will have two options but if you want more file you can of course add them welcome and pizzas now we load the audio file so const audio we will use memo so we will be able to memorize it and not re reload it at each render it will be new audio based on the path though slash audios slash script.mp3 and it will change every time script changes okay now we can do a use effect when play audio change maybe also the script if we want to change it and if we play audio we will play the audio file or if we don't we pause the audio let's also change the default animation to be idle now we have our project hi I'm Wawa Sensei welcome to my YouTube channel let's learn how to add lip sync to our react 3 fiber projects so it's working we need to add the lip sync before loading the Json data and synchronizing our lips with what is heard let's have a look at how we can move the lips with code so we will use something called morph targets if we scroll into our mesh you can go and see on head and teeth there is morph Target dictionary with the dictionary which are what morph targets we can impact and the influences so we tell which one we want to impact and what is the value of the morph Target don't worry if you don't understand yet we will play with it to understand how it works let's add a user effect based on nothing for the moment and what we will do is we will just log the morph Target dictionary if we look at the console log we have only two morph targets which are marked open and mouth smile so those are the values we can impact let's take uh here morph Target influences we will do nots.wolf 3D head dot morph Target influences of the dictionary and the value we've seen its mouth smile okay let's use this one now smile and I will set it to one now our character is creepy smiling but it shows that we can impact his face based on those values currently we only have Mile and mouse open which is not enough to enable lip sync when I prepared this course I was a bit surprised because I saw only Mouse mine and mouse open and when you look at the documentation of Ready Player me they say they are Oculus overall lip sync compatible which means they support all those vision and also those additional blend shapes but it was not in the model so what I saw going more deeply into the documentation is that they have get 3D avatar and you can add some parameters to have maybe a better quality of the model the JB file you generate but also for morph targets comma separated list of individual morph targets if we scroll down we can find morph targets and here you have the different options available and the one we want is Oculus vsems so they have an example uh but we don't need it we will just copy morph targets and Oculus visms okay if we go back here to download we can do copy Avatar link so here how it works we have our Avatar question mark morph targets equal Oculus vsems we wait a bit and generate our glb file don't worry we can use it so just replace it in your public and models now it's in the updated version now if we have a look at the current morph targets we have all the vision from Oculus I didn't add the mouse my but you could add both to understand what it is copy with them or for example now in this morph Target we will set vzmo to one and now if we look at our character it's doing a very pretty nice face it's a bit buggy because we need also to impact the teeth so let's go back to the code and it's teeth that we can duplicate it what is this shortcut I don't know so copy paste and impact their teeth okay and now what you see is a very good looking face of a o so when it feels big and has to say oh it will be this phase now let's load the Json file and try to sync the audio file with the data we have from the Json to do the lip sync const Json file will be use loader from reactory fiber we need to pass what loader we want to use it will be 3 dot file a loader and the path is audio crypt.json then we want the Json from the string so lip sync is equal to Json dot parse from Json file okay now we will add a use frame so our code is called at every frame what we will do is we will get the current time so current audio time is equal to audio dot current time then we'll go into all the data we have in the Json file a quick reminder it's mouth cures it's an array of value that we need to animate the mouth waste so we copy mouth cures and it will be for let E equals zero e is below lip sync dot mouthqs dot lens I plus plus so our mouth Q is equal to lip sync of math queues of I then we will try to see if we are with the current one so if our current audio time is over marked Q I name it Mouse Q so Mouse Q dot start because again it has e where is it here it has a start and end the value so if it's over or equal to the starts and Below or equal to the end then we can just try to log her mouth Q and the data is named value okay by the way we need to import three to be able to load three dot file folder here import three from three okay so now if we do nothing we have X but if we play the audio hi I'm Wawa Sensei welcome to my YouTube channel let's learn how to add lip sync to our react 3 fiber projects you can see we had different values that we don't know what it corresponds to because our vision are those sounds it's understandable like e is e so your mouse is doing e I is I so it it's corresponding to real sound and it generates the mouse shape accordingly but on a rhubarb Library it's a bit different if we go back to rhubarb Library we scroll and there is this beautiful section it explains for each letter so for a it's closed mouth for the p b and M so it's closed and a bit pinched and it's explanation for each of them so G is vowels like a a so it's ah so mouth is open so I used this and created an array well a dictionary for the corresponding letter in rhubarb library and the corresponding with them on Oculus lip sync uh way of working so here is what I ended with so a correspond to p p k k and okay maybe it's not perfect so if you want to try to fine tune it and find a better combination tell me and I will be highly grateful to you now we have it we know how to play with our morph Target we just need to play the correct one so we go back to our use frame first thing we want to do is to reset it before setting a new one so we don't play uh five sounds at the same time with our mouth so let's iterate through each key of corresponding array so object dot values of corresponding and for each on each value we will run the following code so we had uh did I don't delete it they use effect now it's here okay so we can take it remove it by the way we don't need it go to our use frame and on the value so here well it's corresponding value I know it's already the the right one here we will reset to zero and then so this is the reset we copy it here on my Q value we will play it if it's the correct one but here it's not on value we don't have it it's on corresponding of mouth Q dot value so we get the Oculus version from the rhubarb one here and here okay here we need to set it to one and maybe once we find one we don't need to Loop through all of them so we can add a break in our for Loop so now we have X because it's doing nothing and X is a closed mouth let's hit play hi I'm Wawa Sensei welcome to my YouTube channel let's learn how to add lip sync to our react 3 fiber projects so we have that nice animation when our character is playing let's switch to pizzas and try play audio do you like pizzas I mean real pizzas with pepperoni not the ones with pineapple who the hell eat Hawaiian pizza I don't know what you think but I feel it's pretty convincing let's do some Polish maybe we can play the animation we prepared when we are starting a new audio where is our use effects based on the audio here so if the OG well if the script is welcome we can set animation to greeting else we can set animation to angry if we stop it we can set animation back to idle we can also add a check in our use frame here if uh current or no if audio is post or if it stopped [Music] or end it then we can play back set animation to idle so it go back to the default State now we can try it and see what we have do you like pizzas I mean real pizzas with pepperoni not the ones with pineapple who Hawaiian pizza thank you for watching I hope you enjoyed this video if it's the case please hit the like button to have the channel be more visible to other developers don't forget to subscribe to not miss the upcoming tutorials if you still have some energy to continue your Learning Journey I recommend you watch this video here

Channel: Wawa Sensei

Views: 19,768

Keywords: threejs, three js tutorial, threejs tutorial, react three fiber, react three fiber tutorial, three js react, r3f tutorial, three js, three.js, pmndrs drei, pmndrs, webgl tutorial, react, threejs journey, threejs lip sync, react three fiber lip sync, rhubarb lip sync, ready player me lip sync, 3D model lip sync, lipsync, lip sync

Id: egQFAeu6Ihw

Length: 29min 13sec (1753 seconds)

Published: Fri Jul 21 2023