How to make Auto Lip-Sync in Blender 3.3+

hey Sam from 3D here in this video I will show you how to do automatic lip sync in blender first we need the audio from the clip I'll use blender video editor to export the clips audio whatever software you use it's important that you export in doubt wave formats on a new blender file I'm gonna click on this plus icon click on video editing and video editing now click on ADD movie and select the file I have the video that you've seen in the start of the video if you want to you can change the frame rate here I'm going to change it to 24 and if you click on play Same from 3D here you'll see the video playing you can also play with the resolution if you want to cut that extra space there if you want to edit the video a little bit before exporting the audio or the video itself you can select both tracks the video and the audio track and press K to cut now move it forwards okay again and you can select these pieces and delete just like that you have a cut I will show you how I do want to set how many frames this thing will take so let's go to the end and it takes 138 just type here one three eight and now we can just export this since I just want to export the audio I'm going to render and render audio select the folder you want to export these two and remember to change the container which is the format to pay now click on mix down and you're done you should now have an audio file with the voice you're going to lip sync we'll also need a free plugin called blender rhubarb lip sync to download a click on releases now download the dot zip that corresponds to your operating system to install it is the same as any other add-on click on edit preferences add-ons and install now select the dot zip file and just enable it I'll use this character from character Z you can get him and many others at 3D Dot design before using the add-on we need to manually set some pausing we're going to create some phonemes this character already comes with these liners for them but if yours doesn't the process is pretty much the same let's start by making bone groups bone groups are used to select multiple bones in a faster way click on the Armature move into pose mode by pressing Ctrl tab I'm now going to select all of the lip sync options but I'm going to deselect the box into the Running Man click on the plus icon and let's create a new group called face I'm going to change its color so it's easier for me to see and click assign we now have selected all the lip sync options you're going to need a legacy pose Library just scroll down and click on this new icon we're gonna call this facelip and now we need to create the poses for the library and this is how we're going to do this I'm going to click on select on the face here so we select all the pieces related to lip sync and this is my rest position if you scroll down you can see what the add-on is asking us with the bone selected we're going to click on this plus icon click on add new and I'm going to name this the rest that's our first pose now what I'm going to do is Select only AI move this all the way to the front now select all of the bones related to the lip sync click on the plus icon add new and let's call this pose Ai and now we need to repeat this process for all of the others move AI to zero move oh all the way forwards select all the face bones create a new pose and call it o I'm now going to repeat this process for all of the lip sync options that you see here we're using the Legacy pose library because this add-on hasn't been updated to work with the newer pulse Library so for now we're gonna have to use this one if you're not using our characters you probably don't have that slider to use I have this character here his name is snow he's free from the blender Institute this character has the phonemes if you click here you can see his mouth changing to the O shape so the main difference is that you need to do this manually when you're using a character that is not ours also as a side note if you have a character that has the new pose Library like this one has you can click on the shape and let's add a pulse Library add a new one call this o and let's make a new one let's say the F so click on F click here add new f and you can just continue doing this that way you can use the add-on that's right down here if your character doesn't have facial bones it probably has shed keys if not you can create your own you'll need to set up some simple drivers for a bone to control the shape kit since we are using the pose library now that I have all of these created just scroll down and you're going to assign each one of these to their respective places over here on some file click on this little folder and select your audio file and here you can select the start frame now we're going to select the face bones again clicking on face select and now just click on rhubarb lip sync and that's it all I did here was increase the end time and now if we hit play you're going to see him move it's kind of dull we cannot hear the audio so right click here go into horizontal space change this to the video sequencer click on ADD and let's add a sound just move it back to the very first frame and if I hit play Same from 3D here in this video I will show you how to do automatic there's a couple of things we can do first we can press n with my mouse over here and click on display waveform this might be easier for you to see I'm gonna make this a little smaller hey Sam from 3D here one thing that you can notice is that the jaw doesn't move a lot and that's because this character actually has a different bone for the jaw in order to access it I'm going to press n and right here on item I'm going to select face I'm going to click on the Jawbone get back and using this as a reference I'm going to see where I open my mouth so if I hear it hey it's about hey here so before that I'm going to press I to set keyframe and I have Auto keyframing on so I'll just move to where I say hey and just open the jaw after it's done just press alt R to close the jaw again and now we have a much better opening animation so now we just need to match this with the rest some right here that's good a couple frames back move it down we're gonna do this real quick and be back and after just a bit of manual work I'm going to hide this and we can see the results hey Sam from 3D here in this video I will show you how to do automatic lip sync in blender that's pretty cool right you now know how to make your character talk and bring some life into them I hope this video was informative for you and if you bring some life to your characters make sure to share with us in our Discord Link in the description below right next to the like And subscribe button if you have any questions or suggestions leave them in the description down below bye

Channel: ThreeDee

Views: 51,275

Keywords: lipsync, lip-sync, blender, 3.3, 3.4, 3.5, Blender3D, b3D, how to make, tutorial, newbies, beginners, beginner tutorial, easy to follow, open source, open-source, free to use, talkative avatar

Id: 6MN6eui4zpE

Length: 6min 28sec (388 seconds)

Published: Wed Mar 29 2023