How To Create Realistic Lip-Sync Talking Avatar With AI?

hello everyone so I've discovered this pretty cool tool called video ret talking it's an audio-based lip synchronization tool for creating talking videos and animated avatars you can take any video clip of someone talking and use this AI to analyze their lip movements then you input a different audio file and it generates a new video with the Avatar or person appearing to speak that new audio using the original video as the source for the lip sync animations it's an incredibly handy tool and I've seen a lot of people asking about lip syncing capability so I wanted to try this one out video ralking was released a few months ago but it didn't get a ton of Buzz initially luckily they provide a really convenient Google collab option you can run short clips right there in collab or use their Cloud API demo interface no need to install anything locally just click the view and run install buttons and you can easily generate talking avatars or real person videos of course be very conscious about not using footage of others talking without proper authorization for this demo I'll just use the test data they provide and some of my own recorded speech audio files you can install video ret talking locally by following their instructions cloning the project creating a cond the virtual environment installing the required Packages Etc it's a pretty straightforward setup process outline in their repo to run it locally you use a command line interface specifying your input video and audio files along with instructions for the desired output files and locations the AI will then generate the new lip sync video combining those sources but in this demo I'll just go through the collab and replicate options since those are way more userfriendly than dealing with local setup issues whenever I cover new AI models requiring local installs I get tons of comments from people struggling with environment setup problems on their machines so Cloud options are ideal when available kicking things off by running their collab install cell huh looks like the setup is stuck there all right let's not waste time uh Google collab can be finicky sometimes instead let's try replicates playground replicate has pretty smooth interface and API where you just input two things the face video file for the talking head reference and your new audio file that you want to lip sync to that video you can use pre-recorded speech text to speech audio clips anything it's extremely easy to run right in their playground no setup required you you just test and generate outputs on the Fly way more convenient than installing locally for example I can use their test data video clip of this guy talking as the visual reference then I'll input one of my own pre-recorded audio files here let's give it a play first you know those quirky paperbacks you find tucked away in the new age section of the bookstore the ones with corny covers featuring crystal balls and allseeing eyes well this one Takes the Cake the future I see by Rio tatsuki is a real trip folks this Japanese cartoonist claims she's been having prophetic dreams since the 70s and routinely jotting them down like some kind of Supernatural dream journal keeper now I am usually pretty skeptical of Doomsday premonitions and apocalyptic Visions but this book is on another level okay got my third 37 second audio clip loaded in now I'll just hit the generate button and replicate will work its magic animating those lip movements to sync with my audio using their AI model replica.com is honestly a fantastic resource with a ton of capabilities pretty much every latest AI model you can access through their platform and apis the playground is just a free demo but they have paid tires if you need to run larger scale jobs through their infrastructure not sponsored or anything I just really like using their tools to easily test things out all right it's done generating let's see how well it synced up you know those quirky paperbacks you find tucked away in the new age section of the bookstore the ones with corny covers featuring crystal balls and allseeing eyes well this one Takes the Cake the future I see by Rio tatsuki is a real trip folks this Japanese cartoonist claims she's been having prophetic dreams since the 70s and routinely jotting them down like some kind of Supernatural dream journal keeper now I am usually pretty skeptical of Doomsday premonitions and apocalyptic Visions but this book is on another level you know those quirky Pap wow that's really impressive the mouth movements track my speech so naturally the lip syncing quality is way better than the old automatic one on we's stable diffusion extension sad talker for creating talking avatars the motions and mouth shapes seem to adapt seamlessly to my audio file using that original video as the visual reference it produces such a lifelike talking Avatar effect I definitely recommend checking out video ret talking if you need to create lip synced talking videos or avatars just be sure to have proper right start authorization for any footage of people you use their GitHub repo links out to this replicate demo and Cloud API option so that's a quick intro to this awesome AI lip sync tool really powerful tech for easily animating talking characters from any video Source I'll include the links in the description all right that's it for now have fun with this talking Avatar and I will see you in the next video have a great day everyone

Channel: Future Thinker @Benji

Views: 3,173

Keywords: Video-Retalking, lip-syncing, AI-based tool, talking videos, animated avatars, Google Colab, cloud API, installation, Replicate's playground, mouth movements, lifelike effect, lip-synced talking videos, Replicate.com, GitHub repository, AI lip-syncing, ai avatar generator, avatar, artificial intelligence, Lip-Sync Talking Avatar

Id: QlqCWy5hMVA

Length: 6min 32sec (392 seconds)

Published: Wed Apr 03 2024