Lip Sync - React Three Fiber Tutorial

Let's learn how to add LipSync on Ready Player Me with React Three Fiber and Three.js

We'll discover the concept of MorphTargets and Visemes

Live demo
https://r3f-lipsync-tutorial.vercel.app/

Get the starter pack here ?
https://github.com/wass08/r3f-vite-starter

Final Code
https://github.com/wass08/r3f-lipsync-tutorial

#threejs #r3f #lipsync

▬▬▬▬▬▬ Useful L I N K S ? ▬▬▬▬▬▬

My complete course to learn React Three Fiber
https://lessons.wawasensei.dev/

Rhubarb LipSync
https://github.com/DanielSWolf/rhubarb-lip-sync

Ready Player Me
https://readyplayer.me/

And the documentation
https://docs.readyplayer.me/ready-player-me/api-reference/rest-api/avatars/get-3d-avatar #examples-7

Convert MP3 to OGG
https://convertio.co/mp3-ogg/

Eleven Labs (TTS)
https://elevenlabs.io/speech-synthesis

Mixamo Animations
https://www.mixamo.com/#/

LeonardoAI
https://leonardo.ai/

React Three Fiber documentation
https://docs.pmnd.rs/react-three-fiber/getting-started/introduction

R3F Drei
https://github.com/pmndrs/drei

▬▬▬▬▬▬ T I M E S T A M P S ⏰ ▬▬▬▬▬▬

00:00 - Intro
00:31 - Starter Pack
01:01 - 3D Avatar
03:27 - AI Background
04:45 - Mixamo Animations
06:57 - Attach animations
11:04 - Text to Speech
12:37 - Generate lip sync
15:35 - Play audio
17:38 - Morph Targets
21:37 - Play lip sync
24:38 - Visemes
27:46 - Polish
28:46 - Final Result
28:55 - Thank You

▬▬▬▬▬▬ Support the channel ? ▬▬▬▬▬▬
Become a member to get access to awesome perks:
https://www.youtube.com/channel/UCBH-5ctALmmOrm4kczL3urQ/join

▬▬▬▬▬▬ Connect with me ? ▬▬▬▬▬▬

? The Discord Community
https://discord.gg/2wBhwchcWu

? Instagram :
https://instagram.com/wawa.sensei

? Facebook :
https://www.facebook.com/wawasenseiyt

?Twitter :
https://twitter.com/wawasensei

?TikTok :
https://www.tiktok.com/@wawasensei08

? My website :
https://www.wawasensei.dev