AI Characters in Web Browser using ThreeJS and Reallusion Character Creator | Convai 3JS Tutorial

hey what's this video about welcome to the tutorial video in this video we'll explore how to bring reallusion characters powered by convey to the browser using three.js and react let's begin hello and welcome to our tutorial on integrating reallusion characters into 3js with convey you can add AI functionalities to your reallusion characters enabling them to talk to you or other characters in your game perceive the world around them and perform really cool actions making it easy to create your own AI powered metaverse running in the browser in this tutorial we'll prepare a reill usion character in The Character Creator tool then we'll bring it into blender and then finally connect it to a m that we create on the convey website we'll finish up by integrating it all into code that can be deployed let's begin by importing our character in reill usion character creator 4 we're ready to export so go to the edit menu and select export fbx closed character and then select blender next we'll select mesh under fbx options and then remove the check mark for embed textures now we can click the export button and pick the folder where your project is located we'll need some animations so let's explore the animation section in the left column of re illusion for lip sync there will be two animations required one is the idle and the other is for speaking let's use the search search bar and type in talk select the stand talk. imotion animation and drag it onto our character click the play button below the character to test the animation that looks pretty good we'll export this as well so once again navigate to export fbx close the character and then blender this time we'll select motion under the fbx options and select current animation and then deselect first frame in bind pose now click export and save it as talk to your project [Music] folder now we just need to add an idle animation let's search for idle in the search bar and select any of the idle animations listed and drag it onto the character once again we'll use export fbx closed character and we'll do all the same things as the previous animation and save it to your project folder as Idle we're done exporting all our files from reill usion so we'll now check everything in blender inside blender we'll prepare to import our character files first we'll need to install a couple of plugins these can be found on GitHub under ccore blender tools by sup you'll find these files under releases in the right column the first one is CC blender tool and the other one is baking add on after downloading these files you can install them in the add-on section of blender preferences make sure to enable them before you leave the preferences section to show the menu you'll need to press n as in Nancy and then select the pipeline tab next click import character and select Camila from your project folder once the character is imported it should look something like this this format is not compatible with web so you'll need to convert the material by selecting basic followed by rebuild basic materials and then click the rebuild basic materials button now we'll select the bake Tab and click on the bake folder here we'll need to create a bake folder in our project once that's set we'll click the bake button to bake the files after all the files have baked we'll select the character's hair in the settings we'll change the blend mode and Shadow mode to Alpha clip we'll do the same for the eyebrows now let's select clothing and select Alpha clip for blend mode and opaque for shadow mode once these are updated let's convert the file for use on the web to do this we'll go to export gltf 2.0 and then deselect the animation option in the right panel then open the data section to ensure that compression is selected we'll then save this as a glb file in our project folder let's validate that our file is working in the gltf viewer as you can see everything looks fine next we'll go back into blender and open our animations clear the current project and let's import our animations these include the talk and idle animations that we created earlier here you can see the animations we'll delete the unnecessary items from the collection we just need the Armature and pose next we'll need to adjust the timelines to match what the imported animations require in this case we'll change it to 1,100 now let's look at the dope sheet down in the lower left corner of blender change it to the action editor here we can rename this to the animation that we are editing idle then we'll press the push down button now we'll add the talk animation by importing it in the same way as you can see it imported the entire model just select this and delete the hierarchy let's go back to the action editor and you can see that we have our talking animation we'll rename this to talking and once again press the push down button if you expand the hierarchy you'll now see that we have both of the animations once again we'll use the gltf 2.0 export option expand the data hierarchy to select compression before you save the file to your project this completes the basic skeleton setup for our model now we'll move on to set up the brain powered by convey let's get started by creating our convey character head over to convey. comom and sign in so that you are redirected to your dashboard click on create a new character and then give it a name next choose a voice we'll pick Comm us F and voice for this character now for the fun part give the character a backstory we'll say that CA is a researcher and space Explorer when you're ready click create character once it's done loading you can talk to your new friend you can give your character more personality by selecting from a variety of characteristics or providing a custom catchphrase if you want to fine-tune your character's personality head over to the personalities trait Tab and use the dials to make them as feisty or agreeable As You Wish when you're happy with your creation head back to the character description Tab and copy C the character's ID and your API key we'll need them both to set up the convey web SDK so let's Now set up the required code for the code the dependencies we'll need are react 3 fiber react 3 Dre lver for debugging low dash for throttle and the three.js package for more details you can visit our GitHub page under 3. js- world- tututorial here you'll find the main branch in the tutorial branches there's a link provided in the description of this video now let's create the react environment to set that up we'll use npm create V at latest add a period as shown here to bootstrap your react project to create a 3D space that runs in the browser we'll be using react 3 fiber which is an adaptation of react check out the documentation on fiber to find out more about how to use it and to see some interesting examples okay let's start building our project type npm run Dev to run the project on Local Host so this is our environment setup as you can see it creates a 3D space running in the browser the setup includes a file called app.jsx this includes the definitions for the keyboard controls that you can see here [Music] the file also includes the experience component which sets up the gravity for the Rapier physics engine so this is a wrapper for around anywhere where we want to Define physics properties here you can see some definitions for ambient and directional lighting the sky and the grid components come from the react 3 drag Library this creates the sky box and the grid component that you can see here this rigid body will be used to create a fixed platform it includes five cuboid colliders [Music] these can be turned off by removing the debug argument as shown here now we'll comment out orbit controls and enable the convey FPS control as you can see now we're able to move around the space as in a firstperson shooter game okay now we can add our characters we'll add Camila first we'll need to convert the glb files using gltf react 3 fiber there's a link to this tool and the setup files in the description below this will provide some boilerplate code for mapping out the nodes and materials copy and save this code into your local [Music] folder in this case we'll put it in our public folder and then add the appropriate folder and file name to your Camila function so from that file we'll extract the nodes materials and scene on this next line we do the same for the animations next we'll extract the client from props which is passed from the app.jsx file to get the client you'll have to use the use convey client hook which takes an argument for your API key and character id we'll also use the use animations Hook from react 3js and we'll extract actions and mixer which we'll use for mixing our animations this function shows how we fade in an animation and once it's done we then fade it out in this section we're changing the animation based on the state of the client so if the characters talking we set the animation to talking if it's not talking we set it to idle next you can set up lip sync by using the use lip sync function for real illusion characters this takes in the client character reference nodes and [Music] scene you can also use use head tracking which takes in client and nodes this enables the character to face the camera whenever she starts talking you can also import crowd characters from the re illusion website they have some examples that you can download for free like this one to grab some animations just head over to motion and you can look for animations to download once you're finished downloading the files go through the same process that we did for Camila so in this case we end up with a max. glb file for the character and Max animations. GB for the animations if that's successful you should see something like this these characters have the advantage of being pretty lightweight so they can be used directly on the web and they also include morph targets so that the lip sync function should just work if you'd like to try RPM or Ready Player me characters you'll just have to make sure that you download the the glb files with morph targets check the documentation for how to do this for RPM characters you'll need to use the US RPM lips sync. jsx file here you can see we have the same environment but now with a Ready Player me [Music] character hey let's wrap up the video sure it's time to conclude the video thanks for watching and remember to check out our description for links to the products and resources mentioned in the video feel free to reach out to us on Discord with any questions or email us at support convey. comom [Music]

Channel: Convai

Views: 6,155

Keywords: convai, ai characters, virtual world, 3d AI avatars, game development, chat with ai characters, create ai characters, lipsync, generative ai, convai tutorial, AI, AI NPCs, Smart NPCs, Intelligent NPCs, Virtual Characters with AI, AI World, Ai character in games, ai actions, ai animations, ai metahuman characters, blender, metaverse, web browser, ai online, reallusion character, reallusion character ai, threejs, react, AI online project, ai web project, 3d web project

Id: 5tYyeXPo5vg

Length: 16min 9sec (969 seconds)

Published: Wed May 29 2024