Ready Player Me x Unity Live Capture / Face Capture App

all right the goal of this tutorial is to um how to use unity face capture with red player me avatars so you can animate the face of your avatar using your uh iphone with the help of ar kit blend shapes that we have on the avatar um i will guide you step by step and first thing we will do is go into unity hub and starting a new project let's create a new project the version of the project should be 2021 or plus because we will need to add package by name for leave capture which is not supported in the prior versions of the package manager in unity i'm going to use 3d core you can use hdrp or urp if you want i want to keep it simple and i will call this face capture and let's create a project while the project is being created i'm going to go to [Music] docs.readyplayer.me address to download red player unity sdk so on the left side right button you will see unity avatar sdk let's click on download and here is the download url the version i'm using is 1.7.0 it's gonna take us to google drive and we can download this from here closing this window or we didn't download yet so let's do it again then and we got the package so unity has done loading the project let's click on our package and import everything inside the red player me sdk 1.7.0 into our project so now we can import our avatars from url into here when the importing is finished we will be greeted by the sdk splash screen let's wait for it and here it is for now i have this example url i'm just going to get rid of it let's go to red player me if you haven't created an avatar before you will see create avatar button and it is going to take you through the author creation steps if you done it before you will see enter the hop button so i already have an avatar i do not need to create one uh if you have multiple avatars you can go to my avatars tab and pick one i have this for now and i'm going to use it so clicking on this button and copying the url that there is and in unity oops in unity to this url or shortcode area i'm going to paste my url and load the avatar it's going to take a couple of seconds and it's done so i do not have a mouse so i'm going to try to focus on the head this looks good enough i will also rotate the directional light so i can get a better look at face 150 here on y axis and it looks good all right so we got unity red flame unity sdk we got a url for our avatar and import it and this part of the process is done the next step is to go to package manager and this package face uh the package we are going to capture the face with is not in the list of packages unity has in here yet so we are going to edit by name i'm clicking on the plus icon here and click on add package by name so you will see this small window here and the name of the package we are going to add is com. capture i'm going to write this in the description below as well if it's not visible enough so we are adding this package it's going to take a short while while this package is being installed and now we have it all right let's go guys spock too soon i guess all right so let's clean this and if i go to window i will be able to see leave capture menu here let's let's get our connections window and this is the window we are going to use as the last step when we are connecting to our iphone let's for now put it here we will create a server to connect to our phone when we are done with the setup and let's do the setup for um getting the connection to our avatar's face and its plan shapes and everything else in the hierarchy window we are going to create a take recorder game object it's under leave capture menu and let's create this this component is going to help us save the animations we are receiving to our avatars face so this is one of the important parts of it to capture the animations and under it let's create a new empty object and call it face device i'm going to move face device under face recorder and on to this object let's add a arkit face device uh component so it shows up right here if you search for ar kit face device you should see it and on this component you will see uh these channels which are blend shapes head position head rotation and eyes in this case i will only use blend shapes head head rotation and eyes and i'm going to uncheck head position because i do not want my avatar's head to move um away from the body uh since this is a full body avatar another half body avatar only like not a boost type avatar i want it to be still connected to the whole body itself so position is unchecked all the other buttons were checked in here you will see the face devices are asking for a face actor component so this is the component we are going to add on top of our avatar let's click on it and search for face actor it's already here and let's select that and we will see some other settings here this current pose we can ignore it when we start our um phase capturing process we will see some values changing here this the leave values coming from the phone and we have the same enable channels here let's disable position here as well just like that and this component is asking for a face mapper let's go to our project window create a new folder and let's call this face capture as well i'm going to put everything else we have in here and let's go into this folder and create it was called face mapper create menu leave capture on the top arkit face capture and mapper click on it it's going to create us a a new file with the mask logo on it let's remove new and let's call it just face mapper and now you will see it's asking for a rig prefab the rig prefab is our avatar but we only loaded into the scene and it's in the prefab yet so let's drag and drop it into our face capture folder and turn it into a prefab so we already have the instance here our face mask is here and drag and drop our avatar prefab into rig prefab area once you do it it is going to open up new options for us so it's going to try to map the pieces of our avatar into the values coming from the iphone which are head rotational position eye rotations and positions and things related to balance shapes so to be able to rotate the eyes with the bone positions let's search for our left eye let's add the right eye and movement is eye tracking we don't need to change anything here just yet if you are using of course another setting another model it might need some offsetting uh full body avatars already playing it does not for head position and rotation we will use head objects and these all are uh if you want to inspect uh these all are under the armature here in the avatar itself so these are the bone settings and now we have to capture the skin mesh render with the blend shapes on it we click on the add render button and it automatically captures the skin mesh renders we have on the avatar one is the head and the second one is the teeth and these have the both blend shapes all are compatible with the arkit blend shape name convention so we do not need to change anything as well they are already mapped so it eases up our work even more so we don't need to change anything everything is automatically captured but you can see pro down right is broke downright blend shape on the mode so we are also done here so take recorder here face device under it it needs face actor so let's drag and drop our avatar which has face actor in here it is done as well so here is our face actor which requires face mapper that we created let's move it in here and here is our prefab so let's check everything take recorder face device has uh face actor face actor is on the avatar right here face actor asks for a face mapper face mapper we created right here and it requires the prefab and prefab is here so our setup is pretty much done now here i will show uh i will put another video on the screen to show you how you can download the iphone application for face capture so unity has another application that helps you uh phase capture and transmit the blend shape values and eye rotations and head positional rotation into unity uh we are going to use this application i already have it and i'm gonna start it with face screen capture so i can also show it to you first of all i'm going to create a server click on companion app server make sure your phone is already connected to the uh computer you have our server is created let me start phase capture or first screen recording then i'm starting face capture once everything is connected you will see a blue button so uh before that i need to click on this start button here i was about to forget that so start that and you will see on your screen the grayed out uh connect button is now blue and once you click on that you can see your face and probably yeah the avatar moved down right now the um let me hold it like this so the light doesn't get there so since uh it influenced the whole bone structure avatar moved down like that i'm going to uh click on the head again and focus on it so yeah let's click outside so we don't get the bounding box all right so it looks like this as you can see tracks my face and i can stop this start the project so we can see it in action with full body animation as well so it goes back to full body let's focus on the head again it lags a little bit more but let me maximize this so we can see it full screen and i'm going to [Music] start the server again stop the server start it i might need to start application again at this point let's do that connect and yeah now while the game is running the project is running actually it can capture my face so yeah i borrowed an old mech for uh this project so it logs really bad right now and the fan is going crazy probably you will see a lot of uh you will hear a lot of noise at this point uh yeah this is this is pretty bad actually to demonstrate this i'm gonna stop this yeah let's better view it in editor mode let's go here and maybe i can close up a little oh not maximize go back to scene oh well i'm not a mac user i'm having lots of trouble here and maybe if i focus on glasses it could get closer yes it's exactly what i wanted so let's start the connection again and i will need to start the application again let's wait for it connect and yeah this should be it left right up down eyes are like that so i'll try to hold it like so i can see my face on the screen as well yeah this is how it looks um i will try to get better shots at it so i can actually show how the emotion is actually showing up when you do different things with your face yeah maybe hold it like this so oh don't don't like this yeah all right yeah interesting so uh one thing we can do is going back to uh take recorder component uh that we have on the take recorder object so if we click on start recording then it is going to start recording an animation so everything captured eye rotation facial blend shape values and the head rotation those are going to save into a keyframe animation and now i can stop it and i will stop the face tracking as well so now i have this animation right here this uh it's called is in a folder called takes right here and you can click into whichever the name of your animation was so this triangle logo animation is your animation file and it has all the um keyframes in it right here so in an animation controller you can use this animation you can create a second layer separate from the full body animations and trigger the facial animations depending on the events or such uh of your desire and make your character uh play the animation such uh such as you get something from mixamo along with the facial animations you recorded with your iphone i'm i am actually planning to make another um tutorial for it so i will try to capture some short speech and stuff like that and then use these animations along with somebody animations so this was it setup is a little complicated you might miss certain steps be careful with that but it's as easy as it gets um you will need an iphone and i think a mac as well uh make pc um that is all the requirements and unity 2021 plus and then you are good to go and thank you for watching

Channel: Sarge

Views: 27,697

Keywords: made with unity, unity, unity game dev, indie dev, game dev, gamedev, unity3d, devblog, game development, indie game, game ui design, indie game dev, face capture, unity face capture, iphone face capture, ready player me face aniamtion, ready player me, ready player me face anim, live capture, unity live capture, unity face tracking, iphone face tracking, ready player me face tracking, ready player me tutorial, unitr face tracking tutorial, unity face capture tutorial

Id: d1JhZooDHeM

Length: 19min 16sec (1156 seconds)

Published: Thu Nov 11 2021