>>Ryan: Hello! My name is Ryan Mayeda,
and I'm on the Product Team for Unreal Engine where my
focus is on virtual production. So as you might
guess, that's what we're here to talk about today. Specifically, we're going to
cover real-time facial capture with iPhone using our
new app, Live Link Face. To do that, I'm going to enlist
the help of a familiar face, someone you might recognize
from the UE5 reveal. She's going to act as
my avatar, and we're going to do the
presentation together. We'll primarily be taking
a tour of the iOS app, leaving other deeper
topics, like the animation and rigging setup, as well as
timecode and hardcore stage configuration for a future talk. We'll touch on them a bit,
but these are definitely areas we want to cover
later as folks get further into the facial
capture pipeline. All right, let's get started. I'll go ahead and
launch Live Link Face. And you can see that she comes
alive right away with my face driving hers through the app. Check it out. Nailed it. This may go without saying,
but I'm going to say it anyway. One thing to call out
is that the app uses the iPhone front-facing camera. So, my video is mirrored, which
in turn means that our head movements will look reversed
onscreen since she's actually looking at it. If I turn my head to
the right, she's also turning her head to her right. And I think you get the idea. Next, I'll put my marketing hat
on and make sure to be clear. The app is out now. And hopefully, everyone's
already downloaded it. If you haven't, grab it from
the Apple App Store ASAP. It's free. The only real requirement
is that your iPhone has a True Depth camera, which
is needed for the ARKit face tracking. The basic rule of thumb is
that if your iPhone doesn't have a Home button,
you should be all good. Our big goal for the app was
to take the Face AR Sample that tons of people have
already had good success with, including us at Epic, and
productize the iOS portion so teams don't have to build
and deploy the app themselves. This isn't necessarily
that straightforward to do. And we found that a
lot of our clients don't have that
expertise in-house. You know, wherever
possible, we want to empower people
to focus on creating the characters, performances,
and content in general. And that's what led
us down this path. The Face AR Sample is still
very much with us, though. And our old friend,
the kite boy, is still a great example asset
to start with even though he's sitting this presentation out. Let's get into the app. But first, a bit
of housekeeping. In order to use the features
we're showing today, there are few plugins to enable. First up is Live Link.
it might be obvious. Live Link is in the
name of the app itself. But the facial capture
stream is coming in over the Live Link protocol. So you're, of course, going
to need that plugin on. Next up is the Live
Link Curve Debug UI. So technically this isn't
required for the system to work. But this plugin is
super useful, especially if you're doing the animation
and character setup. It really lets you diagnose
the data coming in, figure out how it's being
applied to your character, look at the blendshapes,
that sort of thing. So it's super useful, and we
highly recommend having it on. ARKit and ARKit Face
Support-- again, pretty self-explanatory since
we are using the ARKit face tracking. But you're going to
need those on for sure. Take Recorder is
what we're going to do to record the Live
Link stream in Unreal Engine and get it into Sequencer or
work with it as animation. And last but not least is
the Timed Data Monitor. So, this is a new plugin you
may not be familiar with. It's in 4.25. It's super cool and
it's kind of something I was alluding to earlier, where
this is the timecode workflow is its own demo. But in this case, we're going
to use this new plugin just to visualize the timecode
coming in from the iPhone into on Unreal Engine. All right, let's jump
into the main screen. So, we have kind of
a social-style UX, very much inspired by other
apps that use face tracking. We felt like this would
make it both familiar and fun to people as they were
starting out even though we don't have any filters. So you can do them in Unreal. In terms of the main layout,
the main thing to notice first is the big green
"LIVE" at the top. So, green means you're on
air or streaming data, right? So my performance is being
sent over to her in the Engine. If we tap this, we can pause the
stream, so now, no matter what I do, it's not going to carry
over into Unreal Engine. This can be a handy
thing to do if you need to go offline for whatever
reason, take a break, whatever. It's just a tap away. Let's go back to live for
presentation purposes. Next up in the top right-- it's
kind of some related features, but a little bit more
oriented around battery life. So right now, we have
both video on, right? You can see me. We have the face tracking on. It's being sent over to her. If I tap this once, we're
going to toggle the video off but leave the tracking on. So the video tends to use up
a little bit more battery. It's something you may want
to turn off during a shoot even though you keep
the capture going. In addition, we found
that some actors maybe don't like to see their face
when they're performing, right? It can be distracting. So if that's the case, you
can toggle the video off. And then they can
just focus easier on their lines, et cetera. The last option here is both
the video off and face tracking off, right? So this is super
battery saver mode. Might be something you want
to toggle in between takes, in between setups, or if
you're taking a break, but keeping the actors in
costume or keeping them all rigged up-- that's an option there
to help prolong the life of the phone for the shoot. Let's turn everything back
on for our presentation. Next thing we'll
mention is timecode. So no matter what,
you're always going to see a readout of the
current timecode right there underneath the LIVE button. We're going to get
into this in more depth shortly as we dive
into the settings. But at minimum,
you can always see you're getting
timecode, what it is, and a little icon of
where it's coming from. Down in the bottom
right is face detection. So this is green when the
app is detecting a face. Obviously, it's
detecting mine right now. But if it loses-- if it loses
the face for whatever reason, then that's going to go gray. We can demonstrate this by being
safe, kind of masking up here. So if I put my mask
on, the app no longer recognizes my face or a face. And now we're offline. So even though we are
technically still streaming, the app isn't finding a
face to actually stream. So if I take this off,
we'll be back again-- picks up very quickly. But just note, maybe not
super quarantine friendly. Another thing to mention
is down at the bottom are slate and take,
bottom middle. So we're going to come
back to this later when we do some recording. But I just wanted
to call out quickly you can tap this
and modify the slate and take by hand
directly in the app. Obviously, in general,
it's the best practice to keep this aligned
and matched up to what you have in Take Recorder. But there is the option here
to set it yourself in the app directly. Also down here is the
Live Link Subject Name. And that's kind of a good segue
way into the full settings menu, which is up
at the top left. So let's jump into there next. So, we have a lot of settings. We're trying to keep this as
user friendly as possible. But we also want
to make sure we're offering a robust
set of features that help people ranged from
entry-level folks who may not touch these very much to
professional stages that are going to really get
into the nitty-gritty. First thing we'll hit is Live
Link and the Subject Name, which you just saw displayed at
the bottom of the main screen. So by default, the subject
name is the name of the device or the name of the phone. You can see my phone is
Pandaaaaaaaa with eight As. You can override it if you want. And then now we'll-- let's
quickly jump out and just show what this looks
like in the Engine. So I'll go out of
full screen, pop over to the Live Link panel. So we have a source, which
is the ARKit face tracking. Down here is our subject
name, which is Pandaaaaaaaa, matching exactly what we have up
here in the app for the subject name. And that's how you
make that link. Pop back into the full screen. Next up is network targets. So our main workflow
idea for the app is to multicast the
data stream of the faces to all machines in a
potential multi-user session. And the goal here is
to minimize latency. We've really designed
the app to excel in a sort of collaborative
virtual production with multiple machines. Maybe you have one
that's recording, one that's doing
VCam, one that's focused on stage operation, and
maybe you're scouting, right? So each of those machines is
getting the facial tracking data as fast as possible. And then we synchronize
with timecode. The network targets are based
on an IP/port combination. So in this case, I'm a
single-user, single-machine setup just with
my home box here. You can find your
IP really easily. Just pop open a command
prompt, hit ipconfig, right? So I'm transmitting over Wi-Fi. I'm going to look at
my wireless adapter. And my IP address
here is 192.168.1.17 on the home network. And that's what I've
entered in as the target. The default port is 1.1.1.1. You probably only
need to change it if your IT team tells you to. And if you want to enter into
a multi-user session-- you have multiple machines-- you're
just going to add a target, punch in another
IP address here. Cancel out of that. Close the thing here. Right, so next up is
the Live Link protocol. In Unreal 4.25, we made
some improvements here and added support for
fractional frames which improved accuracy and
helps avoid the possibility of duplicate frames. But this is also
something that you need to be aware of for
backwards compatibility purposes. You can use the app with older
versions of Unreal, right? So right now, I'm using 4.25. But if I wanted to use an
earlier project, like maybe a 4.24 project, then I can go
in here and change the protocol. The main thing to point out
here is that it's very specific. So you need an exact
match between the versions for the protocol. Otherwise, the
stream won't work. So because I'm in
a 4.25 project, if I change it to 4.24 here
and let that setting take hold, I'm losing the stream. Everything looks like
it's set up correctly, but I'm not getting any
facial tracking data. So that's something
to keep an eye out on. If you are using different
versions of Unreal, this is something
that you're going to need to make sure
has an exact match. So let's switch ourselves back
to 4.25, get the stream back-- here we go-- and then close out
our Live Link settings talk with the last item,
which is Live at Startup. This one's pretty
straightforward. You notice that when
we launched the app, we started streaming
right away, and that's based on this toggle. If you don't want
that to happen-- you want it to be, like, a
two-step process where you open the app and then
opt in to streaming, then just toggle that guy off. Next setting to go
through is timecode. So this is a big feature
from the pro standpoint. And we have three different
options for timecode. They kind of build in
complexity in specification. So first up is the
System Timer. So this comes from the
clock of the phone itself. So it may look like a
non-standard, weird timecode. In this case, there's
like a 94 in it. That's not something that
you usually see on a stage. And this is because
the system timer is based on how long
the phone has been on. So I haven't rebooted my
phone for a long time. That's why it says 94 hours. The other two options
are a little bit more of that kind of classic,
expected timecode format. And those NTP and Tentacle Sync. So we'll go through
NTP first, right? So NTP lets you synchronize
timecode with a time server. The default we provide is the
Apple one, the standard one, the same thing that
your phone uses to figure out what
time it is based on what Apple thinks it is. We've seen some productions
rely on using their own NTP server on stage. And they use this to
synchronize all the devices. So this can be a really easy
way to get time-of-day timecode. So note that right now
in LA, it's about 8:20. And that's what you see here. Last up is Tentacle Sync. So this is definitely
the most pro option. And this enables the app to
synchronize with a master clock hardware device on stage. And the way we do that
is with this device called a Tentacle Sync that
we've done integration with. So this is what a
Tentacle Sync looks like. It's a little lightweight
Bluetooth guy. And it connects
to a master clock. So I've got one here,
using the UltraSync. You can kind of see
this thing here, right? So, I know it's mirrored,
but the timecode on this guy should match what
you see there, right? 4:21:27. And the way it works is
that the master clock drives the Tentacle Sync. And then the Tentacle
Sync, through Bluetooth, tells the iPhone what
the timecode should be. So, the Tentacle
device is really cool. We actually use this
on our own shoots. And so one thing to
note is that if you have a shoot with multiple
performers, multiple iPhones, you should be able to use the
same Tentacle Sync to drive the timecode on all of them. You don't need a one-to-one
relationship between Tentacle Sync and phones. So that's pretty cool. Last thing I want to
show here on timecode is-- popping back
out into the editor. And I mentioned it before,
but the Timed Data Monitor here-- so, this is the new
plugin we were talking about. And this is kind of just to
show that the timecode is indeed coming through, and this
is what we're getting. So nothing up our sleeves--
timecode coming through. What you see on the phone is
what you get in Unreal Engine. So I'm going to actually
switch this back to NTP. I always feel a
little bit weird when it's not time-of-day timecode. It just looks odd to me. All right, back to the settings. So OSC is up next. We're actually going to park
that and come back to it. OSC is for remote
control set of features that let you administer
the app externally. But that's going to
be our grand finale. So we'll look out
for that at the end. Next up is Stream
Head Rotation. So, this is something
that you guys want to turn off when you're
in the mocap view. So if I turn this off here-- so now she's very still. Like, none of the head
rotation I had before is coming through. When you're in a mocap suit
or you have a body mocap solution available
to you, you're going to want to get the head
rotation from the actual body mocap. If you have them both on, then
the data you get from ARKit is going to fight with the data
from the body mocap solution. And you're probably
going to get bad results. So if you're combining the
app maybe with like a head rig or something like
that, you're definitely going to want to
turn this guy off. But since I'm at my desk, I'm
kind of more streamer style. I'm going to leave that on. You know, it kind of gives
a little bit more life in this scenario. You can also see how much
I bob my head when I talk, which is kind of embarrassing. But I will leave that in. Next up are some
display options. So first up here is
the Preview Mesh. So if you turn this
on, you're going to get a little bit
more direct feedback on what ARKit is seeing, right? So it sees my face. You can see the exact
placement of it. And it's more an
obvious sign that we have data coming through. It also looks a
little bit creepy, so I didn't want to do the
whole presentation this way. The Record Button lets you
toggle the record button on and off. This is something that
we'll come back to later when we hit recording. But the idea here is that
if you were administering the app externally,
then you don't want it so that the actors or
whoever accidentally bump the record button. So you can just take
it off entirely. Let's leave it on for now. The Blendshape
Data is more for debug. So if I turn this on, I'm going
to see a bunch of data here. And it's actually a good
way to show this is, like-- I don't know. Hopefully, it's going
to come through. But down in the bottom right
is the head yaw, head pitch, and head roll. So if I move my head around, you
can see those numbers moving. But if I go back and I
turn head rotation off-- right, so now you can see
those are all zeroed out. No matter what I do,
they're zeroed out. That means the app is
not sending that data over to Unreal Engine. So let's turn this guy back
on, turn blendshape off. Yeah, so Take Recorder-- this
is the kind of format that's displayed. Like, there's two options here-- you know, Slate and-- there's Slate & Take
and Filename. So the standard here
is Slate & Take. It's the kind of
more editorial style. You know, in this case,
we're 49D, take 1. But you can also change
it to the filename. So this is more of the
pipeline-centric view. You know, some people
really don't want spaces, so there's an
underscore for you. But this reflects
the name of the file that's going to be later
extracted from the phone. But I prefer the editorial file. And we'll leave it as that. And then the overlay
is that we've had it on the entire time. You know, so it's all
the stuff around here-- "LIVE," the different toggles. But you can also
set it to fade out. So if you don't
want to see that-- so if you wait a
couple of seconds, then that overlay
is going to go away. Kind of a better
example of this is when you have video off, right? So this is maybe the
most likely scenario that the performer
is going to see. So we have the video off. We have the overlay off. But we still let
them see what time it is and also the battery. You know, oftentimes
the performer is the one that will catch that
the battery is almost dead, kind of like my phone now. But I'm going to leave that
on because it's kind of easier or better for the demo. And the last thing to mention
is the reference video. So we do record reference
video every time you record. And there is an option to adjust
the quality and file size. We are going to show that stuff
right now because actually we're going to record a take. So with that, let's
record a take. Let's turn the video back on. All right. So we're going to record
a take in Take Recorder and from the iPhone manually. So first thing we're
going to do here is switch over to
Take Recorder. So we've already set our slate
and take to be the same thing. We always want the
consistency between the two. And in this case,
we're just going to record a Live
Link for us here. So now I'm going to hit
Record in Take Recorder. I'm going to hit
Record on the phone. And this will be our take. OK, this is a demo recording
that I've initiated manually in both Take Recorder and
on the iPhone, and yeah. We'll cut. And now let's take a
look at it in Sequencer. So I didn't end up recording
audio in Sequencer. But it is recorded in
the reference video. It's not really set up to
record audio on my computer, but let's just play this back. You can see the
action going here. Maybe I should have
recorded a shorter take, but my expressions at the
end also came through. And now we will pop into the
phone to take a look at it over there. So, Live Link Face has this
take browser feature as well. So in the bottom left,
you can pop into here. So this is showing all the
takes I've already recorded. Not surprisingly,
they're all of me. You can browse them by all
takes, by slate, or by day, right? So you can also do a search. So we know we were 49D. So if I type "D," I
can jump over to mine. And here's the take we just did. The video, as I just
mentioned, has audio in it. But it also has
timecode embedded. And the goal here is to
take the reference video and be able to line it up
to the recorded capture. So let's play it
just as an example. [VIDEO PLAYBACK] - OK, this is a demo recording
that I've initiated manually in both Take Recorder and
on the iPhone, and yeah. [END PLAYBACK] >>Ryan: All right,
it's beautiful. Another thing to
note here is that you can transfer the takes off
of the phone using the iOS Activity Views. So if I hit the
blue arrow up here, it's going to bring up a
bunch of different options that will zip up the data
and potentially send it over. You know, AirDrop is generally
the best and the fastest if you have a Mac. But we also have
some other options. So you can do
Google Drive, Box-- you know, there's a bunch
of different options that let you upload the data
as well as just transfer it over AirDrop. But AirDrop is
generally the fastest. Next up, we're going to
take a look at an example take on disk. So we'll kind of cheat,
but just looking at a take that I've already
moved over, right? So for each take,
you're going to end up with a series of files. Like, there's two
main ones, really. So first is a video, and
then the second is a CSV. First thing we'll show
is that video, right? So, I actually have
that open already. So this is an example of
the reference video you get. I'll play it through now. [VIDEO PLAYBACK] - Here's an example recording to
show that it has both timecode and audio. [END PLAYBACK] >>Ryan: So I'm
going to disable the-- let's mute me. I don't think we need
to hear that again. But it's also meant to show here
that we are getting timecode in the media that come through. Yeah, the QuickTime
itself uses a JPEG codec, so there's no
temporal compression. It's frame-accurate. And again, goal here is to
take the reference video, let you line it up to the
recorded capture for reference. So it's something
that your animators may want to use,
or just something to check that things are
working the way you expected. The other thing that you get
out of the take data is a CSV. So the CSV here shows all
of the blendshape data that's recorded. So it's kind of like a
raw recording or a backup for what you might
get in Take Recorder. So let's make this
a little bit bigger. Right, so you basically get for
each timecode and fractional frame, which you can see here-- you're getting basically all
of the raw blendshape data. This is all the same data that
you saw in the overlay earlier. And so the idea here
is that this is just the reference, the backup. We don't necessarily have
a native importer just yet. But it is possible
to take this data and write a Python script with
Unreal Python API to import it. And this will let you
basically create a Live Link track as if it was
recorded by Take Recorder. In the future, we'd like
to have a native importer. But for now, we're just
preparing the iOS app more functionality later. We've also seen
some people start to automate some of the
pipeline tasks with the CSV. So your mileage
may vary, and it'll be interesting to see
what people build. So now we've reached
the grand finale. I'll close out the CSV,
pop back in to the editor and go full screen. And we're going to close with
a demo of how a Live Link Face can be remote controlled
by an external application using OSC. So to do that, we'll
revisit our OSC settings. It's already enabled. And OSC stands for
Open Sound Control. It's a common command interface
popular in virtual production and, honestly, pretty widely
used across a whole bunch of different industries. It uses the same IP address
and port-style setup as Live Link for
network communication. And the Listener
section here has to do with how the
iPhone app is listening for commands from the OSC
server and then acting on them. We conveniently display the
iPhone's IP address here because you're probably
going to have to enter it into the control application. And this is a two-way setup. We can also go the
opposite direction and send messages out. To do that, you're
going to need to set up an IP and port for the target. And this could be
useful for maybe more of a one-man-band-type scenario
where the performer wants to initiate something, like
maybe the performer wants to hit Record on the phone. And then that triggers
recording on a whole bunch of other devices outside. You know, both
directions are supported. But in our case,
we're going to demo it with a separate remote control
app talking to the iPhone, telling it to record, and using
the right naming convention. So the last thing we'll
toggle is the Record button. So we're going to turn
the Record button off just to show how we record a take
from a remote app on the phone without interacting with
the phone at all directly. So let's bring up our remote
control app called Switchboard. So, Switchboard is a PySide app. It was developed by the
Fortnite Shorts team. So if you saw the SIGGRAPH
talk they did last summer, they showed an
early version of it. And we've kept
working on it since. This is the current state,
kind of a preview of something that we hope to share and
release as part of 4.26. But in the meantime,
it's the best way to visually show OSC in action. It's a little bit
more productiony compared to me just sort
of typing in OSC messages at a command line. So we thought this would
show a little bit better. Quick thing to note here
is the IP address. So this is, again, the
IP address that I got out of the iPhone app-- plugged it in here, and this
is where the messages are going to be sent to by the app. You know, for a future
demo, we'll probably start to touch on
other things here like the multi-user
aspect of Switchboard, but that's for another day. The main thing I want to show
here is just changing the slate and take, right? So this is 49D, take 2. This is the take after the
take we just recorded earlier. I'm just going to go
ahead and change it to something different, right? So 6D, take 1. And you can see that it
immediately updates the iPhone, and I haven't touched
the iPhone at all. Next, I'll just go
ahead and record, right? So here we go. This is a take that we
initiated from Switchboard. It's recording me,
and we're going to take a look at it right now. All right, so now we will
pop back onto the phone, jump into the take browser
here, and scroll down. We can see my new take
6D, or slate 6D, take 1. And here we go. [VIDEO PLAYBACK] - This is a take that we
initiated from Switchboard. It's recording me,
and we're going to take a look at it right now. [END PLAYBACK] >>Ryan: So, I didn't think
I could top my earlier take, but I somehow did. And in any case, this
concludes the presentation. As a reminder, the app
is available and free in the Apple App Store. With this app,
we're really aiming to make facial capture easier
and more accessible to creators going forward. And we're super,
super, super excited to see what you do with it. Thank you so much for
watching the presentation. I really appreciate it. We have a Q&A up next. And I'm looking forward to
interacting with you directly online.
