My Rokoko MOCAP Guide for Unreal Engine and MetaHumans

hello my name is citizenmeta1 today we're going to be doing some full performance capture recording the face and the body directly into rococo studio after that we're going to bring this into unreal engine and apply it to our metahuman hello my name is citizenmeta1 today we're going to be doing some full performance capture recording the face and the body directly into rococo studio after that we're going to bring this into unreal engine and apply it to our metahuman hello my name is citizenmeta1 today we're going to be doing some full performance capture recording the face and the body directly into rococo studio after that we're going to bring this into unreal engine and apply it to our metahuman hey what is up today i'm going to be making a very long video um kind of documenting the process of how i'm working currently of doing a full performance capture for a meta human in unreal engine 5. and performance capture as far as like a term in the industry i've learned means that we are capturing the face and the body at the same time and that is a specific scenario where you're probably not doing backflips and whatnot um and you're going for more of like a natural actors performance versus doing like flips and punches you don't typically record the face or even the hands really during that stuff because it's very physical and you would then capture it separately or animate it separately so it's already a long intro but let's get into it i'm going to go through some of the little details of just generally this workflow mocap and whatnot getting it onto a metahuman and then specifically for rococo people that are looking to get into this or doing it already the little things that i've learned along the way so i'm going to start by going and getting the suit there's a couple little things about the suit and the gloves that i just want to cover that i've learned and if i'm wrong would love to know how to do them better so let me get that suit okay so we have our stuff here a lot of batteries talk about that in a second it's pretty important to this whole workflow so we have the mocap suit itself pretty straightforward you just put it on not really a lot to talk about however the nuance comes when you're powering it and you're using it with the gloves because the gloves are kind of a new addition to the whole pipeline so behind it is the power um center or hub or whatever so that's in here and this is like what's going to connect to the wireless and it has a little cable coming out so you'll see we have a bunch but there's this one cable it's a usb cable here i'm not going to zoom in on it but it's a little usb cable and so you need to power the suit from a usb power bank and they won't send you one so you have to go fend for yourself and get uh your own power bank so that's one of the first things about ricoco that you need to sort out so there's a little pouch here there's a pouch and in it is a battery and what i'm currently using is an htc battery bank that has a usbc and usb a i think is what it's called and this is actually for the htc vive wireless adapter that's these batteries and they're really nice and they're smooth and most importantly they fit perfectly in the little pocket that's back there so you can use whatever battery you want but that's what i'm using so that's step one um so if you're just doing the suit that's enough but if you want to add the gloves which i recently started doing you need to power those each as well so let me grab these things so these are the rococo smart gloves not a great shot of them or whatnot but uh they work really well they have little uh power banks on them as well and wireless transmitters i imagine or something like that and these actually need to get plugged in through the suit and powered individually is what i've found so what ends up happening is you take this very long additional usb usbc cable this plugs into the glove like not a great shot of it but this plugs into the glove and then it threads all the way through the suit back into here so this short cable here or this longer cable is for the powering the suit and then you'll see i have these two cables here these power the arms so you're gonna plug these in like this basically i don't know if this is the right of the left one but you basically plug in the glove like this through one sleeve and then you're going to plug in the other glove like this and so they're both plugged in right and powered in they're kind of dangling there i think that's okay to do they seem pretty strong and then you have to power each glove individually because they really take a lot of power so i originally started doing this with this anchor battery that has two outputs on it and it didn't work very well it's hard to say the power ratings the power draws i don't really know but uh shout out to jonathan winbush who recommended me these these two anchor batteries which are very small and they they go in the back pouch really nicely and they do give enough power each to power the gloves so that's pretty nice so what it normally looks like is i'm going to plug in the battery plug in the body into the battery that's in here again it's the htc one okay so i just plugged in the battery for the suit and i guess i could show it this thing is going to turn on you see the head sensor turned on right there uh and this little thing is turning on here and eventually it's going to be green and then blue i'm not going to zoom in on it if you ever cocoa stuff you'll see it anyway you get a blue status light and that's what you want it means it's on five gigahertz which we'll talk about in a second too and then we're going to plug in each glove individually with these little anchor batteries here so battery per one it's kind of annoying you would think you could use a triple bank i don't find that to work maybe other people have i'm not sure but uh there it is you'll see that this glove is on and connected has green which means it's 2.4 gigahertz which is fine which you'll see and then they're both on and both connected so that's the first step it took me longer than i would like it to uh to actually get that up and running because they don't send you the batteries but then this whole thing just zips up like this i gotta untangle the head sensor hold on and then here's the headsets are flopping around so we have the body turned on we have both gloves turned on and you'd be checking in rococo studio to see if they're connected or whatnot i know they are because i've been testing all day but that is how you have to get the rococo suit set up so power the rococo suit itself its own battery big one the last forever i have i almost never charge these things and then i have a battery per glove and again they last for a very long time but just getting that all sorted at first because if they don't give you the batteries like an xn suit would uh would do um one of the first hurdles to getting it up and running so the next thing that's interesting about the rococo suit and it's the same for exams and i it probably similar for perception neuron is that you have to connect this to your computer right okay so uh if you're just using a normal router um that comes with like your you know your isp whatever like from verizon or whatever is what i have that doesn't work so well and if if it's in like a different room or a different floor it really degrades the performance of the system so what i went and did is i went and bought an asus router which i can link to down low and i'll maybe i'll take a shot of it or something like that over there but it's own dedicated router and that gets an ethernet cable directly into it then i ethernet cable from the router to my computer and then that router which is a wi-fi router i can manage the connection to the suit and then the connection to both gloves so you're connecting three things to this wireless router um through wireless the suit two gloves you're connecting directly to the computer from the router and then you have your phone which is a five gigahertz um connection you can technically do five gigahertz to the suit and the gloves however i've found that they are working for me best with the gloves and 2.4 gigahertz which is still enough bandwidth still fast enough and i forget if the suit is five or four i'm sorry five or two but in either case it's this kind of balance dance and depending on your comfort level with networking which my level with networking comfort level is very low i don't like doing this kind of stuff that took me a while i tried to get this to work i think it was years ago i was working with this and using my stock router and the router was like way far away on like a different floor and that connection was pretty bad the router now is literally five feet behind this camera right now so it's like point blank in the same room and it's called a dedicated access point nothing else is on that wi-fi network and again it has its own dedicated uh one gigahert up and down i don't know if that matters in this case but very fast internet connection nothing else is on it my kids ipads aren't on it it's just the phone just the suit you really need to get that up and running before any of this works together so those are the two main setup points for rococo getting the power set up for this was a little bit of a thing because it doesn't come with it getting the router set up because it doesn't come with a router like exon suits come with a router and they kind of show you how to do that explicitly with the ray hardware uh rococo users we kind of have to like borrow from what they're doing and use the exact hardware so powerful wi-fi router three batteries okay we're kind of uh moving along now so this is a bit niche again not everyone has access to a techno props helmet but it's the one i'm using if you saw my other video you'll see that i have changed my foam layout here so i'm actually thin foam for the top and the top sides thick foam for the front because that's kind of kind of comfortable and then medium foam for the back and so i'll show you the fit now is like this and i don't know if you saw the other video it was sitting much higher on my head because i had thicker foam all over i was like let's just put the thick foam in and it was not sitting well now it sits like really like a normal helmet before it was very high and i was like oh this must be how it works but this is much better and i've also tweaked the arm up so that it's more in front of my face like that and so when i tighten this thing down with no weight or anything on it it's sitting i'm just testing my my eyebrows uh it sits way better so like i was saying in the other video getting the fit right for your helmet is very important and everyone has a different helmet rig you might be using standard deviation might be a skate helmet with like an action mount i have i have one of those two set up somewhere i just don't use it anymore um but getting this fit right with techno props or your standard deviation or whatever helmet you're going to keep iterating on that and testing different things out uh specifically with techno props it's really a battle of configuring the foaming which is you would think it doesn't make a huge difference but honestly switching the foam is like night and day with the balance so i'm going to put the suit on and uh what we're going to be doing that's kind of unique or proprietary or specific domain specific to ricoco is that we're going to be recording the body the hands and the face into rococo studio so completely without unreal engine we're recording the entire performance i'm not going to live stream that into unreal engine and try to record that because rococo locomotion data you need to kind of clean it up in rococo studio similar to an x-sense talking with this helmet on similar with xsens you don't really live stream that to unreal engine you're going to want to record it into rococo studio into um xs and then do hd processing rococo is the same thing you can get ok raw data but you get much better if you process it so the big picture difference here in my first test of it is full performance capture face and body gets recorded to rococo studio clean it up and then we send it to unreal engine to retarget and then clean up again in sequencer and whatnot so i'm gonna hop in this suit and record a quick take okay okay so yeah getting into mocap suits always kind of the thing so i'm in the suit now and again with the gloves which is new um typically with rococo stuff if you don't have the gloves there's a sensor on the hand here but there's also a sensor on the glove so like how does this kind of work so they showed this in other videos i think it's worth showing again the the prescribed method of this is to just simply take off the hand sensor and you just roll it back and it's just chilling there so you kind of just have like two sitting there which feels kind of weird to do but then you just put the glove on over it so that's something i think that was worth repeating and showing over and over again to rococo people i remember thinking about that when i first got the gloves i was like how does this all work and i found the video there is just one video on it though so i thought i would echo it and put more out there but yeah so i have both of them rolled back now and these are just floating luckily the connection is very strong it's like especially um tight the connection so you can really hang the glove from it and uh doesn't seem like it's hurting and hopefully it's not hurting it and then the glove just as i put it on off camera uh goes on like this really simple nothing too much there you do end up having to pull the wrist sensor back quite a bit like this to have it fit and then it's pretty snug um you definitely want to look at the sensors on the glove itself there's one here there's one on each finger and there's one on the thumb uh so you want to make sure that they're not gonna get like jostled really so we have it here and it's like black on black right now but anyway that's the left hand on and i'm gonna do the same thing for the right hand so there it is um both gloves on now it took i'm not gonna lie this took me like a couple weeks not of every day but of like doing it a bunch of times and now i've done it a couple times like anything they're both connected immediately this takes work with their your router uh the suits connected and then you have to use the phone too so we're going to be using uh this is a good point i don't know if i'm going to show footage of it i don't know if you can see this so you can kind of say so instead of live link face which is this app we're going to be using rococo remote and so this allows us to record the ar kit data which is what you call the the data coming out of the phone uh live link face will stream that data to unreal engine right and so you could use take record there that is really good for a not full performance capture if you're that's really good if you're going to do the body separately than the face which i've been doing up until now but if you want to record it all together um in one place and have one backup of your body data and your face data all in one place all synchronized together rococo remote is gonna allow you to do that so that's what i'm testing today and as a solo user just to record everything into one place that's pretty nice that being said i haven't done the process of sending the face data to unreal engine but it doesn't look too hard i don't think that um i think that either unreal engine expects live link face data which is just a live link connection or it will take the rococo data and that's also a live link connection so it should all work out but i just wanted to point that out here we're using rokoka remote not live link face but essentially they're very similar and doing the same thing and now i've got my oh got my techno props helmet very fun to have this up and running uh very comfortable it's also black so it like fit it just like looks um it just like looks right with all the rococo stuff that's also black and let me show the most exciting part is the head sensor where is that thing oh where's my head sensor to hold the counter balance on but the headband depending on your suit can be used to attach the head sensor so we could take this right now the head sensor and stick it right on the back that would be fine it's just for my case i actually have to use a counterbalance with the iphone so that spots kind of taken and i don't want them to kind of like fight for space with each other back there anyway so there's my counterweight on like this and what am i forgetting ah so let's put the phone on then here like this this one slots in right like that and this is my first time using the helmet with it uh set up like this and yeah this is really good now this is better than when i had it before pretty good pretty good i need to be i can't like crank it i can't if i go really hard it'll probably move around but again we're not capturing head banging really that's not my goal so the most important step or not the most important but kind of the final step is where's this head sensor go i'm pretty sure it's the same with xs and probably i don't know how perception works actually but i know for a fact the xn's one has like the same exact thing so you could put it on the side you could put it anywhere honestly but i have a velcro thing right on the top from this helmet so that's maybe a good place to put yours it just goes right here like that and then the cable goes to the side so i don't know if you can see it or not but there's my there's my head sensor up there you gotta connect it to the helmet so that would normally be on your rococo headband if you're not using a face rigging so there it is this is the full setup getting dressed with rococo and whatever helmet you use the helmet is just plastic and you know the phone's really the big thing so to recap before we go into rococo studio and do a performance oh actually well this is the mocap so this is the mocap stuff there's one other thing i'm going to add into this uh you might be able to guess but anyway gloves both connected on wi-fi body connected on wi-fi phone connected on wi-fi and everything's set up in blue lights and ethics there's one more thing i got to add into it let me go get it okay so this is more of like an audio filmmaking thing just so much equipment that i've been like accumulating over time if you follow some of my other channels and projects i have not acquired this equipment in the last like month or year it's been years of getting all this equipment and the computers and all this stuff together you know for my company it has this is not something i just went and bought all of it at once and a lot of it is kind of sponsored too anyway the main recorder i'm using is a mix pre-3 this is pretty much the best one you can get not crazy expensive either really nice has analog limiters so if you're peaking and you're not monitoring and like actually mixing the audio live because i'm not going to be mixing my audio i just kind of generally set the level this will actually protect you from peaking in a lot of cases it records to an sd card and it does a lot of other stuff that i probably don't even know next we have sennheiser uh g3s these are like film industry standard i have had these since i've for a very long time doing interviews and filmmaking stuff sennheiser g3s pretty solid there's nicer ones out there but these work quite well and so i'm basically going to take the transmitter here and rococo has a bit of a pocket on this side so i just stick this right in the pocket perfect transmitter holder and i do not do this very well so any people working in the actual sound industry probably like this is some garbage but hey man i'm trying to do a lot of things at once which means i don't do with them all at like full uh professional level we have a uh sanken lav mic i think that's what it is this is one of the better ones it's a surprisingly expensive little lav mic but it makes a difference and especially because i do very little post work uh i i try to go for the best primary capture i can in most cases so this luckily the rococo suit has like literally like a little and they'll have a close-up of it but has like this little lapel like literally for a lav mic i usually do it in the mirror well it's not on perfectly but i'll go look in the mirror but essentially is sound so this is the first time i've actually had to do it this way because i normally capture this at my desk over there with that mic with the phone mounted to my desk this is the first time i've put the phone on a helmet so of course i need to wirelessly record the sound as well so probably pretty obvious but again i wanted to show what my workflow is show for other people getting into it that don't mind a very long video rococo suit rococo gloves iphone helmet and wireless nav mic and now i am finally ready to do full performance capture so i'm gonna go make sure everything's actually set up and then i'm gonna record a take i'll show you the rococo studio and then we'll bring it into unreal engine see you there okay so it's a lot to record this and the bts so forgive me this might be a little messy it's my first time but i have everything connected in rococo studio so i'm gonna roll on audio here that's rolling uh right i think it is if the audio doesn't record big sad i'm gonna go hit record in rococo studio which i'm screen recording that too and let's do some stuff let's do some stuff finally hello my name is citizenmeta1 today we're going to be doing some full performance capture recording the face and the body directly into rococo studio after that we're going to bring this into unreal engine and apply it to our metahuman and here we go into it pretty cool it's the whole body moving you can see the hands are doing their thing they kind of cross through each other so i'm going to have to clean that up which is good because that's what i want to check out that point looked pretty nice we're just going to have to kind of change where the hands actually land they cross through each other again and there's my ending right there okay so here we are in unreal engine 4.26 right now five is still a little bit new to be doing the retargeting though i'll be doing the cleanup in five i hope and what we basically have is a metahuman here of course and we have our rococo studio sending us a live link for the body and the face which is the first time i've used it i'll link to a tutorial that shows this basic workflow however how sam does the face setup is um unnecessary but that's like a different video to to really diagnose the entire thing but you do not need to do all the reposing and stuff like that for the face uh because the face is separate from the body you really don't have to do that but again here is our little clip from before okay there it is so i'll let this play and unreal engines should be able to handle this at the same time so we'll see the rococo performance on the right again that's all they're really responsible for it is our responsibility to get it onto the metahuman and then to clean it up but you'll see that the default remapping that happens here is pretty good and the things like the hands crossing and i think the elbows are a little bent in a little bit of an odd way uh that could be actually part of other retargeting that i've done a little bit off however i can fix all of this in uh in sequencer which is what i'm going to do at uh so i'm sorry at 30. uh i'm gonna start recording here three two one so all of this is just nothing it's just sitting there and then we're gonna hit play on this and it's gonna start recording and i'm actually gonna so this is kind of a messy take i'm just going to start it over so this is kind of uh yeah i have to clean up that animation but there there we go now we're playing it and now we're recording it so kind of messy and if your computer can't handle this kind of um multiple things happening definitely just turn off the lights in on in the scene or go to unlit and then it won't take up very much uh processing power however my z8 cares not about all these processes and screen recording and obs i can just do pretty much everything it's kind of crazy so there we are at the end we have it all pretty sure right there yes i'm going to stop now that should have stopped the take it's going to do some finalizing um that's fine yeah something about the lights who knows so that should have gotten saved to cinematics takes i've done a lot a bunch anyway so it's in here so this is the body most likely and this is the head something that's very important is that you change this to that body type so it's male tall normal right there that's very very very important it just is i can't i'm not going to go into it but yeah you want to make sure that that is the right body of the body that you are using we have everything as an animation asset now right so we're done with the rococo end of it and now we have data retargeted onto our metahuman and there's a couple little issues that we're going to fix um depending on the level of finish you want to get to i'm going to sort out his elbows i'm going to double check his fingers i am going to um clean up these parts where the hands go through each other there's some default yes there's something definitely wrong with the elbow i'll go untwist it that's this is my fault on my retargeting that i've written um but i'll fix it and there's yeah there's just some like general positioning of the hands that i'm not in love with if we check out the face uh without the audio here actually i could sync up the audio maybe i'll sync up the audio too in a second okay before i go to unreal engine 5 i actually have the audio brought in from the mix pre 3 down here you just bring it in like it's a nle and you just try to line it up as best you can um i'm not sure it's perfectly lined up i don't have my clap sync as part of this take but uh really as we get into like really detailed lip sync cleanup which is what we'll be doing we will have the tools there to diagnose the perfect line up there so let's just check this take out as it is so this is the full raw retarget raw lip sync no cleanup yet hello my name is citizenmeta1 today we're going to be doing some full performance capture recording the face and the body directly into rococo studio after that we're going to bring this into unreal engine and apply it to our metahuman hey what's up editor me here and i'm just kind of time lapsing through a little bit of the body cleanup process i don't have like a nice recap for it um but i did live stream uh kind of figuring out the process there's a lot of new techniques like i'm trying out and different things and there's just a lot of techniques for cleaning up mocap depending on what is wrong what you want to fix etc so if you really care you can watch the very long live stream of me going through that and cleaning up the body and i will link that down below but up next is going to be just a little bit of a recap on how i approached cleaning up the lip sync after and there's also a very very long live stream of that process too so here we are in unreal engine 5 and i have my final lip sync animation cleaned up here and i wanted to just kind of share what this looks like when it's all done it's a little intimidating it's a little crazy but nonetheless this is the process so uh i didn't actually leave markers in the beginning but this all begins by essentially parsing through the audio you can see the waveform here and just scrubbing back and forth and listening to it and the more you do this the more you can instantly decode uh actual audio and uh figure out which one of the kind of standard phonemes matches is i'm actually missing a couple like u h and a y and a couple others but you can still get done quite a bit so to kind of break this one down here what i'm starting to do is um put the the phonemes in red that i know for a fact ar kit is not going to do a good job with it and that is first on the list moba pa ar kit is essentially uh the way that it is now implemented uh live link face with the pose asset it's almost impossible for it to hit a normal speech mobappa you could do an overpronounced one but that almost never happens in actual speech so i've put that one in red and when it comes to implementing this i'm not going to show that whole process now i actually live streamed it but um it's it's pretty heavy you have to do a lot of stuff in the curves to uh actually implement a proper mobile but it's one of the reasons it's hard to get this to actually happen to actually have the lips close like that and then to make it prettier you would add lips press and a couple other things to um add pressure and just have it look nicer because it's kind of generic that way the other one that it will almost never hit is fuva and that requires um let me make sure i'm cleaning my actually this isn't even on at the moment but that's essentially this control right there it's my text messages and upper funnel but you actually have to go and turn off a bunch of other ones because they will actually counter foot which i don't know if i can find a in here there's definitely a couple but that would one that one would be red too and so i'm starting to develop this marker system and i don't expect many people to do this manually so what i'm going to do next is make it so that when you click the button and your timeline your uh your playhead is here it'll add the marker for you so that will make this slightly easier um but ultimately this is all leading towards a bit of an automation process right because i still don't expect most people to go through that that's this is still quite a bit of work to do that let alone implement it which is what i've done manually but i'll start to kind of automate that too really in the end you want something that's going to parse the audio for you and drop these initial keys down there and then have the keys actually be pretty clean um so let's see i'm going to neutral this so if we look at the mubap curves here i don't want to get too into this but um each one of these curves when properly cleaned up has a very distinct uh pattern like this one that's very up and down i know for sure that's lips together uh the actually some of these curves are not completely cleaned up and there's some linear keys in here but essentially i've been cleaning them up by hand just to learn what the proper keys look like so um in context of cleaning up mocap where there's lots of data before and after it's animation it's not just pose to pose um there are inherent patterns to how this has to get thrown down there for it to look natural so this is just like a little peek at my current workflow um if you are a full-on mocap animator for the face from cubic motion ilm weather et cetera you're going to do it this way and you're going to do even more controls than this because there's like 150 and you're going to probably rock all of them you're going to do all of them and it's a lot of just like um look at the jaw open track actually it's just this and your life is animating in curves like this and you have opinions about the curves and what tangents to use i'm just using auto tangents still but for the majority of us we don't want to get into this level i will do this for like final anoms for like my game or things where like you know spending a week on an animation is actually an investment that i'll keep getting returns on and i'll use over and over again most people uh i imagine you make an animation and it's ephemeral content right just goes away um people watch it and it doesn't you know hopefully it does well for you but you're unlikely to use the asset again spending this amount of much time here it doesn't really um make sense even just financially time wise it's just too heavy so i'm going to continue down this path i'm interested in like the core base knowledge of what this is and how it actually works for meta humans specifically uh some of it is generic facts action unit knowledge and just general animation knowledge but i am specifically um looking for the technical mechanics for programming as i keep going onto this kind of path of programming things but on how it works for the meta human so that's a little bit of the journey here if you are interested in watching the entire very painful process of me figuring this out and doing the actual lipstick clean uh lipstick clean up here i think i moved to mark oh well i don't care anymore i did live stream that and i'll link that in the vlog below

Channel: Cinematography Database

Views: 75,447

Keywords: cine tracer, cinetracer, unreal engine, ue4, previz, previs, virtual production, virtual cinematography

Id: E9TpcxtqTEk

Length: 33min 38sec (2018 seconds)

Published: Thu Oct 21 2021