How to do Motion Capture in Unreal Engine 5

how's it going everybody and welcome back to the channel now in this video i wanted to create sort of a comprehensive guide if you will for those looking to do motion capture inside unreal engine so i'm going to include lots of things kind of like what motion capture suit i'm using how to use motion capture suit uh we're going to talk about facial motion capture we're going to talk about all sorts of things including the technical parts inside unreal engine and so this video should be whether you are wanting to do a motion capture this video should be good for you or if you already have a motion capture suit and you want to know where to get started with unreal engine then this video should be useful for you as well now before we get started as usual there's a twitter uh leave a like and leave a comment for the youtube algorithm it it always goes a long way uh thanks a lot to everybody giving to the patreon now first things first how do i achieve this metahuman that you're looking right now he uh was created with mesh to metahuman which is a new tool that showed up for unreal engine right now and i do have a video on how i got this guy actually i have a video on how i got sort of this head and the rest of the body is also in the how to scan the head video how to put yourself with a meta human inside unreal engine so watch those videos if you want to get something like this and uh then just jump back into this one now motion capture why would you want a motion capture suit it is uh really expensive thing to buy it i'm not gonna lie it wasn't cheap for me living in the u.s i can't even imagine what this looks like for somebody living outside of the usa but it is well worth it and i'm going to get into why whether you are an indie developer looking to make a game or whether you're wanting to do film stuff like i am now i've been trying to do things with marketplace animations for a while now and they're great there are great ways for you to make those animations work but the problem is it can only get you so far there are times where you are the one that has to go in and animate with takes a lot of time if you want to get something more casual or if you just want to have something that is not in the marketplace one of the things that you will see in my videos is actually how to use animations from the marketplace and the reason for that is because i have not stopped using animations for the marketplace even though i have a suit there are certain things that i just can't do like just jump rope around i'm not that agile i'm not a stuntman or anything like that so getting a motion capture suit doesn't mean you're probably going to stop doing that you're most likely going to be still buying animation from the marketplace however when it comes to mundane things talking and all that it's just much easier if you have a motion capture suit and to be honest the best way that i can put this when it comes to using a motion capture suit is you don't know that you need it until you have it and i know that sounds nebulous and that probably makes no sense but that's just how it felt for me i spent around um probably over six months pondering over this uh trying to see researching which kind of motion capture suit i was going to get and whether i was going to get one or not actually attempted to make a short film without getting a motion capture system and it it was just horrible the experience was terrible can it be done yes probably if you have a team of animators or if you're an animator yourself you probably can do it but i'm just a 3d artist and i have a day job so i can only go so far until my videos start taking an eternity that's one of the reasons why i jumped in and i forked in the cache to get this motion capture suit that i am using today all right so another thing is why would you want to use a 3d animation setup let's say you're an indie filmmaker why would you want to use a 3d animation setup invest in a motion capture suit instead of just getting a camera and go out and shoot well in my case i am a solo developer but i want to do things with sci-fi i want to do things on a pirate ship i want to do things in medieval times i want to do lots of things to require a lot of bfx and will probably require a huge team and a lot of money to make so that's why i'm staying in the 3d animation room because unreal provides all that for me um i have everything here i have all the assets i have mega scans i have these guys where i can pretty much do any character that i want and since i used to do characters that's also something that's easy for me it's just to make a character in zbrush then turn it into blender do retappo bring it into one of these guys rig it and there you go so that's one reason why i went into 3d animation yeah you can buy a camera for cheaper than what i bought this suit but what about the bfx if you're just interested in doing drama and comedy and all that type of stuff with like things that exist that's probably really easy to do but in my case um i want uh and and the issue is nowadays it's very trendy to hate on cgi for some reason i don't understand that people seem to think that the films that are practical are much better than the films that have cgi ended that's just not true most of the film said most of the films nowadays have cgi even if you don't know it so to get away from that that's why i wanted to jump into 3d animation and i think unreal engine provides a lot of tools for indie developers for all of us to just do this all right now let's get into the suits i do have an in-depth discussion as to why i chose this suit link in the description by the way but uh i'm just gonna go into some of that real quick here in this video now there are three brands hopefully they're yeah they're over here when i'm editing the video there are three brands that i researched when i was doing um all of all my investigation on what kind of motion capture suit i wanted to get and these are rococo accents and perception not necessarily in that order but there are a couple of things why i didn't choose the other brands by the way i'm not sponsored um nothing that you'll see me doing here with the motion capture or the leveling face app so far up until this day i'm not sponsored so everything that i'm telling you right now and everything that i'm showcasing came from my own pocket nobody's paying me to say any of this so rococo i didn't choose for coco because their only solution is a full-on suit and what if i want to pass it to my wife to help me out with a shoe or if i want to pass it to a friend that is not necessarily my size then i would actually need more than one suit and that's just most expensive for me which i'm not gonna go there so the other was accents accents is the one that's regarded as the highest quality in the industry right now and yeah it is very high quality i've actually worked with people that use this i've talked to some youtubers that use accents and it seems like a great setup one of the things that dissuade me from it um i must say the price a little bit the price difference is not so big from perception neuro but they do have a subscription system which i'm not very happy with i don't like subscription system for hardware stuff and yeah you can do the monthly thing and if you do it monthly it's not that expensive so as of the recording of this video you would need to subscribe and to pay up subscription in order to do what i'm doing today and i was just not happy with that so that's why i discarded that option completely and then i came to perception neuron actually also talked to youtubers that really like perception neuron and that's when i jumped in and say well i'm going to get perception neuron 3 which is like their entry level suit i guess because they also have a suit that it's a tier upwards from the one that i'm wearing which is called the perception studio but perception 93 pretty much has uh many of the things that i would need i got this this was about three hundred and three thousand sorry three thousand and uh six hundred dollars with shipping and i also included the sensors for the fingers which come separately but the great thing about this is you don't have to pay any subscription they do update fairly often to their software actually i will right now i'm driving one of their latest updates to their software which actually allows me to get into metahuman right away without having to do any remapping or anything uh weird inside the software or inside unreal engine i can just select metahuman choose the metahuman preset in software and here i am and as you can see everything is awesome if you've seen videos from other people using perception neuro those are probably all videos because as you can see there used to be an issue with the fingers there's no longer that issue with the thumb and the meta humans you don't have to do the t post thing again everything just works now if you're wondering about my facial motion capture setup i'm using a lifelink face app which is the application that you use with ios i find it very very good as you can see i have very good range of motion with this character i am emoting a little bit more than i would do if i was talking normally but i believe it produced really good results as it is all i do is fire the app connected to unreal and that's it we're going to get more in depth into how to use live link face app some tips that i would recommend to get some nice facial motion capture in a separate video now we're going to get into the technical part on how to use unreal with motion capture there is a thing that's very important that you need to get very familiar with which is called the anim blueprint so this is one thing i'm going to show you how to create if you use a motion capture suit you're going to be dealing with these a lot so you're going to be very familiar with it and whichever motion capture suit you decide to get you should be able to follow along because all i'm going to do in this video is related to unreal engine i'll do a separate video in the future on perception neurons specifically their software and how to do some configurations so you get some cool motion capture like i'm doing right now but other than that you should be able to jump into this whatever motion capture suit you're using alright so let's get into the animation blueprint now i know i said we're gonna go into blueprints but before that we actually need to activate a plugin before we actually get into the animation blueprint and that's gonna depend on whatever suit you have whether you have coco accents perception neuron you always have to activate a plug-in first that's going to allow for the transfer of data from the software that's doing the capturing to unreal so usually for that just go to settings plugins in my case i'm using perception neural so i just type neuron and here it is i just need to activate it same thing goes with accents i believe there is called an nbn and i don't know the one for rococo but i'm sure they have one every motion capture solution will include a plug-in hopefully for unreal engine 5 that would allow you to transfer data as far as i know most most of those brands do it nowadays so after you are done with that then you need to locate your character in this case let's try with the metahuman so i already have the metahuman that you saw in the intro is right here and all we have to do is figure out where he is the skeleton of the body is because in the case of the metahuman and we're going to look at any blueprints for other characters in a bit but with metahumans you already have the facial part done for you so you don't have to do the facial part all we have to focus on is the body so we need to look for the skeleton of the body which is here's a skeletal mesh i'm just going to go where it's located double click it and just go to skeleton and as you can see mine has this i don't know why this is going on but it works just fine now if i go back to where the skeleton is located here is the skeleton and here is my animation blueprint now what is the animation blueprint this is an asset that's going to tell the engine how to handle that um capturing that's going on in the external software so the way you do it is you right click on your skeleton you can also right click here uh create the animation blueprint from the animation tab and then go after that i just think it's much easier especially for beginners if you just right click on top of the skeleton of your character and then you go into create and then you go into animation blueprint right so there's my animation blueprint i'm gonna call it tutorial and mvp okay and usually you'll see an icon the skeleton that you were using here now once we're done with double click so we can go into the blueprint and as you can see your character should be here on this screen and we have this output note if you've seen my other tutorials this may look familiar it kind of looks like the material editor if you've done programming then of course you know what this is but uh this is the output node where all the things are going to get transmitted too so what we need is the live link node which is pretty much what's going to be linking the software with this blueprint and then with our character so we just right click go to live link pose it's always cycling post and here you go so in this little box you have a couple of options over here you have retarget asset this is very important for depending on what motion capture suit you're using after you've enabled the plugin if you click on this drop down menu you should see the remap asset for your motion capture suit in the case of perception neuron there are several here and they recommend use the mannequin so that's the usual one that i would use now in that case i will leave it there there's one extra step that i know you have to do with accents but other than that it's very similar all you have to do is connect it and then you're pretty much done so for a demo here i'm going to be using these two characters that come with perception neuron this is just one of their examples and just to show you how it works now once we've done this we need to tell the engine hey go into the source and the source is going to be in my case going to be perception neuron that depends whatever motion capture suit you have but you have to click here if you don't have this tab like i do all you have to do is go to windows virtual production and click on live link in this case has a check mark because that tab is already here now click source and click on the brand that you're using and over here you should see either a port or some ip in my case i can see the ip i just click ok and as you can see i have both of the characters that i have here on screen with their respective names now all i have to do is go here and select which character i want the animation to happen this is exactly the same from what i'm doing live motion capture where i just i'm just wearing my suit and it'll give me a character name and all i have to do is click on the character name then compile and as you can see he starts doing the motion from uh the software that you saw over here in this window now i know you see that there's like overlapping meshes on him this doesn't matter because the final product is not going to look like this it's been doing this for some reason the last few tests that i've done but when i go to the actual metahuman in my viewport it's not going to look like this so don't worry about that now we have set up the animation blueprint for our body now the face in case of the metahumans it already has an animation blueprint and you don't have to mess around with that right now that you have your blueprint ready over here all you have to do is go to your metahuman in this case is this one right here drop it into your viewport wherever you want and now once i'm here i go into my metahuman you just go to your body and over here what you say it see the animation tab under here's the animation mode and you're going to change this to instead of using animation asset you're going to change it to use animation blueprint now when you do that you have to specify where the blueprint is or which name it is in that case we're going to pick the one that we just created which is this one right here tutorial mvp and you know it's an animation blueprint because it has these two um little characters running forward so i'm just gonna click here and you see that he adopts the pose but he is not animating even though these characters are moving and the reason for that is because he's not going to be animating in the viewport unless you have the live link component in this case if you want to leave it at here you can just spray play i'm sorry press play or in my case i'm just going to simulate and you see that he is doing the animation just fine and everything is working now if you don't want to do it during simulate because i don't simulate or play the game what i do is i add a live link component so click on add look for live link skeletal animation you're going to use skeletal animation not controller you're going to use this one right here and you're going to see that he goes live if by any chance your metahuman doesn't go live then you can go into the live link component right here and click on auto activate that usually does it if it doesn't uh go live instantly but as you can see the metahuman is now doing the animation and i don't have that weird mesh overlapping that you saw from the anim blueprint that was just an example from the skeleton but as you can see my metahuman here is behaving as expected it's just moving just like the animation that i have here as you can see it has the open palm we got an animation going in our engine but this is unusable we need a way to get this animation recorded into unreal so you can use it later on unless you are actually doing it in 100 real time which i have done and you can do that you can just start doing your motion capture and add the lighting component and then just start capturing your body here but for the best results i found that it's better to just record it and you can record it here this way and after the fact you can reuse it and then you can just do cameras you can do lighting which again it's still real time but it allows another level of control now for that purpose we need to use take recorder for tech recorder you just have to go here to windows now i already have it over here but it's right here under windows cinematics take recorder and you should get something that looks like this now in order to capture our metahuman there are a couple of ways that i found on how to do this one way would be to go here to the source because you need a source uh and you can select from actor or you can select from live link so if i go from live link right now he's doing the move the motion of the character name gao gale and just click here and you have to make sure that used source time code is unchecked this was something that feeding wolf showcased on how to do uh take recorder before i learned this from her very good channel about motion capture by the way now once you do that all you have to do is get out of there everything is fine and give this your name so we're going to call it test tutorial and we're going to call it take number one and we're just going to hit record now when it hits records it waits 30 seconds and then starts recording the motion and you can just let it be all the way until you are satisfied or you're done with recording your animation i'm just going to call it there and if you click on browse to now you're going to see that it created a cinematic folder for you i'm just going to stop him doing that for a second and over here you have a couple of files now you have this sequence right here and you also have some substance right here this is the one you want to use now you can just double click the sequence and after you have opened your sequence if you have your many human here in the viewport it will in immediately inherit what the sequence is doing because once i go outside of sequence you see that he goes back to his neutral pose but once you go into the sequence he just starts doing that now one of the ways that i would do things and actually that was the way that i did the intro was from this i just add a camera and start recording because what you can do is click on this lock pad over here if you click now you can add things to it so let's say um we want to add a camera that's around here there you go let's say we want to add a camera here we click on the camera and we focus our camera wherever we want to have it go here it's a little bit out of focus let's just fix all that focus settings let's do this let's get our plane there you go and all this it's in my tutorial on how to uh intro to unreal engine for filmmaking beginner filmmakers if you want to look at that playlist but i just did a quick camera with the focus you can add it here all you have to do is just add your camera and play and if you want you can just go into movie render queue and render things from this and you're done you got your sequence with your metahuman and it was that easy all right now that is one way to record things the second way that i know how to record things with take recorder takes a little bit longer to process but it's easier to use if you are doing face and body at the same time what i mean is this one will record a separate take for whatever you tell it to record in this case the source was our motion capture suit but the source from the facial motion capture would come separate from the body motion capture so you pretty much have two sources and then you have to put it together later the other way that you can do things here is if we can delete this and we do it per actor not per source now if we do it from actor because i have him selected you can click here add this guy and then what we can do is we can actually click on him make sure he is using the anime blueprint and make sure the end blueprint is going there you go now he's using the blueprint we're going to start the tag recording i'm going to click here 3 2 1 there you go it's going to start recording and let's just record a little bit and we are done now this one actually took a little bit longer than the other one because it's processing the whole actor as opposed to just process the source which i'm assuming it's much cheaper for the system uh it can take a while so don't be scared it's not crashing for me it actually took like 10 minutes because i did a take that was over 15 minutes long so it's not that bad but just so you know that may happen so let me just pause him so he's really distracting now the way that that's going to look is if we're go into cinematics or text for today and go number two we go into animation you're going to see that instead of a sequence you do have an animation now if i go here you're going to see that it is my default skeleton so again don't mind these um weird parts going on here but he is doing the animation so what this does is it will give you an animation just like the animations that you get from the marketplace so it would be an instant thing that you can use into a brand new sequence that you create yourself now i do have an example where i recorded body and face at the same time which is one of the things that i use for the intro and it's right here so right here we have the body as you can see um they're turning things off and it's a very long animation i'm right there but you can see that he's moving but his face is not moving that's because even though he it records it as one single actor it does it uh separately because it's two separate animation blueprints however if i go here to the next one you're gonna see that my face is actually moving because this is what recorded from the animation blueprint and what you can see is in this case i already have my animations and i can toss them into directly into sequencer and be done with that the way that i would do things right now is i would grab him let's say i'm going to use the second way that i just showed you let's just create a sequence add a level sequence let's call it tutorial test and let's add him to the sequence over here now for this to work uh we need to get rid of our control rig so we delete the body control rig and release would delete the facial control board there you go so he is still using for the body he's still using the animation blueprint we don't want him to keep using the animation blueprint i'm actually going to switch that to animation asset and you can see he adopts a neutral pose now we want him to use this animation that just happened so i'm just going to go and make it easy for me copy this name control c go to body and remember i'm in body not in the first one i'm in the body track go to animation i'm going to paste that here and there you go my first scene and there is my animation and here's my animation where i do all that and if i wanted to add the face i can just go here uh click on this one ctrl c go over here track animation click here paste it grab my animation make sure that everything is lined up and as you can see he is moving and talking at the same time so i got both things here in sequencer very easy to handle the only caveat is that if you're doing really long takes like i'm doing it may take a while to process once you've stopped the recording and it kind of looks like it's gonna crash but it's not gonna crash so if you're patient you can use it this way okay we're gonna start now with the facial part and you're gonna be able to do this if you have an iphone because we're going to use the live link face app i've done this with faceware in the past for unreal engine 4 but in here on real engine 5 i've been using live link face app it works wonderfully and even though i'm not an iphone user that iphone for that sole purpose was a great investment because it completes my setup so let's talk about facial motion capture because we are here in unreal engine 5 you should have the ar kit plugin already enabled by default but in case you don't just go to the plugin section and we're going to type ar kit and these two should be uh checked same thing with if we go to live link you can see that this one's checked this one's also checked like i said they're usually checked by default but in case you don't see all the things that i'm going to show you then this will be the first place to look to see if the plugin is actually enabled now when it comes to our meta human that's on screen right now if i were to scroll over here you see it says default and under default you can see it says live link face subject in here i can actually select my iphone if i have it up actually let me turn on open live link face up so you can see how that looks so if i open live link face app now over here in my live link tab now you can see that instantly as soon as i fire the app on my iphone is reflected right here with my iphone name the other thing that you need to take into account in case this isn't working for you make sure your iphone is uh on your wi-fi and it's on the same wi-fi of your computer in my case i have two wi-fi's at home and i need to make sure that they're always in the same wi-fi so again if you run into trouble this would be one of the things to check for now that i have my iphone over here and as you can see it turned yellow because i turn off the app i usually don't if i'm not actually recording i don't have my phone on because this tends to get hot which nothing happens if it gets hot it's just instead of recording at 60 frames per second it records at 30. again we're going to talk about that when we look at the tips for facial motion capture but for now these are just the basics so if your if yours turns yellow don't that means that you just turn off the app or your phone went offline or something like that now what we need to do to enable this is we need to have the iphone right here however if we if i just do my iphone like that my character still doesn't do anything and the reason for that is because i need to do it at the animation blueprint first so just like we have an animation blueprint for our body we also have one for our face the difference is the phase one it's already done for us in the case of the metahumans so i'm just going to go into the animation blueprint double click this one and as you can see i already activated my app in this one but just to show you where it is it should be right here where it says as soon as you get in you should be greeted with this part of the window and as you can see he just keeps talking which is really funny but over here you should just click here and locate your iphone usually says iphone black something as default but it here is my name i just need to compile and once i've compiled then i'm all done to just start doing mocap but again it's not doing anything because i need to add the live link component otherwise i need to press play so i just need to add the leveling skeletal animation and as you can see he starts um doing facial motion and another thing that you can do if you want to have something for the head there's also live link face head if you click here and you move your head although it's not doing anything if you also want to do head rotation i usually have head rotation disabled just because i use the headband that comes with my motion capture suit so i use the motion capturing from the head so i don't have to do this but if you don't have a suit you can also include head rotation with the lifelink face up so i just click here and as you can see there's a little bit of rotation that i can do with my head um this this is however a little bit more limiting and i don't think is as good as doing it with a motion capture suit but in case that you just want to do a floating head for like an avatar let's say you're streaming doing something like that then you can just do it this way get yourself a stylismatic human open liveling face to head rotation and then you have your character doing this so i'm not gonna compile that because i'm not gonna use it but all you have to do is compile and save like we've been doing so far so we have looked at how to do body motion capture with a meta human how to do facial motion capture with a meta human now i'm going to show you how to do it with a character that it's not a meta human because the meta human already has the facial animation blueprint made for us so we only dedicate to the body however if we have another type of character it's not a meta human you're going to need to do the facial part yourself and i'm going to show you that right now huh hello you wait it's just you what are you doing here we're not transmitting today i just need to use the setup real quick i want to show the audience how to get the system that we're using for your transmissions um okay nobody told me about this gary did you know about this okay apparently gary did know about this and didn't pass the message yeah it's just going to be really quickly and well i thought our arrangement was uh to be confidential with this but i guess it's okay you can show the humans how uh you helped us transmit the news and all that uh yeah go ahead it just you know remember to mention our channel too so all right awesome okay um bye humans all right so thanks alien for letting us do this and his channel is going to be in the description down below if you want to see some funny news and pop culture stuff i think they just watched thor and they're most likely going to do a review on it this week now when it comes to any other character that's not a metahuman like i said we're going to have to do the blueprint ourselves and i'm just going to take you to where this is because it's basically the same thing as uh what we just did only one small difference as you can see i have two live link post notes here one is going to be for the body and one is going to be for the face now the way you put these two together is by using the apply additive note so if you right click here additive you can see that the one of the first things that comes up is apply additive and this is going to be kind of like your multiply if you were doing a material because it's just combining the two together in just one single output there is one thing you have to take into account here whenever you are using this your body motion capture is going to be on the base and your facial motion capture is going to be on the additive we're not going to do anything to the alpha but those are the only two things that you have to take into account another thing that is really important remember when we did body motion capture before in this video we actually changed the retarget acid in this case to the remap acid that perception neuron provides or whatever other motion capture suit you're using when it comes to the facial part you don't have to do that i just put your iphone in here and you should be good to go you don't have to remap to anything else just combine it into additive then everything goes into the results and you are just going to use one animation blueprint instead of using two now another thing regarding custom characters what i highly recommend is that they are rigged to the mannequin skeleton motion capture suits permit other types of skeleton in my case i i've been i've i have even used the maximo skeletons however what make things a lot easier and very prolonging play is using the mannequin skeleton so if you're going to get a character and do the head swap like i've done in videos before or you just want to get a character to motion capture as that character from the marketplace or even from cg trader other sources i highly recommend that you get one rigged with the mannequin skeleton because it makes things so much easier especially if you're using uh perception urine with the mannequin or the metahuman you can just jump in without having to do any other extra remapping thank you so much for watching the video as always a shout out to my level 2 patrons remember if you give to the patreons there are two logs you can participate in it if you cannot do the patreon then leaving a live and leaving a comment helps a lot with the youtube algorithm uh subscribe if you haven't there's a twitter there's the discord uh instagram all those kinds of things and um i'll see you next video

Channel: MR3D-Dev

Views: 39,205

Keywords: Mocap in unreal engine tutorial, mocap tutorial, perception neuron, body motion capture, face motion capture, metahuman, stylized metahuman, ue5, unreal engine, metahuman animation tutorial, animation blueprint ue5, 3d animation, character animation, motion capture

Id: NlnVV35wnX0

Length: 38min 59sec (2339 seconds)

Published: Mon Jul 11 2022