Real-time Full Body Webcam Mocap in Unreal Engine - Dollars MONO Tutorial

I was enjoying a coffee and checking out the latest on Unreal Engine you know there's so much going on in the world of AI right now and we are all expecting big lips in moap Tech then I came across this software dollars moap what they offer really caught my attention Real Time full by motion capture using just a webcam and when I say full by I mean it covers everything your face fingers and the rest of your body I like the idea because it could be super handy for a lot of people so I send an email to the team behind dollar smoke app to ask if I could access the full version to make a real video they got back to missing saying of course you can do it so here we are hey everyone this is dusan and today we are diving into the dollar moap mono it's a tool that lets you do map in Unreal Engine in real time using just a webcam I'll give you my honor review this is not a free software but there's a trial version you can try and also there's is a sample Unreal Engine project that's fully set up and ready to go they have already done all the setup for connections blueprint and you know all that boring skeleton targeting stuff just a$ smoke app and the sample project that's it for download links check the description now let's get started and take a look at the sample project okay so here we go I open the sample project and I'm starting the simulation right now as you can see see real time facial capture is working finger capture is looking good and the body capture is working too I should say finger capture is surprisingly good and I can do wide variety of hand movements right now I'm using dollar smoke app mono in what we call upper body mode kind of like a desk mode it's focusing just on my upper body we're going to take a closer look at this modes in detail and also in the sample project we are using this met she is also pointing me but you can also use different Mets which I'm going to show you shortly let's take a look at the dollar moap features now so this camera icon here opens up the video options where we choose our source The Source can be a camera or video file in my experiments I notice video files tend to be more TB and I want to show you how it works let's select a video file and as you see right now the meta is mimicking the moments from the video in real time and this is another example this is where we select our camera it could be a simple webcam High budget camera or another option to connect your smartphone to your PC and use it as a camera that there are a lot of apps for this both on Android and iPhone just check out Google play or Das store I used an app called IV cam to test my iPhone 12 and it works well for now I'm sticking with my webcam hi again and over here we have a background setting this is a m mode and this is hide mode okay let's talk about the moap modes dollar moap offers on three different moap modes upper body mode jump mode and ground mode Let's see how each one works and when to use them this is upper body mode in this mode movements of your upper body are captured while your lower body is ignored it's ideal for scenarios where you don't need to capture leg moments like when you are sitting or if your project is only focused on the upper body perfect choice for S streamers and the next we have jump mode in this mode changes in your West Position will be inter interpreted as changes in height allowing for actions such as jumping and crouching however keep in mind moving forward or backward is impossible in this mode next one is ground mode in ground mode changes in your W Position will be interpreted as changes in position on the ground allowing for Action such as moving forward or backward so if you're walking running this is your go to mode just remember jumping is not an option here understanding the differences and selecting right modor is is super important for capturing your moments accurately using the wrong one might make it less accurate and miss the action you're trying to capture for more in-depth details you can check out Dollar's moap documentation all right let's take a look at the facial capture first things first you need to make calibration this will help to improve the accuracy relax your face look straight ahead and there you go you can easily adjust the strength of your facial expressions with the slider on the right side just like I did if lighting angles or glasses effect the ey gaze capture you can fix this if you are looking for a completely free alternative for facial capture dollar Mo app offers another software and it's dollar ego it includes all the compressive facial capture features of mono check this out and we have Target IP this is the IP address where the motion data is being sent by default it's set to Local Host which means the data is streamed to the same motion running the software if you are streaming to another device or system you would change this to respective IP address here on the screen you can see the settings arranged for the platforms you might use Unity onreal engine romate and iclone they're all set up and ready to go there is no need to change anything this check box right here confirms that your motion capture data is are set to steam directly into un real engine let's talk about the sensitivity this feature is controlled with a slider that can be adjusted between two modes sensitive and steady when you set the slider to sensitive the software becomes highly responsive even the smallest movements this is ideal when your animation needs capture detailed actions but keep in mind especially if you're are doing real time Moab this might make your animations AIT JY because it's trying to pick up every moment now if you turn it to steady the software will focus on the larger more obvious moments this is ideal when you want your animation to appear smooth and unaffected by shitness in summary if you're aiming for high detail go for sensitive but if you prefer a cleaner more stable animation especially for larger actions study is your go choice I couldn't find specific details on this I think it might function similar to sensitivity so we can skip this all right guys the time has come let's move on D real engine and see how we can use your own meta humans first things first there are three things we need to download from Dollar moabs website dollar smoke mono the Unreal Engine plugin and the control rig for UI metahuman okay I have opened up a blank project and added my metahuman to the scene you might see warning like this popup make sure enable them all we need to check to plugins so open the plugins and enable this plugins control rank andc save all and then close on your engine for now okay guys this setup is pretty important so let's focus find your project in the library right click and select showing folder and click on content this folder right here is where our R engine project is located and these are the three files we just downloaded from Dollar smoke website the first thing I need to do is copy the dollcore medumo folder right here now for the Second Step click into Dollars marketless uee folder and copy that dollars folder then paste it here okay I'm not copying the entire dollars marketless UE folder just the dollars folder inside right if you make a mistake here you'll like to receive an error message like this so be careful with this little but important step okay let's open the dollar Mo monu select your camera and check our real engine is on now we come back to our project let's go the content and find the dollars uncore medum folder and double click on the medum blueprint click yes and select the medum Bas scal okay let's open the metahuman bullet print C click on class settings and in the details view find the parent class and choose BP dollar smoke app actor then click on body find the animation mode and select use animation blueprint switch toim class to doore medum and we are done here compile and [Music] save now let's head to content then dollars blueprint and direct the BB dollar receiver to the scene click on the meta human find the dollars receiver in the details and select the dollars receiver and that's all hit the play and our real time met the human motion capture is ready to go as you can see our body motion capture is working but our facial capture is not just yet at this point if you have an iPhone you can use live link for facial capture the body and face animations will work together setting up facial capture for webcam requires and blueprint work don't worry keep watching it's a simple workflow and I'll explain it in EAS to understand way all right let's get started okay head over to content and search for metah humore Bas click on the one that says dollars Med human base this will bring up a new screen and here go and click on the event but leave it open we'll come back to it now back to the content browser search for face enim and click on the Face animation blueprint now we've got two event graphs open dollar undercare Meda human and face anim graph I'm jumping over to the dollar Med humans event graph what we are looking for here is two variables and these are dollar Moor and owner I'm I'm going to copy and paste them one by one to the face anim graph okay in the next step I copy this notes right here and connect it to begin play and also I copy this function pay attention here I'm adding a sequence note and connected the notes like this let's save it and okay we are done with the event graphs and now I'm opening the animation graphs of face and in Blueprint I'm going to break these connections here because they are related to live link and AR kit noes and we don't need them our workflow okay I'll direct dollar smoke aor variable here let's break it expand the list and find the face data let's break the structure again now we can see the facial capture data here let's add a modifi cure note click on Source post and in the add curve pin option you can see all the curves containing poses and morph targets I'm going to start by adding the iall link left curve and we connect this node to where the arit node was previously connected okay if we go back to the scene and hit play we should be able to see our faal capture is working let's adjust this string okay it's it's just a left ey link that's because we have only set up for left I link post let's add the other poses one by one we can ignore the tangard note and the final step don't forget to check this the mod5 C note has a mod option that needs to be set to blend okay guys we are all set now let's hit play and as you can see everything is up and running facial capture is working finger capture is looking good and this is full body motion capture with just a webcam let's take a closer look at the face not perfect but it's looking good let's try a video file to see how it works and this is real time moap from a video file okay it's time to conclusion dollar smoke app offers an effective solution for full body realtime motion capture there are some notable issues for instance when you are doing real time motion capture with metan the output can be somewhat JY so I'm not saying it's perfect but it works fine overall for complex animation and motion caption scenarios there might be some limitations but for basic activities like vtuber streaming while dink it performs well I noticed more consistent results when using motion capture with video files and also it's worth mentioning that I run this software on a pretty beefy setup and here's my setup I mean this software processes video input to capture motion so having a good GPU and CPU is crucial and keep in mind that metan models are pretty heavy so if your computer isn't powerful you might not get the best result with real time meton motion capture if you are aiming for topn motion capture quality you'll need to invest a good amount in MO cap Suites we are talking thousands of dollars plus there are monthly fees for the software which means it can get pretty expensive so considering the Dollar's mup price take off 99 bucks I think it's worth the price if it were priced around 60 70 it will be an even better deal if you are interested I'd recommend starting with the free version to see if it meets your expectations if it does you might then consider buying the full version if you have any questions or thoughts feel free to leave a comment below thanks for watching and I'll see you in the next one

Channel: Duzgen

Views: 15,633

Keywords: unreal engine mocap, metahuman mocap, webcam mocap unrael engine, full body webcam mocap, unreal engine metahuman mocap

Id: ayTm856fDBQ

Length: 15min 51sec (951 seconds)

Published: Tue Dec 26 2023