Connect mocopi in Unreal and Unity! (Tutorial)

recently the Sony mokopi has come to the US this is really exciting this is a device that can do a lot of things it can record video or motion capture directly on a phone it can give you legs and VR chat and it can connect directly into game engines for real-time motion capture application building and real-time animation I'm Woody and I make content about game engines inside game engines and I'm going to show you how to do what I'm doing right now in Unreal Engine and also in unity thanks to Sony for sponsoring this video the process for getting started with the mocapi is about the same as it always is start the app and put on the device we'll connect like normal you're going to change things over from video mode to motion and you want to make sure the save is turned in to send now go up to the three button menu at the top and go to settings we're going to go to PC connection settings up here you have your IP address an outbound port and an option between vrchat and mocope UDP make sure you choose mocha pudp because we're going to game engines instead of your chat set your IP address if you don't know your IP address for the device that you're trying to Target remember you're looking for the IP address of the device that you're going to in my case that means I will be streaming mocopy data to my Windows PC you can just open up command prompt and type in ipconfig or you can find it in your network settings for now leave port 12351 and press OK you should note that you absolutely have to be on the same network as the device that you're trying to stream to for most people that's going to mean just making sure their phone and computer are on the same Wi-Fi I'd highly recommend connecting your phone with ethernet and if you need to use Wi-Fi I'd recommend using a separate router if you find that you get stuck or you can't connect the makopee site has some really good troubleshooting documents so for starters let's go to the developer website so we'll go to downloads and here you can see all the different Integrations we're going to grab the most recent version of the Unreal Engine plugin installing this is really simple however it doesn't run through the Unreal Engine Marketplace so if you've never done this just hang with me for a second we're going to extract all of these and now we have a folder let's go to our PC and then let's go to the C drive we're gonna go to program files then we'll go to epic games and you can select which version of unreal you want to use I'm going to be using five but 5.1 and 5.2 are also supported double click engine and find your plugins and look here's all your plugins remember that folder we extracted I'm going to grab this version for 5.0 I'm going to look inside it and I'm going to grab mocha B I live link and I'm going to copy it into the plugins now let's open Unreal Engine now I'm picking film and video I like to use these templates because they have a lot of other plugins installed by default call this mocopy and find a folder for it and look we have a new project let's update our project file and then we're being told that new plugins are available if you don't get the little pop-up you can go to settings and click on plugins now if we type in mocopy we should have access to them through live link we'll have to restart the engine but that's not a big deal and looks like we're good to go now you'll find mocopy over in the live link panel over here if you don't have the live link panel I have it by default you can go up to Windows and it's in Virtual production or if you're having trouble finding anything you can always just start typing it in and then it'll pop up so we'll add a new source and we'll add a mocopy live link source we can click on it here here you can see we have a couple of different parameters we can edit including the connection Port make sure that it is one two three five one now we send the data if you have your trackers working in the app and you're ready to send signal press the green button we should have a mocopy live link entry right here with a little Green Dot which means we're sending which is great but where do we find our mocap to do that we're going to need to go into the engine folder and go look in our plugins you might have to scroll a little bit but you'll find mocopy you want to click mocopy live link content and then if you go right here to mocope animation blueprint and you click it we have motion I'm sitting down right now which isn't like the most amazing way to use mocopy but you can see it is tracking my motion so now if we want to be able to use that we're just going to grab a moco reactor and drag it out here if you're familiar with Unreal Engine all the mocopy actor is is just an actor that has access to the skeletal mesh the live link component as well as that animation blueprint you can see that it runs in editor even when we're not running it yet how do we record motion there's a really easy way to do this built into Unreal Engine it's a really fantastic tool go back up to Windows go to cinematics and grab take recorder as always you can also just type in take a quarter so all we got to do is add our Moco reactor in there's some other settings that we might choose to mess with at another time like reducing keys for now let's keep things simple and let's just record some mocap we'll start by playing the scene and I'm going to stand up here just going to record myself standing here making a broad gesture and laughing to myself and press stop when you're done I'm going to do a couple more takes now where do we find our assets all you got to do is go to content go to cinematics takes and here's the date that we filmed this on now if we open this up we should be able to find all of our scenes these will open up as sequencer files however we probably don't want the sequencer files we probably want the animation assets which we can find just by going to the shot and then clicking on the animation this is just a mannequin character right we're going to want to use something much better than this well that's thankfully quite easy to do with Unreal Engine 5's new ik retargeting tool what we need here is to be able to find our default mesh so you can click on one in the scene and I'm gonna click on the browse to button you can also find it the old-fashioned way using this file path this should open our Mo copy mannequin and you can see we actually already have an ik asset so I'm going to open this up we'll find that we already have a bunch of different chains as well as ik goals attached to them this is really easy I mean Sony's done half the work for us we're going to need another mesh let's go back to our content folder make a new folder call it skeletal right click and import an asset and bring in your skeletal mesh once we have your model all you have to do is right click and find an ik rake click on that and then let's find your exact mesh call this Woody rig and open it now I'm going to add a bunch of bone chains this is how the ik system works they can define a parameter for how I want the character to be retargeted so a good example is upper arms before I name them is really helpful if I can look at the other ik first so I'm going to go back to the mokopi section and I'm just going to open up this ik here and I'll use all of these names and that will help the whole process move faster let's start with the spine this typically covers every spine bone before the neck We'll add our left arm starting with the upper arm making our way down to our hand they should look the same between all of these so if you see the upper arm moving all the way down to the hand that's kind of how you know so you'll want to just find every chain and make sure it's getting copied over now let's do the eye case the mokopi rig uses a full body solver so let's do the same thing full body ik and let's make a bunch of points so we can click on the hands make a new ik goal the other hand like another new AK goal same with the feet and that should be pretty good we'll set the exact bottom of the rig as the retarget route you can see we need a root for our solver so we can also set that here however I think we're gonna have a problem with this so let's just do a first run I'm gonna go to animation I'm going to make an ik retargeter it's going to ask me what I want to make a retarget from so I'm going to type in the makopee and I'll call this okopi two Woody Rick I'll save that and I'm gonna open it up it's going to ask for the Target ik rig and now I can bring in my woody rake so these are right next to each other when I double click I should get some sort of matching situation here you'll notice we got a couple little issues and these are pretty easy to work out now let's take a look at the unreal mannequin you can see that its pelvis is the retarget root and it's also the root of the full body solver however the beginning of the spine chain doesn't start at its root it starts at spine 01. so I set my pelvis as the root and the root of my full body solver and then I set the beginning of my spine chain to spino one this is working much better now when you go back to your asset browser you can export selected animations which will allow you to transfer them over to your other asset find your folder and press OK and now when I go over here I can find my newly published asset on my character okay great but how do we do anything with real-time motion capture tracking to characters in unreal this is actually pretty easy what we're going to do is we're going to find our character right click we'll create a new animation blueprint call this ABP underscore ocap receiver now what we can do with this is we can click it we're going to add a specific node here get retarget pose from mesh we're go ahead and just plug this in and then let's grab our asset I'm going to use the mocopy to Woody rig make sure the check box for use attached parent is checked compile and save so far we've been in the animation graph let's switch over to using the event graph we're going to need to create a little bit of code let's get the event blueprint beginplay owner I'm going to get owning actor and then I'm going to type in attach actor to component get actor of class we are going to find our Moco be actor we're going to drag this out and then I'm gonna get the skeletal mesh if I scroll all the way to the bottom here and then I'll just plug it right into the parent and press compile now when we press play we have two characters working perfectly in lockstep but this is really weird so all you have to do is just go to the mocopy actor click on it and then open it up find the skeletal mesh we're gonna we're gonna type in hidden in game let's check it and now type in visibility we want to make sure that we have always tick pose and refresh Bones on and now it's just us if you've ever used Epic Games Live link face to connect to Unreal Engine you can also do this on top of the mocopy to be able to drive facial blend shapes live link face works really similarly to mokopi you'll need another networked connection to your computer you can add the live link pose in between the retarget node and the output node but for now we're just going to stick with the full body capture so let's get started in unity for starters open up Unity Hub and the current version of unity that's supported is 2020.3.33 F1 according to the website but I imagine this will open up soon personally I've been able to get it working in 2021 so check the documentation before you start downloading stuff all right let's install a new editor we go to the archive we'll be able to find older versions of unity so we'll click on this in this webpage you're going to go to the 2020 section and then we're going to find three dot three three and I'm going to download the windows version so you're also able to do this on a Mac I'm going to click the unity Hub button and this will open it right back up in unity Hub it will likely want to install visual studio if you don't have Visual Studio or you've never used Unity before go ahead and install this I just want to make sure I have Windows build support turned on because that's the platform I'm going to build for hit install and now we watch the paint dry alright now that it's installed I'm gonna go to projects and I'm going to make a new project I'm going to call it Mo copy sample scene make sure that you have 3D core I'm going to create the project and let unity set it up let's go ahead and import the asset package that we need we go to mocopy developer from the front page we can go to downloads and we can find the latest plugin here is 1.0.4 which just came out today I'm going to download it let's go to our downloads folder and extract this now we have a regular folder and we can access the package through here back in unity we're going to go to assets and then we're going to go to import package custom package and find what we just downloaded gonna open the package and go ahead and hit import so take a look at what we just got you'll have the mocopy receiver folder you open this up you can find the resources runtime sample folders inside sample you're going to see receiver sample you'll have models and scenes we look through models we will find our standard mocopy stock Avatar and if we go to scenes we'll find a test scene where everything is already laid out for us there we go there's our test scene make sure that your simple receiver here this little object here is set to run on the port that we want that should be one two three five one and make sure that the Avatar selected is this Avatar at least for the sample and now if you've hooked everything up correctly we should be able to see some action if you have your trackers working in the app and you're ready to send signal press the green button press play and here we go that's looking pretty good now what if we wanted to put a different model in there well we just import one it's important to put it in the models folder underneath the receiver sample folder make sure you've got that you can see the default one that's right here and let's import new asset how about a handsome One now I have my model here however it's not quite ready to go there's one thing we need to do first we need to click over here on the rig section and we need to change our type of Animation from generic to humanoid now this is really great this is unity's humanoid system let's go ahead and apply our settings and then now we can configure the humanoid system basically this is a retargeting system or just another way of understanding bone rotations across all characters let's go ahead and make sure that everything looks okay for the most part it's going to guess based on what the names of your phones are mine are all Unreal Engine names but that's okay and uh let's see here chest we could probably change this I'm going to change my chest bone that I include a second spine bone and I'll leave my upper chest blank from the diagram we can see that everything looks good and now we can go press apply we'll go back to our scene here now that we have another model I'm going to go ahead and delete the one that we already have in fact I'm going to delete the entire mocope Avatar pack here I'm going to drag our guy in which is me and I'm going to add a component over to him I'm gonna grab mokopi Avatar and I'm going to put it here this is great now when I go back to our mokopi simple receiver it will tell me that my avatar is missing however if I click the little button over here I have the option to add my character in and now things should be working let's press play and see what happens and you can see we have a character let's go ahead and move this camera closer to him I'm going to turn the sun around as well great let's do a build let's go to our build settings and change a few things now I'm building for Windows so I'm going to go to the player settings here's a handy little guide that I got from the mocabi website that will show us what we need to change right now for Windows we only have to change two things here we're going to find our scripting back end we're going to change it from mono to il-2 CPP and I'm going to take our API compatibility and make sure it's on the latest version on the site they're recommending 4X but for some reason I only have 2.1 so we'll go with that all right you can close this out now and let's build and run the program it's going to ask us for a place to put our build the editor will prompt you to save if you haven't already so let's go ahead and save and we'll let it cook and that's it there we go look here I am [Music] [Music]

Channel: WoodyDevs

Views: 5,314

Keywords:

Id: aMyv9qTlFQs

Length: 17min 52sec (1072 seconds)

Published: Sat Jul 15 2023