Unreal 5.4 Mocap Workflow (For Now!) | Rokoko Full Performance Motion Capture

ooh it's that special time of year when you get to switch to the new version of Unreal Engine hey what's going on everyone I'm Sam Lazarus creative director for Roco and in this video we're going to take you through a preliminary workflow for working with Roco motion capture in Unreal Engine 5.4 and all the new motion capture related tools that are in this version specifically we're going to take a look at automatic retargeting generating automatic ik Rigs and also ik ret targeters as well as looking at modular control Rigs and we're going to do this both for a generic character and for a metahuman I will also say in this video we're just covering retargeting not live streaming for things like V tubing if you do want to do live streaming our plug-in still has to hit the marketplace for unreal 5.4 you can go to our GitHub and kind of jury rig it together if you want to try that we'll have a video coming out about live streaming unreal 5.4 soon but for now 5.3 is kind of the best place to do do that reliably also our current workflows for unreal 5.3 and below our retargeting workflows are really solid those workflows still work in 5.4 they're just not using the new tools that are in 5.4 so all that to say you can still use our old workflows they're up on our YouTube channel go check those out they're very comprehensive if you aren't ready to jump into 5.4 yet and as I said this is a preliminary workflow and that's because this is a brand new version of unreal there's still some Kinks that need to get worked out both on our side and on the unreal side so use this for now but we will update this tutorial when we have a better more permanent workflow for using Roco map in 5.4 with that being said let's jump into it okay so here I am in roko Studio and I already have a clip that I recorded previously and it it happens to include face capture because I want to go through face capture a little bit in this video but this same workflow will work if you don't have face capture at all to export this clip out for Unreal Engine we're going to go to export and we're going to toggle body mesh which we always need to do for unreal we're also going to toggle face just because again this clip happens to have face capture and here we get to presets so as I said at the beginning of the video we are not going to be exporting using the Unreal Engine 4 or Unreal Engine five presets which I know is confusing but it's just because this is a brand new version and our old workflows for 5.3 and Below or for 5.4 if you want to use the that old workflow can use these skeletons for the new Tools in unreal 5.4 we're going to use either the miimo human ik or iclone skeleton so in our case we're going to use mixo I'm going to come down and adjust the frame rate which you can do if you want and then hit export clip here I am in a new Unreal project and I've already actually imported two characters into this project for us to retarget onto one of them this one is a just a character from The Epic Marketplace uses an Unreal Engine skeleton so this character should play very well with this whole new workflow because it's from epic and it's from unreal and I've also gone and grabbed a metahuman and the metahuman workflow is a little bit different so we'll be taking a peek at that at the end of this video but in the meantime we are going to retarget our map onto this character so the way that we're going to do that so I'll go back to my main menu I've just created a folder called Roco mocap I'll enter that go add import to game and then we will go find the motion capture that we exported from our Coco studio and in this case you can see it says mixo on the end that's because we used a mixim skeleton when we exported so I'm going to select this map hit open and I want to do a couple things when we're importing our motion capture we could also batch import multiple animations just by selecting all of them and hitting import first we want to make sure import mesh is checked on then under Advanced we want to make sure to import morph targets these are the facial motion capture blend shapes um from our Coco studio so if you have facial motion capture make sure that this is on and finally of course we want to make sure import animations is on and after we've done that we can click import all we'll get some errors but that's fine uh nothing to be worried about and right here we have our animation you can see it says anim double click this and there we go we've got our motion capture in unreal and it doesn't look maybe very pretty with these bones uh but that's going to be fixed as as we uh work on this workflow a little bit more um but we've got our facial motion capture everything looks great so how do we get this onto our characters the new workflow in Unreal Engine 5.4 is we are going to go find our animation right here rightclick it and go to retarget animations and we'll get this little popup window and you can see in Source we already have our animation loaded in but under Target we need to go and find find where we want to retarget this animation to so in this case I know that I want to retarget onto this character right here and once it's loaded in we can doubleclick our animation and make sure that everything is coming across good and you can see that even the facial motion capture will be visible uh in this little dialogue window just to check it before you proceed so this is all working this looks great and now we have two options we can export retarget assets or export animations and we're going to start with export animations and then I'll show you what this option does as well but let's hit export animations I'll export it to my main level and what do you know it exported the animations just as we thought it might so you can see we have all of our facial motion capture as well it's all baked to our character and this looks great if we wanted to add this to our character we could go add our character to our scene in this case it's right here then we could go create a new level sequence call it map anim we would then add our mesh here to sequencer and under animation if we hit plus we can go and find that animation that we exported full full performance cap right here double click it and it will expand the timeline out so we can change the outo and if we hit play look at this all looks good control and L and we can change the direction of our light and this looks really great and if we want our physics to be running in sequencer as well we can select our character go to physics and and just check this on and off and then we'll actually get some Physics on our character it might not work for every character that way okay so that is our animation onto a character and we've saved a bunch of time with the new Automatic retargeting by doing it this way however remember those other options we had if we go back to our motion capture and do the same thing rightclick retarget animations add our mesh here so the other option is export retarget assets and let's go ahead and do this hit export to the main level and we get three assets from that when we do it the first two are ik rigs one is for our motion capture and one is for our character and if we double click we can see that for our motion capture we've already built automatically a whole bone chain bone map here and we have a fullbody ik rig that's been added to our character as well and this is really cool because it's going to give us a lot more flexibility when we're retargeting our motion capture if we go into our character boom same exact thing we've already got this bone chain bone map already made for us and the full body ik solver has been added as well and you can even open these up um individually and create these by yourself but I find that it's a good way to go into that retarget animations and it will just generate an ik rig for both of them for you the third asset that we get is this ik retargeter asset if we doubleclick this our motion capture and our character on are on top of each other so we can just offset them a little bit here and here we get more options and controls for retargeting our motion capture so this if we double click our Mo app you can see everything's happening in real time but we can also toggle on ik for both of them for instance uh we maybe we want it to be FK maybe we want to toggle off ik just for the arms we could select an arm left arm right arm and toggle on or off ik so I'm not going to go through every feature of uh the ik retargeter but this is just a good place to do more tweaking if you want let's say you had for instance a character with a really round belly and you were going to get clipping from the arms one thing we could do in here before we exported out this asset is we go to edit retarget pose and I could maybe adjust the arms a little bit and now our arms are a little bit further away from our body so we just have a lot more that we can do in this screen as opposed to just exporting out the animations and being done with it so if we want to export out this this animation once we've played around with it we'll select it go to export selected animations and then we can export it out and now you can see we have two animations here that have been exported now I mentioned at the beginning of this tutorial that there are two new exciting things one is that new automatic r targeting system but the other is modular control Rigs and the new modular control rig system is going to allow us to do some map editing non-destructively and really easily here in sequencer so how do we go about making a modular control rig for this character well first thing we're going to do is go and find that character's mesh so for this character it's in here and this is the mesh right here and we are going to right click on this mesh go to create modular rig under control rig and as soon as we do you can see we got a new little asset here it's called control rig and if we open this up we can now create a control rig and an ik control rig from this character and it's very easy compared to how it used to be done which would involve Blueprints and all these other things now if we hover over this first little socket here you can see it says drag and drop the spine module so we're going to grab the spine module and we're going to drag it on and then we get a bunch more sockets that we can start adding modules to so shoulder this says drop shoulder module so we'll do that for both shoulders then in here we put the arm module 1 2 and here we're going to put leg modules let's go find leg one two if we want to adjust the the uh toes and the toe direction we can put in foot modules here which usually you don't really need to do but we might as well cuz we're doing it so currently the neck and head part of this workflow is a little bit broken in 5.4 that will likely get patched soon so for right now we're not going to add a neck socket because it's going to eliminate some of the animation in sequencer so long story short we're not going to be adding a NE socket until this bug is fixed so now if we go back to sequencer if we wanted to do some editing for this character I'm going to go back to frame zero just to keep things clean the old workflow would have involved right clicking on our character going to bake to control rig and selecting our new control rig this is not going to be the way that we're going to do it anymore if you do want to bake down an FK control rig essentially if you want to bake all this animation to the bones of our character we could use this FK baking right here but in this case we want an ik control rig and we also want this to be non-destructive so instead the new workflow what we're going to do is we're going to go to plus not rightclick we're going to go down to control rig I'm going to make sure that layered is selected and then we're going to go to control rig classes and here is that control rig that we created for this character so as soon as we add it you can see that we have all of our motion but we also have all these new nulls that we can use for our control rig so by default our animation in our control rig is set to FK so that means if I to move the hand around we're going to get some very bizarre results I can rotate the hand but I can't move the hand around which is really what I wanted to do when we're doing motion capture editing so in order to get access to ik settings for our control rig we're going to unwiring go to Global and I want the arm FK ik switch so for both of our arms I want this to be an ik control rig and this means that we can now just drag our hand to where it should needs to be and then the rest of the arm is going to follow and this is a very powerful way of doing motion capture editing you can also see that by default because we didn't bake this we can actually make changes let's change this and that change is going to be reflected for the entire entirety of this uh entire layer that we have so we have a whole new layer here right under our animation and we can affect any change will be for the entirety of that animation so in some ways that can be a good thing maybe we just want to broadly make the arms wider or do something that we do want for the entirety of our animation usually though we're going to be fixing specific issues so I'm going to delete this layer you'll see that everything will go back to normal and I'm going to read that control rig and let's say we wanted to do a edit where hand kind of shifted position so let's say right here for instance we don't want our hand coming down as far as it is so I would go to the beginning of my motion set a key frame go to where I want the change to take place let's say maybe I want it to be over here instead for some reason I could add another key frame and then if I wanted to go back to its original location I can copy that first key frame and paste it and we have just done a little motion capture edit that one maybe doesn't make the most sense so I can always delete these and really what I want to do is fix any clipping for this elbow so instead maybe we grab this elbow set a key frame move this out set a key frame and then copy this paste and now we prevented a little bit of clipping so using this system you can do some really powerful motion capture editing right here in sequencer and again right now we have ik legs but if we wanted FK legs we could always just go and toggle that in our Global controls and what we can also do is maybe we want multiple layers of edits I can always just go and you hit an additive layer and this will add more layers on which we can do other edits to other bones if we want to keep some of these uh edits separate from each other so this is a really really powerful new tool and I think it is also good to know that this might not work right out of the box for every character some characters you might not be able to build the control rig automatically some characters might give you funky results uh once the control rig is applied I've had odd things happen with the knees or the hips that's just some of the you know Kinks that'll continue to get worked out in this new version of unre um by 5.41 I'm sure a lot of these things will be fixed but this is a really powerful new way of doing retargeting compared to the old version looks great Okay so we've gone through the workflow for a standard character now what about for a metahuman so I'm going to delete these tracks and my character from the scene and let's go do the same thing for our metahuman here so every time I've got a metahuman I'm first thing I'm going to do is open it up and I want to make sure that my level of detail is static so I don't get things like hair disappearing or changes in the amount of detail depending on how far away my camera is from this character so I'm going to go and go to lood sync change forced LOD to zero that's the highest level of detail and now we'll just be persistent negative -1 is in Dynamic level of detail that changes depending on how far away the character or the camera is from your character so metahumans are split into all these different meshes right and so for retargeting what we want to focus on is this body mesh right here and we can find the body mesh by clicking it in our Global metahuman blueprint and right here you can see it says medium me uh male medium normal body and if we double click this it will actually bring up that metahuman body asset here which looks a little bit funky but that's going to be okay for our purposes so in order to retarget onto our metahuman we're going to go back to our motion capture we're going to right click on that motion capture we're going to go to retarget animations same as before and this time though I'm going to type in M medium and I want to go find that body uh asset here so I'll load that in you can see it looks really funky and even though you can't see it the facial motion capture is also being included on this take um but you know everything looks good so let's just trust that the system works I'm not going to export the assets because I just need the animation but if you wanted to of course you could export the retarget assets and get all of those ik ret targeters and everything like that that it that it produces but we just want the animation for this so I'm going to export this out and here we have our animation and if we double click you can see that we have a little bit better representation of what our character looks like and of course the the uh facial motion capture isn't coming through but it is on this animation we can see it right here it's just not being reflected in real time so when we load this into sequencer we'll see that facial map so let's just do that I want to add my metahuman to my scene let it load up here then we are going to go back into sequencer I'm going to add my metahuman to sequencer when I do that it's going to automatically load in the control rigs for both both the face and the body we do not want the body control rig to be added so I'm going to delete that but I'm going to leave the face control rig because we might want to play with that and it doesn't actually interrupt anything so when in order to add our motion capture remember we've already exported out our animation so now we just need to go and find it performance here it is add it to our metahuman and there we go everything is working right out of the box we've got our animation retargeted to our metahuman as well as the facial motion capture so this is great but what if we want to do some map editing on our Med human as well well the metahumans actually come with their own control rig already so in order to add that control rig we're going to go back to frame zero and I'm going to go to Body Hit plus control rig make sure that layered is selected again and then under control rig classes by default we'll have this metahuman control rig option so if I add that in it's going to add in the metahuman control rig and everything again will be working just great and similar to our other modular control R Rig we made by default this is going to be in FK so we can't move the hands around we can rotate them if we wanted to do our editing this way we could by rotating each limb but we don't we want that beautiful ik system so how do we get that same thing we're going to go back to that GL Global control and here there are the FK ik switches and if we toggle these on same thing we now have an ik control rig that we can use to do map editing and we can key frame this and do all the same things that we were doing with our other character before and again if I don't like this which it looks like I screwed something up I can go and I can just just delete the layer and I can always just add in a new layer if I want and keep moving the same thing is true of the facial uh control rig here we can go and do additive animations whenever we want and these anything we change here is going to be persistent across the entire uh time span of the timeline unless we key frame it so that's just something good to keep in mind so there you go that is how you do retargeting using the new Unreal 5.4 systems as I said in the beginning this is just a brief video because we know that this is going to continue to change they're going to keep fixing bugs uh as they get found but for now if you want to hop in and start playing around with the new tools hopefully this will be a good guide for getting you started put any questions you have down in the comments below or any tips that you might have and we will update this workflow as we find better ways of doing it okay thanks for joining everyone see you on the next one

Channel: Rokoko

Views: 7,190

Keywords: mocap, motion capture, smartsuit pro, motion library, 3d animation, 3d character, vfx, unity dev, game dev, unreal dev, unity3d, unreal 4, motion, film animation, review, demo, handson, blender, maya, autodesk, cinema4d, maxon, behindthescenes, education, student, studentwork, analysis, tracking, capture, mograph, motiondesign, learning, treadmill, drift, fix, magnetic, interference, xsens, noitom, perceptionneuron, awinda, vive, htc, ikinema, nansense, inertial, sensors, ik, locomotion, loop, loopable

Id: t38VZdyON_4

Length: 25min 34sec (1534 seconds)

Published: Fri Apr 26 2024