How to Use Rokoko Video FREE AI to Mocap

hey what's up so I'm going to show you how I made this quick little animation that I posted on my tech talk which if you're not already following consider doing so so today I'm going to teach you how to use a cocoa video to create free mocap that you can put on any 3D character you want you're going to learn how to prepare a model to be rigged how to rig a quick skeleton how to generate your motion capture and Export it how to retarget it onto your 3D character and then how to clean up that animation let's get started [Music] okay so today we're going to be working with this really dope Spider-Man model that's created by hugo9p1 bro this guy makes some insanely good 3D models of Spider-Man and like all the different Spider-Man we've seen on the big screen and the best part is they're free so definitely go check this guy out give him some love I mean he has some just incredible looking models okay so I'm just gonna quickly delete this skeleton here and okay wow he's massive it's really cool about this model is it has eyes that you can actually animate with shape Keys really really really really cool now I want to keep these so I've separated them into their own little mesh object here and I've just put them back in place on the model it's not perfect but I mean it'll get the job done now we're eventually going to take these eyes and put them on the rigged skeleton model of Spider-Man instead so that's why I separated them for now now we have just the Spiderman mesh with no skeleton and a nice neutral pose literally perfect love it it's great so we're gonna go up here to file export fbx Spidey Spidey no rig we're gonna limit it to selected objects and we're just going to make sure mesh is ticked just want to make sure it only exports just that mesh and we can also just do copy so it copies over the textures too and Export once when you have your model exported it's time to open up accurig and automatically rig a skeleton to it so I'm just going to open up Acura you can see right here Octa Core accurate so actor core lets you automatically rig any sort of 3D character you can imagine or want to rig and it's really intuitive really fast really easy the rig is amazing in my opinion it's way better than miximo because you have control over the finger placements it's absolutely insane and what's really cool is once when you have your character rigged in accurig you can then upload it to their little Cloud database thing where you can then use their actor core motion Library which is basically just their huge library and collections of motion capture animations and so you can apply those animations to the 3D character that you then just rigged it's insane I love it it's amazing this is all free it's absolutely incredible so here I'm inaccurating I'm just going to open up the file that we exported right here Spidey no Rigs and boom here we are so all we have to do is Orient it so that the front is facing us and that this line make sure this line is directly in the center everything looks good to me so I'm just going down here to the bottom right I'm going to click rig body so it's doing its thing it's doing a bunch of math and stuff and boom look at that it has all these placements here now if you look over here on the right you have a preview of like where these placements are supposed to be and if you hover over one it tells you it gives you a better idea of where that placement should be so I'm just gonna go and move this one over here just a little bit closer to the center in fact I can actually use symmetry like this and I can move both elbows so they're a bit more centered and more in the elbow joint that I think I'm gonna move these actually those look fine actually I'm going to move this up maybe just a little bit these in maybe a tiny bit adjust the knees I mean honestly it does a pretty dang good job I doubt that these modifications I'm making are going to have any real impact but I'm gonna do it anyway so this looks good to me I'm gonna go ahead and rig the right hand it's asking me how many fingers does Spider-Man have Spider-Man has five fingers he's a human so he has five fingers he's gonna do more math and stuff voxelizing right hand I don't even know what that means and there we go look at this it has automatically placed a bunch of finger joints here for us and again same as before it shows us where the joints are supposed to be placed if you hover over one it shows you exactly where it's really really cool and also if you hover over one of the joints in the thumb it'll highlight in a darker green color the rest of the thumb joints so sometimes for characters that have like maybe not as many fingers or something like that or a weird hand pose sometimes the finger joints can get a little discombobulated messed up and then you have to manually put them in place but it takes a few extra minutes it's not a big deal just wanted to let you know that sometimes it's not 100 perfectly accurate so I'm just gonna go ahead and move these uh fingertip joints they always go off the edge for some reason I don't know why so I'm just going to move them in a little bit and you also see that this one right here this little triangular cone here it says the cone should be perpendicular to the thumbnail so this is a tiny bit off to me so I'm just going to move it a little bit by moving this little blue wheel right here it highlights yellow when I hover over it and just move that around until it looks about perpendicular that looks good to me sick that looks pretty dang good so what's what I like to do is over here there's a mirror options you can mirror left to the right or right to the left so this is the right hand here so I'm going to mirror the right hand over to the left now if all is symmetrical if I go ahead and click over here on this left hand tab right here up at the top right if I click that the model is perfectly symmetrical which I believe it is then all the joints Place automatically right there as well that's all good and fine and dandy to me so I'm going to go ahead and right here in the bottom right I'm going to click finalize character it's doing a bunch more math it's placing the automatic weight weight paint maps and stuff and it's weighing the skeleton to the mesh as well as the fingers and uh yeah let's see where this fits out here we go this is this is our eyeless Spiderman moving around look at him he's just kind of doing a little standard idle animation right now but what you can do is preview different motions and and stuff so right here they have a body Tab and they have different poses and animations like different male poses female poses and stuff they also have like that looks that looks kind of cool they also have a little animation that you can preview so here he is just doing some stretches you know hands on the hip that looks great oh my gosh look at that just stretching that back I also have a hand animation that you can do so you can zoom in get real close and tight on the hands here make sure they look okay some of his hands let's see the fingers let's see the fingers how do they look that's not bad that's not bad look at those finger animations pretty nice pretty nice now another thing I want to point out is that you can actually adjust the rotation of some of these bones here so usually I like to take this the upper arms here and I like to rotate them on the z-axis so out a little bit but this automatic rig did a really good job and I actually like it how it is right now but usually they're in a little tighter than I would like so I often come down here to the upper right hand arms or sometimes the collarbones and I make the right one negative five and I make the left one five and that's kind of just like adjusts his arms out a little bit but honestly like I said this this one did a really really good job so once when you've tweaked some of the bones and accurig and stuff and you rotated them around how you like and you're ready to export this is how you do it so you go down here at the bottom right you click export export they have they have many different ways that you can export your export and fbx file and Export a USD file which is I guess used for NVIDIA Omniverse I have a mess around with that if you want me to I mean I can I guess but they also have it where you can save this as an avatar to use an iclone 8 which is another one of their softwares used to animate characters and stuff so I'm just going to export in fbx and they have a lot of different applications that you can choose we're just going to use blender it's asking me if I want to save the textures and stuff I don't really want to because I already have the textures in blender so I can just reapply those so it's fine I'm just going to go ahead and down here and click export I'm gonna bring up this so down on my files Spidey no rig well actually this is Spidey rig so I'm going to go ahead and click save and exports it out now that you have your model rigged it's time to create your motion capture animation so we're going to jump over here to Rococo video you can go to rococo.com product slash video or just in their little products tab here Rococo video and it'll take you here and it's this is this is what it is it's free AI motion capture and then once when you're on the page you're going to click try it out now you're gonna have to use your Coco ID if you don't already have one you sign in create one it's easy and free so I already have one so I'm just going to sign in so under Coco video you can create different scenes for projects and stuff now to upload a clip or record a clip you just click create scene I'm going to name this tutorial and you can record using your device camera or a local video file so you just upload a video clip or you record with your webcam if you don't have a webcam you can just shoot with your phone and then upload the clip to Rococo video so once when you have your clip uploaded and you're happy with it all you got to do is just click right here turn into animation and it's going to do more math and stuff and it's gonna do a bunch of AI junk and it's going to spit out an animation now over here on the right it says that Rococo studio is their software to process and clean their motion capture data so what you're going to need to do is download Rococo Studio say click download here open up a new tab and then you just choose the right one for your platform if you're on Windows Rococo Studio beta for Windows they also have a legacy but I think for this you're gonna photo cocoa video You're Gonna Want Rococo Studio beta so go ahead and click download there and install it okay so here I am in Rococo studio and if I go over here to project scroll down to Rococo video and I have my Spider-Man dance video double click that to open it up here's the new recording over here I'm going to right click it set Target actor and set to Jiffy already great awesome so now if I just double click the recording over here and here is my animation now one thing I've noticed is that it likes to lean the character forward a little bit I'm not sure why it does that but it's okay I'm going to show you how to fix it when we clean up the animation so let's say you're happy with the animation I'm pretty happy with this I was pretty happy with this for the video so I'm going to go ahead and go over here to the export tab on the right I'm going to export it as an fbx file you can export it with the body mesh I've started to not do it because I ended up just deleting it anyway but sometimes it can be useful for reference skeleton preset I'm just going to choose human ik I'm going to export it as 30 FPS I choose include reference pose it's particularly just a t pose on frame zero and then to set your file destination click these three little dots I'm going to choose my downloads here boom okay and then you click export clips and it exports it out now back in blender I'm going to go to file import fbx first thing I'm going to do is I'm going to import the rig that we created in accurig I'm going to scroll down here to Armature and I'm going to click automatic bone orientation and then we're going to click import fbx so here's our Spider-Man he has no eyes no materials which is fine I'm going to click on the mesh really quick and I'm just going to set the material let me just bring his eyes back six so now we have our rigged Spider-Man and blender now it's time to import the motion capture and then retarget it I'm going to go to file import fbx and new recording two this is it again I'm going to scroll down here to Armature I'm going to click automatic bone orientation it's already checked on and import fbx I'm going to hit n and I'm going to go down here to the Rococo add-on here if you don't have this then you can download it it's free I'll leave a link in the description below so I'm going to go over here to the retargeting drop down on the right and I'm going to click Source our source is the reference that is the motion capture right here reference and then the target is going to be Armature which is created which is the rig that we made in accurig and then I'm going to click build bones list now it probably automatically won't line up perfectly for you sometimes no it does for me because I've done this before but you basically just want to go through and make sure all the bones are mapped right okay and see this right here is a perfect case of an oopsy so it has left arm is left upper arm and then right here left forearm is left up arm again that's wrong we don't want that we want the left forearm so we choose that there we go left hand left hand gray index one two three that looks great middle one two three gray pinky one two three awesome ring one two three perfect thumb one two three and they're all left perfect awesome so we have the neck which is like twice the one that's fine the head the right shoulder is the clavicle right arm upper arm right forearm forearm right hand right hand index one two three middle one two three pinky one two three ring one two three and thumb one two three awesome has the up leg which is the the thigh the left the the leg is the calf the foot is the foot toe base toe base thigh calf foot toe base Awesome everything looks great so I'm gonna go ahead and click retarget animation and see what it gives us here we are now something looks a little bit off his arms are all jacked up what the heck is wrong why is that well this is because of the rest pose that our Spider-Man animation is in now if you look right here he's not in a t-pose anymore our reference is in a Tipo Spider-Man is in an A Pose so that's essentially what what's wrong with this so I'm gonna go ahead and control Z all that all that that we just did so now Spider-Man is back in his t-pos but it's not his rest pose if I go over here into edit mode you can see oh Spiderman goes right back to his a pose because that's his rest pose that's his default pose that's his default position so we need to make this T pose his default position now an quick and easy way to do that is using the cats add-on for blender which is also free link to it down in the description below now it's used for like VR chat and stuff I don't know what all this is at all I have no idea but I just use it to change the rest poses of all my characters so if I go ahead and I select the Spider-Man Armature here and I click start pose mode puts them right back to his default position and I'm just going to go ahead and click on a skeleton I'm going to go to viewport display in front I'm also going to go down to the Armature tab here viewport display and I'm going to change it to a stick it just makes things a bit easier for me to see so basically what you want to do is you just click on the bones and you kind of just rotate them up so that they kind of match the t-pose so I'm going to spend a couple minutes doing this you do the same now if you click on a bone and then you rotate it over and it's not rotating the whole the whole arm uh all you got to do is just press h hide it and then you grab the right bone and it's great and there we go I sort of kind of got Spiderman and t-pose the more time you spend on this and getting the rest posed right the more the better your retarget will be but let's say you're happy with this I'm decently happy with it looks this looks fine to me for a t-pose you can spend more time getting the fingers just right and stuff if you want but I usually don't I usually just clean that up manually and then I go over here and I click apply as breast pose so now we go back to Rococo and we click rest and retarget animation and now if we look at our animation we have Spider-Man doing the motion capture animation just fine and dandy look at that now I'm gonna go ahead and get his eyes sorted and back on the actual Spider-Man mesh okay dope I got them situated real nice and pretty on his face they look great they look just like they're part of the suit awesome so how do we attach these well I'm just gonna use a little child of constraint here I'm going to set the target to the Armature and then I'm going to click the head as the bone set inverse and now if we look the eyes are following them just fine and Danny we can still click on them and adjust the open close however we wish and animate this if we want and there we have it we have our Spider-Man doing his uh doing the motion capture animation he's still leaning forward that's fine I'm going to show you how to clean up all these little inconsistencies and things that you might not like in the animation so I'm just going to go to this little standard frame right here and I'm just going to click on the skeleton here and I'm going to go down here to the bottom left I'm going to click on this little icon here I'm going to change the workspace at the bottom here into the non-linear animation editor and then you can see our Armature right here and then you can see all these keyframes here I'm just going to make this an action it's basically like a boom I've solidified the animation it's like a movie clip that I can edit over here on the right I'm going to change blending to combine if you don't see it just hit n on your keyboard it'll bring up this tab here I'm going to go back to the timeline drag it down so I can see make sure the skeleton is still selected I'm going to go into pose mode click a to select all the bones in the skeleton go over here to the object properties and I'm just going to keyframe location rotation and even scale just in case I want to change any of these properties here and then you can see it created a keyframe down here I'm also going to turn on auto King by clicking this little button right here boom this keyframe over to the beginning go over maybe 10 frames or so and I'm going to try to find the hip bone let's see is this the root bone it is so I'm going to rotate them over about maybe just so he's standing about upright that looks pretty good to me and you could spend some serious time tweaking these legs if you want but again let's say you don't like one of one of these little arm bins here well what you can do is you can since this bone is already keyed you can just rotate it over oh grab the wrong one there whoops oops gotta hide and select the right one and rotate it over and boom now his arm is rotated in in place and then I can make it go over here if I really really want to and do some weird wacky stuff so here it is in place looking good here and then it just rotates over and his arm is broke oh no yeah I don't like that so we're not gonna do that maybe I want this one to go down rotate over like this maybe in a little bit not bad and so yeah you can really go through and really fine tune all these bones and animations and stuff you can even do the same thing with the fingers if you want I did that for a split second I made his hands do the little Spiderman thing so I mean you can really spend some serious time on this but right out of the box this is really great this is really really really great this is an amazing alternative to do free motion capture it's suitless you don't have to do anything you just record yourself doing the motion and do a little bit of manual work and you have something that is totally great and usable so there you have it in the future I'm going to be making more in-depth tutorials and workflow videos about my motion capture process specifically in Unreal Engine 5 and meta-humans for my Batman Beyond CG film so be sure to subscribe so you don't miss those videos also if you learned something new feel free to click the like button down below it helps push it out to other people who might be interested in the video if you have any questions comment down below and I'll do my best to help you out thanks for watching and be sure to follow me on all my socials link below and I'll catch you in the next one

Channel: James Luke

Views: 58,730

Keywords: rokoko mocap, rokoko video, video to mocap, how to turn video into mocap, rokoko, mocap, how to turn video into motion capture, free video to motion capture, rokoko motion capture, rokoko ai video, rokoko free mocap, how to use rokoko free ai mocap, how to turn a video into mocap, how to, rokoko video mocap, rokoko video ai tutorial, rrokoko video ai mocap, rokoko video free ai mocap tool, rokoko animation, rokoko video free ai mocap, rokoko live stream, ai mocap

Id: b4we_0iXC44

Length: 19min 26sec (1166 seconds)

Published: Thu Dec 29 2022