DIY Kinect Motion Capture Studio | Blender

for the past few months I've really been wanting to break off the traditional blender tutorial course and do something a little bit more outside of the box I came up with a seemingly endless list of ideas but I really didn't like any of them and that's when the idea popped into my head can I make my own motion capture studio the answer yes and so can you it's really simple and relatively inexpensive so what do you need to get started well for one you need a system to run it off of this system by no means has to be top-of-the-line however there are a few important things that you have to take into mind one you need to have a USB 3.0 port with a controller made by either Intel or Rene so if you have a different USB controller on your computer it may not work if you're unsure about whether your computer has this or not I'd recommend googling some specs about your system to find out additionally you'll need to make sure that your system is running Windows 10 this unfortunately does not work with any other operating system second you're gonna need to get your hands on a Kinect v2 they're currently going for about $45 on Amazon and I don't expect that to change too much you'll also need a Kinect adapter for Windows 10 which is more or less just a bundle of little boxes and cables that allow you to plug one thing into another it's unfortunate that you have to have this but it's also kind of necessary it's currently going for just over $39 on Amazon and again I don't expect that to change much either additionally having some sort of mounting device is really helpful for the Kinect because you can stick it on more things I personally bought a little TV mount so I can just put on top of my monitor when I'm using it however other people who are working in a more open area may want to invest in some sort of tripod because it does have a little quarter-twenty screw mount on the bottom that you can screw into just about any piece of camera gear that you've ever had the last thing you'll need is just an open space to work in whether that be your garage or your living room it doesn't really matter as long as you have a few feet to move around you're good to go the total cost of all necessary parts is only about $85 totally worth the money especially when you get to mess around with it once it's done as soon as these parts arrive on your doorstep you're ready to get started just take them inside and start packing looks good to me we'll start off with this little doohickey with no cables attached to it it's great because it has a bunch of little symbols telling you where to plug things in for example connect over here DC power down here and USB up here with this block as our centerpiece we can go ahead and grab our connect and start plugging things in mine's all wound up so I'll unwind it real quick and once it's done we can go ahead and set it down and to plug in the cable connecting that into the connect side of the block so we'll plug that in and set it back down additionally if you have any sort of mounting bracket I'd recommend putting that on now they usually snap on pretty easily next up is our oh hello well nice of you to stop by mr. CAD get out of here next up is our power supply box which is the one with the cable with the power connector on the end we can plug this into our main block and then put it off to the side we can then take the USB type B side of our cable and plug it into our block leaving the regular USB side off to the side so we can plug it into our computer later finally we'll take our power adaptor and plug that into the power box that we have over here on the left next we can take the connect and put it somewhere where it's pointing at an open part of our room it's also important to keep in mind that it has to reach a power outlet as well as your computer it would also be nice if my pets would stop sabotaging my videos anyway we can plug our power cable into the wall and our USB cable into a USB 3.0 port from there it's all on the computer so we'll start by heading over to Microsoft's website and downloading the Kinect for Windows SDK there will be a link down in the description all you have to do is click this continue button check the yes box click Next log in to your Microsoft account if you don't have one you can create one at this screen as well and when it loads you can just hit the cancel button because otherwise it's gonna sign you up for a bunch of different mailing lists that you don't want to be part of and it should start downloading while we're at it let's also head over to ni - mate comm or ni mate comm this is the software will you be using for the actual motion capture so let's go ahead and click on the download tab and then we can go ahead and download the windows 64-bit version while we're on this page we also want to scroll down and download the blender plugin right below it we can start by running the Kinect SDK installer first this is by far the most important because it contains all the Kinect drivers that you need to function installation is pretty straightforward once the progress bar is done you can just click close in the lower right and we're ready to move on to the testing phase to test to see if our Kinect is working we can click the start button type in Kinect and then we should have an option that says SDK browser version 2.0 when we open this we can see a bunch of different things that we can do with Kinect and play with but for now we're just going to run the Kinect configuration verifier this will run through a bunch of different checks to make sure it's working all of the checkmarks should be green give it a few seconds though because sometimes it has to double check as you can see my USB controller isn't quite working however it's always been like this this is still something that's heavily under development I know that works for me though if yours for some reason doesn't work I'd strongly recommend researching a little bit more about your hardware and what could possibly be going wrong so if everything is in shape we can close out and have some fun press start type and connect again and open the Kinect studio version 2.0 connect studio is a fun way to play around with your Kinect up at the top we have four tabs monitor record play and log but we only care about monitor over here on the right we have this big thing that looks kind of like a camera from blender and if we click this little icon up in the upper left here our Kinect connects and well look at that that's a 3d interpretation of where I am and you can see that the skeleton is building around me I can stand up move around and the skeleton follows me it gets to be really fun to play with just moving around and trying out different poses and watching this camera analyze you in real time it gives you some insight as to how the Kinect actually works as well which is something I've always wondered we can also switch into 2d view up at the top which kind of gives it more of a camera's interpretation of 3d space we can also see how color mapping works with depth for example if I get too close to the camera my hand becomes black as I step back it becomes red and as I step all the way back I become yellowish green now it also be a great time to open up the SDK browser again and look through some of the fun things to play with in here for example HD face basics allows you to track your face in real time all your emotions your mouth movements it's it's really weird very uncanny but it's it's so much fun to play with another thing I'd recommend trying out is the kinectfusion Explorer which allows you to 3d scan pretty much whatever you throw at it this might be a another project further down the road I guess we'll just have to see how this video does once you're done messing around with that go ahead and install Delhi code and I'm 8 unfortunately the Installer for this one is pretty straightforward as well so you don't have to worry about it installing 8000 random programs on your computer like some softwares try and do whoo thank goodness when you get this Kinect for Windows runtime pop-up click cancel on it because we already installed that and we installed a more up-to-date version however the primeSense drivers you should install for some reason mine always fails the first time but when it tries again it works I don't know why that happens but you know whatever anyway we can click finish with the box still checked and the program should launch keep in mind that and imate is not fully a free software it's it's free and you can use as much as you like it just has some limited features and that's what that's how I use it that's how I've always used it and it works fine just that way for me so when it actually starts it launches in your system tray which is weird so if we just click it down here I ran away from me if we if we click it down here and right-click it down here and click control interface it pops this window up which is where we can actually do what we need to do so we can come down to the Kinect for Windows tab and you can see that it is working already it's using my Kinect right here and we can see a live feed from that if we come down to the output setting here and change it to 3d we can actually get a 3d view of what we're working with and now you can see that my skeleton is being portrayed in 3d space which is awesome if for some reason you don't have a grid floor or camera and your motions aren't popping up click the start sensor button in the upper right mind says stop sensor right now but when you click start sensor that'll actually activate the Kinect and allow you to actually use it with the software as soon as everything seems to be working properly we can exit out of ni mate and we can install the blender add-on I'll go ahead and open up blender I'm using blender 2.78 however I have tested this in 2.79 and it works just fine so don't worry you won't have to downgrade we can exit out of the splash screen press control all you to open up our user preferences if we're not in the add-ons tab go there and click install from file at the bottom then navigate to wherever that add-on file is for me it's on my desktop so I'll click it and then I'll click install from file in the upper right I'll click the little checkbox next to it and I'll also click Save user settings because that will enable the plug-in every single time I start up blender so installing the add-on added this little tab to our tool menu over here with a bunch of settings and this magical little start button up here at the top is what's going to start our tracking however first we have to come back into and I mate really quick click the soil drop down next to the Kinect for Windows tab and check the box that says skeleton tracking this will actually transfer the skeleton tracking data from ni mate into blender so now let's go ahead and delete everything in our main scene just to eliminate any confusion we might have and then up at the top here we have this option that says create and then three or four different options nothing empty spheres and cubes I'm going to set mine to empties and this means it will create empties to represent me I'll move my camera into a position to where I can see the whole grid plane on the ground there and I'll click the start button and stand in front of my Kinect and you can see well there I am now the issue is the camera is not really pointing at me so I'm gonna reach over and try and fix it here but you can see my skeleton is absolutely glitching out so now I'm well in blender represented by a bunch of empties and you can see every single motion I make whether it's walking forward or backward moving my hands around everything sticks so this is also pretty fun to play around with but it's still lacking any use we need to be able to map these motions on to an actual armature and how exactly do we do that well and I make comes with it built in well more or less you'll have to create a specialized rig to work just with and I made but fortunately you have me and I already went through the pain and agony of trying to figure out how that works and I made it available to you so you can download it and use it in your projects you can either click the download link in the description or head over to my website remington graphics net it's under the resources tab once you have the rig download we can go ahead and open it up upon opening it we're presented with a bunch of information stacked on top of each other over here and a connect so I actually have this separated out on the layers down here if we go to the first layer we can we have a bunch of different empties with different names and these are actually the individual data points being transferred over by ni mate if we go into ni mate and click on the skeleton tracking tab we can actually see all these different names that's essentially what I have here except everything's an empty instead of just a name on the second layer we have a rig that transfers the data from these data points over to an actual rig so you can see each bone has a constraint for example here this one will stretch from its origin point the where's which is the collarbone to the left shoulder in this case this one stretches from the left shoulder to the left elbow when we come over to the third layer we actually have a retargeting rig now this isn't retargeted via the motion capture add-on that blender has it's actually retargeted manually because from my experience retargeting doesn't really work in real-time so this is the best alternative to that and it also works with pretty much anything you throw at it so let's go ahead and actually give it a try if this works properly we should be able to take the data from random points into this rig and retarget it onto this rig so when we click start it should transfer the data that is considering an eye mate is open so let's try it start and I'll move into a position where my Kinect can see me and there we go so a little bit of bitchiness off the bat but you can see that I am well moving around in blender as I left my knees you can see all the bones are working my arms work just fine and I should at work with the torso as well and thank you ladies and gentlemen I hope you enjoyed my brief motion capture performance if we switch to layer 2 which is the layer with the bones taking the data from the empties we can see that this works as well and the positioning of the empties themself also works so we'll do some jumping jacks too to celebrate our victory all right so we know that it works but how do we actually map these movements onto our own rig when creating your own rig it's really important to keep it similar to the base rig that we had on layer 2 if you deviate too far from this base rig a lot of bones may go to waste and the end result may not look as good as you hoped so try and stick to that base concept and base rig the simpler the better I always find it useful to move on to a new layer to work on your new rig for my custom rig I'm actually just going to use a human metal rig from the rigify add-on if you guys want to follow along step-by-step you can enable this add-on too in blenders user preferences in the case of this rig there are actually quite a few bones that we don't need we don't need any of the fingers or parts of the feet so we can actually go ahead and delete those just be careful not to delete any bones that you shouldn't for example the hand bone in the middle of all the fingers is essential for our rig to work and there we go even removed all the unnecessary bones and saved just to make sure we don't lose our progress before moving on we also need to check and make sure there are no unnecessary breaks in our model for example with this armature there is a break between the shoulder and the upper arm bone the fix for this is relatively simple just click on the little bone icon up on top here and under the relations tab check the connected box and the bone will automatically connect itself I'll do this for both while we recording this I unfortunately didn't notice there is also another break in this rig between the shoulders and the neck I ended up leaving it unconnected however luckily the break in the bones didn't really affect the final result that much however if you do have breaks I'd strongly recommend fixing them lower body breaks such as the legs in this rig don't really matter as much because there are separate entity one other thing I like to do before moving on is to enable the names for each individual bone so we can see them on this rig I'll also hold shift and select the layer that our other rig are copying rig or our base rig is on and I'll also do the names for this as well so we can copy them easier without having to switch back and forth between the rig will start our manual retargeting process as I like to call it with the torso because it's the most important so we'll select our tour oh I guess actually we should switch into pose mode we'll select the base bone of our armature in order to do this you know it's the base bone because if you press G and move it around it should move the entire rig if it doesn't odds are you're selecting the wrong bone and you should try a different one so with our base bone selected we'll come over into the bone constraints tab over here add a bone constraint and add a copy rotation constraint select capture armature as the target and then with our bone we want to select the bone that we're copying the rotation of in this case the torso additionally we want to add a copy location constraint to that way you can copy the location of the torso so we'll select capture armature for our target and also the torso as the bone I'm also going to move the copy location constraint on top of the rotation constraint just because that's I don't know it's not necessary however it's just a way for me to organize things so I just zoomed out and I also noticed my armature is kind of having some trouble here it's backwards and yours might be backwards too if it's not it's fine it's it's fine the way it is you don't need to change anything but if it is backwards either on the X or Y axis you will need to make some small tweaks mine happens to be backwards on the y axis so I'll switch into edit mode select all my bones press s and then Y to scale on the y axis then press negative one this more or less inverts the armature and as you can see it now lines up properly well kind of properly with my base armature so now we get to go through and add the constraints for each individual bone that can be posed so this is the arms head neck and legs for each of these we'll add a copy rotation constraint with the capture armature as the target and whatever bone they're copying as the bone for the case of a shoulder it becomes the shoulder so you can see the shoulder I guess actually the names are backwards here because I flipped it along the y axis but you can see the shoulder over the left shoulder in the position of where the left shoulder should be will copy the left shoulder we'll do the same thing with the arm here and for the rest of the bones in our armature so just go through and complete this process for the left and right arm left and right leg and also the neck and head you can leave any extra chest bones untouched the reason we you don't want to touch these is because we have nothing to actually modify them however we can switch into the side view here and pose them so our armature isn't quite leaning back because that it can be kind of unsightly so I'm just gonna rotate them upwards so our armature has a little bit more posture to it after this we're ready for our first test run I'll click start to start transferring the data from our Kinect to blender and when I move into the connected zone you can see it works our custom rig is now working perfectly it is a good idea to check every single bone and see how it moves as you can see that's what I'm doing here I'm checking the motion of pretty much everything I can think of and it seems to be working just fine so what's the point of having a motion captures through if you're not actually gonna capture anything first we'll make sure we're in pose mode and we'll select every single bone by pressing a twice we'll click the auto keyframing button down on the bottom here and we'll change the type to lock rot scale location rotation scale and then we'll click start now to actually start recording you need to hit the play button this process can be a little bit tricky if you're alone because you have nobody to click the button for you what I like to do is move my mouse over the play button pick it up stand back and then click when I'm ready to start recording so now all my actions are being recorded to stop the recording just hit the pause button and then go back to your computer as soon as you get back and I cannot express this enough make sure to click the red button down at the bottom again that way you stop recording your keyframes we can also stop the ni mate receiver now when we scrub back to the beginning of our video and hit play we can see that the motion was indeed captured we have our motions in 3d space on our rig doing what we want if you made it this far congratulations you've actually just made your own motion capture studio at home using nothing but blender and an Xbox Kinect having your own motion capture studio at home opens up so many possibilities whether that's just working on something for reference animation or creating an actual animation itself with nothing but motion capture footage now that you guys know how it's done I'm gonna leave it up to you to be the creative ones and come up with a use for it anyway that's all I have for this video thank you all for watching and I wish you the best of luck in your endeavors I'll see you guys later adios you [Music]

Channel: Remington Creative

Views: 714,429

Keywords: blender, education, 3d, cg, cgi, pbr, tutorial, guide, how-to, remington, graphics, lesson, vfx, ni, mate, nimate, ni-mate, motion, capture, mocap, motion capture, mocap capture, rig, skeleton, armature, retarget, retargeting, kinect, xbox, sdk, windows, windows 10, kinect sdk, kinect motion capture, addon, plugin, add-on, plug-in, delicode, adapter, v2, home, made, diy, home-made, do, it, yourself, do-it-yourself, how, to, fusion, scan, scanning, face, tracking, camera, track, prime, sense, primesense

Id: 1UPZtS5LVvw

Length: 19min 25sec (1165 seconds)

Published: Fri Sep 22 2017

Reddit Comments

[u/H4WK1NG]

Let me know if anybody gets this working with ue4 mannaquin. I'm in need of some anims !

[u/monkiebars]

Wow that was awesome! Great work problem solving all that. And, yes, I'd be interested in a video on 3d photo scanning using MS Konect.

[u/[deleted]]

Don't waste your time and money on this. I've tried this sometime ago and it only good for capturing some basic arm flailing motions. The Z depth is inaccurate and the feet are never solidly planted. There's no head or neck tracking. It's best to look for better pro solutions.

[u/MaDpOpPeT]

I don't have a Kinect 2 (I have version 1), however isn't there a free kinect ver2 plugin for ue4? That rig made me grunt in pain. LOL

[u/OrdinaryKick]

Awesome tutorial!

​

For $85 that's pretty damn interesting.

​

Most mo-cap packs are $50 alone so this is a very cheap method of generating mocap animations.

[u/tyduncans0n]

Can this be used with other 3d modeling programs/Unreal/Unity?

[u/Alphey4406]

For anyone asking there is a plugin for both cinema 4d, blender and autodesk maya.