Hand Tracking with Unity XR Interaction Toolkit

I was waiting for this for a long long time and it finally happened we can now use end tracking with the unity XR interaction toolkit that's right before if you wanted to use and tracking with a headset like The Meta Quest you could only do so with their own SDK but with the Redstone Unity XR update this is not the case anymore also in today's video we are going to see how to set up and tracking with the unity XR toolkit and of course go even further to learn how we can use it with the other feature of the unity itar now I'm putting a lot of effort in this video my goal is to make one per week every Sunday and you can help me for free by subscribing and liking this video down below or even join our community on patreon where you can get access to the source code of all of my tutorials and exclusive content but without further Ado let's get started okay so I'm inside a completely new Unity project effect I want to show you the setup from scratch so that you will be able to do this on your own and anyway as with any VR setup the first thing we want to do is to go to the XR management settings with edit project settings XR plugin management there let's click on install XR plugin management and we want now to select the open xr4 Desktop over here as you can see it may ask you to switch input systems so let's say yes and now it should restart the editor as well perfect now let's do the same in Android and click again on open XR so as you can see for both we have a little warning showing here because we need to add some interaction profile so let's go to open XR under the XR plugin management and for desktop click on the plus button and add all of the device that you want to support in my case I will add everything now let's go in Android Droid and when we want here is to click on the plus button and add the Oculus touch controller profile and then enable meta Quest support on here there you go but now this is where things get tricky so be careful because in my case by adding open XR we have downloaded the openxr plugin of course but not the latest version that's right if we go here to Windows package manager select in Project as you can see we have here the openxr plugin version 1.5 install but if we want to use end tracking we need to have at least open XR 1.6 and this might not be a problem that you will have in the future of course so if you see anything above 1.6 you are good to go but otherwise what we can do is simply click on this little plus button over here and click then on ADD package by name right come dot Unity Dot gigs are DOT open XR and click on ADD and after reading a little bit as you can see we are the openxr plugin Now set to 1.7 perfect and also maybe like me you will have this window showing asking to fix some things on our setting so let's simply click on the fix button to resolve this issue there you go now if we close this window and the package manager as well if we go back to the example again management desktop as you can see we have also here a little error showing so let's click on it and it is basically asking us to restart Unity as the open XR has been updated so let's click on fix as well then click on apply and now wait for the editor to restart there you go now the new appendix R is set up of course as I told you this is something that will not be needed to do in the future but anyway from here it is a smooth ride next let's install the unity XR interaction toolkit for this let's close everything go to Windows package manager select Unity registry and search for XR interaction toolkit there it is so as you can see the version I can download in my case is 2.0 but for end tracking we are going to need at least Unity verse version 2.3 so if you don't see it here you can actually download the latest Unity version available like we did just a second ago with the open XR by clicking on the plus button add the package by name and write come dot unity.xr dot interaction.toolkit and then let's click on ADD and voila it is downloaded let's click on yes I made a backup go ahead now as you can see this has correctly downloaded a version above 2.3 and as we always need to do with the unity XR toolkit let's also import the starter asset by clicking here on import perfect but the XR interaction toolkit is not the only package that we need to use and tracking we are going to download Unity XR hand package now you won't find this package in the unity list unfortunately but we can still downloaded it in the same way as as I showed earlier by clicking on the plus button add package by name and write com dot unity.xr dot hands and finally click on ADD there you go if we have a look at all the package that we have in the project as you can see it is right here the famous xrns package so finally let's also click here on import to import the hand visualizer sample which will be necessary to animate a hand model to the position of my hand in real life but now by downloading the xrn package this has done something very very important to our open XR settings now if we close this package manager and go back to the XR management settings with edit project settings open XR as you can see we have now two new little tiger right here that we can enable to enable hand tracking on desktop and now if if we go to Android as you can see we have the same so let's enable it also and now everything should work and we should be able to use and tracking on this project and there you go congratulation the boring part of the tutorial is now over with that we have finished with all the settings necessary to use and tracking with the unity XR so now let's close the project settings and learn how to use it okay so here I'm on an empty Unity scene let's add a Plane by right clicking on the hierarchy go to 3D object plane reset its position over here to zero zero zero and this will be the ground for our little VR and tracking application now we can select the main camera and remove it by pressing on the delete key and at this point we could get started on making a XR origin like I did in my tutorial series about the Unix art toolkits but with the latest version of the unity XR toolkit there is a pre-made XR version that is ready for for us to use wisdom interactor and Locomotion that I will be able to find in samples XR interaction toolkits the version that we have so 2.3 in my case stutter assets prefabs and here it is the complete XR origin setup so let's simply drag it in the ER Key now this rig has a Locomotion system with both continuous and teleportation it can interact by grabbing object and more and I cover it a bit more in details in this video so go watch it if you want to know more about it but anyway to add and tracking to any VR rig it's actually very simple we need to add a component called hand visualizer somewhere we could do so on the XR origin right here but let's instead do it on a separate game object so for this let's right click on the camera offset create empty game object and this will create an empty child of the camera offset as you can see and we can rename it and visualizer there you go then let's click on ADD component and add a hand visualizer component there it is so this component need a reference to the XR origin so let's drag it over there then for the left and right and mesh there are some ready to use and model available by going to samples XR hands the version that we have and visualize our models and now let's go back to the hand visualizer and drag the Left End on the left hand mesh and the right end on the right hand bench there you go next we can say the material that we want to be used by the end so in my case let's go to the asset root on the project I'm going to right click create material rename it blue and simply set the color to Blue this will of course make a very simple blue material that we can now drag back to the hand visualizer there you go we also want to see the hand mesh so let's click on draw mesh is obviously and now we have two of the fitting that can be useful in some case to display the bones and the velocity of our finger joints so for this let's go back to hand visualizer in the project folder and there go to prefabs now we can simply Drag The Joint in the debug draw prefab and the velocity prefab in the velocity prefab and initially in my case I don't want these debug syncs to appear so let's simply set the velocity Type To None and there you go now it is the moment of truth let's click on play to try what we made okay so I'm in VR right now so it is working and by the way I had a weird bug on my end which didn't launch VR in the first time that I tried so I needed to reopen the unity editor this might be helpful if you have the same issue but anyway I can see my controller right now but if I rest them on my desk and that they do not move for a few seconds ta-da my ends appears and end tracking is working now this is always awesome and as you can see this works really well the tracking is doing great and now if we don't leave play mode and that we go back to the hand visualizer component as you can see I can show or hide these hands with the true mesh toggle and if I click on debug draw joint as you can see these show all the joint used to animate my hand next if I go to Velocity and set it to linear for example as you can see this one show the direction of the movement of my finger which can be pretty useful sometimes but now two main problems remain as you can see the controller of the VR rig is on my wrist and so the question is is it possible to hide it and more generally speaking switch from controller to end tracking more easily and then the second big question how can I use this n tracking with the interaction system of unity XR and for this we are going to take a look at a gorgeous demo scene that Unity has prepared for us the end demo scene so let's leave play mode to try it okay so to download the ndimo scene let's go back to package manager with Windows package manager select in projects and go to XR interaction toolkit there you go down here as you can see we have the hand interaction demo let's import it there you go it is downloaded now let's close the package and as you can see this has added a new folder on the unity XR simple folder but more importantly this has added a super cool prefab for n tracking that we can find if runtime prefabs and this is the prefab that I'm talking about the complete XR origin hand setup we can drag it in the hierarchy next to the basic complete X origin from earlier and as you can see these two are very very similar the thing that differ is that it does not have The Locomotion system instead we have here the hand and controller manager and this is the script that can allow you to switch from controller to end tracking now in its children we can see the hand visualizer that we added as well to the other rig but the main important thing with this prefab is how it is used with the interactor system from Unity XR because as you can see if we go under left and right on controller we have three types of interactor poke direct and Ray but we have the same for the left and right hand as well which is the one used by hand tracking now if we go to the direct interactor of the left hand for example as you can see what changed from the setup with controller is simply the input action reference for example the position used here is the pinch position to allow grabbing and placing the object grab between our thumb and index and if we go to the select input double click on it as you can see under select we have this selection that occurs with the grip pressed for the controller but also on the index pressed which is when we are pinching with and tracking and there you go this is how you can hook and tracking to the unity interaction system so anyway now let's remove the old XR origin now that we know more about this one and I think it is time to give it a try by clicking on play there you you go as you can see I can see my controller but when I switch I can see my hands and this works the other way around oh and by the way if the ends are pink in this example this is because in my case I'm not using Universal rendering pipeline but the standard pipeline instead so you can basically leave play mode go to the end visualizer where you will be able to find the end mesh material which is set to none actually so what we can do is just give it the blue material that we made earlier just right there and now if we click on play again it should not be pink anymore and you should be able to see the ends with the same material that we did earlier but this time with this beautiful rig as you can see we can easily go from controller to end tracking in just a few seconds without having the controller showing anymore this is so cool now as you can see there is no interact table on this scene to test all of the interaction so what we can do is actually leave play mode and instead we are going to try the rig in a great interaction scene that Unity made for us in samples XR interaction toolkit 2.3 hand interaction demo runtime and here double click on the end the most scene there it is so as you can see we have the complete exha origin and setup prefab from earlier but there is also a bunch of interactable using the unity XR interaction toolkit like direct grab poke and so on so let's click on play to try this oh and by the way if the material are all pink like it's my case because I'm not using Universal rendering pipeline you can simply go in the material subfolder over there then select all of these and for the Shader select it to standard and this should fix your little issue and there you go guys guys as you can see you can now understand the power of unity XR this is just so cool how it can simply enter hand tracking by changing some input action and as you can see this works really really great they might want to make an End tracking game right now but anyway I hope that you enjoy watching this video but before leaving we have a deal if you made it to the end of this video you can go down below and make sure to like it thank you for watching and also a big big shout out to my patreon which are the one allowing me to make this and see you soon of course for another tutorial bye [Music]

Channel: Valem Tutorials

Views: 53,937

Keywords: beginner xr tutorial, how to make a virtual reality game in unity, how to make a vr game, how to make a vr game in unity, make your own game in unity, unity, unity vr tutorial 2020, unity xr toolkit, valem, virtual reality, virtual reality unity tutorial, vr, vr tutorial beginner, xr unity tutorial, hand menu vr unity, hand tracking unity, hand tracking unity xr interaction toolkit, hand tracking vr, hand tracking meta quest, hand tracking unity vr

Id: mJ3fygb9Aw0

Length: 16min 29sec (989 seconds)

Published: Sun Mar 12 2023