Hand Tracking Laser Pointer & Widget in UE4

[Music] so just a couple of things before we get started well the first one is that i've already imported three assets that i've created in the test project i've put together before starting the tutorial uh they're very simple i said so i don't think we had to create them during the tutorial it's basically a widget an actor and a material uh if you create a material for the hand which is what i've done here it doesn't work so it doesn't show up when you apply it you get the great material it's probably because you didn't set this flag so watch out for that one this is a normal menu uh it's got a few interruptable things in them in it and it doesn't do anything really so it's just copy the empty and then the menu actor blueprint itself is just got a widget based on component smaller that's the actual size and then i set the blend mode to transparent and that's basically everything i've done so far the other thing i was going to say is that we're going to go through setting up again the pawn but in a different way that i've shown you in the previous videos so now you know all the ways you could possibly study up but i'm not going to explain like certain things that have to do with the oculus and that's in the get started guide video so go and watch that one if you want to know more about getting started this is not a get started video so get started video uh for that let's get started then and the first thing we're gonna do is create a blueprint uh in everything from pawn and we're gonna call it ppvr hand on and then we're gonna create another blueprint actor i'm gonna call it bp hand and let's open the vr pawn we'll add a camera and then go here and set the tracking origin to the floor level uh so that we are at the right height and then i'm gonna have here another sequence and then spawn factor from class and we're gonna go and close this three here go to the hand and add a variable called hand of type e controller hand and we're going to make it public and then expose on spawn and now once we select the hand blueprint in here that's gonna appear in there we're gonna make these always spawn in all collisions and then we're going to hook up these two guys here make a transform and use it for buff and then get reference to self and we're going to say that this is the owner of these vector and then we're going to declare this as the left hand and this one is the right hand and we are going to actually attach this to the actor and we're going to say it's not to targets not to target and then keep word and we're going to use again the self here that reference so that we basically attach it to to this actor itself so we oh with this done we should basically have the hand spawn but obviously we won't see anything since the hand uh actually here it's empty so we're gonna actually add a moisture controller and then underneath that one a hand so an oculus hang component and then we can basically set the motion source here so this is all stuff that obviously i'm not explaining because i've already explained that video that's what i was talking about and it's all like stuff that it's also pretty much the basics inside the vr template so this is literally a copy and pasted on the vr template and then the other thing we're going to do is actually get in the oculus hand and set the the mesh type and the skeleton type and we are going to say here select so that we can actually hook it up in here and here and saying that if the hand it's left we're gonna use the hand left and right so these are two different types obviously that can be used interchangeably and that's why we are setting up the select node here and we're going to collapse these into a function and say get oculus hand and then we're going to make it pure we're going to need it later that's why i'm doing this and with this done then if we actually also apply a material override of the handler we should let the hand working here so let's actually drop the pawn somewhere in the world and then say player zero and i'm gonna grab my headset and if it gets tracking confidence here it should display the other hand but one of the two hands is not displayed and that's probably something going on in here and that's actually happening here so i didn't set the hand to right there so that's my bolt and if we go here now we should see the left and right hands working as expected and that's how we get the hands up and running but now we want to get a pointer now the thing about the laser pointer is that when i was trying to set this up initially i thought that i had to actually uh come up with the actual location for the laser pointer which it took before it's not really easy uh and then i basically started to read about it on the oculus developer and actually they provide a way to um to get the proper pointer location and rotation it's not really easy and it's just because it depends on how you basically moving your arm so apparently they'll talk about the forearm stub isn't it and how you're basically pointing things depending on how the elbow is positioned and it's actually really not easy so when i was trying to do it i really didn't succeed at all and then i realized they actually they provide all the functions in the oculus hand library to to use that so let's get rid of these for now we don't need that but we need the begin play and tick and the first thing that we want to do is saying get tracking confidence for the hand tracking so we want to know whether or not we are confident enough to use the hand tracking feature and if we are not we obviously won't don't want to display anything at all so we don't want to display the lassie pointer for sure um and so underneath hand we need a widget interaction so that's the first thing and we can say that we set this to 1000 so it's a little bit longer and then we can add underneath this one a spline and underneath the spline a spline mesh i'm going to make it movable select the beam mesh and then we can assign it the hand material can be any other material i guess we can change it for the m arc endpoint so that it's a different one and uh and then we can create a couple of functions to actually collapse the spline so the first one is set spline actually rather update spline mesh and all we want to do in here is insert start and end and if you've already watched some of my um vr control examples tutorials you know exactly what this is i've already got this one set up in there it's basically the same logic what we're doing is just saying that we want to set the start and then position providing basically position and tangent for the spline mesh so that it basically bends and moves correctly in this case it's not painting so the tangent i guess is not really needed uh and it's all in local space if you're over here it says basically only local space so just make sure that that's local it stays as it is and the second one is basically 0.1 so that's basically just for updating this flying mesh the other one that we need is collapse spline and what we want to do with that is just saying set location let's find point and we want to set the spline point one in word space get the location at spline point in order space again to point zero so we're basically setting the point one of the spline to where point zero is which basically means that this line is completely collapsed they're going to overlap and it disappears and if we update the spline mesh afterwards that means that the spline copy collapses and disappears and we're gonna reuse these later in server presses and that's why we created that function so if the confidence is low we want to get rid of the laser point we don't want it to to see that because ends are going to disappear we're not going to know where to position so it's going to just probably hang in there and we want just to collapse it now the next thing that we want to do is check is checking if the pointer pose is valid so that's another function from the um oculus input function library that's what it was called basically and we can check if uh if that's actually valid so if we have high confidence in the tracking then what we can do is going ahead otherwise again we want to collapse the spline we don't want to do anything and now finally once we're here what we can do is setting the location or location and rotation of the spline as well as the widget interaction so one actually does the interaction the other one is just the visual part so the which interaction is the one interacting with the widget but then this plan is what represents it we could enable the debug uh shoddy back here but it's fairly ugly and it doesn't really work in a package game so we don't want to do that obviously and what we need to do next is saying get pointer pose which is the hand tracking library uh giving us a transform now the funny thing about this is that i've tried to use the terms transform as it is both with the word and local space um so certainly in word and relative space and it doesn't work for some reason and what works rather is getting the actual transform and then transforming if i can spread that the location and rotation that is provided by um this transform here and then using those in more space it works but i don't really understand why if this only locus basically because this is converting from local to word um it should work i guess so not entirely sure why it doesn't maybe i don't know maybe it's just because this is parented supposed to the infuriates is parented to the to the hand um no idea so this should basically set the right location so this works both for where the pointer is going to be and also where it's pointing towards because it's obviously the location rotation but then we need to um actually show and hide the spline if we hitting a widget so we're gonna do is getting the um widget interaction and then we're going to get the heat result we got the last hit pressure from norwegian interaction breaking these and we are going to say that if we actually hitting something then we want to obviously do something otherwise we are going to collapse the spline again and if we hit in something we obviously want to uh do the same thing that we're doing here so setting point one in workspace to be where we are hidden in our space coming from the brake resulter we're going to stack these guys like this and then we're going to get the update spline mesh here and placing in there and now we have everything that we need so this should already work hopefully it didn't miss anything what we can do for now we're not spawning this guy yet we can place it somewhere in the world in front of us and then i can hit play and see if these works and so it doesn't and the thing that we could do here just to see very briefly what's going on is showing the debug just to get an idea what's going on and it's actually so it is kind of working but they point in so depending this is the thing depending on where you're pointing um depending on the elbow isn't it you try to understand exactly where it's supposed to be so it should be with the hands lower and more relaxed and then it works um so it was already working i didn't mess up anything it's just that the i was basically using i was uh placing my hands too high i guess to interact with it so that's already working but we can't interact with it it doesn't do anything that's the next thing to sort out so uh because we want to interact with this with both hands and this is also covering my vr content examples we want to basically set the uh virtual user index to be a different one whether we're using the left and right if it's the same index obviously they come off interact with it at the same time because they're kind of overriding each other but if we copy this over so we should do it only for one for the hands it doesn't matter though we're gonna do it from both one is zero one is one now we should be able to interrupt with both which is great and then we can add a couple of uh events here so press and release there we go and then we can draw this guy here and say press pointer key and we can say release pointer key there we are so we're gonna hook these two up and look for the left mouse so what we're doing is basically faking uh through the widget interaction pressing and releasing the left mouse button that's that's what's going on in there now we can go back here and obviously now we need events coming from the hands which we can create in the project settings under input we can add a couple of events we can say left index page and we can copy these over here and say right index page and then we can look for index dot pinch and we can use the left one here and the right one here there we go so we got two events now to use and if we go back here we can say um let's look for index pinch there we go you got the left one and we got the right one and all we have to do is grab in these two guys and call press and release and there we are and then we can copy this down here and use the right one since it's the same blueprint to actually fire the event and if we so obviously i've got a crash there and that's what happens if you're working with the preview of a and the real release it happens so that's still unstable obviously all the previews and that's it let's actually test it now and see if it works and i'll need to yeah pay attention because it does this little uh funny thing and it's already working so i can interact with the left and right at the same time so i don't know if you can see this but i'm pressing the button while i'm dragging around these this guy is obviously a little bit too far to interrupt with hand tracking especially considering the size but i can basically interrupt with it so that's working now as the as the last thing to do i guess we could probably spawn this one we press a different button so to speak so let's get rid let's get ready let's get rid of that and actually uh the new input here we're gonna call it menu and then look for the middle uh left pinch and the middle right pinch and then we can go inside of the pawn here and look for menu let's go with the way up let's see if i can find it there it is and what we could do is actually spawn an actor from class we can beat the menu itself we're gonna always spawn it and as the transformer need to obviously create transforms let's make actually transform and that's going to be the camera location let's get the camera for vector so it's going to be plus another vector and that vector is going to be the full vector multiplied by certain distance let's say 100 units and that is the location and then we can say that we want to go from these fine we look at rotation so we want to be looking towards the camera so we go from the point that we found where we want to spawn it to where the camera is and we're going to set that as the rotation we declare this as a variable called menu what we do on release so let's do it on release instead of press we're gonna check if this value is valid if it's not we want to obviously spawn it but if it is we want to destroy the actor and then we can grab all these and collapse nodes get menu transform there we go nice and tidy and that should spawn and hide the menu in front of us um we did small problem i can already see but let's let me show you what i mean by this so if i actually do the right and left middle pinch it appears but if i look up it also appears helping the sky there all down below that that we don't want so we're gonna prep these and make a new vector out of that using only the x and y components that we basically flatten in the desert and now we can basically spawn it in front of us uh doesn't matter where we are looking it's always in front of us and i can interact with these i can enter also with these so let's actually press buttons and depending on the tracking obviously it's gonna work or it doesn't but there we go so we can interact with both at the same time and that's basically it so that was it guys that's how you can implement a laser beam in unreal engine using uh hand tracking and then interact with the widget uh next week we'll probably look at some locomotion solutions i guess you can do it in several ways and you can get a little bit more complex than what i've implemented so far but we're to look at a simple method to just teleport around and after that i know that some of you have been asking for how to grab objects instead of pinching so actually detecting some kind of uh different poses with the hands but they come with the solution i think it's ideal but we can look at how you could do that and then we can go from there and if you have any more suggestions about hand tracking features that you want to see i can try to cover them and obviously if you enjoy the video don't forget to like it and if you want to see more content subscribe don't forget also that we got the disco server running now it's free you can join and we can talk all fix above your and that's pretty much it thanks for watching i'll see you next week bye

Channel: Just2Devs

Views: 3,772

Keywords: Unreal, Gaming, Programming, ue4, unrealengine, VR, virtualreality, Oculus Quest, Oculus Quest 2, Hand Tracking, hand tracking, laser pointer

Id: TJpmf4mJhes

Length: 19min 35sec (1175 seconds)

Published: Tue Nov 03 2020