Oculus Quest Hand Tracking Teleportation in UE4

[Music] so before we get started um i just want to mention quite a few things so first of all teleportation using hand tracking doesn't have an official method this is one way of doing it it's one that i've seen online i liked it and i was like okay that's that's you know something cool we could implement and it seems to be working really nicely but this is not the only way of doing it there's probably like several ways uh the thing about hand tracking and teleportation for example or any kind of custom behavior is that we don't have inputs so we don't have any more controllers we don't have buttons and so everything relies on recognizing gestures uh with the hands obviously area comes with its own gestures uh the pinching ones but uh they're not ideal as you could pinch very easily by mistake by just you know trying to grab things so move around so you probably want a very specific gesture for when you are teleporting and the gesture that we're going to implement uh which goes something like this basically you have your hand facing upward like this so the palm of your hand faces the sky and then you close these three fingers the pink in the the ring in the middle and once you have these pointing gesture pointing with the hand pointing towards the sky if you actually also car the last finger the index then that's where you teleport it seems something that you will do very voluntary it would happen like randomly while you're trying to grab things so as for me that's a good gesture to use um now obviously the other thing um keep going with the gesture thing is that i wouldn't implement gestures the way we are implementing right now it's just that uh it covers that single gesture it doesn't cover all gestures so probably what you will do is in editor in the editor you will probably write some code that allows you to actually look at all the transforms on your bones using it one by one store them in an array so you basically make a gesture let's say a point in gestures on it and then you pinch with the other hand you collect all these transforms save them and then run time uh during while the application is running you compare all the current transforms to all the possible list of gestures uh containing transforms you have so that you can recognize suggestions i think that's the best way that's the ideal method that's not what we're going to do today is just so that we keep it easy and simple for now we're going to implement it probably at some point but not today and this is just because we can make some assumptions around the single gesture we want to implement which is just you know pointing this way and then closing the last finger so moving on to what we're gonna do today obviously uh we are going to implement this in blueprints so we are going to actually create our own custom oculus hang component we never from the oculus and component and create our own subclass of the oculus and component and i would never ever on earth implement this in blueprints obviously it's too heavy there's too much going on it's all on thick so just keep in mind that i'm doing this for the video i'm doing this to show you the method but i will definitely implement this in c plus plus i will never uh implement this in blueprint is this that's just too much going on um now apart from that the other things that i wanted to mention mentioned have to do with the project itself so as you can see i've already created a project and i went ahead and created the vr pawn that we're gonna use we're not going to implement any more pawns during the tutorials it just doesn't make sense i've already shown you how this works so uh if you don't if you don't know what any of these is uh you're probably watching the problem video you should go back and watch my previous videos on how to get started using uh hand tracking in a real engine with the oculus quest um so i'll go basically this pawn here you can see all the stuff that i've set up and uh the method that we're gonna use today relies on the fact that we can actually use the uh a custom skeleton mesh with the oculus hand component which doesn't come by default if you just use the oculus and component as it is it's going to spawn a hand for you so you can't obviously add sockets to it that's what we're going to use sockets inside of the hand so if you're wondering how that works you should go back either watch my video on how to get the a custom skeleton mesh running with the oversight component or if you're too lazy to watch it you can go to my website down to code.com tutorials and then you can download the project there and see how that happens uh and once you have the up and running you can basically come back and watch the video so i've already got the hands here working i've uh basically migrated the hands from that project into this project and if you go to the if we go inside of the oculus and left skeleton i'm gonna show the weighty bonds and as you can see i've set up already a bunch of sockets which we are going to use uh to compare basically their location uh to the fingertips location so as you can see i got a bunch of sockets here so that's for the middle finger ring pink pinky index and then the laser beam so uh in terms of how this is going to work we're going to basically get the fingertips and as they move around we're going to compare them to the sockets on the on the palm and when they're close enough we basically have a kind of a bull that gets set and so on i'm gonna show you what that is but that's basically the method we're going for um apart from that i think i've already covered everything uh and the last thing that i keep always forgetting with all these videos is that any of these videos i'm making that always available on my website in one way or the other whether they part of the via contract samples or they are if they own project in the tutorials page you can go and download them so just you know just saying basically okay so let's get started now that i've finished uh bubbling about all this stuff what we can do is create a new um blueprint and look for the oculus hang component that's what we want to do and we're going to call it bpc oculus hand so that's our own custom component and to start with we need a bunch of variables to actually keep state so what's going on and so we gonna create the volume variables i'm going to probably fast forward these and then you can basically see what i've created and what you need to to create yourself [Music] all right so our credit basically all the variables that we need uh and now we're going to actually go ahead and populate the variables that uh the names here so we basically need uh to know the index finger socket so the name of the circus the basic assign in here and then we need to know the uh the index uh sorry the fingertip uh bonds names of the index middle ring and pinky we're also going to get them from the skeleton so i'm going to go ahead and actually populate them all right so we got all of them basically uh set so they're all the right names obviously this is only going to work for the left hand but if you're working on the right hand let's say for instance you probably expose all these variables so that you can actually change at least the l here at the beginning or you could recognize which hand you're actually using on begin play and then uh just change this for an r or a pandan r at the beginning and you could name the sockets exactly the same name on the left and right hand so you don't have to do too much setup apart from that we need also threshold and this is the distance the minimum distance we are going to use to recognize when a finger is close enough to the socket uh while i was testing to seem to be the right distance so that's what we're going to use for uh for now now what we're going to do next is creating a function that we're going to reuse all over the place here and it's called ease finger down and i've already double clicked there by mistake and what this is going to accept is a couple of names one is the bone name and the other one is the uh i guess socket name we are going to also add some a variable for debugging in case we want to debug uh how these is behaving and then we're going to return a value here with that done we can now actually say get one location location uh by name we want it to have in word space and then we're gonna say get socket location and then we are going to say that um if we want to actually debug then we can say druid debug point i'm gonna make red so this is just for you to debug what's going on and we're going to actually hook it up there say 20 and 0.1 and then we're going to duplicate that and make it green and these two together and finally we can actually say print and what we're going to print is the result here so we're going to basically make the difference we're going to uh do the difference uh between subtract basically these two vectors so that we know the distance the offset between the two uh locations so the fingertip and the socket on the hand then we're going to get the vector length uh we're going to absolute the value so that we know hundred percent that it's always positive and then we're going to say if it's less than the fraction then we want to return that value basically there and all we need to do now it's uh basically convert this to a string there we go and append it here and then finally we probably want to also check out what that value is so i did add this extra bit here which may seem like pointless to do in the tutorial but i did it because it's really useful while you're trying to actually understand what's going on if you're trying to uh set these uh locations here so if we go back here get rid of that and then hide again the not weighted ones while trying to get these sockets to be exactly where you want them to be it's really hard obviously here you have no kind of reference but at run time it's going to be easier if you actually have these um little debug there what you can do is just saying okay debug and then it's going to show the two points the fingertip and where the socket is and also give you some extra folder and then finally we want to also hook that up in there when basically it's false because well obviously we want to auction it on something there so with that done now we have a single function that we can use to do several things here uh so to check for several fingers rather and what we're going to do is starting to basically just bring in the index bone name and then the second name and then we're gonna actually say if this is true we want to check if the um index down was false and if it was false we both we basically want to revert its value so we're going to say that it's equal to not what it was equal to so this basically flips the value so whatever it was the previous value is gonna flip it so if it was false it becomes true if it was three becomes false uh and then on the other hand if it was false uh if this is false and this was true we want to flip it again so we want to have this kind of set setup for all the fingers so this is basically just reverting the state if it was open and then it becomes close it says now it's close and if it was close and then it becomes open it changes the state so that's all this is really doing it just checks is the finger down and if it is or not basically this kind of flips the state uh based on that so i'm gonna skip ahead now i'm gonna fast forward and i'm gonna do it for all the rest of the bonds all right so we got all the uh four fingers basically set up so they're all doing the same thing and now what we can do is actually starting to use these variables to understand what's going on and to basically called our behavior so one thing the one variable that we didn't uh create yet which i completely forgot is the uh b was pointing which will let us know if the pointing state was there basically for the last frame so if we actually um you know we're pointing in the last frame and the last thing that we also need another kind of variable or let's say like the event dispatch that we need is on teleport which you want to fire if we actually finally made a gesture and teleport so we are going to add a branch down here and hook it up with the last then pin and we are going to grab the middle ring middle ring and pinky so this guy's here and then we are going to get the indexer and say that if all those fingers were down but not the index so we're going to have an end here and put them up like that like so there we go so if there was the case and we were not pointing now we are pointing so we actually set the stator now we want to obviously copy this down here if that wasn't the case now we are checking for the other possible case scenario which is we are actually we have all the fingers down so this covers basically the first scenario in which we have only few of them and if we weren't pointing out with pointing but if we have all of them down um we have two different kind of scenarios there so if we were pointing then we want to obviously call on teleport and we want to make sure that we actually set was pointing to false and if that's what if that wasn't the case then we still say that the the you know set to false uh which i don't think it's really needed because this doesn't make sense uh and then if this isn't the case if they're not all down that we are going to say again well we are not pointing anymore and we gonna set that to false um so just making sure that that's all looking good and i think it is and now we can finally basically get to use this component is pretty much done we just need to add it to our vr pawn so let's go back in here and we already have a left hand here so we're gonna select the left controller which is misspelled but we don't care bbc oculus and we're gonna call it custom left hand and then we are going to actually select the left hand here we need to make sure remember that we actually um select these guys here because that's part of setting up a custom skeleton mesh so i'm just basically resetting this so that it all works as the other left hand and then we are going to parent these two guys here and get rid of that guide there and now we basically swapped the previous oculus sign component for our own oculus and component um now we can start to basically add the code to actually understand if we are pointing or not and once we teleport teleporting and so on so we're going to reference this and say bye and then to on teleport which is the event that we basically prefer we should teleport that we're going to add a custom event here and say teleport and we're going to create an event here and select that teleport vendor and then we want to say that if we have a valid we're going to use this variable later on tick what we're going to do is saying we want to actually teleport and so we'll have probably a clearly part point uh vector here and we're gonna actually put taylor part on our pawn if that is true and all we're gonna do is basically say that's the destination and we're gonna get the after rotation there we go and we're gonna teleport so that's the code to actually teleport us we probably already want to see something up running here so we're gonna actually speed it up a little bit but uh one thing that we need to do for sure is actually placing this arc in our direction uh where the finger is and the way to do that is by using this laser beam socket so we're gonna actually go inside of here and we are going to say get socket location we're gonna paste that in there and then we're gonna get these two guys here and say set word location hook it up in there and so we set it on both so we're making sure that basically those guys are placed where the finger tip is if we go in here i think we supposed to be able to see that already yes they're really basically placed there for the left hand so that's all working and we need to start to basically be able to say uh when to tell when the hand is facing up or is facing basically down and then at that point we can start to use the teleportation the custom code that we've written so far so we're going to have here a branch and hopefully after this we're going to be able to see something running in the editor and we're going to get the controller here and say get the up vector we're gonna do a dot product against the word uh up vector and then we're gonna say that if it's more than 0.7 and if you don't know what this is basically we are getting two vectors one is the word up vector which is always basically pointing straight up no matter where you are what's going on it's a waste basically pointing just up towards the sky and then we get in the up vector on the hand and we comparing it which you should be shooting from here i guess and comparing to that vector and the dot product will return uh minus one is if they're facing away from each other of uh one if they face in uh in the same direction or zero if they are parallel to each other so uh we are saying that if this is more than 0.7 uh then we want to run some code at this point we can actually for not just print and say um you can teleport we're gonna wrap the code there basically to understand the teleportation otherwise we're gonna say we don't have a valid teleportation point and we can add another message you say not teleport just very quickly so we can test so far what's going on in there and i'm gonna make sure that i have this uh properly yes and i'm gonna hit play so now he says not teleport teleport and if i look at this it doesn't work anymore teleport no teleport no teleport not a part as you can see only works if i actually face the pawn upward as it's supposed to be so we already have that working we just need to keep going with the custom code to actually teleport so let's get rid of this two guys finally we already got something up and running and what we want to do is uh do what the motion control actor actually does internally and we want to predict a project tile path by object type i think that's the right one hopefully and we want to actually make this array we were static that's what's supposed to be for now we are also going to draw the debug information for one frame because we don't have an arc yet we're going to implement it uh later and then we are going to get the custom left hand here and say get circuit so location paste in the laser binder and then we're going to say get socket to rotation again pasting the laser beam and then we're gonna get the forward vector here multiply this by 900. this is exactly what the motion controller actor by the way does if you go inside of the code that does the arc that sexuality was going on so i didn't come up with this i literally just uh looked at how they were doing it and and used it and now what we want to do is um well first of all we can already see this working we should be able to actually see what's going on there so as you can see if i'm facing down it happens i can face around nothing happens but if i face upward that you can see how this is already working we can see the debug spheres there and what we want to do is go back in here and saying if this is actually hidden something that we probably want to break the hero cylinder and we are going to get the location and then project the [Music] just looking for the right one project point to navigation get the actually we may need to get the navigation is that how this works project point i don't think it matters which version we're using i think internally just gets the navigation system it's got a handle to it so that doesn't really matter but we're going to basically pass that in and set these to 500 again this is exactly what's uh what's been done inside of the um instead of the motion controller so i didn't come up with this either and what we're gonna do is that we're gonna actually say that a valid teleportation point is given by the fact that there is a valid point projected on the enough mesh and we actually found something in the word that we colliding against and at this point what we can do is very simply saying well this becomes true in this case it becomes false in this case and finally the teleportation point becomes the projected location and we should already be able to actually teleport everyone if everything went well so we can basically turn these close and then we can already teleport around uh and if i just do this nothing happens if i do this i think if i close very fast maybe it does yes sometimes so if you just squeeze the hand like this it will detect it but it depends uh but you can do it voluntary way easier and you have nothing happens like this as you can see that code is completely cut off when you know facing the right way so you actually need to uh really make that gesture on purpose and the last part so this is already all working and fine uh but we may want to actually get the ark up and running i'll show you how that works um but this is already basically working so that there is a much more that we need to do in terms of the functionality this is already working so let's get the arc up and running as it's actually fairly easy and one thing that i've already done before the video started is creating this update arc spline which i didn't come up with again this code is very similar to what happens in the motion controller and if we're going to sell the motion photo you can see the update or explain this is the function all i've done literally is just copying and pasting this it's exactly the same i didn't change anything i just made sure that i recreated all the references and variables that he needed [Music] that's pretty much it also making sure that if you select this guy set to move there he's got the starting mesh and the spline arc material there [Music] so that's that's pretty much what i've done with that and uh although i think we're missing it's an extra function which is clear uh arc i think it's called yes so it's trying to keep it consistent and i'm gonna copy and paste this code oh i'm gonna copy and paste this code as we don't uh really need to actually understand what's going on uh it just basically goes through the spline meshes deletes destroys all of them clears the um the actual array and then it clears also the points on the spline mesh this is all the code that comes from the motion so i'm not gonna really go through it just know that all it does is basically end a little array of points and are explained and all this this is also coming from here by the way i didn't come up with it again so just so you know so as soon as every time we run the tick we actually clear that spline and then uh what we're doing is somewhere later here we are actually going to run these and saying uh if we actually found a valid point valid location so he wants to know there that's keeps going like that then we want to actually provide a list of points otherwise internally just clears the spline for us and it just creates a very short spline and i think that's pretty much it so hopefully if i didn't screw up anything without spline this is actually working and the actual is it's just very tiny for some reason i think i didn't change probably the when i copy over this code i didn't change the size in here but it doesn't really matter i think that's that's also fine so finally we're going to get rid of these there we go and we have the arc working pointing around and it disappears if we actually don't point in the right if we don't have the hands pointing sorry the hand pointing up there we go so we can basically teleport around and that's pretty much it so that was pretty much it guys thanks for watching and i hope that the video helped you with the uh teleportation using hand tracking in a real engine uh obviously as i was saying this is not the only method to do teleportation there could be several uh i don't really like it particularly it's just that the setup with the hand uh the oculus and components really cure some i don't really like it that's that's just too much going on in there and i think that probably while you're doing it it's actually recording hand gestures uh i don't know what's the next video going to be probably about like grabbing things maybe or maybe we should look at proper hand gestures so that gripping becomes more elegant uh let me know what you think in the comments uh below or you can actually go and join the disco server and you can let me know there so again if you don't know yet we have a disco server it's free you can just join there's a lot of people now i mean it's getting bigger and bigger and we can talk about things above your tutorials future videos we are content examples you can ask questions to other members whatever you want to do it's just a place to talk about vr and real engine and it's only dedicated to that there's nothing else going on in there so obviously another thing i want to say is that all these videos i'm making all these tutorials all the projects are available in some way shape or form on my website whether it's through a direct link to the project itself or through the vr content samples project you can just go and download it so it's all basically for free so just go and get it and finally don't forget to subscribe to leave a like if you liked the video and i'll see you guys again next time bye

Channel: Just2Devs

Views: 2,942

Keywords: Unreal, Gaming, Programming, ue4, unrealengine, VR, virtualreality, hand tracking, Hand Tracking, Oculus Quest, Oculus Quest 2, VR Locomotion, Hand Tracking Locomotion

Id: fsxH-z8qx58

Length: 29min 51sec (1791 seconds)

Published: Tue Nov 10 2020