How to Create Custom VR Voice Commands in Unity

hey everyone this is major batman and lakira studios here to show you something really cool that i found and that is voice activation using wit.ai and the oculus sdk so in this video i'm going to show go over a quick demo show you how to get the sdk started and working walk you through the whit ai configuration uh setup of the scene to do this really cool firing of different canons based upon what we talked about go over that code and then tying in for those vr users into the player input so let's show you how it works so here we are in game mode you can see that it is vr just how we're looking around we can see things we are on a ship with three cannons in front of front of us and we have our hands which we don't really need in this case but let's test this out so the way i set this up is that each of these cannons is named in by a number and that i can with my voice and a press of a button to activate the reception of voices i can select which cannon is going to fire fire first cannon [Music] and that shot so let's look over here and do fire second cannon [Music] and i can even go for the last one fire last cannon and there we have it an easy way to use your voice to make things happen which i think is going to add a lot to virtual reality i hope we see a lot more of this in the future and i hope you do some great things with this tutorial so let's show you how to get started so what originally got me interested in this was reading the oculus integration sdk release notes in november they had it here added experimental voice sdk that you can use to bring voice interactions to your app and for research and development only must not ship product i thought oh well that's really cool let's try this out and then while i was making this video production release of the voice sdk api lets you bring these voice interactions to your app so in the time that i've been working on this video the voice sdk api has been released so i want to show you how it works so the voice sdk has an overview paper um that talks a lot about different things and does this voice sdk tutorial that goes over some how to integrate with with whit.ai and i did utilize a decent amount of it and focus primarily on tutorial 2. however i found that if you go to whit unity tutorials they have a much better more in-depth with pictures and i'm going to leave that link in the description so begin we need to create a new unity project i'm going to use 2021.1 and utilize the vr core as my baseline i'll leave a link up above for how to get started with the movement i don't want to go over that again but if you want to get started and learn how to set up your xr rig to move in oculus quest follow it along there a few minutes later all right next up we will install the oculus the oculus sdk i'll include a link down in below but it's called oculus integration on the unity asset store and you just need to open it in unity and install so you don't necessarily need all this it's rather large without it but you can just keep it as is we are mainly focused on this voice here all right so now that we have the oculus tab up top we're going to go to the voice sdk and open up the settings and it opens up this link here and so what we're going to want to do is we want we want to follow the tutorial all right so you're going to follow the prerequisites which allow mean that you need to go to wit.ai and sign up for an account and then you'll need to go to the sdk voice app settings in this picture here and copy your server access token and you're going to copy it into this this box here all right so i'm going to finish setting up my scene here um the basics of it are i'm going to install polygon pirates scene and set up uh the xr rig for use in this system a few minutes later all right next up is the wit ai so what we're going to want to do is navigate to with.ai and we will create an account i believe it's only tied to facebook so i was had to login using facebook once you're presented with this screen you can just click over here and go to new app after you create your own app you're going to be shown this understanding screen and you're just going to look a little different because you're not going to actually have any utterances the idea here is you want to type in an utterance of what you wanted to do and so in this case we wanted to say an action verb shoot fire blast and we want to choose a specific device and by a number or position and so the way we do that is we type in an utterance such as shoot cannon one and you won't have this intent but that we want to shoot the cannon and see how it says we think this is 84 confident so you're going to create your shoot cannon intent and then within it you're going to highlight your one you can even name this out one and you're going to type in a new entity which is canon number so think of this as a variable for that for storing that and this uses ai to figure out if there are synonyms or something similar for that for that entity and then you're also going to copy the shoot the the the verb that we're going to use and make it like fire status and so the idea is we could create these two entities fire status and canon number and then we can do train and validate you only need to do one of these then you'll go over here to your entities over here in management and you'll have two here are two here canon number and fire status so on fire status we already have the fire keyword and we want to add other synonyms such as shoot attack launch blast you can add your own you can say mickle pickle or whatever it may be but you can add these synonyms here and it'll have a better idea of what type of values will register for the fire status and then for canon number we want it to look something like this right left and middle and we have all the different synonyms now these keywords are the are the storage of the string that will be keeping later so match it up to how i have it on the screen right now now that we did that we just need to go back to unity all right next up is to create the cannons and so i utilized the polygon pirates cannon zero one as the baseline and i lined it up here at the edge of this boat and so i created one here and then i created and create an empty underneath we'll call this the spawn point and what you want to do is you want to arrange it in a way that it's in front of this cannon so it wants to be in front of here because the idea is we need to spawn a ball here and we want it to vector out at on the z forward that way so giving it a upwards trajectory is nice as well and that will be the forward trajectory of the canon ball in addition it's going to be the particle spawn point so that's where the explosion will be and the uh the sound point right so in vr you have a little bit of spatial sound awareness and so you don't want the sound to be coming from the the origin point down here it would sound better coming from the front of the canon so we're going to add an audio source on that canon spawn point and so we'll just duplicate that and do that three times for each of the cameras all right we are almost done so the next thing we need to do is create what's called the app voice experience from the oculus sdk we'll go here to assets go down to create go over to voice sdk and add voice experience to scene and that will create this application here and we'll tie it to our specific width settings which you can do here by this create the wood configuration all right now that we have this we're going to create the response handler so we're going to create a new md we're going to open up the oculus voice sdk understanding viewer it looks like this we're going to open it up and we're going to see that we have our entities which is our canon number fire status and we have one intent which is our shoot the canon which is already what we put so we're gonna go to the canon number and we're gonna open up the zero and we're gonna go to this value here and so this is right which is remember the string we're looking at that we named the right or the third canon so we're going to click it and we're going to say add response manager to response handler and response handler being the game object we just created and if we do that it now looks like this okay so with the app voice experience we have her app voice script so this ties back to wind and then we have the response handler which is going to do the heavy lifting and down here says on multi-value event so this is something that we're going to want to call in a little bit and so we need some scripts to make that work so we're going to add another game object here and we're going to name this canon handler and so we're about to get started but we need to create two scripts the first script is our activate voice script we're just going to tie it straight to the root app voice experience here next we're going to create another c c-sharp script and call it canon multiple fires and we're going to drop this into our canon handler all right so this is the code for the multiple canon fires first we have the audio section so we're gonna have a canon fire sound that gets played whenever the canon actually fires we can have voice clips that we tied to our character in response to whatever we're doing we want to keep a list of the cannons and then we want to keep track of an explosion and the cannonball and then what we're going to do is we're going to create a on fire function where we're going to bring in a string of values so the value in our response handler is going to be left right or middle so if the value if this string value is left we want it to return the canon that's in position zero and if it's in the middle we want to return the cannon that's in position one and off on the right the cannon position two so we've got to make sure that those are set up in the right way and we could do it a little different if we really wanted to but this is rather easy for what we have and then finally just kind of a catch-all it's not working and then once we get the correct on-fire statement so we're gonna get the cannonball spawn point and so what i did was i attached an audio source to the transform of that spawn point so that i can use a git component in children and that's the only one that has it so the left cannon that doesn't have this and the left cannon spawn has the audio source and so that's where we're going to grab this transform position here and so that's where that'll be the cannonball spawn then we're going to instantiate a version of the cannonball we're going to apply a forward direction vector to it and we're going to add this to the ball's rigid body add force then we're going to instantiate the particle effect and then we're going to play it and then at the very end we're going to play the one shot of the fire sound and that is the code to fire multiple cannons using a voice input string all right the activate voice is going to be rather simple but first uh we need to add a using facebook.with.i ai no dot in there and then we're also going to utilize the unity engine input system and this is going to be the button press that we're going to use to activate the wit callback so the idea is that we press a button to when we're going to record our voice to send that back to the servers on wit to calculate the intent and the action and send that back to unity so we're going to create a private capital w width with a lowercase width and then we're going to throw if the width is null we're going to do a width equals get component width and we're going to throw this in the update because the width component gets added at run time and isn't there beforehand so i've tried late start it doesn't quite work so then i've tried on validate and it doesn't seem to work either so the update is the one that makes it work the next step we're going to add is the wrapper for the button press using for the input action and so we're going to do public void trigger pressed and then if if this action is pressed input actions we're going to bring in the context and if it's performed we're going to call this other function with activate and we'll create the way activate down below and on it it's a simple wit dot activate and then when we press play our wit script our activate voice script gets the app voice experience with script here and then we also get this new button here activate and so the is basically going to when the buttons pressed press this activate button so we don't have to press it ourselves within the editor and that's very key for a video game all right we set up the response handler now we need to set up the canon handler so what we need to do is we need to add these game objects here so if you go into the polygon pirates they have an fx for our canon smoke and so we can just drag that over to our explosion particle next up if we navigate to the prefabs props we can look for canon balls and the first one is just a single canon ball and we can throw that into our canon handler as our canon ball object next we want to add three cannons and we're going to go to our canons that we created and we're going to make sure that the left is the first element or zero element the middle is our second or element one and our right is the final and third one and last but not least you need to add the sound effect for their canon shot so i found one from i think a previous humble bundle that i'm going to utilize and you just drag it to canon fire sound ignore these i think this this is something else i was working on to make it a little more complicated okay so let's get into the player inputs all right let's create that player input so that we can tie this into our xr rig we're going to go to create down at the bottom input actions we could really name it whatever we would like and open that up we're going to create a new action map and then in it it's going to have a specific action and this is going to be the trigger press all right so currently there's no bindings so what we'll need to do is we need to add several bindings to this and so you can do that by going to hero version path and since we're using oculus quest we can go to the xr controller and we can go the left hand trigger button and we can do the same thing for the right hand and i always get a little bit of these mixed up so sometimes i add in ones that i'm not positive so sometimes i just go in and i will also put in the oculus touch control trigger pressed so there might be a difference between the trigger button trigger pressed uh specifically in the xr tool toolbox so in your final game you want to make sure that this is precisely exactly what you would like for the purposes of this we just need to make sure that the that the trigger button presses are on here and i even went so far as to add a primary trigger any as that's that seemed to be something that also worked okay so if we go to the xr rig we're going to add the component player input on our xr rig so we need to bring in the input action asset and so we're going to bring in the vr controls that we just made and we have this default map of voice activation then what we wanted to do as a behavior is to invoke certain immunity events and that specific event is this voice activated trigger pressed and so what do we want it to do is we want it to activate on the app voice experience we wanted to go over here to activate voice in this dynamic callback context and call that trigger pressed and so that ties that together and then i'm not sure if this is a bug or something else but i also found that in the input action manager you want to change this or add an additional one for the vr controls or that new input actions asset that's being utilized by player input and this will tie together so now your controller will be able to utilize it and i'm going to show this for a second just so that you can if if you have any issues with your xr rig this is what it's going to look like so i believe that is all we need to do to get this to work so i'm going to jump into the quest right now okay so as you can see we're in we're back in the vr i can see my hands and we can move around and rotate so let's get this started so let's test the cannons out fire cannon one good blast cannon 2 [Music] [Applause] attack with last cannon [Music] so you see that with three different ways of talking we can kind of get it to work different ways and shoot these cannons so you could kind of see how this could be become a minigun with maybe like enemies shooting and you have to choose which one and fire at the enemy so i think they'd be pretty cool and i hope you enjoyed this tutorial and if you like this please consider subscribing to see more of this content cheers

Channel: Linkira Studios

Views: 9,156

Keywords: #unity3d, #gamedev, #indiedev, #learnunity, unity3d, unity 3d, unity 3d tutorial, unity oculus quest 2, unity oculus quest 2 tutorial, oculus sdk unity, oculus sdk, wit ai, unity voice recognition, unity voice tutorial, game development, learn oculus vr, unity xr interaction toolkit, unity cannon fire, unity voice, oculus and unity, unity tutorial, learn unity, vr, virtual reality, oculus vr, vr voice recognition, vr voice activation, pirates, cannons, cannoball, oculus vr games

Id: SJ96P-ZhBoc

Length: 19min 49sec (1189 seconds)

Published: Wed Feb 02 2022