How to: Virtual Mouse with Gamepad (Input System Unity Tutorial)

hello and welcome I'm your Kaki here's a simple tutorial on how to make a virtual cursor which acts like a mouse but you can drive it Using a Gamepad controller this is something that I need to make for my own game dinky Guardians by the way you can go pick it up on seam here's the game and as you can see I can control the character I can move it around with my controller and then as I inspect any building like this grab machine there you go now I've got a nice dot so I can control the mouse using my controller I can click on any of these and everything works perfectly I can do any kind of interaction just like I can if I move the mouse and there you go now I'm using the mouse and everything works and now I swap out the control and everything works perfectly so let's see how we can build a nice cursor that we control either with the mouse itself or with a game pad controller also if you're looking for a guided step-by-step path instead of individual tutorials and check out my complete courses start off with my free course AED at complete beginners which won teach you a ton then watch a followup free multiplayer course after that if you want something more advanced and check out my turnbas strategy course that one involves the making of a relatively complex game and of course the entire thing has a heavy focus on writing good clean code with a good project structure or in general for any Unity developer check out my ultimate unity overview course which has over 16 lectures each covering a different tool or feature of the engine that you might not know about check them all out with the link in the description okay so there are two ways you can Implement Gamepad menu logic in your game one is using the regular built-in gam pad navigation this one lets you define which button is selected and how to change the selection based on game pad input if you just have some simple buttons like for example just a main menu where all the buttons are organized vertically then for that the navigation system would work just fine but if you have some more complex interfaces like for example in my game which is an automation game so some of the UI is quite complex for those use cases having a virtual cursor is potentially a much better option now I was actually going to build this myself from scratch when I thought to search and see if the input system already has it and yep it turns out that it does although there are some really tricky things to get it working properly which took me quite a while to figure out so hopefully this video is helpful if you encounter some strange Behavior make sure you watch the video to the end I will try to cover all these strange issues that I encountered and how I solve them so first you should be using the input system if not if you've never used it then go watch my tutorial on it by the way over here on the package manager if you go under samples and you scroll down you can see the GamePad mouse cursor so if you want you can download their official sample and look at it then once you have the input system set up over here I've got my basic player input actions controller it's really just a super basic super barebone setup so there's nothing in here that is specific to the virtual Mouse just yet then in my Hier I've got my canvas so here it is I have it set up like this so it's in screen space overlay mode then for the kind of a scaler over here it is scale with screen size for the reference resolution I like to put it 1920 x 1080 then match with width or height and over here set the height to one and reference pixels of 100 so this is how I only set up my UI pretty basic now let's make a new game object inside UI so over here let's create an empty game object call it the virtual Mouse UI and now this part is extremely important on the right transform make sure you anchor this one over here on the lower left corner make sure it is anchored onto this one and then put zero on everything so zero on position x y z and also width and height also both at zero again this is very important if you leave it anchored on the center you will get some weird results also very important is make sure that this object is a direct child of the canvas if we make it a child of another game object that does not have the same size the same setup as the canvas if you do that it will not match and again you won't find some weird issues so make sure this one is set up Exel like this and make sure it is a direct channel of your canvas okay so now for the visual technically you can just go into add component over here add an image directly but personally I like to keep things organized and separated so I'm going to create a child UI image game object going to call this the mouse Visual and then for the Sprite in my project files I've got a simple Circle Sprite it is literally just a circle texture so there it is I'm going to move it over here just so it's a bit more visible although the position does not really matter what does matter is the size this is obviously way too big for a controller so let's go with something like 2020 so just a nice little dot in order to make sure that the dot does not become invisible just in case our background is white let's add a simple built-in outline so the outline component let's put it on 2x two and put it on black with full Alpha okay so that's our basic Visual and now another extremely important thing which is over here on the image on the recast Target make sure this one is not toggled this is very important if you only the toggle basically the virtual Mouse will be hitting this image and it will not interact with whatever button is behind it so again extremely important make sure this rast target is disabled and same thing on the parent make sure the anchors are set up correctly like this make sure width and height is zero and on this one make sure the anchor is over there on the middle and with whatever width and height you want again I'm repeating this several times because if you make one tiny mistake then everything will look very strange and you won't be very confused just like it happened to me okay so now let's go into the virtual mes UI game object and over here let's set a component and let's search for the virtual Mouse this is a built-in component from the input system here it is a virtual Mouse input and I let's begin by dragging the actual visual reference so the mouse visual drag this one again drag this one not the container so the underlying Mouse visual drag it for both the cursor graphic and the cursor transform then over here on cursor mode go ahead and leave it as software because we want to control this image and not control the actual Mouse Hardware cursor then for the speed let's put it a little bit faster so let's put it at 1,000 scroll speed let's leave it like that and then down here we have the various actions so you have pretty much every action that you would have on a mouse the most important ones are obviously the STI action in order to move our cursor and the left button in order to essentially fake a click now for defining these actions we have two options we can either add the action directly over here so let's click on the plus icon here and let's add a binding and now I have my Xbox One controller enabled so I'm going to double click this binding and over here on path let's click on it let's listen to input and I'm going to to assign the left stick on the game pad okay so I'm going to use the left stick in order to basically move the virtual cursor then for the left Mouse button again let's click on plus let's add a binding and double click and let's go into the path let's listen and I'm going to put it on the a so let's go button south on the game pad so this would be a on an Xbox controller or for example on a PlayStation controller this would be the cross and so on okay so that's how it is but like I said you can either add the action directly or you can use a reference now if you click on reference and basically you get this field and if you assign this field then basically you can assign any action that you need find in any player input actions so for the input actions over here all of the actions that you got here all of these various actions you can use them as reference in that one so for example in here instead of binding directly to the left sick you could bind directly to this action the benefit of that one would be if you rebind this action then you wouldn't need to update over here the virtual Mouse since it would always go through this action but to keep it simple for this example I'm going to use directly instead of using references I'm actually going to talk about this in a little bit because there was a sneaky issue over here that tripped me up but for now let's keep it just like this so just with this if we test okay here I am the game is playing and if I move my game pad stick and there you go it does indeed move the virtual cursor all right awesome so far so good now again if your cursor is not moving correctly again go back and double check your anchors here is how my Canabis is set up here is how the virtual Mouse UI set up and inside the mouse visual with all this we have everything working except now if I make a button so so I'm going to go inside the canvas create a new uui object let's make it a button and let's make it quite a bit bigger so something like this then just for testing let's modify over here the highlighted color so this is the color when the mouse is on top so let's put this on a green just so it's nice and visible by the way also remember how the Sorting works in the UI basically the Sorting order is based on the order in the hierarchy and things below when the rqy will actually render on top so since we want the virtual mesui to render on top let's make sure to drag this one to the bottom of the hierarchy so that it's above the button okay so with this let's test okay here I am and I can move and if I go above the button and nope it is not working the button is not changing color if I use the regular Mouse and I go above yep you can see it does recognize but the cursor the one by the game pad nope this one does not work so this is one of the main tricky issues that I encountered thankfully this one has a simple fix the fix is actually on the Hy let's go in the Advent system and up over here you do see a nice warning message basically it's telling you you should be using a different event system for the input module thankfully it's super simple literally just click on this button and there you go it replaced with a different component also before we test one very very important thing here which is on the game view over here make sure you are under full AG this is very important make sure it's full HD and not 16x9 or any other it needs to be full HD and on the K scalar over here make sure it is also 1080p so it needs to be 1080p both up here and down here I'll cover why this is extremely important in a bit and in the end of the video I'll cover the the solution to the problem but for now just make sure you have it set up exactly like this and now if we test again and now I move and yep now it does work if on your end it doesn't work then make sure you have full HD 1080p selected over here on the game View and also on the canvas same thing canvas scaler make sure it is full 1080p like this it is working however if I were to swap this out to 16 by9 note how over there the canvas scale that one is now at point4 instead of one it is under one so the canvas is basally scaled and now if I try moving the button over it it does not work but for some reason it works when I go onto this corner so that's a really strange issue that took me a lot of time to debug it has to do with the canvas scale I'll cover the solution a little bit after I cover some more potential problems but right now as long as you set it up here as full HD everything should work so I can move the cursor and yep it does work although in your case maybe it won't work I'm saying that because when I tested in my game actually for some reason this did not work now in this case it works so that's great but just in case it doesn't work for you like it didn't work for me let me go over what I did to to fix it I believe the issue may have been because I was already controlling the player with a player input actions whereas over here I really just have an empty project so basically when we click on that button it swapped out to this component and over here on this component you can see it used the default input actions asset so this is the one that is down here inside the actual packages so this is the default one now just in case you have issues here's what I did basically I just went into the over here this default input actions so the one inside the package I just open up this one then I just grabed the UI action map so just right click and let's copy this one okay close and now up here I went into my own player input actions which again I only had the player action map so then over here right click and paste and there you go now we have the UI input actions then I also went here onto the point action and for this one I added a new binding and for the path I found the virtual Mouse and yep I add this one the position for the virtual Mouse same thing over here on the click add a new binding go and search for virtual Mouse and for the left button on the virtual Mouse so just this and then save asset and then on the hierarchy back into the event system back into this component instead of using the default input actions let's assign the player input actions and just like that since we copy the action map everything should already be automatically matched so it should be using dii point and so on so all these should be matched exactly perfectly and now here if I move the controller cursor and if there you go it does work I go above and it does indeed turn green now again if you have isues on this one the main thing is on the mouse visual make sure rast Target is disabled if I enable this one and now I try moving above it nope you can see it does not change that is because basically the virtual cursor is hitting this image instead of hitting the button behind it so if this is the case make sure the visual has rast Target disabled and make sure there are no more visuals over here under the virtual mous UI make sure there are no more images nothing else that can basically block the virtual Mouse and if you do that then yep you should see this working so over here it works with the virtual cursor and it works works with the regular cursor okay so really that's pretty much it but let me cover some more potentially sneaky issues now there's one that is really very strange and it's actually on the virtual Mouse component as you can see down here we assigned the actions directly so just the move and the click action and as you saw in the editor this works perfectly there are no errors nothing no problem however if you make a build just like this you will get a ton of errors and not just errors but actually crashes as well now when I made the build just like this for the steam deck for my game it completely broke everything because it was triggering non-stop errors so basically the build would run on the steam deck and then immediately crash just because way too many errors and the error itself was some really strange argument exception now this one took me a real long time to figure out the cause and it turns out the cause is because all of these actions are not assigned apparently some of the backend code for the input system apparently that one assumes that these fields are always assigned for some reason that logic runs differently in a build than the editor so here in the editor there's no problem but apparently in a build it assumes these are always assigned so in order to prevent that error just make sure that you assign all of them so for example use the reference and then just drag all the references so inside my player input action for example for the middle click I'm just going to drag the same one again even if I don't use them because for example in my own game I never use middle clicks or right clicks but just go ahead and sign them just so they don't have that strange error okay so this should solve that strange issue now the next issue is potential conflicts with the menu navigation like I said in the beginning there are two ways to handle menu controller support is either with a virtual cursor or the navigation system if you don't even know what it is you can basically look at any button let me actually just take this button and duplicate and put another button over here off to the side now if you look inside the button component over here you see something called navigation and by default this will be set to automatic and then you've got this button saying visualize and if you click on it and you look on the scene view you can see a bunch of arrows so you can see there's one Arrow coming from this button onto this one another one from this one onto this one so basically what that means is if you have this button but selected and you move to the left then it's going to select this button and if you have this one selected move to the right and select this one now in order to easily visualize this let's select both buttons and over here on the selected color let's put something more visible so let's say a nice blue okay so now let's test this now here I'm going to use a little trick in order to select a button so basically I'm going to left click inside this button and then I'm going to release it outside the button just so I don't trigger a click and just like this Yep this one is now selected and now with this button selected and the navigation enabled now for my game pad stick I move to the right yep there you go it select this button and I move left and there you go select that one so that's how it works that's how the automatic navigation system works except of course if you have both this system and the virtual cursor obviously you've got some conflicts between the both of them so really you should pick just one or the other assuming that you want to use just the virtual Mouse and disable the automatic navigation for that you can either select the button and down here under navigation just set it to none that's one option however obviously it's quite tricky if you have tons and tons of individual buttons having to go and set this to none for all of them that's quite tricky so a better method is to just ignore that and then just go into the event system and over here we can see the action for move this is the action that actually moves which button is selected so for this one we can simply swap from navigate and go into none so basically we're going to have no input in order to trigger the navigation menu and with this now if we are and again let me click hold and release outside so that one is selected and now I move right on the stick and there you go it no longer selects that one so the navigation menu is no longer working and I can now just go up here click onto that one and I click onto that one and so on and everything works next we have one more issue and that is the game pad going off screen now this is a bit of a strange one because it seems like something that the input system should really have by default but apparently it doesn't so here I've got the game running and I've got side by side My Scene view so there's my cursor and and if I move it on to the left and I keep moving and there you go now my cursor was way off screen and now if I was player and I was only seeing this and now I move the game pad to the right there you go I can't see where's my cursor why is my cursor invisible that is because it is way off screen obviously you don't want this you don't want the cursor to ever leave the game screen now fixing this was actually quite a bit tricky it's another thing that took me quite a while to figure out we're going to need a script to do this so let's do that let's go ahead and create a new C script let's call it the virtual Mouse UI let's select the virtual Mouse UI and attach the script okay like this and let's open over here let's remove the regular code and let's write a simple private void late update we want it on late update just because we want to run this at the very end of the frame after the input system has done its movement and now here basically we just need to clamp the mouse position and we move it so first let's actually grab the virtual Mouse so let's go up here make a private make it of type virtual Mouse input which again this one exists inside using Unity engine. input system.ui so if your Visual Studio doesn't add automatically make sure you add this line so this is going to be our virtual Mouse input so that is that component that we saw so let's grab it so we can just grab it over here on an awake Let's see we get component of our virtual Mouse input and now that we have this over here on our late update we can go into this one and first we need to grab the current position so inside the virtual Mouse input let's grab the actual virtual Mouse so this is the actual mouse that that component drives and we can grab the position this one is a vector to control so let's grab the value which is the actual underlying Vector 2 so let's grab the vector 2 for the virtual Mouse position okay so we have this and obviously we just need to clamp this within the valid values so let's simply validate first the X so we can use math f. clamp clamp just in case you don't know this one takes a certain value and make sure it is within a minimum and a maximum so that's exactly what we want so let's clamp the virtual Mouse position. X and for the minimum let's put a minimum of zero and for the maximum let's use scre do with wi so there you go that makes sure that the virtual Mouse position never leaves the screen and let's do the exact same thing for the Y but for the Y we're going to clamp it to the height so that's e and that's e also by the way if you wanted to add some kind of Border just make sure it doesn't really reach the edge you could just add it here just some simple math so then when this position is clamped when this position is now validated then we can use the class input state which again this one exists inside Unity engine input system L level so make sure you add that and then we can call the function change and then this one takes a certain device so for the device let's go into the virtual Mouse input and inside virtual Mouse input let's grab the virtual Mouse and we're going to want to modify the position for that virtual Mouse and the position is going to be the new one that we saw so the virtual Mouse position okay so that's it basically we grab the position we clamp it within the valid values and then we set it again so let's test okay so here we are and if I move all the way over to the left and nope it no longer leaves the screen and I go up nope does not leave go down nope does not leave go to the right and nope does not leave so now the cursor is always inside the screen okay great now the next bug is the one that we saw a while ago sof up here is set as full HG and the scale of the canvas is one if I do that then yep it works look that I'm mousing over the buttons exactly as I should be but if I now Swap this out for 16 by9 and now I try to pass the cursor over there nope does not work and for some reason if I go let's say over here and there there you go the button is somehow for some reason all the way over here this is really range this is another issue that drove me insane for quite some time how the cursor is not on the actual cursor position so hopefully this video helps prevent you from going insane just like I did the problem has to do with scaling if the canvas is not scaled which happens if I set this one to f g like this there's no scaling and yep everything works but for some reason if there is scaling of any kind so over here it's at point4 and now for some reason it appears that the cursor itself that one is also being scaled so this strange position just so happen to be that one is at 04 so this one is about 40% of the actual position the difference from here to here thankfully the solution is pretty simple here in our virtual Mouse UI class let's just add a field let's make it a serialized field make it a wrecked transform for the canvas wed transform then here in the editor let's drag the reference of the canvas itself okay there it is and now we basically just need to do some math to correct it so for that let's do it on a regular update so on update let's grab the transform. local scale so that's the scale of the virtual Mouse and and for this one we're going to basically make it the inverse of the canvas R transform so we can invert it just do 1f / by the canvas Rec transform let's grab the local scale and since all of them are exactly the same the X Y and Z let's just grab the X but of course this is just a regular float so to convert this float into a vector 3 Let's just multiply it by Vector 3.1 also by the way since you're here just another quick tip if you want to make sure that the cursor is always on top you can use transform. set as last sibling and this will basically always move it to the end of the hierarchy just making sure that the virtual cursor is always on top okay so this should pretty much fix that issue basically we're going to scale this virtual Mouse up or down depending on the scaling of the canvas which should put it on the correct position let's see so here and first let's verify okay with f g with no scaling on the canvas itself so over here no scaling and as I move and Yep this one works and that button is over there okay works now let's scale it and now there's my cursor and as I move over that one yep it works and move over that one and yep it also works so not like this even if the canvas is scaled you can see that the cursor is now in the correct position so if I put the cursor visually over there over that button then yep it does trigger that button now of course then as we scale the virtual cursor this game object you can see that the visual scaled upwards so again if you want to counteract that you could do another basic math so you could just go inside the actual Mouse Visual and over here on this one let's say put this one on point4 to match that one and yep now we have the exact same size cursor but the important thing is just by doing that math now the cursor is visually exactly where it should be so then another thing is how to detect Mouse versus Gamepad so for that I'm going to Showcase code from my game in my game I can seamlessly swap between Mouse and gamad controls note how the virtual cursor only shows up when I actually touch the game pad so here's a code for how I identify which input is active in my game I've got a general game input class that does lots of things regarding input but the important thing for identifying that one is I simply have an inom so just a game device so it can either be the keyboard and mouse or the GamePad and then I simply got a field for the active game device then down here on the awake inside the awake I subscribe to this event this callback the onaction change inside the input system this one is going to get triggered pretty much anytime the player touches any input so that is going to call this function over here and then on this function this one takes two arguments and one of them is input action change so here I'm basically just testing the external phase for the action if it is an action performed if it's not then I'm just going to ignore it so if it is an action performed then I'm basically going to ignore if it is a virtual Mouse because again I'm only trying to chest between the GamePad and the actual Mouse so if it is a virtual Mouse and I don't care about it I'm going to ignore it if it isn't the virtual Mouse then I'm going to check if the input action action control device if it is of type Gamepad so if it is an Xbox controller or a PlayStation or switch or anything if it is a Gamepad then I'm basically going to change the active device to Gamepad and if it is not then I'm going to say that the keyboard mouse is active then this function over here this one just sets the active game device then I also either show or disable the regular cursor based on whether the keyboard mouse is enabled or not and then I'm going to fire off this event on game device changed and then I have this other script which is pretty much exactly the same thing that we saw the virtual cursor UI script and then on this one over here on start I am listening to that event on game device changes and when that changes over here I'm really just updating the visibility and I'm really just checking the game input instance if the game pad is the one that is active then I'm going to show it if not then I'm going to hide it and that literally just enables or disables this game object so y this is how I handle swapping between Mouse and gam pad in my game also one more note instead of using the built-in component you can actually just build yourself manually from scratch if you want to learn how to do that I suggest you look at Sam yam's excellent tutorial on it she covers how to create the underlying input object and how to manually move it okay so the virtual cursor is finally working and here is the exact same thing in my game so I can inspect any buildings whilst in here I am controlling the character just as usual and as I inspect the building yep I've got my virtual mouse cursor and I can move around I can fake click to select anything and I can do pretty much anything that I can do with a mouse this is actually a big part of the reason why it was so easy to make my game playable on a steam deck I just need a regular controller to control my character and then as I inspect any of these buildings I just need the virtual cursor controller just like I would with a mouse with all this and yep the entire game everything works perfectly with the controller if you like games with automation Colony building and defense mechanics then check out ninky Guardians on scene all right hope that's useful check out these videos to learn some more thanks to these awesome patreon supporters for making these videos as possible thank you for watching and I'll see you next [Music] time

Channel: Code Monkey

Views: 10,429

Keywords: code monkey, unity tutorial, unity game tutorial, unity tutorial for beginners, unity 3d, unity, game development, game dev, unity 2d, unity 3d tutorial, programming, coding, c#, code, software development, learn to code, learn programming, unity tutorials, how to make a game, virtual mouse, virtual cursor, virtual gamepad, fake mouse, fake gamepad, fake cursor, interact ui with gamepad, ui gamepad, ui controller, ui steam deck, ui xbox, unity virtual ui

Id: j2XyzSAD4VU

Length: 25min 9sec (1509 seconds)

Published: Thu Nov 02 2023