Learn Augmented Reality In Unity - Introduction and Placement Indicators

[Music] all right welcome to this next tutorial in unity this one's gonna be a special one this is an introduction to augmented reality or ar in unity you've probably seen the term x r x r is a an encompassment of multiple alternate reality uh systems augmented reality is one of them virtual realities another mixed reality is another one and in a future tutorial we'll jump into those other two but for today we're going to focus on augmented reality so this is a fun one this is a fun fun little exercise i'm going to do today in in the final product you're going to see that we're going to move our camera around we'll have a little placement indicator and we're going to click on that little placement indicator and have one of the characters from monkey kiss and donkey 2 and and so it'll be a lot of fun uh we're gonna have a lot of fun with this project here so first of all we need to create a new project and i must warn you uh before attempting anything uh if you're intending to build for android especially if you're attending a build for android there's a lot of pain and suffering going into trying to build for android it took me a while a lot of research a lot of redoing my my installations not in uninstalling but just it was just such a horrible experience to to get things working for android because uh the newest unity uh ar plugins uh seem to break a lot of things in my opinion and so i had to do a lot of major updating to different things so first off as you can see from the screen here we're gonna use unity 2019.4.15 so that's the first thing we you need to update either 2020.1 or 2019.4 to dot 15. now that fixes a bug that comes up because you also need to update to android sdk 11. and i'm going to drag over my sdk manager so you can see here android 11 it it needs to be installed okay so you need android 11 for this i mean i'm targeting nine because my phone is a nine right now and that's okay uh it'll still work fine but in order to get things working i found i had to update everything to android 11 which leads us to another discussion uh another another thing that pops up uh with with android and that is you need to upgrade your build gradle and in fact uh google gives full instructions on how to build with android 11 for unity and so they have a full web page here they're they're saying android gradle plug-in 3.6 is sufficient i disagree uh i had to grab the latest and greatest which was gradle 564. now what's interesting and we'll get to this in a moment when we're actually building for this so you'll notice that you need to specify in your your preferences gradle your path to gradle five six fours you have to find out where it gets installed we're going to go through these steps today because we had to do that one another thing we're going to have to edit our main gradle and launcher gradle files and basically copy and paste what's here at the top of those files now i mentioned uh gradle 5.6.4 but you notice that the line here was it specifies gradle 360. and i i foolishly thought okay well this should actually read five six four and so for the longest time it still would not build then i just from dumb luck switched this to 3.6.0 and everything started working so leave this as is and i'll mention this when we get to it don't don't change it just leave it as 360. all right so there's all these steps for for 2019.4 and 2020.1 all right so so we're going to be using android 11 and that the third dependency of course is uh gradle 564. so uh so upgrading that as well so there's a lot of upgrading you should do in advance before actually building for android and so now let's get into building our project so we have here our our unity hub we're going to say 2019.4.15. let's get upgrade to that if you don't have it and i'm going to call this i call this m11 so module 11. and we'll call it um intro to ar and we'll give it a home okay now that we have our project open here first thing i want to do is i want to rename our sample scene to uh game scene i just like it better so go to rename okay and we'll just let it reload okay all right i'll just resize a few things here okay now that i'm ready to go so the first thing we're going to do we're going to take a a bit of a different approach to building this time around is that we're going to start with configuring ios and android so i'm going to file build settings thing i want to do is want to add my open scene it's the only scene we're going to have in this game so might as well add it i'm going to switch to ios first we'll build for ios after uh we'll build for android first but let's get the settings for ios out of the way now there's a few settings that we have to mark here in in ios okay so it should take a moment or so to uh to transfer over all right so we see that uh ios has finally uh converted over well the project's finally converted to ios let's go to player settings i'm going to stretch this out now let's start with uh actually no we'll come back to xr plug-in management after because we stopped to install the xr plug-in which we'll get to we'll come back to these settings let's go back to player settings and what we want to do is uh we need to do three settings here which is uh starting in other settings let's give this thing a name let's call it com.something dot m11 intro to vr or ar all right give it some versions and build numbers all right so do that it's important to do that i've noticed that that actually burns me if i don't do that so scripting back-end il-2 cpp i'm going to switch it to net 4 all right and scrolling down minimum ios is 11 not 10. so let's do 11. all right requires ar kit support yes we need ar support which jumps us back up to camera usage description put in a message it populated with something like required for augmented reality support so sure uh good enough so and last thing is architecture my computer's thinking a little bit there we go i think i just did something and yeah accidentally unchecked optimize mesh data let's put that back in and finally we define the architecture and set that which most likely is sometimes when things are just right in front of you they're not really right in front of you so architecture universal we're going to do only arm 64 for this uh arm 32 or arm aka arm 7 is not supported for for uh ar processing so these are the main settings and we're not building for ios yet we're going to build for it second now let's switch over to android and do a switch platform there's a fair number of settings in android that we have to switch over and so now we're making sure we're under android settings i'm going to change again to be the same package name right uh version one yeah build one's good now uh minimum uh api is seven for android uh target api you can do automatic android 11 so i'm going to target all right uh this is where you run into a lot of problems and this is why i had to un upgrade to dot 15 for unity and upgrade my build gradle and so on because it wouldn't it wouldn't read this it would just say api 30 if it says api 30 and not android 11 it's not gonna work it's not gonna it's not gonna build for android 11 okay so you need to to uh do the settings you need to do for for this uh so all right scripting back end will be il-2 cpp dot net four okay uh uncheck arm seven and check arm 64. okay and uh we'll scroll up to here we go graphics apis falcon is not supported so gotta remove vulcan and finally multi-threaded rendering we're going to uncheck that once it's done that's thinking here so it's done thinking let's uncheck multi-threaded rendering or is it done thinking it's still thinking let's see there we go unchecking multi-threaded rendering and that's it for settings for now we'll come back and modify a few more settings but now we can close this we've got our game scene let's close this up now the next step is to go to package manager so we're going to open window and go to package manager all right now we should also set view preview packages so once it's done loading switch advanced to show preview packages okay so it's going to do some thinking now let's scroll up to where we see ar now there's ar foundation ar subsystem ar core xr plugin ar kit face tracking ar kit xr plugin we want all four so i'll start with ar foundation and hit install now you'll notice that it installed ar foundation ar subsystem at the same time but if i expand ar foundation and i click on i notice that there's a 4.1.1 and it says update to 4.1.1 so i'm going to do the update as well if it automatically went to 4-1-1 great you have to do this but if you didn't and i only showed you two 110 then upgrade to 4-1-1 so it's doing his thinking okay so it updated to 4-1-1 ar subsystem we'll just double check is up to date to 4-1-1 okay let's look at ar core xr plug-in that's what we need for android actually i'm gonna expand i'm gonna see if i can go straight to four one one instead of clicking on the two one twelve so if i highlight four one one there hit install it's gonna do this installation of the ar core xr plugin so what this plugin is so first off ar foundation ar subsystem they are basically the core packages the core apis the classes and and so on that that are used for just doing your ar work in unity so uh being able to have an ar camera have some ar ray casting uh you know plane detection like detecting a floor taking a surface and so on that's ar foundation ar subsystem ar xr core plug-in ar core xr plug-in is is the plugin that interacts with a ar core uh android's actual library for ar okay and then since we're since we're talking about the all these plugins here so we got below it ar kit face tracking an ar kit xr plugin so ar kit xr plugin is the same idea as ar core xr plugin but it's for ios so it's a ar kit xr plugin will interact uh with ar kit ios's package for ar on iphones and ipads and finally ar kit face tracking it is also for ios it uh it interacts with the face tracking face detection functionality that's built into ios all right so now you understand what the five packages are all about in earlier versions they only had three packages they only had ar foundation ar core xr plugin and ar kit xr plugin now you're dealing with the latest and greatest here there's now five packages here all right so we'll just wait and see this thing finish installing okay so now that ar core is done let's expand the last two i'm going to start with the last one arkit xr plugin 4-1-1 hit install all right and finally ar kit face tracking finally so we'll click on the 4-1-1 for ar kit face tracking hit install and let us do its thing all right looks like we have all the packages we need installed now we've done a lot of configuration but we haven't done anything yet now let's actually do something so i'm gonna kill this off let's start by just trying to get things up and running all right what i mean by that is let's just stick a very simple cube on the screen and let's see how it interacts in the ar world so first off uh let's expand our game scene let's get the uh ar objects into our scene so i'm gonna go to game object you'll notice that xr is now an option uh for you so i'm gonna say xr we need ar session so we're adding ar session we're going to add one more all right go to xr ar session origin now let's expand that if i if i expand ar session origin you notice that there's an ar camera built into it which means that we do not need our main camera anymore so i'm going to hit main camera and i'm going to delete it okay now if i go to ar camera and i go to the inspector we'll just tag it as main camera so now our ar camera is our main camera all right now let's throw a cube into the game so i'm going to go to game object 3d object cube all right now it's throwing a cube let's let's put it visible to our uh to our camera so we can actually see it here but i should also note that in our space here uh the cube is uh it is too big we actually have to scale it down i'm going to scale it down to 0.1 0.1 0.1 and that's because one square so if i zoom in further one square here represents a meter and a meter is too big so we want to scale it down to be so we can see the remaining area of our camera so i did 0.1 0.1 0.1 and what i'm going to do is i'm going to try and move it uh to be in front of the camera so let's see here put it about there go to our camera see so you can see it there maybe i'll move it back a bit oops not the camera the cube all right still too far away i wanted to be closer all right oh much better so we have a cube there all right let's build this thing and see how it looks in the ar world all right but before we do that we need to go back to project settings i said we had to do a couple of project settings before we before we move into things now you notice that xr plugin management has been expanded all right to include arcore and ar kit all right so click on xr plugin management initialize xr on startup plugin provider will be ar core now let's click on ar core it should say required okay now i want to switch to ios for a sec because i don't want to forget and i'll do the same thing plugin provider ar kit and just double check ar kit is required you could put face tracking down uh we're not going to get into that today uh so i'll leave it unchecked but okay so all of our settings are now done okay so now let's build this thing so let's do a control s to save our project now we'll go to file build settings and choose your device build and run and call it test dot apk for lack of a better name and let's go okay so uh this was expected uh man if this is actually the main reason why building for android is a lot of pain and suffering manifest merger fail with multiple errors see logs see the console for details i already kind of know what it's going to say it's going to complain about some error package for key which is somewhere in this pile of goulash all right so we're going to do is we actually have to make another change which i mentioned earlier at the beginning but we didn't do it so i'm gonna clear all this out and we'll go to uh player we're under android still and we're going to go to actually publishing settings there it is uh so like in the instructions uh we that we saw on the website we need to check off main gradle template and launch your gradle template okay now it'll actually generate these two files and these locations so launcher gradle template.gradle main template gradle.gradle and so if we look in our project we'll see plugins android and we see these two files all right now what i'm going to do is i'm going to open these in visual studio to edit them but double clicking on it doesn't exactly do that for me so i am going to start by right clicking and going to open with and we should have this main visual sorry visual microsoft visual studio version selector i'm not going to say always notepad wordpad i don't i don't trust those notepad definitely does not deal with line breaks properly so you don't want to do that so visual studio seems to be the safest bet for loading this all right so it says here we're going to remove this comment because it says generate by unity remove this comment to prevent overwriting when exporting again i'm going to uncheck that it also said in the instructions i'm going to call up the website again this is the website i'm gonna find so it says remove that comment it also says copy and paste this at the very top of the file all right so i'll make sure it's all there ctrl c to copy ctrl v to paste hit save leave that as 360. like i said before even though we're using 564. now same thing for main template.gradle open with version visual studio selector hit okay it's going to automatically load in 2019 again for me unfortunately in a separate one but that's okay i'm gonna remove its comment paste as well and save that and try again all right so build settings and build and run all right all right and eventually what you'll see is uh you'll see your a cube in the middle of the screen and as you move around your camera uh you'll find that you'll you can actually get the different perspectives of the cube it's it's uh you know it's actually it's as if it's real but when you look in front of you it's not there it's actually so cool like just to be able to see like this this little object right in front of you all right now that we've seen a simple cube in action on on our on our in our in our world there it was pretty cool to see we can walk around it now let's actually take it a step further and let's put a placeholder a place mark in front of it so place marks are useful for detecting surfaces and and indicating to the user hey this is a floor this is a wall this is this is a table you know and then you and then when i take the final step and put an object on in place of it so what we're going to do is uh we are going to uh well i'm gonna keep my cube in play for now uh what we're going to do is we're going to add an empty game object and i'm going to call this i'm going to rename it i guess in the inspector we're going to call it placement all right and then what we're going to do as well is uh we're going to add another 3d game object but this time we're going to add a quad all right and this quad uh will be rotated on the x-axis by 90 degrees okay and we'll have we'll make it 0.2 0.20 all right and the y level of it will need to be adjusted actually well let's see why at zero is fine it's zed well first off let's drag it inside the placement it's gonna be a sub object of placement there we go zero zero zero all right so that looks good okay and it's flat on the on the ground i'm going to hide cube we don't need cube anymore you can see behind the shadow there there's our little quad there we're going to replace it with an image so i'm going to do is i'm going to add an image to my project so i'm going to add a new folder i'm going to call it textures inside textures i'm going to bring in an image i already have the code llama logo i'm going to use that all right and i'm going to bring this into my project i'm just going to copy the file over from another part of my hard drive okay so there it is called codelamalogo.png i'm going to pull this away and let it think there we go now in terms of configuration first off we're going to bring it up in settings now i don't really need to do this because this is a simple square but i'm pretty sure you're probably going to try this out eventually with with a png file that has a transparent background like an arrow or something so what you want to do is you want to set alpha is transparency i'm going to do it anyway and i'll hit apply i'm not going to change anything else and there we go okay now we're going to do is we're going to create a material for this logo this texture so i'm going to add a new material there it is i'm going to call it placement all right now its shader needs to be unlit transparent okay so unlit transparent then you can do is you can drag your logo into it okay now we have our material i'm going to bring the quad closer it's going to drag my material onto the quad all right and so now i have my logo there okay so now we need to visit or revisit ar session origin make some modifications there so if i click on ar session origin it only has the ar session origin script we actually need two more components so click on component we need so we need ar raycast manager and we also need a our plane manager we need both of these okay now in terms of what to drag over well uh plane pre sorry raycast prefab is simply your quad and plane prefab is also your quad okay so now ar session origin is configuring hit control s to save what we have so far now we need to create a new script so i'm going to create a new folder in my project called scripts and inside of it and add a new c sharp script and we'll call it tap placement now let's load this inside visual studio and we'll write some code to support this all right so first thing we'll do we're going to add a couple of usings we're going to say using unity engine dot xr dot ar foundation and the second using is going to be using unity engine dot experimental dot xr all right so we have two usings that we're going to bring in now let's add some variables okay so first thing we're going to do we're going to define a public because we're going to connect to it game object placement indicator now the second one is a public game object you'll call it object to place we're going to do this later this will be our final step we're going to use this this second game object to place a new object where we tap on the placement indicator but since we're here let's just add it anyway let's add our remaining variables so we're going to say private ar session origin call it ar origin and private ar raycast manager call it ar raycast manager all right private pose equals or call it pose placement and private bool valid pose placement equals false so if there's no place to there's if there's no surface to to latch on to then we want to hide our placement indicator so we use this boolean to do that and pose placement will will basically manage our orientation and location for our placement indicator so first thing let's start into start and we're going to say ar origin is equal to find object of type ar session origin okay so ar session origin second will be ar raycast manager is equal to find object of type call it ar raycast manager all right now inside update we're going to start with calling a method called update placement now of course this doesn't exist so let's let visual studio auto generate it for us all right and it's going to throw will it let's remove the throw and replace it with actual code so first thing uh we're going to set up a location where we want to put the placement indicator which is going to be dead center in our screen so we're going to say var screen center equals camera dot current dot view port to screen point and this will just be a new vector 3 of 0.5 f 0.5 f so halfway x halfway y dead center now we need another variable this variable is called hits so we say var hits this is uh it's going to be an empty list for now so i'll explain in a sec what it is a new list of ar raycast hits okay so this is essentially to retrieve hits on the screen a hit is a plane that that a placement can be placed on all right so that's uh that's the idea is that hits will appear hits will be saved uh anytime you find a surface a floor a plane a tabletop whatever uh whatever we can place our placement indicator on now it's an empty list right now so let's go and see we can find something by saying ar raycast manager dot raycast oops sometimes typing can be uh an unfriendly thing so ar raycast manager dot raycast and open it up we're gonna pass in screen center we'll pass in our hits array and then finally what are we looking for unity engine dot xr dot ar subsystems dot trackable type dot planes look for planes and save them inside our hits array now if there's anything to bring back let's update our boolean so we'll say valid placement pose equals hits dot count greater than zero and check to see if kit stock count is greater than zero the length is greater than zero valid post placement is true otherwise it's false so now assuming it's true let's go ahead and and say if valid pose placement now we're going to basically say pose placement is equal to hits of 0 dot pose now we need the camera camera orientation because we're going to update that right now camera position camera orientation so we're gonna we're gonna say var camera forward equals camera dot current dot transform dot forward so this is good enough on its own this will place the placement indicator on the screen everything after this line is to fix the orientation because it's not going to look very proper all right so we're going to add a few extra line of code to actually orient the placement indicator to still continually face us and and uh and to point in the direction of the camera of the which way we're looking so it's a it's like i guess gravy on the train essentially so we say camera forward now we're going to say var camera bearing is equal to new vector 3 for camera forward dot x zero for y and camera forward dot set okay we don't care about the height or the y finally pose placement dot rotation is a quaternion dot look rotation for camera bearing right now the second method now the second method we need inside update is called update placement pose of course it doesn't exist let's let visual studio generate it for us there it says throws not implemented exception which we're going to correct and so now in this method what we're going to do is we're going to based on finding a plane to display this indicator on we will make the placement indicator visible if there's a plane to to to display it on or invisible if there's nothing to display on so we're going to say if valid pose placement so hits were found right then we're going to say placement indicator dot set active to true and we're also going to orient it properly so we'll say placement indicator dot transform dot set position and rotation to be pose placement dot position and pose placement dot rotation otherwise we're going to say placement indicator dot set active to false so we don't want to see it if there's nothing to display it on don't need to see it all right let's save this file we're going to go back to the editor and it'll update and refresh now we need to do is add a empty game object once it's done thinking okay so now let's add an empty object so create empty i'm going to call it tap placement manager and add a component add this new tap placement script it's going to say placement indicator so what we're going to do is we're going to drag our quad and object to place we can leave it as none for now we're not using it at all so it shouldn't crash if it does we'll we'll jump ahead and add the add our object in in a moment but for now let's just save and let's give it a run we can just do a build and run we don't need to go build settings anymore all right and so when you run this finally you should see the code lama logo indicator or whatever graphic you're using you'll see different sizes different different placements you know as you move closer to a surface as you move away from a surface as you rotate uh you'll see you'll see different you'll see it follow and track you two all right so now that we have our uh our placement indicator up and running you may have noticed uh in the in the video uh there that uh it um it was showing multiple placement indicators because of course my my floor isn't the greatest it actually it has those planks so you can see that there's multiple possible surfaces going on so it was actually displaying multiple possible placement indicators but in a nutshell it's working it's actually displaying a placement indicator for us to uh to track so now the last step is we're going to set it up so that when you click on the placement indicator it displays a 3d object okay so i'm going to take one of the characters from my game monkey kissing donkey 2 or n code llama the two separate games they use the same characters and i'm going to bring them in so i'm going to take one of my characters so i've got here we're going to do is we're going to create a new folder and i'm going to call this a folder robot for lack with better name all right and what i'm going to do is i'm going to bring my characters in so first off let's do a show and explore all right and we enter robot there and i'll just copy the files over okay and so we see them there now i'll just take this away and we should see project thinking as it imports these new assets okay so he's been so this character is named ion it's a it's a lion and he's been brought in here so what i'm going to do is i'm going to add ion to my project it's pretty massive so i'm going to scale him down to 0.2 0.2 0.2 all right uh he might still be massive let's actually drop them down to 0.1 0.1 0.1 now remember these squares represent a large distance in the real world so let's see here if i bring him in he's still going to look pretty huge but you know what i think it'll be good enough for our application today now i have him here i don't want him be to be in in the middle of my placement indicator so i'm actually going to move them off to a far obscure area i like this doing this because what i can do is i can actually bring my game objects in this little technique i do is i bring my game objects in i scale them down to the size i want and then i set them off to be at 999 999 99999 and what i can what i do in code llama is i do this with a lot of npcs and then what i do is i do some cinematic some uh some cinema machines so that uh in in the game when when i want my character to talk to the npc it actually uh it actually zooms into a camera view of of the of the character in a far off distance but this still looks like it's talking like right in front so it's a little neat trick i like doing and i get all the things i want out of it okay so we have here ion i'm going to click on tap placement manager object to place we're going to drag him over we hit save now our last little bit of code modification and then we're ready to go so now we're going to jump back to our uh our our script here and we've already uh we've already defined object to place all right so now we're ready to just go ahead and add uh this ability to click on the placement indicator and and have them appear so click on update there add a little space there i'm going to say if valid pose placement and input dot touch count is greater than zero and input dot get touch of 0 dot phase equals touch phase dot begin if it has we're going to call a method called place object of course hasn't been created yet so i'm just going to tap generate it and we'll replace it with instantiate object to place pose placement dot position pose placement dot rotation all right and that's all now we'll just save our script and we'll go ahead and run this thing so let it refresh we're basically done with anything else we need to do here so after it's refreshing we're going to do a build and run all right and so what you should see now is your code llama logo or your texture for your postman placement you tap on it and you should see ion in my case or your 3d object in my case it's pretty big uh and as it detects more planes you can tap more and add more of them right and then you can move away move back you can you can do so many different things and you'll see that yeah that you can place 3d objects wherever you want all right and so there you have it you have now a fully functioning ar application all right and and uh now you can take it a step further play with it and maybe you'll build the next pokemon go now i did promise some work in ios if you've been following along hopefully you've been building an ios along with me uh as opposed to android but just in case if you're following along and you're using an android device and you want to do for ios well uh the easiest thing to do is just go to your build settings and you're going to switch over to ios hit switch platform and let it think it'll do a re-import it might take a bit longer than before all right now that it's re-imported the project we're just going to do a build so i'm just going to create a new folder call it ios build now of course if you're using a mac you don't go through these these loops and hurdles uh to do that of course you can just do a build and run directly uh but because i'm using windows i don't want to move my project over o mac just simply build the project export the project copy it over to xcode build it and run it there [Music] you

Channel: Jawaad Sheikh

Views: 641

Keywords: learn to code, game development, unity, augmented reality, ar, xr, arcore, arkit, android 11, virtual reality, mixed reality, app development, android, ios

Id: b4QSoidE9pk

Length: 51min 27sec (3087 seconds)

Published: Thu Dec 03 2020