Augmented Reality (AR) tutorial for beginners using Unity 2022

hi i'm alistair and in this video i'm going to walk you through how you can create your own augmented reality app using completely free software so augmented reality is where you take the video feed from a camera on your mobile phone or tablet and then you superimpose virtual objects into that real world view and there's lots of different applications of augmented reality one of the most common examples is where you have face tracking or body tracking so this is where you detect facial features like the eyes nose or mouth and then you wrap a virtual mask over those features and this is how snapchat filters and face swap applications work you can also use ar to detect the geometry of the environment around you for example you could detect vertical or horizontal surfaces and then place virtual objects on those surfaces a certain distance away from you but in this video i'm going to create an ar app that uses image tracking so that's where you detect an image or a texture in the video feed and you use that to position your virtual objects and unlike a qr code or a barcode scanner that image can be absolutely anything you want so here i've got a regular playing card it's the king of hearts if i load my ar app in my phone and show you that at the same time hopefully you'll be able to see i'm able to replace that texture of the king of hearts with a texture of a different playing card instead so here i've got the two of clubs instead and it doesn't have to be a simple image replacement either um here i've got a little printout of a robot i made and if i show you that on my screen here i've actually inserted a 3d model on that texture instead and you can look all the way around it and see the robot from all sides now i already published a tutorial about augmented reality a few years ago but one of the problems with tech like this is it's constantly changing and improving so that previous video is probably a little bit out of date now the advantage of the method i'm about to show you is that this is now completely self-contained it no longer has a dependency on the third-party vue fourier components and you don't need to use an external website to manage your tracked images it's totally self-contained on the device the other advantage is i'm using native ar functionality provided by the manufacturers so if you're building this app for an apple ios device then it uses apple's ar kit library and if you're building for an android device i'm going to use google's ar core library i'm going to use the latest versions of both those libraries and i'm also going to be using the latest version of unity which at the moment is unity so let's get started now if you've not used unity before the first thing to do is to head over to unity.com and click on get started then you can select the appropriate license if you're an individual you can use the personal license for free and then you want to select your operating system now this will download what's called the unity hub which is kind of like the setup wizard for the unity program now once the unity hub is downloaded if you load it up you'll be able to see different unity projects and versions of the unity editor installed on your machine and you can also install new versions of the unity editor by clicking this button up here now we're going to be using the latest version of unity which is 2022.1.5 f1 i've already got that version installed on this machine so my button is greyed out but when you click to install that you'll also have the option to install additional modules depending on different target platforms that you want to deploy this app to now i'm going to build this app for my android mobile phone and tablet so i've included android build support uh together with the sdk and open jdk tools if you wanted to build this for an apple ipad or iphone instead then you would include the ios build support when you're installing unity in fact you can include as many additional target platforms as you want but each one you include is going to require more disk space to install so there's not much point including more targets than you actually plan to use once you have installed unity and all of the required platforms you can come over to projects and create a new project uh make sure that you are selecting the latest version of the editor if you've got more than one version of unity installed here and then we're going to select a template which we'll use as kind of the the base from which our project will be created so unity can be used to create all kinds of different games and applications it can be used to make virtual reality mobile games online games so we'll use the ar template which will set up the project with most of the functions we'll need to use if you don't already have that template installed you'll just want to download it there and then give your project a name i'll just name this playful tech ar and then we'll create it and here's what you'll see when that new project loads in the unity editor it's had some assets already added into the project and there's also some elements already been added to the scene hierarchy and behind the scenes some of the project settings have already been configured ready for you but there are still some settings that you might want to check and modify yourself as well and in fact there's a dialogue window appeared that's reminded you about one of them in particular so we're going to go into the project settings and i'm just going to talk you through some of the important things you'll want to check out so first of all click on xr plugin management at the bottom now xr is extended reality and that's unity's word to refer to both augmented reality and virtual reality plugins and we need to select the plugin that we want to use in this project so we're targeting the android build platform so i'll click the android icon and i'm going to use ar core which is google's ar library if i was targeting uh apple device here i'd have the apple logo and i would select ar kit instead remember you'd need to have installed that module when you're installing the unity editor as well i'm also going to make sure that that xr plugin is initialized on startup and then if i go into the module specific settings here i'm going to require ar core on the device some applications have ar as an optional kind of add-in that can be enabled but in this particular case it's absolutely crucial to the front lamp of this app so we want to make sure that ar core is actually required and supported by the device then we're going to just check through a couple of the other settings of the project here so in the player settings is where we'll find most of them we can give our project a name and we can actually say the name of the company that made it as well this will just mean that when it appears on your device it will be named correctly and if you wanted to you could insert an icon here how it will appear on the start when we get to uh graphics i'm going to remove opengl es 2 that's an old graphics api which is not going to be supported going forwards anyway so i'm going to remove that there and going down here i think most of this should have been set up correctly already that was the point of choosing the ar template at the beginning but i just want to check because sometimes i found the templates aren't quite up to date i don't want to overwrite the package name i'm just going to use the default package name which always follows the syntax of com then company name and then application name version one i want to make sure that we're targeting a minimum of api level 24 on android devices because you need that version to support ar core uh scripting back end that's correctly set to il2cpp if yours is set to mono you want to change it and we also want to make sure that we are making a 64-bit build so we've got arm 7 and arm 64 selected that's a requirement if you want to publish the app on the google play store now it looks like all of the rest of the settings are correct but if for whatever reason you have a problem uh building your app or deploying it to your device i recommend perhaps comparing the settings you've got set in your player settings to the ones i've got here now the next part of the setup we just need to check is the packages that have been installed into the project already so if you come to window package manager you'll see here that i've filtered for the packages in the project and we've got six ar packages already installed for us but one thing we just want to make sure is that they are all up to date even though we used the template that was available you can see already here the xr plugin management we've got 4.2.0 installed but the recommended version is 4.2.1 um so one thing you can do is just to go through all these and every time you see this little icon here it means that the update is available to that package i'm just going to update all of the packages in the product to the latest version and that can help with some issues you might find if you find that you deploy and you get a black screen or something like that that can fix that problem so that should be it for the project setup now we can actually take a look at what's been provided by the sample template we're using so we've got three objects in the scene we've got a light an ar session origin and an ar session so the ar session has a script on it called ar session and that manages the overall life cycle of the ir application you're going to have to have that script attached to an object in every augmented reality app you create the ar session origin well that acts to map the real world objects and the virtual objects created in the scene together i think in future versions of unity that's going to be renamed to xr session origin so more unifying the idea of virtual reality and augmented reality together so if you're seeing this in the future and you see that object called exile session rather than ar session don't panic it's the same thing what it does is it provides the common frame of reference between those virtual objects and real objects based on an origin point here and you'll see that it also has some other scripts attached to it at the moment as well we've got a plane manager an anchor manager raycast manager and anchor creator and this is actually the example i showed earlier on in the video where you saw me place a blue sphere on surfaces that i could see in the video feed so the plane manager here allows the detection of horizontal or vertical planes and the video feed and then you can fire a raycast to determine the intersection on those planes at some distance away and then you can create an anchor so an anchor is like a physical point in space that is tracked by the application and allows you to create virtual objects at that location so this is an example of a particular application but it's not the application we want in this case so i'm actually going to remove these components until we've just got the origin script which we're going to leave behind and instead i'm going to add a new component so if i start to type ar you can see that there's a whole load of built-in ar scripts for all types of functionality that you might want to play around with but the one we're going to use is the ar tracked image manager so this is what's going to enable us to detect images in the real world and track them to allow us to position virtual objects at their location and it needs to have a library of images to track so this is a serialized reference image library here so the next thing we'll do is to create that library so i'm going to come to assets right click create and then right down the bottom where it says xr i can create a reference image library that looks like that and then this is going to have to have some images in it as well so i'm going to find an image on my drive so in windows explorer here i've just got a regular jpeg file which is my image of the king of hearts i'm going to drag that into the assets of my project and then in my reference image library i'm going to click add image and drag that texture into the slot here that's called k hearts that's probably fine now optionally you can tell the ar engine how big the image it's looking for in the real world is and this tends to make detection a little bit better so i am going to tell it this is a playing card and i'm going to tell it in meters well this is a standard playing card so in the uk that is a little over six centimeters wide and it's 8.8 centimeters high and then if i go back to my origin object again so my ar track image manager i can now drag my reference image library into that slot there so what we did there was add a texture into the project and then add that texture into the reference image library and you can add as many images as you want you just keep on clicking the add image button here drag more textures in and this will form the set of images that the ar app is going to be looking for in the real world and then i dragged that reference image library into the serialized library slot in the ar tracked image manager okay so just doing that the app is now going to be finding and tracking those images that's great but at the moment it's not actually going to do anything when it finds those images for that we're going to have to write a custom script so we'll do add component again and this time we'll go new script and we'll give it a name so we'll call it place i don't know place tracked images or something like that create an ad and then if we double click on it we can edit our script in visual studio now unity scripts are written in c sharp um the majority of the programming examples i've done in this channel are using arduino c or c plus plus so if you're not too familiar with it don't worry what i'm going to do is i'm going to paste in the code here and i'll include a link in the video description for the code itself but i'm just going to step through it anyway to explain what's happening here so we start at the top with some using statements this is much like in an arduino code where you can include references to external libraries and then we define the class itself which has the same name as the script we just created at the top of that class we've got a handful of essentially global variables we've got a reference to the tracked image manager script which we just attached to the object in the scene hierarchy that's what's going to actually detect the images for us and then this script is going to have an array of game objects which we're going to call ar prefabs now game objects are unities kind of bundled assets everything in a game you can think of as a game object whether it's a character or an enemy or a prop or a piece of the environment or something like that a game object consists of a model and possibly some scripts that affect its behavior as well so this is what we're going to create in the scene corresponding to each image in the reference library and that game object could be a simple 2d texture applied to a quad that's how i did the replacement of one playing card with another playing card or it could be an animation unity has a movie player installed as part of it so you can play an mp4 animation instead of a static image or it could be a 3d object or an animated model or something like that so game object is the generic name for something that is going to exist in this augmented reality scene and we're going to have an array of them and the idea is that each element in that array corresponds to one of the images in the reference library that is being tracked and in fact they're going to be called the same name as well so for every tracked image we called it k hearts that image texture we will also have a prefab in this array which we'll call k hearts as well and the prefab with the same name will be placed and will track the image with that name in the scene that's how this script is going to work okay um we'll keep a dictionary which is a type of keyed array of all of the prefabs we've created and then we've got a handful of functions so awake that is synonymous with um the arduino start code this happens once when the script first starts running um so we will grab a reference to that ar tracked image manager component and we will store a reference to it in our global variable that we defined up here and on enable and on disable well this is where we're going to attach a function that's going to listen and respond to events that happen on that tracked images manager so the tracked image manager kind of does its own thing it runs in the background it's constantly monitoring that video feed that's coming onto the device and it's looking for those images in the reference library that we told it to look for and it fires off events when certain things happen and the particular events that we want to respond to is this thing called trapped images changed so this is when a new image is detected in the scene that's on that list of tracked images or perhaps an old one has left the scene it's no longer able to track it anymore or it's moved or something like that so when this script is enabled we're going to attach our function here called on track images changes we're going to call that every time the tracked images manager fires off this event and when this script is disabled well we'll simply remove that event handler instead so we've only really got a single function here on track images changed and that is the function that fills up the whole of the rest of the script it's not very long so this is what's called whenever that manager says haha something's happened in this scene with one of those images that i've been told to look out for so the first thing we're going to do is we're going to manage the case when a new tracked image has been detected so if this is the case uh we've got an argument that's passed the function here event args and you'll see that that has an array called event args.added so we're going to loop through all of the tracked images that have just been added to the scene and we're going to grab the name of the corresponding reference image that was in the library that corresponded to that tracked image and then what we're going to do is we're going to loop over all of our prefabs so for every image that was found we'll also loop over our prefabs and this is where we do the comparison to see if we have a prefab in our array that matches the name of the tracked image that has just been found in the scene so we're going to do a string comparison we'll use an ordinal comparison and i'll ignore the case so if your texture is called k underscore hearts with lower case and your game object prefab is called k under hearts in uppercase or something that doesn't matter we'll we'll ignore the case in the comparison so if we found a matching string in our array and also if we haven't already instantiated that prefab in the scene then what we'll do is we will make it and we will parent it to its tracked image in the scene so that's this idea of anchors i was mentioning before we'll create this virtual object and in the second argument to the instantiate function we'll say make the parent the transform of the tracked images so it will move around with it in the virtual space and we'll add it to our array as well just so we keep track of what's going on so that's the first thing that might cause this event handler to be fired is when a new image has been added to the scene it's also possible that one of the existing tracked images has been updated in the scene if that's the case what we'll do is we will set the prefab to be active or not depending on whether it is currently being tracked or not so some objects that briefly leave the scene they don't get deleted straight away it's just the tracked image manager will say okay well i'm not going to currently track that because i can't see it and if that's the case so if it's left the field of view in the real world what we'll also do is we'll set the corresponding prefab in the scene to be inactive as well likewise if something had left the scene previously and it's just reappeared again well what will happen then is that we'll get notification that has been updated and it is now being tracked again well let's make the corresponding prefab active in the scene again and the third possible event that can happen is that an object has actually been removed so this is where something that we were previously tracking has left the scene completely we've lost the ability to track it at all in that case that image appears in the removed array of the event arguments that's sent to this handler and we simply destroy the corresponding prefab in the scene so this is keyed by the name of the image and we'll say okay we can no longer track this reference image with this name let's destroy the instantiated prefab in the scene that had that name as well destroy removes the object from the scene and remove removes it from our dictionary that was keeping track of the things as well and that's it so you can save that script and then come back to unity and you'll see that the inspector window here has updated our ar session object with our new script and it's currently got an empty array of prefabs in it so what we're going to need to do is to populate that with one prefab for each of the images we want the reference image library to track now if you recall at the moment that's got a single image in it which is called k hearts so what we need to do is to create the object that we want to appear when the k hearts image is detected in the scene and to do that we're going to come down to the assets window at the bottom here right click go create prefab now this prefab is going to be created in the scene and track the location of the k hearts texture and for our image tracking script to do that correctly this has got to be named the same so i will call this prefabricate hearts and then double click it to edit it and now you can insert all kinds of content into this prefab so for example if you wanted to create a 3d model we could go to 3d object and we could insert a primitive like a cube for example and this prefab now would display a 3d cube in the scene that would track the playing card now just as when we were creating our reference image library we told the augmented reality engine what the size of the texture to look for was we also need to tell it the size of the object to create so my cube here at the moment is going to be one meter in all directions it is rather big to fit on a playing card so if i change that to just point one instead this will create a 10 centimeter cube instead for example i can then come back out of editing my prefab and now if i go back into my ar session origin and my place tracked images i'm going to drag my new prefab into the array here and that's all you need to do what we can do now is we can actually deploy the application to our phone and test it so we go file build settings and we need to actually target the android platform uh we're also going to add our currently open scene we've only got one scene in this particular application so we'll add the example scene we're working on at the moment just need to switch platform and now if you connect your android device to your pc via usb cable and if you have debugging mode enabled you can build and run the application all in one go there's just one step you need to do this you have to enable developer options on your phone and you do that by tapping on the serial number i think it's seven times in the device information screen to start with that will enable developer options and then you can enable usb debugging after that we can just click build and run name our package a file name so we'll call it build.apk for now and this will build and run it on the device now of course just displaying a 3d cube is not a very exciting augmented reality application so if you want to have a more interesting 3d model instead well that's easy to do here i've got an fbx model which i downloaded from the internet i'm just going to drag that into my assets window in exactly the same as i dragged the card texture earlier and now if i edit my k hearts prefab and go into here go to the source i can place a strong knight there instead now the way that the orientation of the axes works uh what you need to do is you need to position your 3d models so that they look correct when viewed from the top so it's sort of the wine the the z-axis are flipped over slightly so to just tilt this guy around what i'm going to do is i'm just going to spin him 180 degrees on the y-axis first and then lean him back 90 degrees there and just as with my cube i'm going to want to scale him to the appropriate height so let's make him 15 centimeters high in all dimensions there and these 3d models kind of have animations on them as well they can even be interactive and respond to input on the screen another thing i mentioned is you can have uh movie textures so instead of my knight model i might want to create a simple quad object so a quad is the the most simple plane you can have in 3d space i just need to flip that around as well so it's facing the correct way up there we go and on to that i could add a video player component and now i can add a video file to my project so i've got an mp4 file here which i'm just going to drag into the assets of my window and i can assign that into the video clip slot of the video player component here and now every time this prefab is created in the scene the video player will play this clip on the quad and if there is an audio channel attached to that video clip as well it will play that through the device's speaker as well so even though we're only looking for static images and textures in our real world scene we can replace them either with a straightforward image so an image swap or we can replace them with an animated video clip or we can create a 3d object at that location as well we're simply using the texture in the real world as the anchor to which those objects will be attached and if you want to find out more about any of the kind of objects which unity can create i recommend looking up examples or the unity documentation because at this point there's nothing special about augmented reality that we're using here this can simply be any game object at all that unity supports so i hope this video has given you some ideas of some of the ways that you can use augmented reality i could carry on for ages because there's so many different examples and people that follow this channel will know that i'm particularly interested in escape rooms and i hope some of the ways i've kind of teased the image replacement or the animated images or the 3d objects give you some ideas of how this can be used in an entertaining way to present new information to players that may not have seen it before perhaps they have a handheld device that is a spectral scanner and when they inspect certain objects in the room with it it is able to reveal past memories of events that happened or items that have changed over time and things like that there's really loads of different examples you can use and i hope you have some fun with it if you've got any questions about further ways that you can use the ideas which i presented this video please do write them in the comments and i will try my best to get back to you as ever i want to say thank you so much to all my patreon supporters who enabled me to make these tutorial videos each month i will put the code download available on my github account as normal but if you'd like to access the downloads for any of the other projects which i've done on this channel please do head over to my patreon where you can find out more information there other than that i just want to say thank you all very much for watching and i look forward to seeing you next time okay cheers bye

Channel: Playful Technology

Views: 148,588

Keywords: Unity, Augmented Reality, ARCore, ARKit

Id: gpaq5bAjya8

Length: 32min 11sec (1931 seconds)

Published: Mon Jun 27 2022