Create an IKEA app clone with Unity and ARCore

Hey IKEA would like em Hey hi everyone sorry for the convening inconvenience I am really sorry about that so basically what's happening is that this is the first time I am coming to a livestream so this is kind of difficult for me at least I try to figure out how to do how to figure out all this stuff you know so I apologize for all the inconvenience that you guys have to face so today what we're going to do is create an AR application that is going to be kind of similar to the IKEA place app right so um if you haven't seen that app before I can play this video and we can first of all see the view and see what they are basically doing in the app and then we're going to go directly to the practical coding session where I am going to show you the step-by-step process to create and similar kind of app like this and obviously we cannot possibly make it that much finished in only one class right but we can get the basic from here and you can use the code to polish this app so you know you can develop similar kind of app in the problem space so let's see what they're doing everyone to know about the place our new augmented reality app [Music] they are key we can easily place to to scale 3d models of IKEA furniture in your place scan browse select and place so that could mean less of it [Music] and probably more of it [Music] we want to make it easier for people everywhere to imagine a better place share this place [Music] right place in your place right so as you saw that this application helps you to put different kind of furniture in your own place and you can possibly do some more interaction with that you can rotate that you can place it in different places right you can scale the furniture to see which of which are scale suite to your environment and so on so we can develop this same kind of concept with the current AR technology and this is extremely easy and doesn't require a lot of coding experience I am assuming that you know a little bit of unity but even if you don't know that's also fine but just going to start from various scratch from a brand new project and then gonna go slowly so that everyone is in the same page ok so let's get started first of all what we need to do is create a unity project okay so let's open the unity hub if you don't have any unity installed in your computer just go to unity dot unity 3d calm and you'll find out the download link somewhere in here I think if you click on the get started you should find a option called individual and then you can sign up for the student or the personal account and unless you earn about I think 100k in 12 month the personal unity is free for everyone ok so yeah I'm not going to show you the whole process of installing unity there is a lot of tutorials out there where you can e like search and install unity in your computer but if you're struggling just text me I'll try to help you out with that okay so first thing first we need to create a new project so I just opened up the unity hub and I am going to create a new unity project I am using the images now to do the 19.3 point for version and here I'm going to name the project as a our furniture and we do wait for a while to open the project I think the computer is taking a little bit long because I am running the game engine and the streaming service together I don't know okay here it is so now just before you start developing into a our application I'm just going to talk a little bit about where the current state of art of this mobile air technology is right there is obviously a lot of advanced headset like Microsoft has follow lens right listen we magically stopped making new devices but I am hoping finger crossed they will come back so there is this kind of advanced level headset which can track actual environment which can use advanced computer vision algorithm with some kind of you know sensor fusion to detect the surrounding environment but there is again mobile apps which are now really really gay like taking off in the current industry because of the SDK the SDKs are making the development process so easy for everyone that it's not that hard to create a air application of your own from you know very scratch without knowing much about coding so there are basically two popular AR SDK one is AR code which is for basically Android devices and all there are certain amounts of Android devices which supports air code and there are there is another one which is AR kit which is for Apple devices right and AR kid again has some specific device requirements that you can find out in their website if you search for AR core support that devices you can obviously find the supported devices list in their website so you can check that if your phone is basically supported with this technology right there is a list of all air supported devices in that page now what we are going to do today is a little bit different we are not going to use our code we are not going to use air kit but we are going to use both of them together how we are going to use the power of air foundation so a our foundation and I'm gonna go back to Google again a our foundation is basically a unity tool that combines this cross cross plot from compelled compatibility that unity always provides us right - so is its combines the Air Corps and air kit together and you write your code once and then you can put your game or app both in the Android device or in the iOS device okay so we have talked a lot now let's get started with the coding session where we can actually write the code to develop and air app so first thing first we need to do some installation I am assuming that everyone who is watching this live has the unity software installed so if you don't have the unity installed maybe just do it quickly and then try to you know rewind the video and just try to cope up with us so I'm just going to assume that everyone has the unit installed now what I'm going to do next is going to install some package that will help me to initialize this air application to do that I am going to go to window and package manager and in here you will find the option called all packages right so if there is too often all packages in project we need the all packages option and then here I'm gonna search for a our foundation right so when you find the plugin just hit install and it's going to install the necessary assets into your project right next because for this example I am going to build the app for Android so that's why I am going to install that AR core SDK okay so as you can see there is this a our core XR plug-in so I am going to install that that's it so now when we are done installing this if we right-click on this hierarchy in our project and if we go to X R we are going to see a lot of new options that was not there before okay so if you are brand new to unity if you don't know anything about unity I'm just going to give you a real quick introduction of the editor so by default the editor should look something like this right and every part of this editor has a particular name for example this portion in the left hand side of your screen where I am just moving my mouse cursor is called hierarchy right the name is written on the top of that tab similarly this is called the scene window this is called the game window this is called asset store and so on so everything that we do in our game should go inside this hierarchy okay so everything that you put inside your scene or that like stays in your game is also going to be stay in the hierarchy so hierarchy is the place where you put all your stuff ok so if we try to create a new object or if we try to create a new camera we always start with this hierarchy number one number two is that when we click on one of the object in hierarchy we can see the property of that object in the right-hand side of the screen this is called the inspector so currently we are just going to talk about three windows which are really really important hierarchy which contains all the game object if you click on one of the game object in the right hand side we have inspector where we can see the particular property of that game object for example I am clicking on the directional light and that shows me all the properties of the directional light in the right-hand side of my we know right then this scene window is basically kind of an editor right so we can use this to change different kind of stuff in our game or in our environment finally we have this project tab where we can store all the asset that we are going to use in our game okay so for example currently we don't have anything we only have a scene so that is only as that we have so that scene is combined with one main camera and one direction right okay so I don't like this um I don't like this I like format so I am going to go to my the to 2 by 3 is gone okay so I don't know I'm just gonna go to for split then and then get you down this I'm just going to do a quick you know configuration because I don't like the default formation of the unity engine so I like I am more familiar with the older one the older unity layout so this is what I prefer but your you can even do whichever you prefer you know like whatever you want okay so yeah what we should do now is going to first of all delete the main camera because this is an air application and we are going to do a like we are going to use the camera of our device we are not interested in the camera of this you know engine so we can simply delete this delete this and as you can see there is no camera rendering right now next I am going to right click go to X R and then in unity AR foundation in air foundation the first thing you need for an air project is basically a session and an origin okay so we are going to create both of them we are going to create a session origin and a session okay so a session origin contains an AR camera that is going to be popped up automatically when you first install your app and it's going to ask for all the permission you don't have to do anything manually you don't have to write any code it does it for you so we are just going to take the power of that particular game object right okay next if we now save and run our project let's try that and try to build the project you know Android device and see what happens if I go to file and then build settings I need to change the platform from PC to Android and then click on switch that and then I'm going to just add the open scene right so then if I try to go to build settings and player settings there is a few things that we need to take care about before we hit the build button first of all we need to give a name of the build right so we need to give a company name I'm going to then do it justic lab and then in the publishing settings because we are not publishing it now so we can put it empty that's fine in the resolution and presentation I am going to put it as portrait and also I am going to go to the other settings and in here we need to make sure that the Vulkan API is disabled right so the Air Corps does not support this API this graphics API so we need to work with the opengl so we're just going to delete that next aerco doesn't support markets ready so we cannot use the multi-threaded rendering here we need to uncheck that as well finally here where you have the minimum minimum API level you need to make sure that it's at least under seven so I'm just going to do 7.1 when that's done I am just going to hit the build I'm going to give it a name I think I I thought I have the JDK let me check okay let's go to unity hub and I'm just going to go through add modules and draw it right so we can just install that so if you see the same error like I got you can go to Android build support and then you can just install that this is new to unity the unity 2019 point three point four has changed a lot from their previous version I don't like this because they're moving on so fast and it's hard to cope up with right but yeah I like unity so when everything is installing I'm just going to connect my Android device with my computer so I can build and run that game okay restart now I'm going to close the unity and open it again right so then I'm just going to yep so everything is configured now that's good I'm going to go to file and then build and run hope there is no error okay nevermind I'm just going to build the game first let's see if there is any build error not safe okay that's good so it's building there is no error till now right so we have successfully built our first built with the air foundation let's run this in my inner game and see what the output looks like I think I have visor installed here okay so I I think that the the build and run was not working because I didn't mention that the run device should be the one that I am connecting with so let's try it again that's weird the debugging is obviously one just gonna try for the last time if it doesn't work then I'm just going to copy and paste the app yep so it doesn't work it's good all right so I'm not going to bore you guys with all this and I'm just gonna move on so yeah so what will happen that if we now run the app it will open the camera and it will ask for the permission to open the camera and we can just move around the camera nothing else will happen okay so we're just going to move on and I'll probably hopefully will show you the demo for the actual air app later on when we build it from here so we're just going to fix that later okay so what did we do till now we just created an main camera for the AR app right we just add the game object for a session and then we hit the build to build the game the next thing we need to do is basically detecting a plane so the current AR apps mostly works almost like the same way so they use some feature extraction method to find some feature or point cloud from your surface where you're pointing your camera T right so when there is a lot of feature points then it can find out that okay I can create a geometrical shape from these feature points right so it can match those feature again and next time when you track that let that same position it can you know do some mathematical calculation some complex algorithm and like put the Augmented component on that particular surface so it it all starts with simply finding the surface of the plane so this algorithm of finding the place is basically a localization problem in computer vision so the algorithm is called the odometry okay so we are not going to talk about the odometry arm algorithm and not to go in depth of the background of the algorithm because this is this needs a lot of you know vast knowledge of computer vision to understand the algorithms probably in the future classes is in some of the days we're just going to talk about computer vision and then we can talk about odometry okay so yeah so I I'm just going to show you the like the papers that tell algorithm that the Air Corps is using to find out the the real time you know understanding of the world tracking so these are the paper that you can search on Google if you're interested to learn more about this you can go to this nice color article and just search for visual inertia odometry okay so this is the actual algorithm that has been used to do the world tracking so long story short we need to do something to track the world okay so this is the process we can't create a camera we track the world when we get the tracking information of the world by obviously feature instruction from that algorithm that we don't have to implement then the computer or the device has some understanding of the world itself right then it's easy for the computer of the app to put some stuff on that world so to do that first of all we need to obviously find the surface find a plane to do that we first of all need to create a plane object of that XR and you'll see that all the stuff that I talked about just now sounds so complicated right but a our foundation has made it so simple and we can just do it with a bunch of clicks so to do that first of all going to create a default plane so I'm just going to create a plane I am going to call it a our lane and I'm going to create a new folder name it as prefab and drag and drop the airplane on the prefab okay awesome next we need to make this be fab actually you know work with the plane tracker so this this plane this airplane it is just a visualizer when you find a plane like I said you track the world through your mobile camera when you have enough tracking information when a our foundation have enough tracking information it produced a plane right and that plane is this plane so this is how the plane looks like okay and this is not going to work unless we give a way to track the actual feature informations right so next we need to do that I'm just going to delete this we already have the prefab so we don't need that we just need to go back to the session origin and Here I am going to create a plane man there you can search for a our plane manager and here you can see that there is a prefab asking for the plane prefab so we're just going to drag and drop the plane prefab here that's it now this is going to be a IKEA kind of app right this is going to be a furniture app when you put some furniture in your room flow so that is why the detection mode should be only horizontal so we are just going to click on the nothing first and then we're going to select horizontal okay that's it if we now save the project and billion I'm just going to build I'm just going to build this replace this and in the meanwhile I am going to download Android file manager okay awesome so I got the connection to my Android device there's copy the app and then I'm going to install it and show you the output if I can I I just finger crossed if I can if you allow me to show or share that like screen here then I'm going to show you guys that output files are right so again I'm really sorry about all these things I should have done this before but basically I I thought everything will run so smoothly so I didn't even care to think about all this so I am I'm just trying to do it now so I applaud justice for all that stuff okay okay guys so yeah so now you can see the screen of my device I think okay that's better I'm going to install the app okay so now you to hit allow and then if I try to move this back and forth as you can see the thing is getting rejected right so yeah so we can basically did you guys see the output or no I'm not sure right so I'm going to move on and now I'm going to show you that okay I'm just going to move on I'm just going to move on maybe maybe later on I don't know I'm just gonna put some demo video and you can see that how it's actually working if your project is not working you can connect me back like ping me in my email or in the discard Channel and I'll try to help you out with that okay so it should be working so now by this time you can see the plane right but the problem is that the plane is not the the color of the plane maybe we are not satisfied with the color of the plane in that case what we could do is like we can go to this plane prefab right and in here we can go to the material of this plane right we can go to this mesh renderers material and if we change the material with our own picture we can find out that particular texture when we you know try to get a plane so I'm going to give you a link in the chat box where you can find out all this primary knowledge of the air foundation just the one that we just discovered right so I'm just going to go to my account there is this this video right so I have this series I started this series and we are going to go forward with the same series so I'm just going to pass this in our chat so we can go through that video and you can like follow the same process to see that how you can change the plane structure okay so send you guys that know okay so that's great now we have a plane the next image to do is basically add some object in the game when we tap on the screen to do that we will need a c-sharp script so let's right-click on the asset folder create a new folder and I'm going to call it script and in here I am going to create a new C sharp script I am going to call it as input handler maybe a good manager right now the input manager is going to take an input from the user and try to generate a augmented object on the environment where the user has clicked so I am going to create a private game object actually we don't maybe okay just let's do it public game object augmented object so this is the object that is going to be augmented okay and if we go back to unity if I go back to energy and then drag and drop this anywhere I am just going to create a new game object call it input manager drag and drop the input manager here and as you can see it's asking for an augmented object right so we can use this link to basically create an Augmented object into our environment so we can drag-and-drop that object piece up here and then we can generate that object when you click the button so this is the first thing that we need to do next we are going to create a setting so that when you click on somewhere that particle location is basically tracked by the computer right so here what I am going to do is just going to create a option to check that if the user is like clicking anywhere in the Itachi anywhere in the screen okay so we can do that by finding the gate mouse button down zero right I don't know why it's not giving me solutions everything has to be wrong now okay here I'm going to when I when I click anywhere in my screen that is going to give me a true on this condition okay so when you get that particular click what you are going to do is get that position by input dot Mouse position right so this is going to give you the actual you know the actual mouse position where you clicking but basically what we need is get the pasta position from the screen to array okay so we're going to go there in a while so I'm just need to comment that so what our idea here is going to be is that we're going to throw array from the camera right so this mean to throw a virtual ray from the camera which is called ray cast so it is going to throw that ray and if that Ray has hit any of the object in the environment then where it is going to spawn something in that particular location so basically that means we need a ray caspo so let's start with creating a ray cast first so we're just going to do public private a our brain cache manager so to do to get that we obviously need the XR name space provided by unity and here we're just going to do a our a test manager and they cast manager so we have a recast manager now we can throw array from that records manager and try to find out that what did we hit with right so that means we next need array and like I said we are going to throw away from the camera to the location where the user has clicked in the screen right and then we're going to pass that tray to our records manager so let's see so we are also then going to need a camera because like I said we're going to throw that away from the camera so create a a art camera and in here I am going to do ray ray is equal AirCam dot screen point - ray so we're just going to take the input dot Mouse position which is inside down inside the screen right the screen space and we're going to convert that to a ray right so why we need the rain because we need to pass this ray to the raycast manager okay so to do that we're just going to do if the raycast manager this is so annoying I don't know why jet brain is not training any kind of you know any kind of solution ray cast ray and then we need the heat information right so just bear with me okay so you guys see that we need a few things to configure all this work together so first of all we need the input location of the mouse and when to pass that even with location to the screen point array so we can convert the screen to we can convert the screen point to a ray and that ray is going to be passed to the recast manager and finally we need a hit information that is calculated through this ray cast manager okay so we are going to create a private list of AR ray cast it and video hits I will initialize it with the raycast hit class and we can just pass that hips information here right so that means what will happen if it finds any kind of hidden information by throwing a ray which ray is basically calculated through the input position of the users mouse location or the touch location then we are going to store that hinge hit information in this hit list okay and then we are going to initialize that is for instance yet understand yet instantiate that augmented object right we will instantiate this guy wherever we basically hit right so how can we get that location well we can get that location and the rotation with something called the poles so we can do it of 0.4 so from the information of these hips right we're getting the poles of that particular heat and we're just like we're just going to use that to look at the instance instant object okay for example here we are using this augmented optic so rather than putting it inside this bracket this is going to make our code ugly so I am going to create a new poles class instant instance and then I am going to store that here right that will help us to do this for start position hold on rotation awesome right so yeah that's about it so now as you move on if you know again build and run the game I cannot do it I don't know what happened if we run the game obviously by assigning some component I will show you that in a while it should generate new object whenever you click somewhere in the screen okay so it will throw a ray from the point you clicked to the environment and then in the environment you will see the object instant okay so I'm going to go back to unity and then I'm going to assign some object that I want to instantiate I'm going to go back to Google find some object from sketchfab this is a brilliant website we can also use poly right so we can either use poly that is from Google right so we can use this poly to get some furniture right I'm going to use all of this download this they are all gltf file this is going to be me we can obviously use all of this but I am going to go to sketchfab because they have direct support for unity so you can go to sketchfab and then you can search for furniture and grab one of them there is free options as well I'm going to grab one of the free asset probably this one this one maybe it's dead is downloadable yeah it's downloadable so if you click on there you will need an account I thought I have an account already but this login you you

Channel: Joystick Lab

Views: 4,436

Keywords:

Id: onLtMBkHtjc

Length: 68min 3sec (4083 seconds)

Published: Fri May 08 2020