How to make a Mixed Reality app for the Quest 3 - Part 1: Passthrough

so you want to make a mixed reality experience for your meta 3 well you've come to the right place this video is the first part in a short tutorial Series in which I will take you through the process of creating a mixed reality app for your quest headset we will be building our app with the unity game engine when using Unity we can currently choose between two sdks for developing mixed reality apps on the quest 3 our first option is the Oculus integration which is a complete software development kit developed by meta themselves the other option is to use unity's native XR interaction toolkit since many of my previous videos cover the XR interaction toolkit I will go for this option for this particular tutorial series however I may well cover mixed reality development using the Oculus integration SDK in the future in this first video I will show you how to set up a Unity project for Quest 3 mixed reality development I will then show you how to activate pass through pass through in case you don't know is the ability to see the real world through your VR headsets cameras anyway in the last part of the tutorial we will drop a virtual character into our mixed reality scene a couple of points before we get started although I will be using a quest 3 in this tutorial the resulting app should work just fine on the quest Pro or even the Quest 2 although the pass through on the Quest 2 will be monochrome and low resolution I will also be using Windows 11 throughout this series however you should still be able to follow along on a Mac one caveat being that you will not be able to preview your VR scenes using Quest link and unity's play mode Quest link is unfortunately not supported on the Mac you will need to fully build and deploy your apps to your quest headset in order to test them finally this tutorial assumes that you have activated developer mode on your quest headset if you do not know how to do this I have dropped a link to a short howto video in the description okay let's get on with it open Unity Hub click the new project button and select the most recent LTS version of the unity 2022 editor I'm using 2022 do312 here although it may be a higher version by the time you watch this select the 3D urp template give your project a name I'm calling it Quest 3mr now click the create project button the unity editor will now create and open your new project the unity editor has now opened and our project is ready to start working with the first thing we need to do is set up our build settings go to the top menu bar click on file and then build settings in the build settings window select the Android platform set texture compression to ASC take a look at the Run device setting if your quest 3 is connected you should be able to select it from the drop- down menu here now click the Switch platform button switching the platform will cause Unity to repport the project assets once this is complete click the player settings button this will open the project settings window with the player settings section selected in order for your app to run smoothly on the quest we need to change a few settings let's start by setting the graphic API exclusively to Vulcan next make sure that you have a valid package name you need to use reverse domain name notation for this set the minimum API level to Android 12 API level 32 set the scripting backend to il2cpp and under Target architectures untick the arm 7 checkbox and enable arm 64 instead close the project settings window and then the build settings window now we are going to to be using open XR to develop our mixed reality app open XR provides a common framework for developing XR apps on different headsets and devices however to access some features specific to the quest we will need to import The Meta open XR Unity package let's do that now go to the top menu bar and click on window and then package manager on the project manager window click on the plus sign in the top left corner a drop- down menu will appear select add package by name next a small panel will appear with two text fields for entering the name and version of a package in the name text field type the following com do Unity do XR do meta hyen open XR now click the add button the package titled Unity open XR meta will now be imported into your project after which a dialogue box will appear stating that the new input system requires some native platform back ends to be enabled click yes to enable them and restart the editor once the unity editor has restarted you will notice a number of additional packages have also been installed this is because they are required dependencies of the unity open XR meta package let's quickly go through these new packages the most foundational of these is the XR plug-in management package this provides a userfriendly interface for easily configuring XR settings it uses a plug-in architecture in order to support a wide variety of XR platforms such as meta Vive or Windows mixed reality for example next we can see that the open XR plug-in has been installed the unity open XR meta package provides extensions for open XR and these are needed for us to build mixed reality apps on the quest lastly we have the AR Foundation package this provides some additional components that we will need for mixed reality to work there is is one final package that we need to install here and that's the XR interaction toolkit in the package manager go to the packages filter menu at the top of the packages list select packages Unity registry now scroll down to the bottom of the list you should find the XR interaction toolkit select the package and install it once the XR interaction toolkit has finished installing you can close the package manager go back to the project settings window select the XR plug-in management section from the left sidebar in XR plugin management make sure that the Android tab is selected tick the open XR checkbox and then underneath it tick The Meta Quest feature group checkbox go back to the XR plug-in management item in the sidebar make sure that it is expanded and then directly below it click on open XR in the main panel with the Android tab selected go to the interaction profiles list click on the plus sign at the bottom of the list this will open a drop- down menu where you can choose from a number of controllers select the Oculus touch controller profile now go back to the sidebar and underneath XR plug-in management select project validation the project validation panel will list any issues with the current XR setup there are three outstanding issues here which are clearly highlighted with yellow warning symbol [Music] click the fix all button in the top right hand corner Unity will then try to resolve these issues automatically there are two issues remaining so let's tackle each of these manually the first one is warning us that screen space ambient occlusion is enabled in the render pipeline this can impact performance on meta Quest devices significantly so we should really turn this off before we do this however let's quickly optimize a few other things regarding graphics rendering in the left side bar select quality notice that at the top of the quality section there are a number of quality levels various platforms can be assigned a specific quality level we are working solely with the quest 3 which is an Android platform we can see that the Android platform has been assigned the balanced quality level so for the sake of clarity we will remove the other quality levels we will also change the vsync count we want to set this to don't sync since the meta runtime handles its own vsync okay we will leave the rest of the settings alone for now go ahead and close the project settings panel go to the project Tab and make sure the assets folder is expanded inside of assets open the settings folder the settings folder is where all the render pipeline files live I'm going to clean things up by deleting the unused render pipeline assets that don't relate to the balanced quality level now select the urp balanced renderer asset go to the inspector panel on the right side of the editor under postprocessing untick the enabled check boox where possible you should try to avoid postprocessing on the quest as its Graphics architecture makes it expensive to render notice that under renderer features there is a component called screen space ambient occlusion remove this component completely by clicking the three dots in the corner and selecting remove from the drop- down menu since we are not using postprocessing it is also a good idea to remove any post-processing elements from your scenes for this reason I am deleting the global volume game object from the scene hierarchy there is one thing we need to check in Project settings with regard to our render pipeline go to the top menu bar and select edit and then project settings in the project settings window select Graphics from the left sidebar right at the top of the graphics section under scriptable render pipeline settings there is an empty slot which requires a render pipeline asset drag the urp balanced asset from the settings folder directly into this slot a dialogue box will appear warning you about changing the render pipeline click continue in Project settings go back to the project validation section notice that we still have one outstanding issue it is warning us that we need an AR camera manager component in order for pass through to work correctly we will need to remember to add the AR camera manager to our main camera once we set up our XR rig which is exactly what we are about to do now close the project settings window first off we are going to remove the current main camera from the scene hierarchy we don't need this since we are going to be using an XR rig which contains its own virtual camera let's set up our XR rig right click inside of the hierarchy view this will bring up a context menu select the XR submenu and then XR origin VR an XR origin object will appear in your scene hierarchy make sure that XR origin is placed at the scene origin which is World position 00 let's also expand the XR origin so we can see what it consists of the XR origin is in effect our XR rig some of you may be wondering what is an XR rig let me briefly explain an XR rig is the interface between your VR Hardware which is typically your headset and controllers and your VR scene the XR rig contains virtual counterparts to your VR Hardware devices and it tracks their position orientation and input actions so for instance the XR rig contains a virtual ual stereo camera which tracks the position of your headset this is basically the main camera inside XR origin your touch controllers are also tracked and represented in Virtual space so that you are able to manipulate virtual objects through them here the touch controllers are represented by the leftand and right hand controller game objects inside of XR origin with XR origin selected go to the inspector panel on the XR origin component set the tracking origin mode to floor go back to the scene hierarchy nested within the XR origin you will find the main camera select it go to the inspector and scroll down until you find the environment section set the background type to solid color below it is the background color click on it to bring up the Color Picker set the color to Black crucially you need to set the colors Alpha to zero if you don't do this you will see a black background when you run your your app instead of the pass through image close the Color Picker the other important step is to add an AR camera manager to the main camera let's do that now with the main camera still selected scroll down to the bottom of the inspector click the add component button a component selection panel will appear find and select the AR camera manager an AR camera manager component has now been added and enabled on the main camera this is Cru crucial for pass through to work in your meta app we should also add the AR session component to our scene AR session controls the life cycle of the augmented reality experience by enabling or disabling AR in our Quest headset rightclick inside the hierarchy view to bring up the context menu from the menu select XR and then AR session an AR session game object will be added to your scene there is no need to change anything on this component I've discovered that having HDR enabled in your render pipeline settings will prevent pass through from working correctly on the quest let's remedy this go to the project assets panel and in the settings folder select the urp balanced asset in the inspector under quality untick the HDR checkbox additionally let's disable terrain holes in the rendering section since we are here go to the lighting section and also disable additional lights finally scroll down to the Shadows section and reduce the max distance to 2.5 okay we are more or less done with setting up our project to work with pass through on the quest let's find a character to drop into our scene the unity asset store is a good place to find free assets so let's go there while keeping your Unity project open open a web browser and go to asset store. unity.com the asset store has a number of free character models that we could use however I've tested out a free zombie character that I think will work well for our purposes go to the search bar and type zombie a free zombie character by pxl Tiger should appear in the search results if not I've provided a direct link to the asset in the description click on the thumbnail you will be taken to the product page make sure you are logged into your Unity account now click the add to my assets button accept the terms of service finally click the open in unity button back in the unity editor you will notice that the package manager has been automatically opened with the zombie asset selected download the zombie and then import it once the zombie asset has finished importing close the package manager notice that there is a new folder called Zombie in your project assets open this folder and within this open the prefabs folder there are three zombie character models in the folder drag one of them into the scene view you will notice that the zombie is a uniform bright pink color this is because the zombie was designed for the built-in render pipeline not the universal render pipeline which is the pipeline that we are using as a consequence the character's material is not compatible with urp not to worry though we can use a render pipeline converter to fix this I'll show you how to do this now go to the top menu bar and select window then rendering and then render pipeline converter at the top of the render pipeline converter window there is a drop- down menu make sure built-in to urp is selected below this are some checkboxes relating to asset types that can be converted let's just tick all of the checkboxes click the initialize and convert button and and wait for the conversion to complete close the render pipeline converter notice that our zombie character is now fully textured I'm just going to reposition the zombie so that he is right in front of the XR camera and facing towards it okay we are ready to build and deploy the app to our headset make sure your quest is switched on and connected to your PC via USB from the top menu bar select file and then build settings in build settings make sure your quest 3 is selected under run device click the build and run button once your app has been built and deployed you can Dawn your headset and play your mixed reality app you should be able to see your real surroundings however you should also be able to see the virtual zombie superimposed onto the pass through image giving the impression that the zombie is in your real space let's return to the unity editor before we finish I just want to show you one more thing you may be aware that you can preview your quest apps in the headset directly from the unity editor without the need to compile a build you can do this by launching play mode in the editor for this to work however you will need to change a couple of settings in the XR plug-in management system go back to project settings in the project settings window make sure that the XR plug-in management section is selected make sure that the PC tab is selected tick the open XR checkbox go back to the sidebar and under XR plug-in management select the open XR subsection go to the interaction profiles list and add the Oculus touch controller profile close the project settings window okay with this setup you can now trigger play mode and preview your quest app directly through your headset let's test it out make sure your quest is switched on and connected to your PC via USB cable in your quest headset launch Quest link once this is launched go ahead and hit the play mode button you should now be able to put on the headset and play the app unfortunately pass through does not work in play mode however and that brings us to the end of this tutorial we will continue to add mixed reality features to this app in upcoming videos so if you found this one useful please like And subscribe but for now goodbye and happy questing

Channel: Ludic Worlds

Views: 19,740

Keywords:

Id: ibBzydrumc0

Length: 19min 33sec (1173 seconds)

Published: Fri Nov 17 2023