How to use Lights & Shadows in Passthrough Mode | Passthrough Relighting w/ OVR Scene Manager & MRUK

hi XR developers in today's video we're going to look at meta's pass through relighting pass through relighting allows us to show realtime Lighting on our room mesh I will show you how to achieve this effect using the mixed reality utility kit or mruk for short which we looked at in a previous video if you are not using mruk I will show you how to achieve the exact same effect using the OVR scene manager which you will know for sure if you have worked with a scene API before or used a room model building block we will also shortly look at how to use realtime shadows in our scene and so-called blob shadows as a performance saving alternative to realtime Shadows furthermore we will look at how to reduce and increase the brightness of our pass through layer reducing the brightness and increasing the lights will make for a very nice effect which I will show you in this video If you like this type of content and you would like to support me please take a second to like And subscribe to this channel if you'd like to get the source code to each video please consider to join my patreon if you have any questions feel free to join our Discord community and now let's get started with pass through relighting we start by setting up a Unity project with the meta XR SDK and the mixed reality utility kit if you're not sure how to install them just watch my last video that explains the utility kit in detail I am going to continue with the same project because we only really have to import One More Sample package which is the pass through relighting import it and go ahead and open the pass through relighting scene first L we have a OVR interaction component there is nothing special about it if we open it we can see that it simply contains the controller visuals and a controll array interactor on the right hand so we can interact with the control panel that sits on the left controller we can see this by opening the tracking space on the OVR camera rig and opening the left controller anchor to clean up the hierarchy a bit let's just move the OVR interaction onto the OVR camera rig let's also move our directional light out of the way and drag the pointable canvas module onto the control panel since it is just a component we need to interact with our panel I think the scene looks a lot clearer and easier to understand now next we have a simple pass through layer that we need to activate pass through mode back on the OVR manager we just need to make sure that the scene and pass through support are enabled next there is OPI a character we can let walk through our room we choose the Direction with our joystick and jump by pressing the a or x button poy carries a flame with him that contains a regular Point light let's also drag this game object onto Opie to clean up our hierarchy even more you can see we can already play our scene in the editor and see how the light from OPI is reflecting on our room surfaces on the OPI game object we can see another game object that is disabled at the moment which is called blob Shadow those are simple blots of color that do not take into account the geometry of the object and are more performant blob Shadow objects are designed using the projector component along with a specific material and Shader from Unity but let's first finish our scene overview next we see a regular Muk component that we also explained in detail in the previous video keep in mind that this serves as a replacement to the OVR scene manager lastly we have two effect mesh game objects both are applying the transparent scene anchor material to our room meshes this material consists of the highlights and shadows Shader that comes with the utility kit the difference between those two game objects is that on this first object we apply the effect to our anchor objects or in other words to our furniture walls and Floors we can also see that this will be our default mesh because we are creating this mesh M after starting the scene and after the room has loaded the second game object applies the mesh material to the global mesh which is basically our raw Point cloud from scanning our room and could potentially be more accurate and also take into account things like the PC on your table or the pillow on your bed perfect this seems like a pretty simple setup let's now take a look at our control panel again before we give this scene a test as you can see we have several toggles and Sliders we can switch from real time to blob Shadows we can turn off the highlights or turn them on again which refers to our Point light next we can toggle between using our regular effect mesh or the effect mesh for our Global mesh the button simply lets us respawn opy in case it gets stuck somewhere the light Alpha multiply slider increases and decreases the strength of our Point light and lastly the pass through brightness slider lets us adjust the brightness of our cameras let's quick L open the script to look at those features under the hood we can see that we simply add a listener to each slider and toggle this allows us to call a method as soon as the value of an element changes so for example if the value of the global mesh toggle changes we call the global mesh settings toggled method when the global mesh setting is active and the application is not in the editor mode it creates or destroys each mesh in the effect meshes collection based on whether they have the global mesh label as we have seen on the effect mesh component before conversely if the global mesh is inactive or the application is in the editor mode it applies a more specific destruction process using a label filter or creates the meshes that don't have the global mesh label followed by a respawn action through opont controller. respawn Also let's look at the Shadows settings toggled method it simply tells the renderer on the OPI object to not cast shadows and enables The Blob Shadow lastly in the awake method we have a very neat implementation of the pass through brightness slider we simply call set brightness contrast saturation from the pass through layer and apply the float value that comes directly from the slider and should be between 0 and 1 enough talking let's finally visualize these examples like always if you want to use your own room for testing make sure you set up a room scan and added your furniture as accurately as possible inside your quest device since we cannot visualize the global mesh in the editor we will build an APK and test it on our device as you can see Opie spawns into our scene and we can let him move around while Opie walks around we can see the lights change on the wall we can see it much better by increasing the light strength and decreasing the pass through brightness now it looks like Opie is walking through a cave with a torch awesome we can switch from realtime Shadows to blob Shadows or turn off the lights completely lastly we can switch from showing the light reflection on our scen object to showing them on our Global mesh scan as you can see if we created a detailed scan of our room before we can now really accurately see that our complete room mesh is reflecting the light amazing guys to quickly recap all we need to achieve this pass through relighting effect is to add a m UK component as well as a effect mesh component that applies a material consisting of the high lights and shadows Shader to our room models this together with one or more Point lights will allow us to achieve a nice lighting effect inside our pass through scene now I want to show you how to achieve the same effect even without the Muk component in our scene let's create a new scene for that let's use the building blocks to quickly get going if you don't know what building blocks are feel free to watch this video where I explain everything in detail I will add a camera rig pass through and room model building block to my scene the room model block contains the OVR scene manager which allows us to scan and visualize our room mesh just like the Muk component so if you are still using the OVR scene manager or just don't want to use Muk all you have to do to achieve the same lighting effect is to search for ptrl in our project window under packages ptrl stands for pass through relighting here we can find new prefabs for our plane and volume anchors as well as for our Global mesh simply drag the plane and volume prefabs into the inspector for the global mesh we create a new element in the prefab overrides list and select whichever labels we want to apply this effect to in this case let's select Global mesh and drag the global mesh prefab into the field however keep in mind that these prefabs only come with the mixed reality utility kit and that's it the last thing we need is some some kind of light source for that we want to shoot a ball into our space that has a point light attached to it let's create a new sphere and scale it down to around 0.05 let's not forget to attach a rigid body to later apply Force to the ball we then add a point light as child we reduce its range to about 0.3 and I would like to have a more yellow light let's create a new folder and call it materials and another one called prefabs in the materials folder fer let's create a material and set the color to Yellow I also want the projectiles to bounce a little bit so let's also create a physics material you can adjust the bounciness however you like we lastly apply those materials and are then ready to drag our ball into the prefabs folder now we need a simple script to instantiate the ball and shoot it into our room let's create a scripts folder and create a new script called spawn ball in this script we want to firstly declare two fields that we can modify in the inspector which is the reference to the ball prefab and the force with which we would like to shoot it we then simply have an update method where we first check if we press the trigger button on the right controller which is also known as the secondary index trigger if so we get the position and rotation of the right controller and instantiate the ball prefab at this position and rotation lastly we fetch the rigid body of our spawned ball and apply Force to it to shoot it from our right controller into the forward direction of the controller at this moment amazing guys let's create a new game object and assign our new script to it let's reference our ball prefab and give this scene finally a try you can see the ball in our room and how the lights show on our surfaces however you will notice that if you spawn multiple balls the lights suddenly behave weirdly in order to show highlights from more than one point light source make sure that the desired amount of per pixel light sources is specified in your project settings for the built-in render pipeline the setting is called pixel light count and it is in the project settings under quality let's set the value to something like three and give it another try now we can notice that we can finally see multiple light sources in our scene for the universal render Pipeline on your Universal render pipeline asset in the section lighting set additional light to be per pixel and set the per object limit to the desired number all right guys and that's it for this video as you can see we can achieve a very nice effect by lowering the brightness a little bit and turning on the lights and it almost looks like Opie has a torch on his head and is walking through our dark room I can't wait to see what you guys are building with this again please take a second to like And subscribe to this channel consider subscribing to my patreon or joining our XR developer community on Discord thank you so much for watching and see you in the next one [Music]

Channel: Black Whale Studio - XR Tutorials

Views: 2,277

Keywords: meta quest passthrough, develop app for meta quest, unity vr, unity meta, unity vr tutorial, unity tutorial 2024, meta quest tutorial, oculus integration, unity installation, meta vr, meta mixed reality, meta quest developer tools, mruk, mixed reality utility kit, meta utility kit, meta mixed reality utility kit, meta quest pro, meta quest 2, meta quest 3, passthrough relighting, ptrl, meta passthrough relighting, meta ptrl, lights passthrough, shadows passthrough

Id: p8LYtvNxcf0

Length: 12min 8sec (728 seconds)

Published: Sat Jan 06 2024