Hey, game dev enjoyers, PixTrick here. Recently, well not that recently because I
was busy taking care of the discord server and slaying enemies in Darkest Dungeon, but
more seriously I made a video about the stencil buffer in Unity, and some of you asked for
a tutorial. As usual, I will consider that you know the
Unity basics, otherwise I would recommend checking out the BlackThornProd’s channel
for example to get used to Unity in general. Anyways, here’s how the stencil buffer works
in Unity URP and some of its applications. We will first see the theory, then how to
cut holes in things, then how to make an impossible geometry, and finally how to create a magic
card. First, let me quickly explain how the stencil
buffer works in Unity. The stencil buffer is basically an integer
between 0 and 255 associated to each pixel of the screen. When rendering, each object can then check
and manipulate the stencil buffer values covered by its mesh. For instance, for each pixel that an object
geometry would cover, we can check the value stored in the stencil buffer and render the
object accordingly. When I say “accordingly”, I mean that
for each object that is rendered and for each pixel that its geometry covers, we verify
an equation involving a value used for the comparison and the current stencil buffer
value in that pixel. This is called a stencil test, and if the
test is true, then the pixel of the object geometry is rendered. Otherwise, if the test is false, the pixel
of the object geometry is skipped, which means that it is not rendered at all. The value used for the comparison AND for
writing into the stencil buffer is called the “Ref” value. I insist on the “and” because the ref
value is used for both reading and writing. Besides, the comparison function used in the
stencil test to compare the Ref value and the stencil buffer values is called “Comp”,
and here’s a list of the comparison operations we can perform between a current stencil buffer
value and the ref value : Never, the comparison is always false so we never render the object
geometry pixels; Less, we render the objects geometry pixels only where the ref value is
less than the current stencil buffer value; Equal, these two values are equal; LEqual,
which is Less or Equal; Greater, the ref value is greater than the current stencil buffer
value; Not equal, these two values are not equal; GEqual, which is greater or equal and
Always, the comparison is always true so we always render the object geometry pixels if
that’s possible. I put you a link in the description for the
comparison operations documentation. As I said before, if the comparison is true
for a pixel, then the object mesh renders on that pixel. We say in that case that the pixel passes
the stencil test, and we can configure the operation to perform on the stencil buffer
in that case by referencing the Pass command. Otherwise, we can also configure the operation
to perform on the stencil buffer if the stencil test fails by referencing the Fail command. Here’s a list of the 3 main operations we
can perform on a stencil buffer value: Keep, which is keeping the current stencil buffer
value; Zero, which is replacing the current stencil buffer value by a zero and Replace,
which is replacing the current stencil buffer value by the ref value. You can check the other stencil operations
on the Unity documentation, I put you a link in the description for that. As an example, you can declare that if the
value stored in the stencil buffer is less than 10, then the object should not render
on these pixels but instead store a zero in the associated stencil buffer values. So, in that example, we would check if the
stencil buffer values are greater than 10 to render our object, so we’ll use the Greater
comparison function because we want our stencil test to pass only where the ref value is greater
than the stencil buffer values, then if this stencil test fails, which means that the object
is not rendered on these pixels, we put a zero in the associated stencil buffer values
using the Fail command. Finally, a more advanced method to configure
the stencil tests and operations is by referencing the ReadMask and WriteMask values, which are
also both integers from 0 to 255. Before jumping right into it, just a disclaimer
to tell you that this part won’t be useful for the tutorial so you can skip it to that
timestamp if you’re not interested. The ReadMask value is used as a bit mask for
the ref and the current stencil buffer values only during the stencil test. That means that the final compared values
would be then the resulting values after masking them. For example, let’s say that we want to write
a value of 3 into the stencil buffer for every stencil buffer value that is an uneven number. An uneven integer always contains a 1 on its
first bit of its binary representation, like 3, our ref value. Then, we can set the ReadMask value to 1,
so that the resulting value after masking every uneven integer, such as 3, our ref value,
or 5, 7, 9, or 11, would be 1. The stencil test would then test if the stencil
buffer values covered by the object geometry after masking them by the ReadMask value are
equal to our ref value after masking it, which is also 1, and if that’s the case the stencil
test passes, otherwise it fails. Regarding the WriteMask value, it also acts
as a bit mask for the ref value, but only during a stencil operation. Then, instead of writing directly the ref
value into the stencil buffer, you can write another value into the stencil buffer that
would be the resulting value after masking the ref value. For example, let’s say that we want to check
if the stencil buffer values are equal to 7, but we also want to write a 4 when the
stencil test fails. Then, our ref value would be 7, and we can
set the WriteMask value to 4 because the resulting value after masking the ref value would be
4 too. The stencil operation would then use 4 instead
of 7 for writing into the stencil buffer. To summarize, we have to use 7 as the ref
value with the Equal comparison operation, then use a value of 4 for the WriteMask value,
and lastly, we have to reference the Fail command to write a 4 in the stencil buffer
values where the stencil test failed. Okay, let’s calm down a bit now. Another way of controlling how an object should
be rendered is using the depth buffer, i.e., the distance between the camera and the object
mesh pixel that is rendered. You can check for example if that distance
is greater than other rendered object pixels, which means that the pixel of the object we
want to render is behind other objects. This test is called ZTest, and is executed
after the stencil test. If the ZTest passes, the object geometry pixels
are rendered. If the test fails, the object pixels are not
rendered but we can perform a stencil operation on these pixels by referencing the ZFail operation,
which is a stencil operation such as the ones used in the Pass and Fail commands. By the way, we cannot ZTest transparent and
invisible objects, and this is basically due to how Unity handles the transparent rendering. We will see why this is important later in
the video. Anyways, now we will see how to interact with
the stencil buffer, how to set values in the stencil buffer and how to read them. Before anything else, we have to do a very
important thing. We have to go in Settings, then into the URP
renderer settings your project is configured with. By default, this is the HighFidelity one. Now, disable the depth priming, because otherwise
we won’t be able to use the stencil buffer. This is an issue someone on the discord server
encountered and for which we spent a lot of time trying to fix it. Anyways, to set values into the stencil buffer,
two approaches are possible. On the one hand, we can use shaders to do
that. For instance, a common case of the stencil
buffer is a simple shader that always sets the stencil buffer values covered by its mesh
to a certain value, while being invisible. This is especially useful for our applications,
because it works as an invisible mask for object geometries. To use the stencil buffer through a shader,
we simply have to add the Stencil section in the Pass section of its subshader. Every command related to the stencil buffer
will then be placed here. You can actually add a stencil section in
whatever shader you want, like this one. This shader is an opaque URP shader that is
simply about referencing a value we will write into the stencil buffer values covered by
its mesh, regardless of the values already stored in the stencil buffer. Finally, we set it invisible, so we also have
to disable ZWrite because we cannot ZTest it. By the way, this shader is in description,
just in case. On the other hand, we can directly use the
render features in the URP renderer settings to render an object and set the stencil buffer
values covered by its mesh to another value. This prevents us from writing a shader and
allow us to use any material we want for the object material, but then we have to make
sure to use the right settings for that. Also, it is not possible to configure the
ReadMask and WriteMask values using directly the URP renderer settings, but we will never
need them in this tutorial so it doesn’t matter. Anyways, now we are ready to start our first
stencil buffer application: cutting a hole in the ground! For this tutorial, I will use the shader method
but feel free to use the render feature technique too. Now that we know how to set values in the
stencil buffer, we will see how to interact with them with the first example: cutting
a hole in something. The idea is extremely simple: we first render
a mask for the hole and set the stencil buffer values covered by the mask mesh to a value,
then we render the ground where the stencil buffer values are not equal to the mask stencil
buffer value, and eventually we render the hole sides for a better look. For that, we first have to create a layer
for the ground rendering that we will use later. Now, we have to create two different 3D models:
one for the hole entry, and one for the hole sides. You can perform this either by using a 3D
modelling software such as Blender, or by using Unity tools like the handy ProBuilder. Don’t worry, I will show you both. In Blender, you can actually create 2 models
in one, which is even more convenient. To make a simple through hole, create a cylinder
and set its position to -1 on the z axis. In the Edit mode, select the top and bottom
faces and separate them from the cylinder by pressing P then Selection. Now select the side faces and invert their
normals by doing Mesh -> Normals -> Flip. Now export the object in FBX in Unity, and
that’s all. Regarding ProBuilder, this is also fairly
simple. First, we have to import it from the package
manager, by doing Window -> Package Manager and install it from the Unity Registry. Then, open the ProBuilder Window by doing
Tools -> ProBuilder -> ProBuilder window, and create a new poly shape. Create the shape you want, for instance we
can create a crack. Extrude it in the negative direction, invert
its normals, then select the top and bottom faces by pressing Shift plus selecting them,
and click on detach faces, and invert the normals of the created object again. You should have now 2 different GameObjects,
one for the hole entries and one for the hole sides. You can then parent one of them to the other
to make it more convenient when moving the hole around. That’s all. By the way, it is not necessary at all to
use only a poly shape, you can use for instance the cylinder shape or the sphere shape as
well. Just be creative! Anyways, now we have to create the mask material
for the hole entries. We will use the mask shader we talked about
earlier for that. After recopying or importing the shader, create
a material from it by right clicking -> Create -> Material. Now apply this material to the hole entries
and set its stencil ref value, for example with 1. You shouldn’t see any difference, and this
is because the ground is currently rendered regardless of the stencil buffer values. So, to make sure the ground will not render
where the stencil buffer values are equal to the hole entries stencil ref value, first
we have to apply a layer to the ground, and then we have to create a render feature for
the ground. For that, you have to go in the URP renderer
settings, then to disable the default rendering for the ground layer. For that, in the Filtering section, then by
unchecking the ground layer in the opaque and transparent layer masks. The ground visual should now disappear, and
this is intended since we are not rendering it anymore. Then, we have to create a render feature to
render the ground. Still in the URP renderer settings, click
on Add Renderer Feature, then Render Objects. Call that render feature as you wish and make
sure the event is set to AfterRenderingOpaques, because we want to render the ground after
the hole entries. Now, in the Filters section, make sure the
queue is set to Opaque (unless your ground is using a transparent material), then select
your ground layer in the Layer Mask field. In the Overrides section, check the Stencil
check mark and set its value to the value you set in your mask material. Now, select the Not Equal comparison function. For a better visual look, I would also recommend
disabling the ground shadow casting. Now, you may be wondering how to make the
physics related to the hole. This is basically the same method as the one
used in the interactive shadows tutorial, go check it out if you haven’t see it already. The only thing we have to do here is to create
a triggerable collider for the hole, then create a script that deactivate the ground
collider on trigger. For the collider, we can create a GameObject
that has a mesh collider component which uses the hole mesh. We also need to set it convex and triggerable
to make it work, as well as setting its position correctly. For the script, create a C# script, apply
it to the hole collider GameObject and open it. Okay so this script will be extremely simple. We just have to reference the target collider,
and disable it when on trigger function is called. If you check the tag of the collider that
triggers the shadow collider, don’t forget to add the tag on your object. Congratulations, you now know how to make
a hole using the stencil buffer! This is one of the many implementations of
hole cutting in Unity. If you want me to make a video about other
or more advanced techniques, such as making a hole that follows the curvature of the ground
or that does not require a special layer for the ground, let me know down in the comment
section. Now let’s see how to make an impossible
geometry with the stencil buffer! If you followed the previous part, it will
actually be the same process but instead of rendering an object when the stencil buffer
values are not equal to the ref value, now we will check if the stencil buffer values
are equal to the ref value to render our objects. First, we have to create the box. We can do that by using Blender for example,
but I won’t explain how since it would make the video too long. You can find the 3D model in the description
if you’re interested. Next, create as many quads as we have views
for our impossible geometry and set their position according to the box but in a way
that they are facing toward us. I will go for 3 quads and leave a blank view,
but it’s up to you. Now, we have to create and apply as many mask
materials as we have views. Again, we will use the mask shader we talked
about earlier for that. After recopying or importing the shader, create
as many materials as there are quads by right clicking -> Create -> Material. Now apply these materials to each quad and
set their stencil ref value, for example with 2, 3 and 4.
Make sure to use different materials and ref values for each quad. Now, create and place the objects to be seen
in the different views into the box, and create as many layers as there are views. Assign each layer to each object of each view,
and go in the URP renderer settings. Disable the default rendering of the layers
if that’s not already the case by going in the Filtering section, then by unchecking
the objects layers in the opaque and transparent layer masks. Now, we have to create as many render features
as there are layers. They would have exactly the same parameters
between each other, except for the stencil ref value and the target layer of course. Still in the URP renderer settings, click
on Add Renderer Feature, then Render Objects. Call that render feature as you wish and make
sure the event is set to AfterRenderingOpaques, because we want to render the objects after
the quads. Now, in the Filters section, make sure the
queue is set to Opaque, then select your view layer in the Layer Mask field. In the Overrides section, check the Stencil
check mark and set its value to the value you set in your mask material. So, we want to render our objects only where
the stencil buffer values are equal to the ref value they are associated with, which
means that we can see the objects of a view only through the quad associated to this view. To do that, we set Equal for the comparison
operation. We can then leave “Keep” in the Pass,
Fail and ZFail command since we won’t use them anyways. Repeat this process for each view and voilà,
you have an impossible geometry! Just a quick tip before the next stencil buffer
application: if a layer contains a mix of opaque and transparent objects, you have to
add another render feature to render the missing queue for this layer, which is generally the
transparent one. To do that, you only have to repeat the exact
same process but instead of setting “Opaque” for the queue, you have to set it to “Transparent”,
and also set the event to AfterRenderingTransparents instead of AfterRenderingOpaques. This is an issue someone on the discord server
encountered as well. Now, for the final stencil buffer application,
we will see how to make a magic card. This one is actually a mix between the first
two: the idea is that we first render a mask quad, then the card itself but only where
the stencil buffer values are not equal to the mask stencil ref value, then we render
the card inside but this time only where the stencil buffer values are equal to the mask
stencil ref value. So, to do that, first we have to create the
card model but also what’s inside. Create whatever you want, and once you are
done doing so, create an empty GameObject that will be the parent of the card’s inside. Now create and apply a layer to the card and
another one to the card’s inside GameObject. For the latter, make sure you changed the
children layer as well. In the meantime, we have to create the mask
material. Again, we will use the mask shader for that. You can find it in the description. After recopying or importing the shader, create
a material from it by right clicking -> Create -> Material. Now apply this material to the mask quad and
set its stencil ref value, for example 5. You shouldn’t see any difference, and this
is because the card is currently rendered regardless of the stencil buffer values. We will fix that later no worries. Now, go in the URP renderer settings. Disable the default rendering of the layers
if that’s not already the case by going in the Filtering section, then by unchecking
the card and card inside layers in the opaque and transparent layer masks. The card and its inside should have now disappeared,
and this is intended since we are not rendering these anymore. So, we only have to create two render features:
one for the card and one for its inside. Still in the URP renderer settings, click
on Add Renderer Feature, then Render Objects. Call that render feature as you wish and make
sure the event is set to AfterRenderingOpaques, because we want to render the card after the
quad. Now, in the Filters section, make sure the
queue is set to Opaque, then select the card layer in the Layer Mask field. In the Overrides section, check the Stencil
check mark and set its value to the value you set in your mask material. So, we want to render our card only where
the stencil buffer values are not equal to the ref value. To do that, we set NotEqual for the comparison
operation. We can then leave “Keep” in the Pass,
Fail and ZFail command since we won’t use them anyways. You can repeat this process for the card inside
layer, but this time we want to render it when the stencil buffer values are equal to
the ref value, so we put an Equal in the compare function field instead. Also, if you are rendering transparent objects
in your card inside, like some UI elements for example, make sure to add another render
feature to render them as well. To do that, you only have to create another
render feature and repeat the exact same process that the card inside, but instead of setting
“Opaque” for the queue, you have to set it to “Transparent”, and also set the
event to AfterRenderingTransparents instead of AfterRenderingOpaques. Congratulations! You know how to make a magic card using the
stencil buffer! That was quite a long video. I hope the stencil buffer is clearer for you
now. Please consider liking or subscribing if you
enjoyed. Also, if you have any question or comment,
please put them in the comment section, i read and answer every of them. Also, do not hesitate to join the discord
server of the community if your interested in discussing about game dev in general or
if you’re seeking help for your projects. Everyone is welcomed! Anyways, that’s all for the video, I hope
you enjoyed it and I’ll see you next time! Bye!
