Creating an Outline Effect as a Post-Process in Unity

i don't think it's too outlandish to state that when designing a video game giving feedback to the player's input is super important and whether you're designing a first person adventure game third person action game or a top-down strategy game at some point during development you're going to want to allow your player to interact with things and so giving the player feedback when they try to interact with things is by association really important there are of course many ways we can give the player that feedback but it's usually in the form of some kind of selection outline and potentially paired with a button prompt initially you might think that it's just as simple as adding an outline to your mesh's shaders and so you create the outline effect as part of the rendering pass but if you've ever toyed around with shaders you'll know that outlines can sometimes be kind of messy it's often tricky and restrictive to get mesh-based outlines to look good so let's use the alternative and do it in post hi there i'm matt and welcome to gamedev guide and in this episode we're going to take a look at how we can extend unity's post processing stack to add our own outline post process we're going to write a screen space shader that will draw an outline around anything our camera can see and write a script to allow us to control the parameters of that shader within a post process profile and as a bonus we're also going to write a very simple shader that will highlight our character when they're hidden behind something in our scene let's go ahead and get started i have this scene here with a character in a little environment and my usual go-to camera controller rig in this instance let's say i'm building out a little strategy game and i want to be able to mouse over my player character here and have them highlighted when i do so there are a couple of ways i could add an outline to my character including building an outline shader into the character's mesh themselves but i'm choosing to do this as a post process because not only do i think that that will yield the best looking results but it will also give me a lot more versatility and enable me to add outlines across various objects in my game regardless of their shader properties so the basic premise with how this will work is that when we mouse over an object we'll push it to a separate layer that layer will be rendering to its own camera which will have our outline post process enabled and then it will draw a black image with just the outline we'll then combine that outline with our original image using another post process the first thing we'll do is generate our outline itself and get an outline around our character as i've mentioned we're going to be using the post processing system for this so if it isn't already installed into your project head into the package manager and grab it i've already got a post process on our main camera here just to make the scene a little more vibrant we're actually going to write two different shaders for our outline and two different post process effects for now we'll start with the outline itself and deal with the compositing later in the video post processing scripts are weird because they're actually comprised of two different components a settings class and a renderer class we'll start by creating our outline settings class these settings are essentially everything we'll want exposed to the user in our inspector and we'll control properties within our shader here we'll create some float parameters for settings thickness depth min and depth max and then next we'll create our renderer class using the settings we just created this class requires a render method where we'll tell our post process what it needs to do we don't need to do anything complex in this method essentially we just want to grab our settings send them to our shader and then render the screen image using our shader to do this we'll grab the property sheet class from our render context a property sheet is just a wrapper that automatically makes a material for us to pass into the command buffer so thinking about it in the same way we'd set up any other material we'll set out our properties here like this and then we'll call the bullet full screen triangle method on the command buffer using our property sheet aka our shader which we now need to actually write post processing shaders are a little different from the old shader lab ones you might be used to unity have started to move away from shader lab and are now leaning further into hlsl instead if you're like most of the population and find yourself a little overwhelmed when it comes to shaders it shouldn't be too difficult to get your head around as thankfully hlsl is incredibly well documented across the web including the always excellent documentation pages from microsoft additionally the resources over on unity's post processing github page are actually a great place to start as all of the code for the post processing stack shaders are written in hlsl and are available to view there as well as all the utility and helper functions that they use anyway back to our shader at the top of our shader here we'll start by adding our includes and referencing the standard hlsl library functions within the post-processing package we'll then set up our main camera texture and the camera's depth texture as well as the samples for them and then set up our custom properties we'll create our struct with properties for our uv vertex and screen position and we'll also create a method for calculating the screen position of a given vertex using the projection params property from the standard library we included in our vertex shader we'll populate our struct making sure to flip the image around if the graphics api has its uvs flipped and then we'll start with our fragment shader to build the outline essentially this works a lot like a blur shader we're going to take our pixels and offset them up down left and right by our thickness we'll then sample our depth texture returning the length of its vector and smooth step that between our min and max value settings this will allow us to more closely tune the outline around the depth of our objects and finally we'll return the result okay so now we're ready to add our new post process if we actually add it to our main camera here nothing will happen but the whole screen will go black this is because the entire world is being rendered to our camera so the outline is actually being rendered outside the bounds of our frame here what we want to do then is create a new camera specifically for rendering our outline let's duplicate our main camera this will be our outline camera which we'll want to keep in sync with our main camera so we can write a little script here to update all of its settings to match now let's go ahead and add a new post process and our outline effect then we'll need to make sure that our camera is rendering to its own layer which we'll call outline and if we go in here and set our character onto this layer we now have an outline if we move other objects in the world into this layer as well we can get an outline drawn around them too so moving a game object to this layer causes their outline to be drawn which means that we need to have our character swap to this layer when their mouse over and then back again when they're not i've gone ahead and set up this script on our main camera here it shoots a raycast out into the world and gathers anything it hits with a selectable object script attached when the user mouses over it the on select method of the script is called and simile when the mouse exits the on deselect method of the script is called and so in the select method of the script we can tell the game object in this case our character to swap to this outline layer and similarly in our deselect method we'll move them back to our main layer this swap layer function here is just an extension method i use to more easily move objects between layers just so we can see what we're doing i'm going to disable the outline camera for a second but if we interact with our character and take a look at the inspector they should be swapping between layers when we interact with them so now that we've got our character swapping into the layer and our outline rendering to our camera the final thing we need to do is composite this outline image on top of the original image from our main camera to do that we're actually going to want to render our outline camera's output into a render texture and then mix that render texture into the output of our main camera so let's go into our outline post process and change a few things to get it to render to a texture instead we'll add a static render texture into our script and then tell our effect to create the texture and render to it [Music] now let's write our shader so that we can composite our outline with our main image this one is a pretty straightforward shader we'll simply take in and sample a texture and then combine it in our fragment shader along with a color and finally let's create the post process script for this we'll just add some settings for outlines color and then we'll also need to retrieve the static render texture from our other post process script and then assign it for our shader to use here and now if we add our composite post process to the stack on our main camera we should be able to see both our character and their outline composited together we can change the color in our composite settings and of course play with the thickness in our rendering settings what i really love about doing an outline as a post process like this is the level of versatility it creates for us we now have a really simple setup for drawing clean outlines around objects with relative ease as i'm sure you can imagine there are a lot of avenues you can take this now i could leave it here and end the video however something i've noticed is that if our character is entirely or somewhat obscured by another object we can still get our outline if we select them and i really like this but i'd love to also be able to see a better silhouette of our character when they're obscured by things so real quick let's tie this whole thing together and make some adjustments to our character object to make that possible rather than messing with our character's shader and making variants or anything crazy like that we're actually going to cheat a little bit and have a super simple clone of our character that's always there but only visible when our character is obscured by something so let's duplicate our character's mesh and make it a child object so it will remain in the same place as our character and we'll call this ghost then let's make a new shader which we'll call unlit occlusion now you can do the work to style this however the heck you want but i'm happy enough with a default online shader with a color property the key difference is that we're setting the rendering flags here to only draw when obscured so we'll set our z test to g equal let's create a material for that shader and add it to the parts of our ghost object here as you can see our ghost is intersecting with itself and causing some weird results to fix this we basically have to make another camera that renders our character independently from the world so the main camera will render our world layer only and then our new camera will render our character and the outline layer with the clear facts set to don't clear and the depth above our main camera then our character's main object will either always be on the character layer or the outline layer but our ghost object should always be considered part of the world layer and now when our character is obscured by something in the world we can see more of them through this ghosted silhouette effect it's a super super simple hack so it obviously won't work in all use cases but when it does i think it adds a lot to the visual presentation that was otherwise missing and that's it for today's video this is just a starting point but hopefully this has been useful to you and i'm curious to hear if any other applications you might be thinking of if you've enjoyed the video be sure to hit the thumbs up button and let me know your thoughts in the comments down below if you're new to the channel please do consider hitting that subscribe button as you'll get to know when new videos from me go live if you'd like to see more from me first however please do consider checking out the recommended video on screen now as always thank you very much for watching and i will see you again next time

Channel: Game Dev Guide

Views: 59,271

Keywords: unity, learn, games, gamedev, how to, making games in unity, tutorial, unity tutorial, learning unity, unity3d tutorial, programming, making games

Id: ehyMwVnnnTg

Length: 10min 18sec (618 seconds)

Published: Mon Aug 22 2022