Outline Stylized Material - part 1 [UE5, valid for UE4]

let's be real who hasn't ever dreamed about being able to render images that are indistinguishable from hand drawings all of us had at least once a wet dream about making a game that looks like an interactive Studio Ghibli movie of some sort I'm pretty sure about that well why not starting to learn how close we can get to that then let's start from the line work and let's see how far we can go with it in this video I'm going to introduce you to some of the basics of image processing while showing you what's the let's say standard practice to create outline serial time rendering in the video that will follow this I'll walk you through my personal take on it which will be a more accurate way to pass through the handmade feeling of the ink so did you ever heard about kernels and convolutions filters in the context of image processing they all refer to the same thing a way to manipulate the values contained in a digital image to change modify or extract some data contained in its pixels what they do in practice is per pixel to look at also the Nate bars multiply them with certain numbers and add all the results together you can imagine it like a little grid with a number in each cell you can Center it over a pixel and then you multiply all the pixels covered by the grid by their respective number then you add all the results and assign the final value to the output image at the position that corresponds the one of the grid you repeat this process for every pixel in the starting image and you obtain a new one with the filter applied that's it building a convolution kernel in a rail is as simple as tedious as mentioned a few moments ago we just need to sample the current pixel multiply it by a number and add it to the final result and we have to do the same on all its neighbors which means doing a nice column of almost identical nodes 8 more times the only difference is that these ones will have an offset to the UBS to offset each one of these nodes to sample a different neighbor pixel we need to multiply the Texel size by the direction of where they lie relatively to our current one and that the result to the UVs with so many nodes it's a bit hard to create a tidy visualization but we can visualize our kernel by positioning all the scaling coefficients like this so how can this thing generate lines everything depends on which numbers we set here in the kernel there is a ton of different well-known kernel values that have different applications and lots of them are built for Edge detection as well as far I'm concerned the simplest one is also the one which Returns the best output the laplacian without going too technical what this filter does is to return a value either positive or negative further from zero the more quickly the pixels inside the kernel change in value which can essentially be considered the definition of the concept of line in this case to understand why I'm saying this try to imagine to draw a scene you're most likely to draw a line along the silhouette of an object and where that same object has maybe a sharp edge or a part that causes self-occlusion which leads to the next thing I'm going to talk about at the moment our filter is generating something that gives the feeling of a line work but it's not that accurate since we are applying it to the final render of the image it's reading as I just thinks like the Shadows too moreover is also picking up some ambient occlusion and other stuff that make the result a bit dirty luckily in a 3D environment we have much more available to us than just the final image you'll see in a moment why but everyone's favorite data to apply this filter on are the depth and normal passes it may seem kinda obvious once you're staring at them but applying the laplacian here will focus The Edge detection on the structured parts of the image a much more accurate approximation of the lines that an artist will draw let's see how they look like by applying the filter to the depth pass we can highlight very well all the Silhouettes of the various objects there are some cases where it falls short though for example you can see how we are not seeing some edges that we may want to and for this reason this filter is usually applied to the normal pass too this time we can clearly see the edges where you are missing before in this case they're colored because the normal pass has three Channel and the filter runs on all of them detecting different edges to merge all of them into one we just have to take the maximum value of every channel result however as you can see this one falls short in other cases where the depth 1 worked perfectly like when two different objects have their faces oriented in the exact same way and that's why you usually want to apply the same filter to both of them at the same time together they can complement each other and you can generate an overall better quality Line work now I know what you might be thinking all this looks good on the Cubes but do you want to ignore all the ugly stuff that's going on around them nope definitely not the reason why I'm shamelessly showing you the filters doing a crappy job is to raise the quality bar for this tutorial see what I often see when other people illustrate this technique is that they only show you it working on some basic shapes and that's it then you go and apply the stuff on a real case scenario and you discover that the matter is not that simple after all and what's the most problematic thing you could apply this filters on for the realistic assets of course their geometry is much more intricate than the normal Supply to them much more noisy that I bet you can see why they represent a very good stress test so let's talk a bit about how to manipulate the filters output to have a better result before doing any weird manipulation we need to understand the range of the values we're working with otherwise it becomes just a random tinkering of the values in the hope to magically find a good setup as in almost everything related to computer Graphics the deal scenario would be having to deal with values in the zero one range in the normal pass case this is very possible to obtain we know that every channel in this bus can range from -1 to 1 which means that we can figure out which are the edge cases for the laplacian filter the highest possible output can be reached only if the central pixel is either 1 or -1 and all the neighbors its negation which leads to the conclusion that the highest absolute value we can possibly see outputted here is 16. so let's divide everything by that number and see what comes out you might think that by dividing the values I just made every Edge less bright which technically is exactly what I did but the point is that now we can read this output as a map of how sure the filter is that every pixel represents an edge where white means definitely an edge black absolutely not an edge and all the Grays in between range from there is a remote chance to kinda shorts on edge this now allows us to fine tune the convolution outcome and Define some threshold to reduce or increase its sensibility then we can append the 1 minus node at the end just to preview how it will look as black and white as far the depth convolution is concerned the matter is a bit more complex than that since a pixel can potentially be at any distance from the camera there's no initial range of values to use for our normalization but we can still do something let's have a look at the filter if we separate the neighbor's weights from the central one we notice something all the neighbors are equally weighted and all added up together which is the first step to calculate the average of a list of values so if we then divide the result by the number of elements eight we get the average depth of the neighbor pixels then if we add this average to the depth value of the middle pixel we get the average distance of the evaluated pixel from its neighbors which finally gives a meaning to the filter output and a hint on how we can work with it all these convoluted explanations simply translates to dividing by eight the final output of the filter now that we know what the values contained in this image are we can work out a way to manipulate them we can remap them to a zero one range of our choice by thresholding them so that we Define how much difference in depth units there has to be from the pixel and its neighbors before starting to consider it an edge and after how much we consider it to be one for sure you can see how now we are losing some important edges where different geometries are closer than our threshold but we are also removing all those ugly lines that the filter was drawing on mesh triangles as with the normal filter we can preview it as an income paper image by adding a one minus node at the end let's combine their scaled values together and let's invert their values too now what if we'd like to have thicker lines one may think about increasing the offset to True three or more pixel to expand the reach of the kernel that wouldn't be an entirely wrong solution let's say its outcome it still holds pretty well but we can see that something weird starts to happen and the higher the offsets the more evident it becomes this happens because by doing this we are creating gaps inside the kernel so the total area considered by it starts to have blind spots inside to remedy that we are forced to just use bigger kernels so if instead of using the current 3x3 kernel we use a 5x5 we will decrease the thickness of our Lines by 2 pixels the saddest part about this is not performances is that increasing the kernel size increases exponentially how tedious the node implementation becomes look at this [Music] foreign [Music] at least it's not that difficult to figure out which ways to use in the bigger kernel just so I told the neighbors to -1 and may call the central weights equal to the number of neighbors we are evaluating and of course this question now must be asked what if we want even thicker lines I bet you already know the answer but now we Implement an even bigger kernel like a 7x7 and yes if a 3X3 meant evaluating 9 pixels and the 5x5 needed 25 a 7x7 will need a whopping 49 evaluations that will mean doing an even taller stock of these nodes plus manually setting all the offsets and weights that's not gonna happen yes I enjoy making these tutorials for you in my spare time but there's a limit to everything even though you know there's one thing that could convince me to do more and that will be money that's why I opened the pattern so if you want to support me in a tangible way you can do it and I don't want to say anything about this in this video because that may become updated so follow the link in the description and give it a look advertising Society is not that if you don't pay me I'm Gonna Leave a result with questions if you wish to implement bigger kernels with nodes I bet you already know very well what to do as far I'm concerned I'll go down another route now way more manageable and I'm efficient I'll convert everything to a single custom node I mean why not show off some hlsl code it's less effort and more importantly you look way more expert than you are to non-programmers huge boost of confidence if only mentioning code made you almost have a stroke and lose faith in humanity don't despair you'll definitely be able to keep following the video I'll explain everything supposing that the viewer has no idea of what I'm talking about the greatest advantage of writing this shadering code is that we don't have to manually tell the material to evaluate every specific pixel we need but through a thing called for Loop we'll be able to just Define the size of the kernel we want and make it automatically go through all the single cells of the grid it defines let's start by defining all the variables we need like the kernel UVs you can see this action has creating an image where root node in the material editor whenever you call that name you're actually reading the value that went into it the difference is that in code we can't just plug in anything we want but we have to explicitly say what kind of data we want to store in it let's set the variable value to this thing which is just the code equivalent of the text chord node as it works inside the post-process material it Returns the viewport to this then we have the textile size which you can get through this line of code and the pixel UVS which will hold the coordinates of the cell currently evaluated in the kernel grid during the for Loop now let's finally Define the thing we're doing all this for the kernel size and let's set it to an odd value we like then let's create the variables that will store the final filters values which will have an initial value of 0 on top of which will be accumulated the outcome of HP pixel evaluation and as last thing we have to calculate the weight of the central pixel which will be the same value used to normalize the filters at the end too so what do we put here as previously mentioned the laplacian filter has a very simple rule to define the weights -1 to all the pixels except the middle one which will be equal to the numbers of all the neighbors evaluated and since the kernel is a square we know that the total number of evaluated pixels will be the kernel size squared and since there's only one Central pixel the total number of neighbors is the squared kernel size -1 stay with me good we're now starting to build the actor logic of the convolution the task now is to write some code that automatically iterates through all the pixels contained in the kernel without knowing in advance how big it is to do this kind of things we use the for Loop a structure that repeats specific lines of code as many times you want until certain conditions are met it looks like this and you can read it like start with this variable that is initially equal to this value until it is less than this other value keep repeating all the code contained between these two brackets after every execution increment the variable value by 1 and check again if the condition is still valid of course depending on what we need to do the variable name its type its starting value the condition and the increment can vary and so what do we have to write in here let's study a bit our case since the structure we have to iterate through is a grade we need two coordinates X and Y to identify the position of a cell but let's start simple and say that we just care about identifying the rule so let's only consider y we know that the kernel has kernel size number of rows and they can be either above or below the middle pixel this splits the kernel in two and makes us understand that the rows can be at any position between negative half kernel size and positive half kernel size we should save it to another variable shouldn't we consider one thing though since the kernel size is always another number the half of it will always not be an integer so we have to make sure to remove the fractional part of the result in this way we don't risk to evaluate a point that is right in between of two different rows so our for Loop becomes something like this Define the coordinate y with an initial value of minus half kernel size until such coordinate value is less than or equal to half kernel size keep executing the code between the brackets after each iteration increase the coordinate value by 1. now that we successfully understood how to iterate through all the rows the step to get how to iterate The Columns should be pretty short once defined the coordinator on one axis we just have to repeat the same logic but on the other one so for the x-coordinate we'll just duplicate the exact same for Loop we just did and change the defined variable from y to X this second for Loop is nested inside the first one so that will check for every row selected by y every column position with X and that's how the Shader is going to automatically check every pixel inside the kernel independently of its sides now it's time to tell it what to do every time it resets a cell of the grid here is where we have to convert to code the sampling of the depth and normal passes along with their waiting and addition to the filter final result actually we have to first separate two different cases we know that we'll have the same weight for all the pixels at accept the central one so we have to tell the Shader to do a different thing in that case to do that we use an if Health statement which is pretty straightforward if this condition is met execute the code between these brackets otherwise execute the code in these other brackets in our case the condition will be that if at least one of the two coordinates is not equal to zero then we are not in the central pixel and what to do in this case well first of all we calculate the current pixel UVS like we did in nodes secondly we use the pixel UVS to sample the two buffers by calling this function twice you can see this line of code as the hlsl equivalent of the scene texture node where the first argument is the UV spin the second one is the identifier of the buffer we want to sample and the last one is this checkbox how can we know at which number each path corresponds just open the list of the buffers in the node and Count Their position starting from 0 so depth path is 1 and normal pass is 8. good now it's worth noticing that these two lines of code as they are now are completely useless or we save their outputs into new variables for later use or we use them directly to update the value of existing ones since what we have to do with each pixel is to negate it and add it to the filter result and nothing more I'd say we should go for the second option we can compress the negation and addition in a single single subtraction that we apply to their respective filter results minus SQL is just a short form to say decrement this variable value by this other value last thing to do to complete these two lines of code is to add what's the equivalent of the component mask node in the material editor now what do we do if we are in the central pixel nothing much different than the other case we just change few things around we still have to accumulate the sampled values to the filters but with a different weight this time though we don't subtract but add the result we don't calculate the pixel UVS but we use the kernel ones directly and we multiply the sampled value by the centered weight we calculated at the beginning of the code with all the weighted samples accumulated on the filters the only thing left to do is to normalize and return the final value since this is an operation that comes after we went through all the pixels and we have to do just once we now place of threads after the for Loop is ended let's divide both filters by the center weight now we have to tell the custom node to Output those values and for that we use the return command the problem is that in every code language functions can only return one single variable usually in this case we have the result of two filters to return one with three channels and one single so we can append the second to the first and make the node return a float4 which will contain the filter run on the normal paths in its RGB values and adapt one in the alpha Channel keep in mind that we have to also specify in the custom node itself which type of value we wanted to return let's set it to float4 Let's finally copy our code to the custom node erase all this madness and run everything to see if it works ah um right there's this dumb thing to account for when sampling G buffers inside the custom node all the functions made specifically to do that don't exist in the compiled version of the Shader if we don't first put a syntax or node into its input pin and we give it a name there you go now it's working and we can freely set the outline Suite as we prefer and that's the end of part 1. if you don't want to go through the pain of implementing this thing by yourself you can follow the link in the description to download the node ready to go hopefully you now have a good enough understanding of how lines are made in real time Graphics to also fine tune them to your personal artistic taste in the following video we'll discuss where and why this approach falls short both visually and in terms of artistic flexibility and then I'll run you through my personal take on the problem see you next time

Channel: Visual Tech Art

Views: 30,552

Keywords: cel shading, post process, linework, ue5, ue4, ue5.1, studio ghibli, ink, stylized, stylized rendering, hand made, convolution, kernel, filter, custom, hlsl, gpu programming, shader, material, anime look, real time rendering, character, overlay

Id: Ptuw9mxekh0

Length: 23min 1sec (1381 seconds)

Published: Mon Dec 19 2022