Why You Should Be Using Stencil / Render Layers - Unreal Engine 4.26

hey everyone welcome back it's so nice to see your faces in the middle of this pandemic today we're going to be talking about a really exciting feature called stencil layers now if you have a bit of a vfx background and you've used render layers before then you know exactly what i'm talking about so render layers and stencil layers are more or less the same thing there's a few key differences here but i'm going to go over that in this video so the reason this is so exciting is because contrary to crypto mat or object id stencil layers actually support depth of field and motion blur right out of the box so you get a perfect alpha mask right in the exr file and and you get way more control over how you want to split your scene up contrary to kryptomat so without further ado let's just get started all right so now that we're in unreal i've got a very simple setup here i'm using the unreal apartment scene very simple camera nothing too fancy so the purpose of this lesson is to jump into render layers and how we would set these up okay so in this case i'm going to want to separate both the foreground and the background so everything that's on the table here is going to be one render layer and everything that's in the background will be its own layer as well so before we get started i'm going to assume that you know how to use the movie render queue if you don't know how i suggest you go watch this video up here right above because i don't want you to get lost okay so i'm going to be going a little bit fast if you've never used movie render queue before so go check that video out so you know what i'm talking about before we get started so moving on let's go to the window tab up here go to cinematics movie render queue so i've opened a movie render queue i've already added my sequence i'm going to go to the settings tab here now what you want to do okay in this start setting up your render layers is you're going to go to the deferred rendering tab click on this and in there's going to be a stencil clip layers tab here okay you're going to want to click on the stencil layers and click on the little plus okay that's adds element okay so it's going to say none click on none and you can do browse layers this brings up the layers tab up in the top right hand corner here once that's there you're gonna see there's nothing there so you're gonna right click and create empty layer and i'm gonna call this layer foreground right so let's hit accept and close the movie render queue now i'm going to select everything in my scene that i want to be part of the foreground okay so i'm going to collect these plants like these like this the books and this i'm going to go up to the foreground right click click add selected actors to selected layers click on this now just to make sure that everything works you can click the little toggle visibility here click a little i and you'll see oh whoops see i forgot the one of these plants here here and here i'm going to add these to myself like the layer and you know just for good measure i'm going to add this chair as well to my foreground layer all right so now we've got my whole background here and we've got the foreground layer here okay so what we need to do next let's go back to the movie render queue go to cinematics movie render queue back to our settings deferred rendering and in the none layer here click on foreground the layer that we just created okay now it's also very important to have accumulator includes alpha checked right here you may need to adjust one in your project setting it's going to tell you a message is going to pop up saying like hey this project that needs to be enabled make sure we go ahead and do that now you may think that you need to go ahead and add another render layer for your background fortunately you don't need to do this so if you see here it says add default layer and you check this what this does is everything that's not in a layer it's going to be rendered as its own thing as its own layer so let's say for example in your background you have hundreds of thousands of other objects you don't need to select all of those and add those to a new background layer okay so just checking this it understands that if something is not in a layer it's going to be part of its own default layer so very handy tip to know so before we hit accept there's one more thing we need to add okay so we need to go to the settings tab up here and we need to add color output click on this now color output here it may be minimized at first click on this and you want to click on disable tone curve okay so the reason for this is epic states that if you don't disable the tone curve you're going to have some black haloing or some black halos happening around the render layers in your mass and your alpha is not going to come out right okay so it's very important to disable the tone curve this means that your image will be rendered in linear srgb space which is great for compositing you should be compositing in linear space to begin with so once that's done okay i've already set up my console variables for sub sampling i've got my anti-aliasings temporal sample count set to eight override anti-aliasing set to none exr 16-bit it has to be exr it's not going to work if you just use png or jpeg or whatever so make sure esr sequence is checked with multi-layer and the output tab make sure that you have your output path set correctly and once again okay going back to the deferred rendering tab here make sure the accumulator includes alpha super important otherwise your masks are not going to work so a few caveats that you should know about here so using stencil clip layers drastically increases your render times okay so for each layer for each extra layer that you have it's going to increase your render times by 100 because it needs to render the entire scene for each individual layer so in this case we've got the foreground and we've got the default layer so it's going to take already by default two times longer to render on top of which it takes even longer because you're including the alpha this can increase it says here this adds 30 percent to cost to the accumulation so you should not enable it unless necessary okay so it's going to take 30 longer because of the alpha and it's going to take 200 percent longer because you've got two render layers all right so now that all of this is set up we can click the accept button we're going to render local and we're going to send this over to nuke i'll be using nuke but you can probably use any compositing package of your choice it can be fusion or after effects whatever so here are a few things that you are really good to know these are the best practices when it comes to using the stencil layers you should be assigning your desired actors to your layers you should be adding the layers to the stencil layers in the deferred renderer tab so selecting default layer checkbox will contain everything that is not in the specified layer each layer increases render time by 100 you need to disable the tone curve in the color output tab because layers will not add together correctly in nuke if the tone curve is applied a black halo will appear around the edges and generally will not look very good at all you also should ensure that you disable auto exposure in the post process volume you should also disable screen percentage or set it to 100 because using a screen percentage resizing does not support passing the alpha channel through to your renders so some of the pros of using the stencil layer include the scene remaining visually stable between many many layers okay the main advantage here is that the stencil layer is a much better alternative to object ids or kryptomat because it supports depth of field motion blur etc cryptomat or object id do not support these things and you need to render separate depth passive motion vector paths and assemble this in nuke afterwards it's a lot of trouble and if you don't have nuke it's a real pain to work with another advantage is the alpha channel contains a mask of pixels that are actually written to a given layer and you have much better control over what layers get written compared to object ids which have zero control you can assign whatever actor or shape or object that you want to any given layer whereas the object ids you have no control over what gets assigned to what now the cons however is that the render time increases with each additional layer and if not possible to get an exact match to what you see in the engine there is no perfect way to convert back to unreal's look when you're working in post-production in nuke or any other compositor this has been stated by epic time and time again if it comes from them i'm going to take their word for it okay so we're in nuke now and i've already brought in my new render here that i just got from the movie reniq so the first thing you'll notice is the colors might seem off and this is totally normal because we disabled the tone curve which means we're getting a linear srgb image okay so and you have a tendency of linearizing everything you bring into it so don't be surprised this the render the colors look a bit different that's fine the first thing you want to do we got our exr file here so we want to add two shuffle nodes next okay we need one shuffle note per render layer so i'm going to go shuffle bring one here and shuffle again bring to the next one here so i'll get double click my shuffle layer here now i'm in my input layer i'm going to select on rgb i'm going to go to final image default layer and here i'm going to click on final image foreground because the foreground is the render layer that we created that's how i named it so it'll be named whatever you named it in unreal now this click so you'll see right away we've got render layer a and render layer b so we got the foreground and we've got the background we got nicely separated elements here so if i could click the alpha we got a perfect alpha for both of our renders so this works fantastically we can see right away this is a great way of combining things together now the next thing you want to do is we're going to create a merge node okay so i'm going to do foreground over background and you'll see right now this looks pretty good but there's a major thing that you should know about unreal states that you need to be merging with plus and not over so right now we said this by default it's set to over and you'll see in theory the results should look the same for under beauty pass so here we've got a beauty path and here we've got our merge node so you'll see you've got this we've got this odd haloing over the basically over everything around everything this is because we're using over and not plus so epic clearly states you need to use plus so we're going to go ahead on my merge node i'm going to click on this i'm going to set this to plus okay and right away now we're getting a better much more similar result now it's not perfect it's not exactly the same toggling between the two are beauty pass and our merged nodes this is normal epic themselves has clearly stated in their live stream on twitch that the results are never going to be a perfect match okay so i'm putting a link to this twitch stream in the description below this is where epic clearly tells us that things are never going to be a perfect match okay so don't take it from me take it from epic as always when it comes to compositing and post-production with unreal footage it's always a compromise it's it's not there yet um there's it's unreal still has a long way to go but for you know proper compositing work but they said themselves sorry epic has said themselves that you know it's not perfect but it's good enough so as you can see comparing again it is in fact you know not too bad okay and i'm pretty sure that an advanced compositor who's not me i and then by all means not a professional compositor okay i barely know my way around nuke but someone who's very good in nuke or any you know a full-time compositor will probably know how to work with these alphas a lot better than i do so so one of the major advantages of using the merge node here is the following so i'm gonna just for simplicity thing i'm gonna set this merge node back to over okay if you can deal with the black haloing around the edges which to be honest if i didn't know where to look i probably wouldn't notice if i hadn't if i didn't have the comparison with the original beauty pass it would probably be okay and like i said earlier an advanced compositor is probably going to know how to work around with this because i'm a dumbass when it comes to compositing work i'm not very good i know the basics but i'm not going to pretend that i know what i'm talking about when it comes to compositing so we're going to just first just for fun here i'm going to go add some text so i've got some text here with hello world written and i'm going to go ahead and merge this okay with our background okay so i got this overlaid on top of our background i'm going to uh move it around a bit i'm going to move it like right here let's say and i'm going to merge this over this and now you'll see this is the one the major advantage you can go ahead and like sandwich some text or other elements between both of your render layers and you get a pretty nice alpha so you can kind of see here the text you get the perfect alpha cutout between the text so if i want to add you know a person in there between the table and the back of the living room i could do that so this is just one of many reasons why i prefer using stencil layers over kryptomat because you have full control over how you can split your scene up so once again just moving this around and it's it's pretty nice to just have this really nice alpha especially you know it kind of fades in nicely it's pretty good it's not bad right so the stencil layers feature here really start bringing unreal one step closer to vfx territory some of the main things you should know when recombining these layers in nuke is that you should color correct each individual layer by using the alpha channel as a mask for each color grade node and when you're merging the layers you really need to merge with plus or also known as add merging will plus will provide a much more accurate result but the typical over a over b can be used in some cases if you know the result is acceptable it is totally your call you have to use your creative judgment on this so if i wanted to go ahead and grade the foreground separately i'm going to add another shuffle node okay and i'm going to set this here set this to foreground and i'm going to go ahead and add a grade node right here for my foreground so in the mask i need to make sure i connect my grade to the mask of my foreground layer here and in mask here i'm going to select final image foreground alpha now in my gray node i can go ahead and add a tint you know make this blue i'll make my whole foreground layer blue or you know orange whatever but we get a nice masking right so this is uh how you would grade it because we're using a plus node and not over or it's not a correct merge it's not a over b it's a plus b um you need to use masks on any of your grade notes okay that's just something you should probably keep in mind so using render layers is almost more powerful than using unreal's new crypto mat because it supports that the field that supports motion blur all those post process effects will be rendered correctly in your render layer passes okay well whereas kryptomat or object ids with the movie render queue will not support depth of field they won't support motion blur or any other post process effects like that so you need to actually render a death path you need to render motion vectors and to get the kryptomat masks to deform correctly having rental layers is also super beneficial because it allows you to actually choose what kind of id you want so you can choose i want if i wanted just to plant just the plants to be in one layer if i wanted a flower pot to be in the render layer you can do that okay so i'm going to delete this because it's really ugly but this is how you would go ahead and setting up your render layers in unreal engine this is a very powerful new tool super handy it's not perfect epic themselves has clearly stated that it's not perfect you're never going to get a perfect one-to-one match but you know what it's it's a great start it's something that i can work with so guys once again i hope you've learned a little something i hope this has helped you out leave a comment down below if you have any questions whatsoever don't forget to like and subscribe and i'll see you guys next week in the next video

Channel: William Faucher

Views: 26,408

Keywords: Unreal Engine, UE4, Ue4 4.26, Unreal 4.26, Unreal Engine 4.26, unreal cinematics, render layers, stencil layers, Stencil Layers, realtime cinematics, unreal vfx, unreal cgi, cgi, 3d, movie render queue, Epic Games, vfx, stencil clip layers, stencil clip layer, aov, unreal

Id: QUyznLlnchA

Length: 16min 57sec (1017 seconds)

Published: Mon Jan 25 2021