VFX Reflections: A Compositing Case Study (Blender)

[Music] today we'll look at how to make cg objects appear correctly in any reflective surfaces that are in our footage i'll share three techniques i tested out we'll cover everything from onset filming tips blender scene setup and rendering and then of course compositing let's get started [Music] so you can follow along i've put together this reflection research kit which includes all the assets for this violin shot from the raw footage a blender file for the 3d scene and another blender file for the compositing all done in aces i also threw in a bonus compositing file for the free version of nuke along with a video breaking it down at a beginner level this kit is five dollars on gumroad or free for any patrons okay with that out of the way let's start our reflection research by looking at the real world now every solid surface in the real world is either rough reflective or usually a mix of both one of the main jobs of a vfx artist is to make cg elements interact with these surfaces in a realistic way let's take a very rough surface like concrete if we put an object on this surface the object creates shadows which simply darken the surface so if a vfx artist wants to add a cg object into this footage the first step is recreating this scene in 3d here in blender i created a concrete material and i've matched the lighting of our footage now with the cg object added we want to capture the shadow the object creates we can extract or catch this light information with something called a shadow catcher or shadow pass in blender 3.0 the shadow pass looks something like this we can take this shadow pass and multiply the pixel values by the pixels of our footage then of course the cg object is put on top not too tricky right on the other hand with a reflective surface to add a cg object we can't get by with simply darkening the footage if we put a real object on this surface since the surface is partially rough the object still creates a shadow like before then since the surface is partially reflective our object is blocking out the background reflection and replacing it with its own we can see these changes more precisely if we recreate this scene in 3d with a 3d render engine we have the power to split up our image into the different types of light that make it up this is like breaking down a cake into the individual ingredients in 3d these ingredients are called light passes this first light pass shows us how the light bounces off the partially rough nature of our surface this is the diffuse light pass diffuse meaning rough this second light pass shows us how the light bounces off the partially reflective nature of our surface this is the glossy light pass glossy meaning reflective if we add these two ingredients together we get the final image let's go back and study the effect our cg object has on these light passes here in the diffuse pass we see the shadows created by the object and here in the glossy pass we see how the object's reflection is completely covering the old reflection on the surface now since we have to cover the reflection in the footage with those of the cg object we also have to then recreate what the rough diffused part of the surface would look like in other words rough surface easy reflective surface not so easy and thus we arrive at our first method of compositing vfx reflections the surface replacement method here's a great example of this method from ilm in the original plate there's this very reflective surface that they want to add the cg characters onto so they just cut out the parts of the real reflection they want to keep and otherwise the reflective surface is being fully replaced with a cg surface this is a great method i just didn't use it for any of my test shots so let's get into the methods i did explore i'm going to first give a broad overview of all the methods i tried then afterwards we'll get into the technical details so the next method is what i call the reflection adding method this approach works well when you have a reflective surface with a dark background behind it as always we'll recreate the scene in 3d with the proper materials and lighting with the clay pots added to our scene how do we capture just their reflection well remember how we can break up our render into the diffuse and glossy light passes we can actually further break these passes up into what are called the direct and indirect light passes and we can see in the glossy indirect light pass we have the reflection of the mug similarly in our scene the glossy indirect light pass will capture the reflection of the clay pots for more on what these labels direct and indirect mean check out my previous video from our 3d program we'll export the main objects a shadow pass and the glossy indirect light pass of the surface now for compositing we'll start by taking the glossy indirect light pass and use an add operation onto our background then we'll take our shadow pass and multiply it by our background finally we have the clay pots which we'll simply put over the background again this only really works when the reflective surface has a dark background on a bright background if we try this same technique that is we use an add operation for the glossy indirect light pass it doesn't look right instead we want to fully cover up or replace the background reflection this brings us to our third method the reflection replacement method here you can see i've set up this scene in 3d from our 3d program we'll export our main object and a shadow pass just like before then instead of just exporting the glossy indirect light pass we'll export our entire reflective cg surface finally we'll create this special element called a reflection matte this is a black and white render that shows us where the reflection of our mug appears on the surface remember i'll show you how to make this later in the video next over into compositing we will use this reflection mat to cut out the mug's reflection from the render of the reflective surface then we simply put this over the background next we need to multiply our shadow pass on top we already see the shadows in the reflection that we added so we'll only multiply the shadows everywhere outside the reflection finally we will put the mug over the footage now this approach is decent but not ideal since the shadows are split between those inside and outside of the reflection mat this is one of the things improved on in this next method the hybrid reflection replacement method this method requires some extra work when you're actually filming your footage what you want to do is capture a reference of the reflective surface with no background reflection we'll call this the no reflection image one way to create this is to use a large black blanket or black flag and cover the background light source you don't want to cover too much light only enough to cut out the bright reflection where your cg object will be a similar way to capture the surface without the reflection is to shoot a top-down photo this may or may not work depending on the lighting of your location but it worked well for this shot when we create this no reflection image we're almost acting like a 3d render engine but in the real world we're blocking out the glossy light so that we're just left with the diffuse light information with our scene recreated in 3d we'll again render out the violin and the shadow pass then we'll only export the glossy indirect reflection of the violin next we'll export our no reflection image aligned and tracked into the scene and finally we have the reflection mat of our violin then in compositing we're going to first use our reflection mat to cut out the no reflection image we'll place this over our footage next we can add the glossy indirect pass on top then we'll multiply the shadows and put the violin on top now there is one more approach i tested out for doing water reflections it differs a bit from the rest so i decided to leave it out of this tutorial but i've provided the blender and nuke files for this shot as well as a video breaking them down for any patrons linked below but now let's get into the technical details now i'm only going to break down the violin shot since it's the most complex and the other ones use similar techniques so let's start by looking at how i set this scene up in blender our scene is pretty basic we have the violin here in its own collection we also have a floor plane with a material i created with the top down reference photo i took at the location i tried to roughly match the glossiness and roughness of the floor finally our scene is being lit completely by this hdri now it's time to create some different render layers our first layer will be for our violin we'll place our floor in a collection and set the collection to indirect we can make this setting easily accessible like so and then we can just click here to make the collection visibility indirect next we'll render out the reflection of the violin so let's create a new render layer and now set the violin collection visibility to indirect it's in this layer where we'll enable the glossy indirect light pass and remember we can always preview any render pass in our viewport by using this drop down menu to make sure we've enabled the right pass next we'll create our shadow render layer we want the floor to catch the shadows so let's duplicate the floor collection we'll hide the original floor collection then we'll go into the object properties and check shadowcatcher and let's set the violin collection to indirect in the render passes tab let's enable the catcher pass this is a new render pass in blender 3.0 and let me take a quick moment to describe how it works with this render pass enabled we'll see in our viewport our shadow catcher is visible just like the old shadow catcher but when we render this image our shadowcatcher disappears to see it again we have to change the view to the shadowcatcher pass we just created now this looks a bit odd what's happening is that not only are we catching the shadows but we're also catching the reflections so if we can just catch reflections is this the answer to all our problems well not quite when we multiply this pass over our cg floor the shadows and reflections look exactly as they should but change this background to anything even slightly different and the reflections really fall apart so how do we use the shadowcatcher pass without including the reflections the workaround i found was to duplicate the floor material and set the roughness to 100 now we have just the shadows phew next we want to render out a tracked version of our no reflection image so we'll create a new render layer and turn off all the other collections for this shot i took this top down photo of the floor and you can see i have four tracking markers in this photo which correspond to four tracking markers in the footage what i can do is create a plane with this photo as an emission texture then in edit mode i'll create edge loops to intersect the front marker and one of the back three as well let's select this vertex snap our 3d cursor to this location and then in object mode we'll set our plane's origin to the 3d cursor location now we can align that origin with our front most tracking marker in the footage and then easily scale and rotate the plane to align the other three markers we'll then put this plane in its own collection alright we are on to our final render layer our reflection mat first i need to apply a new material to our violin but in order to preserve our original materials i will duplicate the violin collection and then parent this duplicate to our original [Music] then we'll apply a white emission material to one part of the new violin we can then select the rest of the parts and go to object link materials then we'll duplicate the floor collection we'll also duplicate the floor material and set the base color for this material to black then we'll need to turn off any lights in our scene i don't have any lights but i do need to disconnect my hdri in the world shader we'll set our violin collection to indirect and there we have our custom reflection matte now we need to go back through the previous render layers to make sure we're not seeing any of the new collections we've been adding when it comes time to render everything out we need to reset our hdri and scene lights back to normal we will disable this reflection matte layer and render our other layers first as usual we will render everything out as a multi-layer exr file then after we switch everything back to the reflection matte settings we have to render this reflection matte separately now let's talk about the compositing for this shot in breaking down this shot you should be able to understand how the other test shots were composited i actually composited all shots for this video in nuke again which i break down in the research reflection kit but now i'm going to show you how to do it in blender to show you just how straightforward the compositing really is i'll start with a fresh blender file and go into the compositing workspace we'll hide this panel and check use nodes let's hit shift a and add a viewer node [Music] then we'll add in all the image sequence elements we need [Music] by the way i'm working completely in aces for this entire tutorial so make sure your blender is set up for aces all our elements should automatically be set in the asuscg color space which is what we want let's quickly create a new window here and set this to the image editor and select the viewer node and now if we ctrl shift and click on a node in our graph we can view our composite in this window i prefer this to having the composite behind our node graph okay first we want to use our reflection mat to cut out our no reflection image so let's add a set alpha node and connect our no reflection image as the image and our reflection mat as the alpha if we view the output we see it's semi-transparent this is because our reflection mat isn't quite bright enough if we right-click the viewer we can preview the pixel value and see that it's around 0.3 and we want to make it closer to 1. so let's add an rgb curves node and drag our output like so brightening our map we don't want to go too far or it will obviously look pretty bad now we see our values are above 1. to cut them off right at 1 we'll add a math node set to add we don't want to add any value but we do want to click the clamp checkbox this will cut off all our values at 1. now we will see our no reflection image has a nice solid alpha we'll now put this over our footage with a simple alpha over node next step adding in those glossy indirect reflections for these we'll simply add a color mix node set to add to add these reflections on top of our composite [Music] next we want to multiply our shadow pass on top so we'll add a color mix node set to multiply and we'll multiply our shadows on top of our composite finally we'll put our violin on top we'll add an alpha over node with the violin in the bottom socket now there's actually one more secret element i added that i haven't mentioned yet that is this subtle layer of surface scratches i essentially just took this scratch texture and exported a tracked version like the no reflection image we want this to show only where the reflection shows so we'll use another set alpha node and use the reflection mat again then we'll add a color mix node right before our shadows are multiplied over the image we'll set this node to screen hook up our surface texture and then lower the factor to a nice level now we have a nice amount of surface imperfection next let's adjust the levels of our cg elements to match the footage we can see the highlights on our floor are around .85 while the highlights on our violin are around 0.7 so let's add an exposure node to boost the overall brightness of our violin now things are looking pretty good let's come into our render properties tab and all the way down under color management we can crank up this exposure to increase the exposure of our preview it's always a good idea to check your composite at a higher exposure as it reveals some issues you wouldn't otherwise see first off our violin is looking a bit too saturated so let's add a saturation node and take down the saturation a bit next if we sample the black levels of our footage they're around .006 and our violin is at about half that so let's add an rgb curves node put a point on the lower left and raise this output black value ever so slightly now you'll see this changes our entire image not just our violin to prevent this we'll put a mix node before all of our color correction we'll set the node to divide and divide the violin image by the violin alpha then we can hit shift command or control and d to duplicate this node and put it after our color correction we'll change this one to multiply our image by our alpha so in this first node i'm essentially telling my rgb values to forget about the alpha channel then here i'm reapplying the alpha channel to the rgb values this is a very common and smart thing to do before and after color correcting a cg element it helps avoid introducing color issues with any semi-transparent edges there's actually a blender node just for this called the alpha convert node here we'll set it to straight and we can see it's doing the exact same thing then after the color correction we can replace our multiply node with the alpha convert set to pre-multiply and then we can delete our old color mix nodes okay now our violin is looking pretty good color-wise let's check our shadow pass really quickly remember our dark values in the footage are around .006 whereas our shadows here are nearly zero so let's add an rgb curves node and start to raise the black values there that's a lot better and we don't need to use the alpha convert nodes here because our shadow pass doesn't have any alpha issues okay all our colors are matching pretty well now we'll just add a bit of blur to our different cg elements so they're not quite so sharp and that completes our composite you could add some final color grading in blender but i did my coloring in davinci resolve so i'll simply hook everything into the composite node and set an output type of open exr delete everything in your scene except for the camera and hit render animation i then brought this exr sequence into resolve for the final grade [Music] i've released a tutorial for how i did the color grading on my patreon along with this patrons will get the gumroad reflection research kit and the water reflection breakdown video and files remember all nuke files are for the free version of nuke which is great if you're looking to step up your compositing game and thank you so much to all the current patrons i also want to thank gabe who helped with some early research and development curtis hope for actually meeting with me in person thanks for the help you're welcome and julius elay for his nice blog post and personal help with my research so as you can see there isn't really one right way to composite reflections and i'm sure there are many improvements on the workflows i've shared but my hope is that this can serve as a starting point for you and your work so thank you for watching and i'll see you in the next tutorial [Music]

Channel: InLightVFX

Views: 32,818

Keywords:

Id: PVSVVUChRP0

Length: 18min 54sec (1134 seconds)

Published: Mon Jun 13 2022