Unreal Engine 5.4 Green Screen Workflow (Blender to Unreal)

all right let's learn how to turn something like this into something like this using the power of blender's camera tracking Tools in unreal engines USD pipeline now I couldn't find a single video or documentation thread on this so naturally I spent the past several months developing what I consider to be the perfect workflow that I use on professional VFX work all the time last month I went to Korea to VFX supervise for a K-pop video XXL by Young posi in the entire video was shot on a green screen so I figured it would be the perfect opportunity to choose a complex shot to demonstrate this on load your footage into blender it helps to use image sequences for frame Precision set your scene frames and hit prefetch this is not a camera tracking tutorial but I just want to highlight that I didn't set my frame rate literally doesn't matter because down the road we're going to be snapping our key frames to the frame rate and Unreal Engine sequencer if you're going to have specific interactive objects in your scene it definitely helps to have tracking markers on reference geometry like these apple boxes we set up the cool part is this will give us visual points of reference in 3D space so we know where in the world these objects will be a lot of these tracking markers obstructed by the boxes so you can see that I discontinue the track until they pop out the other side it also helps to have a couple tracking markers on the floor if you can that way blender can do its best to put the floor where it needs to be now a bajillion and a half tracking markers later it's time to solve our camera motion but first it's a good idea to reference online sensor guides the footage was captured with the red rptor VV we shot in 6K with spherical lens so according to this chart 28.8 mm is our sensor with put that into blender oh and it's also a good idea to have a dit onset notating which lenses are used for each shot that way you can input the exact focal length but honestly who cares we're going to let blender refine that anyways focal lengths are not always accurately represented there's a lot of like weird Optical nuances that can cause discrepancies here let's let blender do the work and choose our key frames and also refine the lens Distortion I'm going to set the lens Distortion model to Brown I don't really know what it means but it lets us refine the tangental Distortion and honestly it seems to help in most cases so I don't ask questions let's solve our camera in 4.92 that sucks so I'm going to spread the Reconstruction initialization key frame so the camera solver has a wider range for determining the motion you can make this as wide as you want the wider the better as long as there at least eight common tracking markers let's solve again and uh 2.52 that's better also not the best but hey if you set up your tracking scene it looks like the movement is sticking well then there's really no need to keep going what can I say sometimes your eyes are a better judge than the solve error tell blender where the floor is start blocking out your scene in this case I'm starting with the television fiddling around with the orientation to make it feel right notice how I'm using the tracking markers in 3D space as a reference for its position then I block out roughly where the window is going to be so I can start to get a better feel for it now this is where the magic happens go to the constraints tab under your camera and click constraint to F curve this will convert your camera path into key frames that are specific to each frame in your footage go ahead and select your ground your reference messes and your camera it's a good idea to copy the camera before doing this in case you have to go back export your scene as a USD file check selected objects and make sure you tick animation super important checking animation will create a level sequence for our camera when we bring it into un rail engine hop over to un rail engine and click window virtual production and USD stage click open then load in the USD file and crash classic we'll see a new folder in our content drawer conveniently named after our USD file click the level sequences folder and open that on up select the root Prim in the inspector which in this case is called TV track pull out since that's what I Nam the USD file change the location on the route to line it up with our scene make sure you only change the location of the route here's a quick tip if you import your USD stage and your scene is way off from where you want it to be rather than manually dragging it to where it needs to be instead here's what you're going to do add an actor or mesh or whatever into your scene copy the location and paste it into the root Prim now the good thing about exporting the ground plane from blender is that we're able to line that up with our ground in 3D space if the height of our camera in relation to the ground feels a bit off simply scale the root Prim till it doesn't all right once we have a place that feels right I hop over to Cargo by kit bash 3D and I begin looking for a corner store type building all I really care about is that it has a storefront window for the TV then I just do some simple modifications to the geometry and placement of the building I'm viewing the camera so I can make sure everything lines up like it should oh here's a cool thing I found out recently that I wish I knew like a year ago go into your content drawer and search for media bundle you might have to turn on the composite plane add-on for this to work to be honest I I literally don't even know anyways you should find a media bundle blueprint which you can add as a component to your cic cam and your USD stage this will allow you to input a custom media player onto your plane that sticks to our camera as it moves all you get got to do is open up the blueprint click on the plane component and drop in your material that contains your texture media thingy if you don't know how to do this look it up image media source to Media Player create a texture media thing just look it up now in order to see this playback as it should you're going to need to add a media track and sequencer which I'll show you how to do shortly but now we're able to watch frame by frame this allows us to add a front plate to compare to our 3D match move and we can use that to further reference placement in our scene great for lighting continuity and it looks like the footage makes perfect sense in our scene or at least almost there's one big caveat how do you even say that word and that's that our frame rates don't match and their key frames are misaligned because that's just what happens when you import these USD files I don't make the rules but lucky for you there's an easy solution I worked out first you're going to want to change your frame rate to the original frame rate of your footage then select all the linear key frames that were imported click them and hit snap to frames you might have to realign them after you do this it's essentially the same as quantizing notes in like FL Studio or whatever if you don't do that step you're going to have a really tough time talking from a place of self-sabotage here but you can see now when we play back our footage it's now perfectly aligned frame to frame and all microv vibrations are accounted for I began dropping in the large Electronics into the storefront since that's what the director requested oh yeah and another cool thing if you want to add to the beginning or end of your match move you can just slide over the key frames and add new ones with smoother interpolation and customize any part of the shot that doesn't have the front plate in it you'll just have to remember what frame the tracking actually starts on for proper alignment and compositing final thing I did was add a new image media source and a media player linked to that source with a media texture sample then we create a material from that and apply it to a plane fix the orientation and scale till it fits your object in unreal because unreal is not going to get that right the first try I made this material a missive with a multiply node to better control the brightness and look at that it's starting to look like a screen but you're probably wondering like how are we going to get this to line up with our shot in terms of playback it's simple go to your level sequencer and click add track make it a media track click the plus sign and select our image media source for the video right right click the media track in the outliner and click the check boox that says use external media player we're going to select the media player that corresponds to our image media Source now we have a track that we can actually slide to get the timing right before we render and the cool part is it'll always play back at the same point in the timeline I added some actor core models because digital doubles always add a little life to the scene and construction workers are a Vibe and just one last Quick unreal tip if you ever see some of your textures tweaking especially like the kit bash assets simply identify the textures in your material open them up and search for never stream this will prevent undesirable myip levels in your textures it's not like we're optimizing for a video game here so real-time performance doesn't really matter I also added some lights into the storefront to help better emulate the practicals we had on set there were a couple people in the grip crew with light tubes throwing cooler tones onto the girls faces that ended up playing pretty well with our TV in the digital environment this is an extremely effective way to get your match moves into Unreal Engine from blender or really any tracking software that exports USD now just do the whole process like 50 more times and you have a video you're proud of I've tweaked and refined this process by learning the hard way and I really hope I save some people a headache by sharing the workflow if you don't miss any steps you should end up with a pretty solid match move

Channel: Andrew DiLeo

Views: 5,186

Keywords: unreal engine, XXL, Young Posse, Behind the Scenes

Id: 4D9HKxxsDSs

Length: 7min 39sec (459 seconds)

Published: Wed Mar 20 2024