How To Set Up A Virtual Production Shoot

what does it take to set up a virtual production shoot well I'm going to show you every single step of the way so you can do it yourself I've been working with this studio for the past month to get them set up for virtual production because tomorrow we have their first client so the scene that we're going to be running is here at Bourn Studios on their brand new green curve we're going to have our Unreal Engine scene playing back while the green is keyed out live they're going to be lowering the blind out the end of the scene just like that because it's semi-transparent we should be able to see our unreal scenes straight through the window we are going to be doing our camera tracking with the lightcraft Jet Set app running on the iPhone they are the sponsors of this video but I wouldn't be using it on professional set if it didn't work really well you've assembled this yes um where have all the pieces come from so these little bits like the phone and things were were bought we've put together the walls and the windows I made the framing blender and then brought that in to Unreal and then we've Boolean the shape out so that you can see out into the seat and we just put a glass panel through Jack sent me that scene that we've just had a look at and we're going to then get that and put it in the phone what we need to do first of all just add a basic actor and drag it in so Jack tells me that this middle window right here is the one that we're going to be using for the main scene in on the green screen so what we need to do is place our actor just on the floor now I'm going to call it it's very specific name it's going to be scene loore Bridge the scene Lo underscore is the most important part of the name because that's how the jetset app will know that it's a scene locator when we set our origin when we actually start filming our scene it will match up the position of the real world environment and this scene locator now that we've got this all we want to do is export part of this scene and then send it via autoshot into the phone sounds more complicated than it is let's see how that works in Unreal Engine we have a layers tab so I'm going to select everything in my scene and I am going to drag them from the outliner across into the layers tab so we'll have all of the scene components and the scene locator all in this one layer now this scene is relatively simple so I can do that if you have a very large complicated scene only select the bits that you desperately need in order to compose an image when you're looking through the phone but in this case we can do this and then we can go to our file menu export selected and then I'm going to make sure it goes out to USD and save now I'm in autoshot autoshot is going to find my bridge. USD file and we're going to turn that into usdz file and then send it into the phone under the models tab there's my bridge. USD file all I need to do is Click make usdz Auto shot looks a lot more complicated than it actually is when you know where all the buttons are it's no bother at all so that says finished successfully down the bottom there that means that when I refresh down here we have bridge. usdz and all I need to do now is Click push usdz to Jet Set and then that's going to be sent over the local Wi-Fi straight to the phone this is actually that simple as we've been investing into virtual production we found the hurdles along the way one of the main elements that we were ready to invest in was the live tracking on top of our cameras and we want something a solid solution that's not going to let our clients down but also not let us down in any of our Productions or of our approaches let's set our origin so we're going to have a look around here we go look around on the floor here draw some grids with the phone and then we're just going to tap to set an origin so you can turn and rotate the origin point just by swizzling it around like that so if it's not quite perfect then we can always adjust it after the fact and there's my window there you go so it's currently backwards I'm just going to turn it around 180° there just like that and there's our window I also have the ability to work with it as a green screen so I can set my green color by just tapping on the screen in the Green Key menu and there's our window floating in midair there so now you're looking [Music] out see through you can you yeah yeah amazing so this can now clip right in here but if we need to make any tweaks that should stay nicely in position position nice okay we're ready to do some lens calibration sounds pretty scary but it's really really simple in this setup so I've got the axon Simo Pro mounted on the camera now that's getting a signal from the main cinema camera and piping it through to the iPhone the Jet Set app is going to pick up that video signal and then allow us to both see through the main cine camera lens and the iPhone lens it's going to look at both those images and find common points between the two so that's how it knows that we're tracking for the main cinema camera position and not the position of the phone go to the recording menu then select cine calibration and then select new and you can see under c one it says ready so let's click Start we've got both video feeds coming through the top one is from the phone and the bottom one is from the uh Evo one I think I need to take about 15 images for this to work nicely so let's take a frame here click test frame it can find 55 common points between those two images if I want to keep the frame I can click keep frame but in this case I'm going to click ignore frame let's find something else with maybe a little bit more detail let's have a look at the floor down here down here I'm getting 85 common track points between the two images which is obviously better so now I just need to move around the same area getting lots of different viewpoints with varying degrees of powerlax so it can calculate everything we need it to the next thing we need to do is click save enter a file name so I'm going to name this after the lens so we don't forget what our calibration is called and then we click exit if you look at the reticule on the phone here that's showing you the field of view of this camera which matches what we currently have from the camera view so that's calibrated now so we want to get tracking in it should be coming from here straight over the local IP local Wi-Fi so this is connected to the same Wi-Fi as the computer so let's go to auto shot if we open web page and then go to settings we'll be able to find our IP address and quickly check that it is set to this PC save it we just need to copy this IP address that's what we're going to use in Unreal Engine let's quickly go over to Unreal Engine and enable the lonet 2 plugin this is a free plugin you can get online at loled I'll put a link in the description for that restart the engine and when the engine's relaunched just go to live link Source loan it to live link and paste in the IP address that we copied and you can see straight away we are getting tracking in the plan was to render everything after the shoot using the path Tracer but we still needed a live preview for the client which we'd play back on the TV we used an ultimat to do the live key which we would record for post and also feed the footage across to Unreal via the decklink 8K for the preview so with footage going in we also needed the tracking offset and lens calibration data from earlier for our unreal cine camera we recorded a very short take with Jet Set brought it into autoshoot where it does the full Optical solve then we used a script generated by autoshot to create the C camera in unreal with the correct offset and Distortion there you go so that's our tracking I created a simple translucent material in unreal for our ultimat footage the RGB input is linked to the base color and the key is our opacity mask this was then played back on a plane in front of the cine camera with everything working we just needed to dial in a live link tracking delay to bring the video and tracking into sync we weren't running genlock on this shoot so this was a bit trial and error but worked well enough so I've only just found this out apparently the guys over at lightcraft have developed a way so that you can always have your origin exactly at the same point so let's say you had to close the app and or or turn the phone off and on or change your battery or something when you relaunch you can just use this marker that we placed down on the floor here if I just zoom in on this here and then click origin I can just say a line to marker and it will reposition the axes throughout our shoot if anything goes wrong we don't have to reset our origin in any more than a simple click on the phone the client's going to be here in 5 minutes so we're going to get some behind the scenes while we're working but I'll catch you at the end of the shoot there we go it's been a hell of a journey um and not lie there's been a lot to navigate someone like Josh come on board as a consultant has been um Second and on uh he's been absolutely amazing it's gone well in fact quite quick really quick actually uh we've still got hours left of the day and we might have to cover everything but come on we're going to ask Chris how it went what we're going to do with our spare hours now see to crack through everything yeah it was almost 10 videos done in 6 hours so not bad at all the alternative to that would have been us doing a lot of different fix positions it was one of the things that we were split between before we pressed live on the camera do we still keep it on the tripod and just Embrace that time or do we take it off the tripod and flex and we flexed and it worked out really well fantastic right well I'll see you on the next one

Channel: Joshua M Kerr

Views: 9,069

Keywords: virtual production, virtual production unreal engine, virtual production unreal engine 5, virtual production setup, virtual production studio, greenscreen, green screen virtual production, virtual production green screen, virtual production green screen unreal, virtual production greensc, real time virtual production, real time virtual production unreal engine, real time camera tracking, virtual production camera tracking, lens calibration

Id: 9u_8LHj9rEQ

Length: 9min 42sec (582 seconds)

Published: Mon Jun 03 2024