Virtual Production with an iPhone

i've been trying a lot of different things out recently especially revolving around unreal engine and rather than learn these things and then bring them to you on the show we decided we're just going to show you as we go as we play around with these things and learn new things we'll just bring it to the show at that time and today we're breaking out something that we have used a bit on the show before but this time for a different reason and that is a mobile app called cam track ar which we can use to live key and see a virtual background while we shoot on an iphone or an ipad but before we get into that let's just have a quick refresher on what camtrak ar is this is an app where you can record your video with the iphone while collecting the tracking data that you'll use later it uses the ar tech and iphone and the ipad to pull this off and it's incredibly effective you set the floor and anchor points and the app finds the walls the ceiling and it just does the rest then in the end you bring in your data to after effects blender unreal engine or hitfilm and you can use that tracking data to place your assets into the scene so what you would normally have to do in post is done right in camera while you're shooting you can also live key and bring in 3d models which is what we're going to be doing today so we have this green screen set up in our studio and if we turn the app on and move it around to collect data of the room we get a solid track pretty quickly then i can use these controls here to lock my exposure white balance and focus then to pull my live key i'll hit this icon here and the app keys out the dominant color which here of course is green but we can dial that in with these settings here to get it as solid as possible but it does a surprisingly good job right off the bat however this is just for reference the key will not be baked into the footage we will still have to do that later but with it keyed out we get this gradient background and we can see that everything worked out really well and now we can start throwing things into the background like our 3d models or environments or even an image one of my personal faves for instance we just select that and move it in place spin and size then lock it where we want it and then we can add multiple images as well just multiple wonderful images so if we couldn't do a 3d model we could take a few images from our setup and place them in the scene for reference that way and here you can see that it almost looks like rear projection i was far more interested in throwing an image back there than i thought i'd be i actually would have loved to have used this for our chongqi episode since i think the best use of this tool right now is with images i could get a still of what my end background will be and then i can light to that while seeing that background this will be massively helpful for matching environments i plan to comp into later like when we did our episode where we placed ourselves into the original avengers trailer it would have been massively helpful to use this to see that background that we intend to use so that we can match the light as closely as possible but we still can place a 3d environment here so let's take a look at that first thing that we do is build out what we want in unreal engine we talked about that a few weeks ago in our unreal engine episode link to that below but thanks to mega scans we can build out something fairly quickly with really solid assets all of which are free what you can do is so insane so definitely check out the episode if you haven't seen that yet but now we can send this model out and bring it into the phone here as a usdz file to place in the scene and we now have a small version of virtual production style setup right on your phone again this does not bake into the shot this is only for reference but this reference allows me to see my final shot and find what i want to do then when i do it the app is tracking my movements perfectly to use later but now we can bring that footage and data into our system first i'll handle my background so i'll go into unreal open the environment that we used and i'll import the tracking data by creating a new level sequence then in the actions icon i'll go to import and then i'll select the fbx file that camtrak gave us i'll leave everything unchecked except create camera then i'll make sure my frames per second is correct and set my timeline to the correct length there's a few other details that you need to do to get this working correctly but i've added a link to a video from fx home that goes into a good amount of detail on how to get this working properly it's actually how i learned this too so check that out below and show them some love but now i can export this camera move of my environment as a background that i can bring into after effects so we toss in our actor footage do our typical compositing key out the green and we have it and again since we translated that tracking data straight from the unreal project the movement matches really well so this is a really interesting way to test ideas do quick passes for prep and so on and while this is insanely cool as you can see there definitely are downsides right now first is the difficulty of getting the virtual background into the phone to use i ended up reaching out to fx home to help me figure this out i couldn't really find any info of the process online and right now it's not super straightforward so after getting the background that i wanted in unreal i selected all my objects in the scene that i want to export then go to export selected and change my type to obj then i'm going to export all the materials as well then i had to download a beta app from apple called reality converter this is a mac only app too but i'm sure there are others that you could use for windows but inside of reality converter i can bring my obj in by dragging and dropping and as you can see there's no texture so i can go to that folder and drag and drop and place my textures where they need to go here i had to convert the bmp files to jpegs for reality converter to recognize them but once i had all of that in i can export this obj to what's called a usdz file which is what ios uses to read 3d models like what we are using after that we can send that file to our phone and open it in camtrak ar so as you can see it's a bit of a task to get from point a to point b then there's the bigger downside for me which is what you're shooting to not that it's to my iphone or ipad i've shown on the show before how good i think these have gotten and they just keep getting better especially with the new iphone release but within the app you are limited to shoot either 30 or 60 frames per second no option of 24 and no 4k hd only and there's no image stabilization either so to avoid very jittery shots you'll want to use a gimbal or build out some kind of rig to smooth the movement again i asked fx home about this and they let me know that these are all limitations imposed by ar kit and that they are working with apple to improve all of that so while this isn't something i would use for a big project it is an amazing tool to try ideas and expand how you think about creating because for me it's incredibly important to dive into things like unreal camtrak ar and all the other bits of new technology and ideas that are coming out things like camtrak are in their infancy so i'm not just looking at what it can do right now but what it clearly wants to head toward so by jumping into these things early wrapping your head around them how they work what they can do for you you put yourself ahead of the curve and prepare yourself for things that are still yet to come there's always people too that will point out that it should be about the story and that focusing on things like technology is why we get bad films and to me that's just a ridiculous way to think it's the understanding of technology that allows the filmmaker to tell their story at the highest levels think of steven spielberg and janus kaminsky it's their knowledge of technology around lenses and cameras that led to them being able to craft unique and immersive looks and feels to saving private ryan and that really helped tell the story on a whole other level or alfonso quran with gravity san mendes and roger deakins with 1917. if you don't have the depth of knowledge and understanding of the technology that's when it controls you instead of the other way around the idea is to be the master of the tech to get it to tell the story the way you want it to not the other way around so again although all this stuff is in the early days it's all incredibly fun and exciting so do try this out for yourself and see how it might benefit you right now or maybe in the future links to everything we talked about in the notes below and if you are into comics you have to check out astonishing times i mean granted i'm a bit biased because it was created by my friends eris quinones and frank barberry but it's honestly a badass new superhero comic with great art by rory coleman it's a bit of a new spin on the genre in a brand new universe and you can really tell that they put their blood sweat and tears into this thing i'm super proud of the guides and this great accomplishment and again it's out now it was released by comixology and their third issue is releasing on the 28th so if you are interested check out the links below for more of that and episodes of variant where they dive into the making of that story but that is it for today and until next time don't forget to write shoot edit repeat [Music] [Music] you

Channel: Film Riot

Views: 159,697

Keywords: film riot, triune films, ryan connolly, filmmaking, tutorial, how to, adobe, film school, iphone, virtual production, vr, ar, camtrackar, unreal engine, ue, ue5, iphone 13, new iphone, vfx, easy

Id: KYZy-MSHnQo

Length: 9min 7sec (547 seconds)

Published: Fri Sep 24 2021