Using an iPhone as a tracker for Virtual Production

so I recently got really interested in virtual production if you're not familiar with the term it deals with anything where you're using real-time graphics such as video game engine like the Unreal Engine to create graphics for filmmaking or to do something related with filmmaking it's not a real great definition but you'll probably understand what I mean if you look at some examples or just search the term the best example of this right now is probably the mandalorian if you've seen anything behind the scenes you'll see these gigantic LED walls that they use both as sort of the background sets and to help with the lighting of the actors and people have been figuring out techniques on how to bring the cost of this down to do these sorts of things for much smaller budgets than they have on a gigantic television series by Disney but you still need to put in a fair amount of money to be able to do this well at the moment not only do you need a camera and either LED wall or green screen and some way of getting that camera signal into a computer and a strong enough computer to be able to calculate the graphics in real time and do the compositing in real time you also need virtual reality equipment that can be not super expensive but it does add to the budgets if you look at the vive pro setup you're looking at probably at I mean I'm talking Canadian dollars but I think about $1,200 for the equipment that I would need to be able to do this and I was looking for a way to do it for cheaper this is by no means something you could just follow this lesson here today and just go and start making movies tomorrow it's work in progress my hope with this is that if I put this information out into the world other people to have this equipment lying around their houses or can get one or two of these things can maybe start experimenting as well and we can figure out a way to make this work to really let anybody with an iPhone a camera and an input make a movie using virtual production techniques let's talk about the equipment that I'm using for this test rather than using a vive pro with its base stations and sensors and all that we're going to be doing all our tracking using a iPhone 8 so unreal allows for you to use your eyes as a sensor to move a 3d camera around it's mostly been used for adding sort of a handheld camera field to cut scenes and video games so my idea was that if I can have the iPhone mounted on a camera so using just any kind of camera mount I could then use the position and rotation data from the iPhone as a means for telling unreal hey this is where the live action camera is that way we could marry a computer-generated background with live-action footage the other thing we need to do this in sort of a real-time way is a some way of getting your camera's signal into the computer and in my case I'm using the cam link LOL ghetto cam link HD there's a 4k version of this you could use a lot of the Blackmagic tools odd-job makes a bunch of things as well those are usually more expensive than this which runs about $200 Canadian so yeah this is sort of a low-budget way of doing things just a note I am using unreal 4.2 4.3 in this not the more recent 4.25 that's simply because I haven't tested in 4.25 which I only downloaded yesterday it should work exactly the same way I don't think anything's really that different between the two versions in terms of what I'm using here I don't know necessarily that the iPhone is the only way to do this sort of technique there is a virtual camera plug-in or a sort of a virtual plugin by a company called buy owl studio for the Android to connect to unreal it's about 40 bucks Canadian and another 30 bucks u.s. probably I don't have it my Android phone is pretty old now so I don't know that it would work even if I did but that might be something to explore if you are a brave Android user so wanted to show a very brief look at how the hardware is set up here and so we've just got the iPhone sitting in a mount on top of the gh 5 on a tripod that I can move around and there's an HDMI cable going out of the gh 5 into the elgato cam link which goes into a USB port so we're gonna be doing three things here we're gonna first be bringing in the live video signal from our camera via the capture card second thing we're gonna do is set up the virtual camera which uses the iPhone for the tracking data and the third thing we're gonna do is use composure the Unreal Engine compositing engine to combine those two layers and add the two effects together so we're gonna start off with creating a new project and it could really be any of these I'm gonna go with games and third-person just give me a place that I'm familiar with and leave that all as it says and I'm just gonna change this to phone track will be the name of my project and create that project so here we are in a brand new project and to start off I'm going to head over to the Edit plugins and we're gonna look for a couple of things so first off we're going to go to compositing and make sure that composure is enabled and next we're gonna go look for virtual camera and here we go virtual camera now that is to control the actual virtual camera but to get it to work with the iPhone we need to install another two plugins the first being the apple AR kit so let's enable that and finally we're gonna look for something called a remote session which is what a let us interface with that iPhone here we go so we're gonna restart now a few moments later our first order of business is to get our live action camera into Unreal Engine so to start that off I'm gonna head to content browser and I'm gonna right-click and make a new folder called movies I believe you should name this movies that's what I read somewhere not a hundred percent sure why but hey it's not gonna hurt to just name it movies I'm gonna head in here and what we're gonna do is in the content browser see add new and go to media media player now this is gonna create a media player that's gonna allow us to bring in our camera feed or it could be used to bring in a video clip of somebody against a green screen but as we're trying to do this live we're gonna be bringing in a live camera and it's asking us if we want to create a video output media texture asset and we're gonna say yes so now we have a new media player we're going to call this input and let's double-click this now so it's created a media texture here and that media texture is going to be our actual video clip we're gonna head up to this little folder here click that and look for in this case I'm bringing in the elgato cam link and there's a little Deckard that I have set up if I move the camera we see that that's happening in real time so I'm gonna save this now just to get this guy in the scene so we could just see how he's doing every once in a while make sure that our camera feed is going I'm gonna minimize this for a second and I'm just gonna create a plane just a nice little plane bring it up and we're actually going to create a material from our input video by right-clicking on it and we're gonna call that video material oops and I'm just gonna apply that to my plane here so weird if we were to rotate that now we'd see a little Deckard and obviously mm-hm just gonna resize that a little bit so we don't really need this in the scene it's really just to have a look at whether our connection is going and right now if I move the camera you'll see the live video is changing in on the plane there we go so we've got one problem half solved because if I were to open up this input window again and you'll see that he's live if I were to close this down even though I've saved it the feed into unreal seems to go dead so even if we move the camera around you can see on the input video the thumbnail here nothing's happening so what we want to do is make sure that that connection is established in the brew at the blueprint level I don't understand why but this is the way it seems to work so I'm going to bring that cam link back and we're gonna look here and see this little URL that starts with a vid cap and I'm going to ctrl copy that and let's minimize it again so the next thing we're gonna do is head into the blueprints and go into the level blueprint we're gonna have a events begin play to start things off and that's gonna tell everything to when when we press play these things will happen we're gonna create a variable I'm gonna call this media player I'm just gonna hop over to the variable type and we're looking for a actual media player variable type so I'm gonna find that down the list we have to change the default value but because there's nothing here we're gonna head over to the compile button and hit that and now from the drop down we have our input media player which is the one we created over here so that's like that and I'm gonna now control drag this guy into the blueprint and now what we need to do is drag this out and place a new node and what we're looking for here is open URL so let's click that and the URL we're gonna use is the string we copied over from the input media player in our live video which is vid cap USB whatever so there we go now we're just gonna connect the event begin play over here and probably compile and save this and let's shut this down so now nothing's playing here despite Deckard still being live on the camera but if we were to press play and now this is what I'm doing live I'm moving the camera around and the video of Deckard is moving around so that's our step one we have our video into the software step two is going to involve our iPhone and setting up the virtual camera our next step will be to go into our world settings and we're gonna switch our game mode over to virtual camera mode now that that's done we have to connect our actual iPhone to be used as an interface so what we're gonna do is go to we're gonna go to our command line CMD bring open the command prompt and what we need is our IP address so IP config we're gonna open up your remote and type that IP address in here I've already done that so I'm going to hit OK and connect and now when I press play I have control through the iPhone of the camera so we now have a virtual camera working and we have a live video feed how do we bring these two together well we're gonna need composure so let's stop this playing put the iPhone down our next step is to go to window composure compositing and open that up I've already got to tuck down here so let's have a look at creating a new composition if you're familiar with After Effects it's exactly like creating a new comp there so we're gonna name this guy Oh I'm gonna say yes to this empty cop shot and I'm gonna call this composite and it's popped up right here and we can see that it's empty so what we want to do is add two layers to it we're going to be adding the virtual camera layer over here and we're going to be adding the real-life camera and then pulling the key on the real life camera and combining everything together so I'm going to right click on the comp and say add layer element so we have a couple of options here media plate which would be used our video layer CG layer which will be our virtual camera and CG mat so what we're gonna do first is let's bring the CG layer in and if we hit this drop-down we see there is something called CG element so at the moment we're missing a camera as it tells us right here so we're gonna head over to the cinematics area of the modes and bring out a cinema camera actor drop that into the scene get a nice wide ash view here of everything that's happening and Steve select that camera now and go back to our composite and it's automatically put that camera in but we also want to go into our composite and under the input let's say the target camera and pick that Sena camera actor so we have our cg plate and now we're gonna go back onto the cup and add a new layer element which will be our media plate and let's just keep that name media plate 1 and what we're gonna do is head over to the inputs and pick our media source which will be our input video so there we go it's actually not live at the moment if we wanted it to be live we would have to head over here into the input and reconnect that cam link I don't know why it does this but that's how it works as long as it's minimized you can see the live feed so we have our green screen and what we need to do now is key this green there are obviously going to be better versions of this as a tutorial for how to actually work with the king I'm gonna do the basic bare minimum key here just to get us moving and we're gonna head into the transform compositing passes under the multi pass chroma keyer we're going to go to key colors and hit add elements and now this gives us a little color selection tool we're gonna hit this and just pull a key there we go got a nice clean ish key Deckert is still alive here obviously there are lots of options you can choose to tweak this better and there's d spills and erodes we don't really need to deal with that for what we're doing today so I'm gonna move on and if we actually head back in composure and look at the composite it's still empty and that's because we need to create a major material to blend these two elements together so we're going to go under composer we're gonna go to the transform passes and we're gonna say a custom material pass if we drop that down we're gonna see make a new material here or we can head to our content browser I'm just gonna hop up to top layer and create a new material let's call this comp mat and I'll double click that to open up our material maker so the first thing we're gonna do here is look at our actual material and we need to switch this over from surface to a post process material and what that's done is taken away some of the options here but we have an emissive color what we also want to have is the opacity so let's just type OPA alpha output alpha when we click output alpha the opacity lights up here so from here what we want to do is bring our two elements together and we're gonna do that using an over so let's add each element as a material we're gonna look for something called Purim 2d texture sample parameter 2d and I'll call this one live and we'll get another one and we'll call this one CG element and we're gonna pipe the rgba oh no we need something to actually blend them together so what we're gonna do is right click and we're gonna look for something called an over and what an over does is it just combines the two so on top we want our live-action plate into slot a and we want our CGS into slot B and our RGB a is now gonna go into emissive color let's just save that and close this out so we've got this comp mat here we're just gonna drag that onto the material of our complex empty we get a nice green blob well we've got to now add the two elements into our green blob so for the live material we're gonna go and pick our media plate which is dead at the moment again because this disconnects so much hours are so now if we make that live again we can see hey we've got Deckard standing in front of this green blob so let's go add the CG element and here we're gonna choose the CG element one and now if I move the camera well that's all well and good Deckard is moving but the background isn't moving with him so we have to now connect the virtual camera and we'll do that by starting up our iPhone so while we have Deckard here in the window and we have a background we can't move it unless we play our level here and unless we play our level and when we do hit play tada we're losing our comp so what we need to do is output the composite and we're going to do that by going down to the output settings and adding an output we're gonna say player viewport so now what this is doing is taking our composite and outputting it here in the player viewport so now if we hit play we have our view from our camera or was for my camera but we do have a background moving and we have Deckard moving as well so we're gonna grab our center camera and just rotate it see if that fixes our problem put in 90 degrees in other direction and let's hit play again there we go we have a little bit of a better view in the sequencer I'm gonna actually create a new master sequence I'm gonna say let's give it a 15 second duration and I'm going to head over to the composite composite and bring that over into my sequencer just drop it in here and now if I were to render out from here I'm just gonna use my default settings I'll change its 1080 say yes to save selected we're gonna get a tiny bit of a window here should get a better view on the camera can get a more finalized render although one of the many mysteries to be still is why why do those green things appear 15 seconds of Deckard in his little virtual world so there you go there's obviously a couple of issues still mainly that they're not totally in sync I'm not sure if it's frame rate issue if it's an issue with just my computer being five years old and having a couple of 970 gtx cards in a world of 20 80s but if somebody out there has some ideas either posted in the comments here or maybe over at the unreal virtual production group on facebook i want to give a shout out to two youtubers whose information was absolutely essential for me to be able to do this and ii Blondin who's got a great tutorial on composure fatty bowl is someone else i've linked to both of them below i also linked to matt workman who i believe started the facebook group and who's whose youtube channel has been invaluable for learning just about this field in general i knew it existed before i found his channel but seeing his channel really was was eye opener on how you can actually do this stuff another little note is i turned off the image stabilization on the GH 5 and on the lens and i turn down the motion smoothing on the iPhones you remote unreal remote as low as I could because I think that's necessary to make sure that we get the smoothest closest motion as possible but who knows really I'm just making this up as I go that's it okay it's gonna say something else but I don't remember

Channel: hyperbolic films

Views: 70,645

Rating: 4.9542422 out of 5

Keywords:

Id: zZQu9f6U8zw

Length: 28min 15sec (1695 seconds)

Published: Thu Jun 04 2020