Using an iPhone as a tracker in Unreal Engine 4

all right ladies and gentlemen welcome to another green screen tutorial in unreal engine 4. now this right here is definitely a work in progress i am still working a lot of kinks out but i decided to make this video because i want to see if people can solve what i'm running into so what we're going to be doing today is we're going to be keying a live screen footage using a dslr plugged into an elgato cam link 4k and then we're going to be using my iphone as a camera tracker mounted on top of the dslr so that when i move the dslr the camera inside unreal engine 4 moves as well i know it's a lot so let's go ahead and get started let's create a new blank template we're going to name this mom all right create project all right let's go ahead and get this interrogation environment added to your mom and then go back and unreal it's only 80 meg so it shouldn't take that long okay so next thing we're going to do is enable some plugins built in we're going to do apple ar kit we're going to do virtual camera all of these right here yes yes yes restart now okay so let's go ahead and minimize minimize let's go to our content browser interrogation maps interrogation and that should load it up oh my goodness this doesn't look right so what we're gonna do is go to our world outliner we're gonna type in post process click on that go to exposure metering mode manual i always check manual and then just you know turn it up as so all right let's slow down the camera the default is so freaking fast let's turn it up a little bit more alright alright so let's move this cup and that's you oh man okay it doesn't matter it looks like the um the imprint is on the table but it's okay so the point of this video is we're gonna put a live green screen footage on top of this table key it and then move it around using the dslr and the iphone app unreal remote too okay so first things first let's go ahead and create a new folder right click new folder footage double click right click media media player click output text texture asset press ok leave it by default is alright we are now going to create a plane drag and drop it to the scene move it up towards the table rotate it 90 degrees now i'm not going to rotate this just yet because something weird happens it happened here sometimes so go back to your media player we're gonna turn on our dslr real quick okay the dslr is on link it to the cam link perfect i'm gonna turn the brightness up a little bit let me change the aperture all right that is pretty much one of the nice things about doing this live is you can actually adjust it on the fly all right save that we're going to copy this control minimize go ahead and drag and drop this media texture and it's going to create a new media texture and that's why i said i wasn't rotating it around just yet click on the plane 1.78 is the aspect ratio of my canon dslr so that's good we're gonna go to blueprint open level blueprint delete these two create a new variable media player right click event begin play drag and drop that media get click the media we're going to change the variable type to media player and then we're going to go object reference okay that's fine it's going to search for any other object reference sometimes it does this but that's okay so minimize is fine all right we're going to drag and drop it here and then we're going to do open url we're going to paste that url we just copied we're going to connect this little diamond bright diamond baseball thing and then compile save and then minimize right press play okay it's not there we missed one and i realized it right after i saved it go to the media drag and drop media player compile save minimize play voila now our live footage is in the scene and this is live i'm going to show you change the uh okay all right so that is a live footage feeding in using the elegato cam link 4k all right okay so the next thing we're going to do is we are going to create uh a keyer we're going to key this out so we can only see the green so double click the texture we're going to disconnect alt left click we are gonna click the rgb drag it and we're gonna go with mf stands for mf chroma keyer put that in the middle put the emissive color to emissive opacity opacity and opacity to opacity masks and then we're going to save we're going to click on this right here the video mat we're going to change this to transparent or translucent press save go back all right and what we're going to do now is create a material instance right click material instance i'm not going to change the name actually let's change the name first instance okay now we're going to double click that and now the options that we just created with the original are all enabled and let me just move this away for now okay we're going to turn on the keyer we're going to click this color get the eyedropper and then we're going to try and key the screen voila that looks good and now we're going to mess around with some options here now i'm not going to try and get the perfect keyer here guys this is we're just going to do something quick and i'm going to save it and as you can see nothing's really changing well it's because we have to apply that new material instance to that footage okay so let's go ahead and zero this out there you go now we can mess around and see if we can get a good key out of this now i do have two tones of green um i'm using two different green screens so that's why it's not keying as well on the ground plane but that's okay let's go ahead and crop it crop right and then crop the top oh i said top not left okay that's fine just like that all right so let's go ahead and um get some key out of this okay and if you want to mess around with it more obviously you can but we're just doing a quick and dirty key here okay now the only good thing about using composure is is that it has more keying tools than this okay let me go ahead and save this and obviously you can spend a lot more time than what i'm doing here okay see change the luma mask again go increase the luma mask shrink it again okay so that's pretty decent that's a pretty decent key let's save okay and now we can move this around which is awesome we'll press q and i'm going to just try and see if i can have it touching the table there okay obviously it is a 2d element so you know okay save it all all right press play it's there that's amazing all right so the problem number one i'm gonna be using the unreal remote too with my phone i'm not gonna be showing you how to set this up because it's really easy if you need help go ahead and check out the video in the comments below on how to set this up but basically we're going to be using the unreal remote 2 on my iphone and we're going to control the camera inside here but what i'm going to show you here in a second is i have the iphone mounted on top of the dslr so theoretically what i want to happen is when i move the dslr moving this footage the box i am also moving the camera and the environment inside right here the problem is and i can't figure it out every time you turn on the virtual camera mode in unreal engine 4 it creates a new camera and that is a big problem because i have to match the dslr to the view of the unreal engine for virtual virtual camera mode so let me show you what i mean so with unreal remote 2 make sure you're connected to the same exact wi-fi as your phone and your computer that you're using so let's go ahead and go to world settings we're going to change the game mode to virtual camera mode i'm going to go ahead and set up my phone okay so what we're going to do now is turn on unreal engine remote 2 on my cell phone i'm going to wake up the camera i'm going to wake up my phone all right and now when i press play it's going to connect to my unreal engine 4 app on my phone and you're going to see that the field of view will change you see that how screwed see how that just kind of moved and it's not right it's a little bit tilted and you can see it right here on the bricks so basically what we need to do is since that camera if i go here it created a new camera inside so what we need to do is we need to control that default camera that the app is spawning so that it looks right all right and that depending on your scene that could get very complicated so to fix that we're going to press stop we're going to go to view options i think i already have show engine content on make sure this is on click on this scroll the way down virtual camera content and now you're gonna see where this camera is pulling from so basically these are the components that the app is pulling from and i accidentally just found this so let's click on double click virtual camera player controller and it's going to open this all right and you're going to see right here at tracker post offset the rotation is what we're trying to fix because it is a little bit skewed so right here is trial and error until i figure out how to fix this okay so press play and this is going to connect to my phone again you're going to see that it's going to change view here there you go all right so that's a little bit better so we're going to go ahead and stop and like i said depending on your scene this can take a while so and it doesn't look like it's letting me do that so let's close it right here it looks like it's bugged so open it back up let's go to negative 13 compile save minimize play but basically what you're trying to do is matching the camera to the dslr all right and as you can see that is still off because what you want is actually the box in front because the box is in front of the dslr as you can see with this camera right here now let's turn on the ui on the phone so press edit project settings because you can technically control the left and right and up and down on the app so you can't technically do that but you wanna still get the rotation right so if i watch what i'm gonna do here watch so i'm able to move that um camera using the app by using the the thumb wheel thingies but it's still crooked so that's why we're having to do this and the only thing is you can't rotate in the app itself you can only do left and right but not like rotate completely so make sure so pretty much rotation you're gonna have to rely on doing it just the way we've been doing it by just trial and error okay so now that we have that live i'm gonna minimize it and i'm gonna show you that it is live it just turned off so it's not live what i'm doing right now is i'm stabilizing it as you can see it's kind of shaking a little bit okay so now in theory i can now move my dslr and it should move the phone camera or the camera inside as well in unreal engine 4. let's do that so you can see what the theory is you kind of see what i'm trying to do here because when reviewing items such as cameras and so and so i'm actually trying to go ahead and move to unreal engine 4 because i'm getting tired of all the b-rolls starting to look the same online on youtube so that's my plan with this um but you can see it's not perfect whenever i am tilting up and down there's some kind of rotation going on that i really don't like um additionally it's it's just not as smooth as i would like that's pretty much it guys obviously it is not perfect there's still a lot of stuff that i need to figure out uh but it's pretty crazy that a game engine can do something like this with a couple of workarounds now this is what really excite me with the upcoming unreal engine 5 because i can guarantee you i'm hoping that it's gonna be much better as far as trying to do stuff like this so if you guys have any questions let me know and i'll try to answer it if i could but just so you know that i've only been using this software for two months so i might not have the answer for you and additionally if you guys don't mind make sure to vote for my cgi short film that i made inside unreal engine 4 in the links below i would truly appreciate that you

Channel: JSFILMZ

Views: 20,307

Keywords: Using an iPhone as a tracker in Unreal Engine 4, Unreal Engine 4 Green Screen Tutorial, ue4 green screen, virtual production unreal engine, vfx unreal engine 4, unreal engine 4 composure, ue4 for beginners, unreal engine 4 tutorial, unreal engine 4, unreal engine 4 demo, unreal engine 4 beginner tutorial, unreal engine 4 animation, unreal engine 4 photorealistic, unreal engine 5, unreal engine vive, ue4 composite plane, ue4 vive tracker camera, ue4 vfx, jsfilmz, htc vive ue4

Id: bk8fK6n9Ui4

Length: 17min 44sec (1064 seconds)

Published: Sun Aug 30 2020

Reddit Comments

[u/jsfilmz0412]

Trying to try out live chroma key with ue4 using elgato camlink 4k