In depth Tutorial - Virtual Production with Unreal Engine and HTC Vive

all right welcome back it's been a while and I'm super excited that my video got such a warm welcome so let's just continue and today we gonna do a more in-depth tutorial kind of how I set it up the whole live-action compositing with Unreal Engine and what we'll cover in this video is setting up all the gear and what plugins you need the equipment and how to record and how to export the material short interim of me is I'm a self-taught director and producer of various films commercials and music videos for the last 15 years and I always been thrilled by making mixed reality with combining 3d and CG elements together with live-action and I think this year is really where it all comes together with all you have the high end graphics card that's are available and to a really nice price actually and also that unreal is really focusing on making their software work with our purpose and I think that's pretty cool so that's great we're gonna go look at some example shots and some behind the scenes for from my test videos I've been doing over the last month's and we gonna go through the pre-production of making freed world in Unreal Engine I'm gonna show you one of the world's I made we're gonna set up live tracking set up the base stations the plugins setting up the project with the blueprints preparing for recording setting up live view with tops and the syncing synchronized with timecode and recording yeah and we're probably gonna have time to look at the post-production how to export the CG and how to combine everything in After Effects but since I don't have a green-screen we're gonna do it quite a little bit different but I think you're gonna get some ideas how you can do it with a green screen some must-haves you need to have the HTC y pro with at least two base stations and you need to have the wide tracker and also you need a computer with a high-end graphics card i have a gtx 1080 probably gonna buy the Arctic's 2080 and and some kind of film camera DSLR or other I guess you could use an airy or whatever I'm using a black magic or some mini pro only because that's the one camera I use for everything else and if you want to do the live compositing on set you probably need something similar to what I have I have a black magic do PCI card that that I put SDS in and SD out and from the main PC the SSDI is going to my MacBook which has two black magic ultra studio SD capture and that's the computer that does live on set compositing and this is a flowchart on how how it works so basically in the studio you have the HTC wipe tracker and the camera mounted together and the white tracker of course sends the tracking signal to HTC Y which goes into Unreal Engine on the main PC and from the camera we have an SD output the goes into the black magic do which them from Unreal Engine I have two SD outs one with computer-generated background and the one with just the video feed going out and all this is going into ops which is pretty much open source streaming software and there I used to do a quick key and we can watch it live as it as we shoot it and we output it to some kind of monitor or television so everybody can watch it that's pretty much it so I was thinking of used I don't know let's use yump into it let's have a look at some examples the video is released one of the videos I did and it looks kind of like this but maybe you should pause this video and go have a look at that video right now I'll be here waiting so this video is referred and one of the first videos we did with this setup so it's kind of experimental video and we noticed we had a few problems with it it's the one thing that's really hard with this is just to get the talent to really stand on the C on the freedom for your background so you really have to have time to experiment and find the right position I'm gonna show you how how you do that later and since this is new tech and I don't have a clue what I'm doing I used X Miren thing we had some problems with keyframes that got up on two shots and unfortunately we we didn't notice until it was too late so we had to use it we have to release it but but the stuff you have to count down when you're working with non production ready the technology I guess so so be careful and take your time and really test things out before you could do go out and do live stuff these guys were in on it and they knew that this is really new tech and it could go straight to hell or it could be really good I think we we did pretty good actually but we have a few problems but overall I think it looks really nice and of course you have to be really good at compositing the the standard after-effects stuff to make it look good in the end but overall I think it's turned out pretty good and the problem I was talking about this you can see it here where the background is a bit yellowish on some shots but overall the shots are smooth and working really good here you see the shot so what happened is that the keyframe I think that I think the computer were I think background the freed where were too heavy for the computer to handle so you couldn't bright all the keyframes when I was recording the light when I was recording live and Sept so after that we probably gonna record in much low lower quality to have to ease it up on the computer so right let's have a look at some behind the scenes material here's here's us testing out you can see it's really close together it works really good and here you can see quite a shame to show you this but the green screen is not it's probably worth screwing screen I ever did but we were a small team it was just me this is my my friend Shirley and he has no experience in the film business at all so but it was good having him lift stuff stuffs and whatever and I had the pleasure to having happy Martin who is a really really great photographer and friend here you can see us building the I don't know why it's doing that we rented a studio here in Stockholm that has a trust and everything so it was quite easy to get everything up but the green screen setup made it really hard for me to get in in the post so here you can see the setup here's the computer with Unreal Engine on and this we Swedish smells really important and this is the ops with two channels I don't know why it's doing that it's not working really good ok you can see we're testing the alignment of the camera and the free world and it's smooth and working the theater you see there is because the tracker gets blocked by someone or something from from the base station so it will lose its tracking and so we built a screen screen just for today three of us and we used every sky panel RGB lights to to fake the neon lights of the world and here's Martin it's a proof filmmaker and has there's a meaning with with the HTC y on it and for this video issues to rent an jib to make some smooth pants and camera moves as you can see it fits really well in the world and you see the heater from we also have some problems with the HTC tractor because since the base station are sending out a small IR light if it's too bright in the on the sensors or it can throw the tracking off so you can't use it in two bright environments and here's the whole setup with Jib the airy sky panels and we these are the lights that lights the green screen and you can see the really crappy green screen and now you understand why I had such a bad time trying to key it but it's alright and here's one of the talents getting prepared for the first take I guess so this was really good because she could see where she were were in the world and could act after she got a better feel of how the video is gonna look it's another Talent some more shots of marking work in the crane so here we have the all the cables going down into the computer and the third one as you can see the camera the video feed and a background our perfect sync and that's the well moment moment and we did something like this put them on our roof crappy green-screen again me there are some dancing stuff you can really see oh I did so that's a few balancing shots and you can really see the camera is locking into the 3d background which is awesome so let's then I guess jump into I'm gonna show you I'm gonna do the whole setup again it will take a while and it's quite complex so if you don't understand something you shoot the comment and I try to answer as much as possible I probably should do a live live live stream too so we could so I can answer your questions in a real time but let's or maybe if we should have a look at 3d background first let's do that let's have a look at this world finished world still loading all right here we go it's quite a big scene so it will not be smooth showing you this but basically what I did because we were short on time is that I bought Unreal has a really great marketplace so I bought quite a few assets and also kid bash has a great shot for buying cool stuff so here's the stage with everything in it and so this is how I did it is that I did a preview for for the client and then we did a session over where she logged into my computer and I showed her the screen and we worked together for it and did small stuff like this change the text to to the name of the group and change that can change color that can be a add a house there and it's really cool to be able to do it straight on the spot and and one thing that's really awesome too is that you can play in your own world still a game engine you know well yep so this is really cool but I did something more to add a bit of depth I added a train going by in some shots you see it and it's at a bit of more realism to it or something so that's cool but basically I use bought ton of stuff if you are a true 3d modeler you could probably use make your own stuff but and it's just a matter of you know pulling stuff in we need a new building here let's put a bill in there and probably doesn't have a shader for this it's needs to compile the shaders or something but it's super fun to do and it's an easy workflow super fun but anyhow let's create a new project new project let's do this from the beginning I used you can use whatever a blank I guess great project don't say so the key part if you if we start with equipment I'm not gonna show you how to install the HTC why and what not that's something you have to figure out yourself it's not hard but in here in my small office I have one base station there that's almost by the ceiling and pointing down and I have one over here pointing down but on on the set of the video I showed you I had two more that were in a square so I could cover as much as much area as possible to have more freedom to move the camera for the next shoot I probably gonna buy two or three more base stations because we had some problems with the tracker getting blocked by something so we need more base stations to pick up the tracking signal but anyhow alright let's jump in the first thing we do is start an m2 actor and this is going to be our scene mover where we can move the camera position without affecting the camera relative camera tracking data camera tracking position that's good because we want to be able on set to move the camera around in the world because in scene 1 we may be gonna shoot at that angle but in scene two we're gonna shoot at this angle so it's quick way to move move the camera around and now after that we starting we were doing the camera tracker and put it as a shot to the C mover and lastly the camera tracker is going to be the one that has the camera tracking data and let's name this live the camera and put it in camera tracker so let's add some scripting to the camera tracking and that's this let's leave that one and remove everything except for event ticket begin play and the first thing we do is get tracked device says position error and orientation and from out orientation we drag it in to combine rotators and this is only a backup because sometimes the track a white tracker is flipping axes so you have to on-the-fly be able to fix that and that's what you do with this one but at the moment it doesn't do anything it just in case something happens and it's going out to a set relative location night rotation default scene route which is what we want and from this one we're going out into a set relative location and also defaults in route so from out position into this one and to make everything tick and here we have a complete blueprint it's super easy let's see if it works something happens when you put simulation it's not this one I haven't figured out a way to get the right device C D it depends on in which order you turn stuff on and early today when I test things out I had number six for a wife tracker let's see if it's divided six number wife tracker no it's not then we have to do is go backwards I guess till we find the right ID for compile something happened it's not the right one free compile it's not working let's see if number two it's not that one maybe you need to turn off and turn on simulation no let's go back to six see what happens nothing happens what what I think we need to do then is restart save everything and I have a Mac in Apple image on a PC so let's start this up again all right open our level up again and open the blueprints let's see if we are now working then you just keep trying more compiled no let's go all the way to here serious to rule everything out it's not that one compiled I don't know why it's not working it should be working get track twice to feel defaults in route defaults in route to compile nothing nothing I don't know why it's not here we go number five perfect number five today my not lucky number okay we probably need to see we would like to see better what's going on so let's add a mesh as a child static mesh and here we can add axis and you guys are probably not seeing the axis I'm gonna show you how to do it you go to view options and show folder show developers content and show C++ classes so if you have that up you're able to choose the access guide so let's have a simulation and it seems to be working pretty good the important thing is to have the green LED light pointing forwards then you should everything should be correct so the x-axis is forward for me right now and it's I guess that's okay I don't know probably cool so now we have the tracking after working and you have to do move everything camera tracking blueprint into C mover and live camera like that and just asses just clean this up play your start assets so we know what we're looking at so here's our maybe you could do it like this create folder what see let's move everything into scene and if we choose simulate we should yeah we have everything ding ding ding ding ding ding so let's mount this on the camera great and what we have now is we need to install a few planes we need black magic mirror player boom and that's something you need to install from the unreal marketplace wait with restart now we need timecode synchronizer yes we need media at the profits right at it so it's a new one and me at the framework and what do you need more composure let's let's start with that and see if we need to add more plugins so this camera has an SD out going into the Blackmagic capture card so that's what we're gonna do now we need to get our time code and our video feed into Unreal Engine and what we do is I guess used we have to create a new media profile profile Blackmagic media profile just configure I guess go back edit so mera sources we have a black magic mirror source and I know that it's tech linked to one and 30 frames per second time code format after weeks of trying I know it's widths and I used use RGB 10 bit synchronization we not want to sync with engines timecode since we we we are feeding the engine our time code from the camera and when I want to burn in the timecode timecode provider over it project settings I choose black married SD as the I in input and shoes tech linked to one again timecode format widths and you want a gen lock it to the same BAM and if you look at our timecode provider up here it should change yeah and now it's time code of the camera and if you don't have this little bar you could go up here the developer tools and choose timecode provider Oh so you can see it so now we have a timecode we have a tracked HTC Y going in what should we do then we need to combine the 3d picture with the video feed so what I do then is I need to add a major bundle a black magic media bundle and choose black magic black magic made a source and the same the input 10 bit RGB output and here you can add the lens parameters for the lens on distortion or lens distortion but haven't played around with it yet haven't it's something or my to do I like to burn time frame code so I can know what's going on and then we should have yes so then we have the camera with perfect sync just leave it there and what we want to do next is you can do it in several ways now you know you're pretty much set up you can do you can record a tracking and you can but you can't view the output so my way of doing stuff is I need to get two outputs from Unreal Engine 1 with the video feed one with the video feed and one with free the background so let's you start with getting the free the background out yeah and we do it either way you can do it like this and go up to window and developer tools and major capture and you can use use viewport capture cameras and choose the live camera and we need to make an output media black magic black magic media output and we can choose 3d world output SDI and I know that one I put my SD out on the third port and let's just choose 10 bit widths and that's perfect so if we go back to Mira capture you can choose Freder world output SDI and if you start capture now we should have we have a picture so we have a picture from Frieda background going in I think you guys are capable of managing how to use ops but later on I'm gonna show you the process but let's just focus on Unreal Engine otherwise it's gets to jump if back and forth but we have an output that's working and then we need to have the SDA the video feed go through Unreal Engine into ops to get the SD l io out we need to create a new Blackmagic output let me call it video feed output SDI and I know that that's my fourth my shoes everything here ten bit good burn timecode so here we need to feed the SD i-- and by doing that we're gonna use the composure component so go window developer tools I guess or where do we find it I don't know it's just there for me yeah it's probably there let's create a new comp empty and create add layer element and we want to use the media plate and go back to details and choose input maida texture and we can go to blackmail a bundle inner assets and used put it there Bom we have it and we can do output output pass me a capture but we have a render target as set not yet we do could you made a capture and choose video feed output SDI and now we have we have two channels one with CG and one with black magic device going out and that's great so now we have sync with everything so we have it free the background moving and we have cool did you get it quite complex and what now so that's pretty much it but the thing you need to worry about is that you need we need to go back to seen live camera and probably add a blueprint so we can add on mesh static mesh we need to see the camera in the viewport and let's add an scene cam now we need to align this one up as close as possible and we need to change the snap value to one so we can get really tight it's pretty much it and like that and that's just for to be able to show let's move it back to Shia love that one because now we need to align the camera because the tracking data is quite far from the monitor so what I did a thing around here is good for my camera but you need to you need to figure that out for your own camera and it's just a lot of trial and error so let's see what's going on here I have a mere to capture the free world is going out we need to choose the live camera blueprint capture and if we play now and then we maybe could turn off the axis on the camera tracker blueprint and just make it non visible right now compiled this is pretty much how I shot the the video I showed you earlier and I'm gonna show you how to record what we want to do then is go to it's quick sale developer tools and shoes sequence recorder over here and choose which three elements we want to record let's add those here boom boom boom and to record a camera movement week let's choose test sequence perfect I think we should doesn't need to do anything more let's cry let's do a recording record and play so the first thing I do because there's a problem to sync the video and the free the background in post because you need a marker that shows both in the video feed and in the free world so what we have been doing to get that marker you still have to do a manual sync but in the beginning of each take you just do a quick tilt and hold it there until backup because that's something you are able to see in both the video feed and this free world sequence so we are recording right now let's see if this gets recorded cool stop recording don't test sequencer successfully recorded open test sequence and as you can see we have the camera recorded and but you have no keyframes and that's because if we we would like to be able to maybe change some settings in the camera after the fact and that's why we are recording into the camera tracker blueprint that means that you can also move the camera afterwards cause you're on set and you're recording but then when you get back to your office and start doing the post-production you maybe want to use a different angle or different position of the camera then it's possible to move the camera without moving the tracking without messing with the tracking data so here it is it's fully recorded and with the marker boom and that this is how I do it and afterwards I just go up here render this movie to a video image frame sequence choose my options you probably should ah now you can use app approvers which is great I mainly use pngs or a which is say the English a B I ve say are we in Swedish we say all the e but in English I guess is maybe I used render it out and then compete in after-effects with the real data so that's how I do it nope that's the whole process let's I would like to show you how I sync it with the live calm so I gonna do something special now since I don't have a green screen here and that's used to be able to show you how to do the live compositing so this is another step first we can we can remove remove the sequence so now I have the original camera we go back to composure and we add a new layer element because what I want to do now is output just a box and composite it here on my desk so you can see how to align the camera and make the sync work so let's add a single a CG layer and do and we need to add a little cube let's make that smaller boom and just create a new layer the layers you get from here let's just name it CG CG layer and we want to move the box into the CG layer go back to the comp composure or CG element and down here at input capture actors don't choose to include from the layer CG I'm also gonna hide the background so we got it it's black and white black and white school and this one we need to output 2 outputs how to put add an output media capture yes output free the world and it's stopped the free world a clean layer like that so now we have the box over here outputting and we need to output the camera put that on there we go the camera and let's have a look where the camera is lost the tracking see if the camera tracking yums back yeah there it is so I'm gonna try to align the the Box on my desk by eye let's see how cool like that say and one thing that's more important too is to have the right camera settings and this is a DSLR I have Sigma 18 to 35 on and it's roughly around 25 because this is a smaller sensor and this is a bigger sensor on the camera let's see what we do all right so there's a box this is just for testing purposes we need to add a filter so we can remove the black I have a luma key here that's perfect close and as you can see it's not really really in sync right now there's a few stuffs that's not working and it's probably a delay let's see if it's I had a video delay for 42 milliseconds but start with that see what's happening yeah pretty pretty darn good but you can see we have some problems with when you move the camera up and down let's see if you can figure that out I'm probably gonna have to align the box a bit better move it back let's see something over there let's see how are we doing so we probably gonna move the box a bit closer okay we still have a problems with moving the camera up it's something that's not aligned and it's probably and this is why we need to do the calibration this is by eye we should figure out a way to do it is much better so probably we're gonna move the camera a bit forward let's see what's if it's better the preview enable this keeps getting an ticked I don't know why but let's just deal with it what's the tracking let's see if the track comes back there so it's not so either they are yet but maybe we need to move the camera back I don't know it's pretty pretty close but not not yet let's see if we move the camera back so we're moving the camera back on the camera tracking axis let's let's already shoot it and see you see what's going on I don't know why it keeps turning itself soft but so this is this is worse so let's keep moving the camera forward just overshoot it keep pressing the output I'll put the Nabal now we maybe need to move no the MOU talks a bit as pretty close so this is what you have to do for a while to find it out the right position of the camera but luckily I did it earlier today so I can show you let's go back to the tutorial switch yes save everything and open up our earlier let's see if he just works out of the box of course not let's see what we need to do in a vault enabled and made it capture maybe No here we go play let's see if it's yeah who's here I did the calibration of the camera it's almost like it's there Lucia Trish because something is hiding the sensors they probably have to tweet deal a bit but it's it's almost there so that's how you do it pretty much and what did we say that we were about to talk about yeah so what I have hearing in ops is just two devices you told us to do with the background and with the video feed and and now we're not gonna see anything but with the free the world and to sync everything so what I noticed is that the video this is the 3d background is taken for this project roughly 42 milliseconds to render and that creates a delay that you need to add in ops to get a live sync and this is a number you need to experiment with because you maybe have a heavier scenes that takes a lot more time to render and you need to add more milliseconds but that's pretty much it that's it I guess and this is rather complex so feel free to shoot me a comment on youtube or if you have better ways to do this this is just me what I come up with so so just to recap we have we have looked at how to make a 3d world and how I did it and how to set up the live tracking and post-production how to export CG and compositing and syncing CG and live-action I think that could be another video but now you know how I sink it I do it with the tail drink before each take a tilt the camera which gets the tail both in the camera tracking data and on the video feed and I guess that's that's what I come up with and I'm eager to hear what solutions you guys will have for everything so good luck and have a great day

Channel: Richard Frantzén

Views: 88,005

Rating: 4.9687042 out of 5

Keywords: virtual production, unreal engine, virtual cinematography, ue4, htc vive, aximmetry, virtual camera, cine tracer, virtual reality, the mandalorian, unreal, vive tracker, camera tracking, led wall, behind the scenes, filmmaking, mixed reality, vive pro, virtual studio, jon favreau, visual effects, star wars, vfx, tutorial, unreal engine 4, unreal engine 5, real time, bts, compositing

Id: XWQZNw9iMNs

Length: 58min 18sec (3498 seconds)

Published: Fri Dec 06 2019