Virtual Production with a Projector & Unreal Engine

hey [Music] i've been going around with unreal engine for a bit now not as much as i'd like to thanks to my schedule but as much as we can and as most do i think unreal engine and similar are going to be a massive aspect of the future of filmmaking which again i won't go into detail on since we did a whole episode on exactly that which you can find in the notes below but the main thing that has interested me the most is the possibilities for independent productions there's a few people doing some amazing things with this like matt workman js films and corridor digital all these guys are really killing it and doing all kinds of interesting things so definitely check out their stuff in the notes below as well but one thing that i've been trying to figure out recently is setting up a small mandalorian style thing and of course mandalorian and similar productions are shooting on this massive led wall this 240 degrees of shooting with those walls surrounding and a ceiling of leds sometimes that will help with lighting these led walls are made up of several led panels creating this array that the image is split across which is far more complicated than what i'm trying to do we also don't have the landscape they have as we showed in the episode last week we're using the cinebeam 4k laser projector from lg and we set this up on this big screen in our studio and once the projector is on we don't even have the full distance of that screen since our studio ceiling is too low so we don't have a ton of room to work with but we do have enough but we have been able to get some solid shots with this setup just using stock images and video so we'll see how it goes but to get this working there are several things that we need to get working and right up front i'll say that we'll be talking about several different specific products that we use to do this none of which are sponsoring this specific episode but to get this on its feet i needed to figure several things out tracking my actual camera so that the movements i make will control the movement of the world setting the image to display on the screen accurately and getting the final image on the actual projector screen and it wasn't easy at all and this is a good time for me to remind you that this isn't an episode from an unreal expert telling you how to do it i'm a writer director first trying to figure this stuff out and bringing you along for the process so there's plenty of missteps ahead but first up was figuring out how to get the camera tracked to control the virtual camera inside of unreal to do that you would often use vibe trackers or controllers made specifically for vr i've seen people do that and it works really great but recently i was able to connect with htc and they sent over a system that is in early stage development so this product might not reflect the final look of the system that they do eventually release but this is the mars system which is made specifically for virtual production with this i have the actual mars body and another piece called the demos i still need the vibe tracker and some base stations and these are going to help my trackers know exactly what their location is within the space then i have the vibe tracker here which gets connected to the damos piece and now it becomes one single unit that gets attached to the top of my camera which will then track and translate the 3d movement of my camera back to the mars system then that gets connected to the same router that my computer is connected to as well then i power it on make sure everything sync which it did without issue htc will be sharing more information about mars in the next couple of months and we'll be doing more episodes as well so let me know in the comments if you're interested in learning more and everything here is running off just one pc and like i showed last week that is my new puget system pc i've had it for a few months and without a doubt it's the best pc that i've owned it's screaming psychotically fast and there's a lot going on in this scene too and what we are trying to accomplish so for something like this the system that you work off of is incredibly important the correct motion and quality of that life rendering within the scene is going to rely on that so the big caveat with this sort of work is that you need a beast of a machine to keep up with it to be able to execute what you're looking to do and that's exactly what this puget system is i have the amd threadripper and the nvidia geforce rtx 3090 so this thing really flies you guys are often asking me about all the specific specs of the systems i talk about so if you do want to know all the specs just jump over to the link in the notes and check that out and yes i do have a puget page but i won't be making any money if you go to the site or if you buy anything from their site but back inside of unreal i needed to go to my project settings then plugins then in udp i add the ip address to the mars under static endpoints with an added colon 666 at the end it's four sixes so we're safe finally i open live link which of course we have to go into our plugins and activate our live link plugins but then inside of live link we will add a source message bus source and then select our mars system here and that's it everything was connected and ready to go so compared to what i've seen other people have to do to get something like this set up this new system is making it insanely simple to get things working but that wasn't all yes all of that is synced to my system and ready to go but now i have to get the virtual camera inside of unreal to accept the commands of this new tracking system but before we do that let's thank our actual sponsor which is hellofresh we talk all the time on the show as filmmakers or anyone really finding extra time to do just about anything is really tough which is why hellofresh is a killer service hellofresh delivers food directly to your door and offers a wide variety of quick and easy recipes including 20 minute meals easy clean up and low prep options and if you go to hellofresh.com and use the code filmrite14 you can get up to 14 free meals and three free gifts and there's a lot to choose from each week with 50 menu items including vegetarian calorie smart and gourmet options the holidays also don't make it very easy to find time either but again hellofresh helps keep things simple with recipes and ingredients that cut out grocery shopping and limit meal prep so that you can spend more time with your family they also have more five-star reviews than any other meal kit and this stuff gets delivered right to your door like these epic burgers that they sent us everything is included right in the bag including all the instructions to make the food and the meals are really delicious and the icing on the cake is that the packaging hellofresh uses to ship your food is made from entirely recycled content which is you know great so if you want to try it out and you should go to hellofresh.com and use the code filmriot14 for up to 14 free meals and three free gifts wow look at that look at it but jumping back into unreal and getting our camera connected to do this we're gonna need to make a new blueprint make it an actor then inside here i have to build out my nodes to set up this system this is a lot for this episode and i'm not familiar enough to confidently talk about it so instead check the notes for a link of the tutorial that i learned it from he does a great job of explaining it and he actually knows what he's saying so now i can drop that blueprint into my scene and now you can see that the cam track actor is actually being controlled by my tracker so now we just need a camera i'll add in a cinema camera then if we click on our cam track actor inside details we can go here to sin camera and select the cinema camera that we created and that's it we now have a tracked camera that we can move around our world to perform moves inside this virtual world but that camera is locked to its position relative to the tracker so we can't adjust our starting point in the world to fix that create another actor which i'll drop my cam track actor in this new anchor actor so that i can reposition the camera actor in the scene similar to what you would do with the null inside of after effects and now if all we want to do were cinematics were golden i could set up a scene with a character animated and use this system to operate the scene and with this connected to my projector right through hdmi this by itself is just the coolest thing ever having the scene up on the screen and then exploring the shots in this virtual world is just well insane and again it's on this projector that we've talked a lot about too really great color and contrast so even for this much it's excellent to work with but we want to go further we want that led wall vibe so now we have to get an end display setup to do that first we're going to again make sure our plugins that we need are enabled then i'll right click in my project here and then select end display and then end display config and then we're going to create a new configuration then we can double click that here go into our editor and this is our 3d configurer where we can get everything set up for where the projection would be taking place for this i measured the size of the screen that i'm projecting on then recreated that inside of cinema 4d then brought that mesh into unreal engine here like setting up those camera nodes the details here are very long and i'm still figuring it out myself so i'll put in the links below the other tutorial that really helped me get all this set up and wrap my head around it but once this is complete we can put our projector screen inside of the scene to find the area that we want to cover after that the last thing we need to do is send this out to a screen with a clean image to do that we're going to be using something called switchboard and as with everything in unreal the first thing you have to do is make sure your plugins are enabled then i'll get this here and i'll click on the switchboard icon here to open it up then i'll open the listener as well because it has to listen and i'm not gonna lie i don't 100 understand how this is working just yet i just know it does so there you go but inside of our switchboard i'll need to add a device which will be our end display device and since i have our end display config already set up it will find that right away and i'm doing everything off of that one system so i'm going to be using that system's ip if you don't know how to find your system's ip just google it but now i can click here to connect to the listener and here to launch the level and it comes right up beautifully with one big problem my camera track no longer works and after slamming my head against the wall for two days it seems to be an issue with running the end display and the live link off the same system i have found some people who seem to have used workarounds but i haven't gotten to that just yet so it's doable is just the next step that i'm gonna need to take but like i said earlier this is all work in progress stuff figuring it out as we go and certainly not experts but very new to all of this but we did find a cheat very much a cheat here that gave us the opening shot we just used the projector as the monitor and made the viewer as big as possible hilarious i know but it actually worked for the test at least and like i showed last week we are using the lg cinebeam to project this image it's 4k projector that is very bright and has killer quality which is obviously paramount to what we're trying to pull off here but it's also a short throw projector so it isn't getting in the way of anything i'm able to move all around without worrying about getting my shadow on the screen we've talked about the projector in detail more than once so if you want to know more about the specific projector check it out in the links below but man was this fun being able to step past my actor and keep moving into this digital world to come up to this wall was just insanely cool and one thing we did have to figure out was proper parallax the virtual camera was at a different distance from our physical one so we had to make those adjustments to get a more correct parallax going but once we did it worked really well especially in the close-ups just getting that sense of the parallax behind the actor was just so damn cool and this environment wasn't one i created i bought this right off the unreal marketplace to use for this test so this map while very cool isn't even all that photo real but still is looking pretty badass as you can see we have loads left to learn and this is just the starting places next up is to get end display working with the live link situation that i found out and if you have any suggestions for us pop them in the notes below that would be much appreciated but man i am endlessly excited about all of this all the possibilities around it even just setting up a scene as previz and getting in there to physically operate the moment you're testing or are prepping i'm just really stoked about the tech and having a blast learning it and like i said in our unreal episode the craziest part about unreal is that it is free so while i am using a bunch of pricey gear for what we're pulling off today to get in and get started will cost you nothing but the time it takes for you to learn it but that is it for today list to all the things in the notes below including all the amazing tutorials that helped me get this going today a huge part of the frustration for me was just figuring out the right tutorials to help so i put those together for you down below so that you can get rid of that headache at least this is not intended to be a walkthrough of how to do button by button everything we've done but rather to show you what could be done and where we're at in the process we'll get to more detailed stuff once we get much more comfortable with it in the future but if you aren't subscribed consider doing so we're going to be doing a lot more with unreal and showing our progress and what we learn and what we fail with along the way so hit the bell button to be notified when we put up new stuff too and until next time don't forget to write shoot edit repeat [Music] [Music] you

Channel: Film Riot

Views: 54,574

Keywords: film riot, triune films, ryan connolly, filmmaking, tutorial, how to, adobe, film school, virtual production, htc, mars system, lg, projector, mandalorian, unreal engine, filmmaker, filmmakers, motion capture, vr, tracking, walkthrough

Id: ZJzcO_eXqgI

Length: 14min 23sec (863 seconds)

Published: Thu Dec 09 2021