Dev Tutorial: Live cameras in virtual studio in Unreal Engine

what's up what's up thank you guys for dropping in uh today we're gonna be talking about my studio um so i've been getting some comments asking about how my studio was set up how it was made you can see that um we have kind of some uh standstill textures here to show how it works but essentially uh let's see if this works the way i think it's gonna work yeah so we can actually see that um it's a little laggy right now because i got a lot going on at once but all of the movements are tracked in real time inside of the studio so it's pretty interesting how that's set up and it works really well assuming you have a computer powerful enough to do it because mine is obviously struggling a little bit with um having all this open at once but it's not too bad it it really isn't too bad considering everything it's doing at once including we can see in the background here um i believe it's what is my switch key let's see let's exit that real quick let's go into my blueprints and i have everything organized really nicely here so that i can make some videos on how everything is done but the nintendo switch toggle is l okay so if we hit play let's go over here and you can see that we have a nintendo switch running there's some lines on the screen because my lighting is having some issues but um that's just a really temporary issue and this is a live switch so i have my switch controllers here and i can actually go in and load up a game if i want to uh oh there's an update of course let's go and update that uh yeah let's close it and update so this is actually my live switch if i hit l we can see that i have a nice little interface for the switch here i really need to fix those lines though because that's really annoying how that looks like that let's wait for it to reload thanks for dropping by by the way there we go so it's reloaded uh let's connect my controllers again load up animal crossing and see if it loads this time and this is all streaming in from an elgato game capture so it's actually able to take any kind of hdmi device so i could plug in my xbox i could plug in my switch ps5 or even my computer which i do often for zoom calls so i can have a guest on the screen in the back and myself here in the studio so that works really well and we can see that the screen is actually duplicated all around the scene so we can actually look at the same scene up here and it's very very high quality um if we zoom in it is a full 1080p resolution screen in here so if we zoom in we're getting all the definition that we normally would get from the game if it were actually on that screen so yeah we can see that it's on all the screens in the studio including this little one down here and these ones up here and i can control it with my gamepad the way i got that working and i'm going to combine that with how i got my camera working in my little desk set up here and you'll notice that any direction we go in the camera turns so that i'm always facing the camera i'm running into things the plane turns so that i'm always facing the camera and no matter where i move it'll still be appropriately facing the camera which is really useful so let's exit this and let's take a look at how it's actually done if we go into any of these textures i'll go into this texture first because it's the most difficult i'll open that up and you'll see that we have a texture material and the texture sample is actually a media player file and the media player file i'll show you how to make but it's essentially a direct link to our webcam and then it takes that and applies a chroma key to it to get rid of the green screen i have painted my wall in the back uh green so the green screen actually gets taken out using this chroma key alpha here so the chroma color is green which we can adjust to make it more closely that green but chroma key does a pretty good job at understanding the range and then um the image color goes in here and we output the raw comparison to the opacity mask and then rgb to the emissive color and that makes our screen actually emissive so that if we had something in front of it whatever color was on the screen would actually reflect off of the other object as well um so yeah that's a pretty simple media material let's go ahead and take a look at the the media file so this is essentially a playlist of one media player and it's webcam mp2021 that's just what i happen to name it if we open up that media player we can choose so let's find it here i'm using snapcam player that's an awful that's not it and then what you do is you change to your chosen webcam so i'm using snapcam there it is and then um we just save this under the playlist name once you choose your video source it should look fine but you can go to playback options and change video to choose different resolutions most webcams will have 720 1080 all the different choices and you can also change the video track um and it's the same kind of deal here so you have to find the right track and the right video source and make a note of those numbers if i go to my other webcam there's actually like 40 or 50 different ones to choose from like 1080p at 60 frames a second 1080p at 30 1080p at 15. um 720p at 30 frames a second 720p at 15 frames a second so it's just there's a bunch of different ones um so you basically choose this as a media source and that's how you get the webcam in now you notice i didn't choose my actual camera my camera here i actually chose snap camera and what that lets me do is uh let's go ahead and load back into the scene so to add a an added layer of immersion making it seem like i'm really 3d um i added snap camera support so i can do things like this and add a helmet on me so it's still live but i have a helmet on same thing with like a little glitch texture i can make myself look uh like a cowboy a knight this here ah a demon so it adds more to the illusion of being in a full 3d space so that's why you snap camera instead of the direct camera feed for you hello hello thank you so much for dropping in i appreciate that so yeah that's how this texture is made and then i have a blueprint attached to it um let's go ahead and edit the blueprint and this just rotates the plane to point towards the camera so we have our set actor rotation and then we get the location of our plane we get our camera we get the location of that camera and then we find the look rotation using the find look rotation function and then we break that into the x y and z directions and um we go to make rotator and set the axle rotation to the new rotator the reason we have this 90 degrees here is because it might not be directly facing the plane so for mine i added 90 to the rotation so if it's this way or backwards you can add 90 to turn it or 180 to turn it all the way around so that's what i did here and then um we set the new rotation and that causes the plane to follow once i load back in come on there we go so no matter what direction i look at myself in it rotates to face the camera and this will work with any camera that is the current camera so if i change to this one it's still facing the camera if i change to this one it's still facing the camera i could point and it's always facing the camera no matter where i go so that's how i get that rotated and then we use a similar technique for the uh video screens in the back um if we open up the material i just don't have a chroma key on it i just have straight texture sample to the media file so we open up the media file and we open up the media player and we choose where that input's coming from in this case we go to video we choose our game capture hd and there we have our video file this is where we have some playback options to choose from uh we have video it goes from 1080p all the way down to uh let's see i guess it's just 1080 but we could choose all these different uh frames per seconds between 60 and one so i'll choose 60 there and then under video frame there's two different frames we can choose between them to see the difference they look pretty similar so we'll leave that and then you just save it and that'll apply the material to those video screens in order to get them to play in game there is a level blueprint blueprint you need to add so in our level blueprints i have the beginplay and initialize webcams function so here we use open source and we choose our webcam and uh we choose our webcam media player we delay it three seconds because it actually takes a second to open the media a webcam source so after three seconds it sets the video track we set the frame rate to 30 and we set the track to zero and format to zero because in snapcam zero is the default then we do the same thing for the game capture device we open source we make a new variable for our media player and then we set the video track to 60 and the frame index and track index to zero once those two are started they load up the source in-game and it works on any object that the material is applied to and that's about it that's how you do the cameras in unreal engine and get them to pull from live sources including webcams and game game capture devices including other screens if you have them connected to a game capture device i'm going to continue this series in the future with information on how some of the other effects that i've added to the to the project have been created uh but yeah thank you guys so much for watching and have a great day what you've never heard of stream savers and you thought pewdiepie was the only youtuber to make a game [Music] i made a game too and it's called stream savers it's available for pre-order right now for 9.99 and that's a great price

Channel: MikeTheTech

Views: 20,800

Keywords: mikethetech, unreal engine, virtual production, virtual camera, virtual set, unreal engine 5, virtual cinematography, unreal engine 4, virtual studio, mixed reality, real time, camera tracking, game dev, green screen, realtime compositing

Id: Hv5yZNZ3bjg

Length: 12min 45sec (765 seconds)

Published: Fri Dec 10 2021