UE5 Guide: Blackmagic DeckLink Video IO, Genlock, Timecode, Vive Tracking, & Live Compositing

okay this video is kind of for reference for anybody who needs to know how to get from xero to getting video gen lock time code motion tracking and camera calibration all working inside of unreal five so we got tanner in the background [Music] can launch unreal engine from the epic games launcher and it does its thing um if you go to film video and live events and choose blank uh basically all these different presets do is it loads you in with different um assets right off the bat so different meshes and it'll also um activate different plugins that you need so blank from film video and live events is the closest one that i found to having all of the different versions or all of the different plugins that you need built right now you don't need starter content and you don't need ray tracing you can always turn that on later i have a couple different windows here pulled up like you probably won't see live link gen lock or time code provider initially when you boot up you can find those by going into window and then virtual production and that's where all these live cool steam vr booted back up so now what you got to do is you have to make sure that all the plugins that you need are activated so you're going to go to edit plugins and just for reference the different ones that you need to have active to make all this work is composure steamvr blackmagic media player live link livelink xr openxr media framework utilities media i o framework and cal camera calibration and i think the only ones that aren't active when you start this are live link xr and camera calibration yes there's that one yes it's going to make you restart your engine and camera calibration hit restart and also you can minimize this it's not important exit out of this so in the real world the signal flow that we have going right now is you have the blackmagic ursa mini 12k going out via sdi into the blackmagic video card inside of our desktop we're using the blackmagic decklink 8k pro so how that works is it has four different sdi connectors and you can configure those to be inputs or outputs or basically accept or receive any sort of like video coming in at a different setting so i have three and four disabled so you can see that right here if you open up the blackmagic desktop video setup um in the first one you can see the videos coming in from the ursa 12k at 2160 p 30 frames and then on the second one i have a sdi cable going out into an sdi to hdmi converter which is then going into the atomos inferno monitor which is how you're going to see your final composite after you've like keyed everything and have your i o all set up so that's going at 1080p 30 frames so that's good you can confirm you're getting video signal into your computer into the card by opening up opening up media express and going to login capture and you can see right now that's the real time video feed you can see the levels peaking over here but that's good for reference you didn't actually need that right now so you can exit out of that okay so to get started you're gonna have to pull in your video data into unreal engine so to do that press control space bar to open up your content window right click go to media media bundle i'm just going to call this black magic design underscore video input double click on that and here's going to be all of your details for all of your camera settings and it's really really important that the output settings on your camera match the input settings in unreal 100 or it's not going to work at all um and you'll spend a lot of time pulling your hair out trying to figure out what's wrong when it's just your settings aren't aligned so go to media bundle choose black magic media source twirl this down go to black uh configuration and then it's our we know our video is coming in the dec link 8k pro on the first uh card it's coming in at 4k ultra hd it's progressive at 30 frames and that's all set up inside of the camera as well time code format we're using vipc video input time code um and i believe actually one more thing go down to video this is 10 bit yuv and it is srgb so we're going to save that exit out of this so now if you go back into your content drawer pressing control space bar and take this video input and pull it into your scene you should be able to see a live feed of what's actually happening inside so if i go and wave my hand around you'll be able to see it cool so that works i found i don't know if this is an engine thing or what exactly the deal is here but i've noticed that you have to keep this in your scene for your video to remain active which there's probably a workaround around that but as far as i'm aware it doesn't get in the way it doesn't affect performance or anything so i really don't care about it cool okay so now we have video confirmed in here now we want to set up time code and gen lock so to set up your time code what you're going to do is you have to make a new blueprint class go to your content drawer right click go to blueprint class and you're going to twirl down all classes type in black custom time code or blackmagic timecode provider who is called sdi input select that i'm going to call this blackmagic design custom time code that'll be good double click on that and you'll see the exact same settings as you saw in your video input i o pop up right here so decline pro you're going to want to use the exact same settings that are coming out of your camera again so 4k ultra hd progressive 30 frames time code format vitc that's good compile save x out of that and you're going to want to do that again for gen lock so blueprint class blackmagic custom time step now unreal engine calls gen lock time step which essentially all it really is is the frames inside of unreal engine and the frames on your camera are firing at the exact same time gen lock and time step are technically different things but they work the same way for our use case so it doesn't really matter rename that [Applause] and you're going to go into here do the exact same thing alter hd everything's all good compile save and then what you're going to do if you don't have these windows right here time code provider and gen lock if you go into window virtual production you'll see them pop up right here and you can just click on it from that so this is not actually accurate right now what you have to do is you have to go into your project settings let's do time code first so type in time code it'll pop up change your time code provider from none to the blueprint that you just created uh blackmagic is on custom time code change this to whatever your frame rate is and this one is 30 frames and you will see it pop up on here blackmagic custom time code 30 frames um this is the absolute time right here so 9 30 38 there's actually an led display on the camera that will show you this exact same number so if these don't match then you have a problem but these do match so you know that your camera and unreal engine and your black magic card are all reading from the exact same absolute time code so to get gen lock in here go to edit project settings search for time step not gen lock time step twirl this down go find the blueprint you just created should pop up automatically sometimes you'll have to re-initialize this but you can see right now it's online and it's firing so that means gen lock and time code are set up oops so now to get video out of unreal engine actually no we're going to do first is we're going to create our virtual camera that is going to be eventually mirrored of our physical camera so you're going to go to add to project cinematic cinema camera actor and you can just plop this wherever and i already know right now that like a couple of settings that are on the camera so the current focal length is 20 apertures 2.8 custom sensor size so i know that the sensor on this camera is 27.03 millimeters by 14.25 millimeters so you're going to plug that in so your sensor size matches your aspect ratio all this stuff you can get into it later not super important for the time being until you calibrate all your camera settings i only change that to 2.8 just because i know that that's what my current aperture is and that's what my minimum aperture is on the specific lens which is the dzo picture zoom 20 to 55 2.8 so let's see what am i doing to get this cinema camera viewport and to get this blackmagic uh media import to be showing you the same thing you're gonna have to use a plug-in called composure which will come automatically activated if you chose the same starting setup that i did so go to create new comp uh empty comp shot and then you can rename this this should show up it's a little bigger close vp for virtual production composure this also creates a physical object inside of your scene so it can be referenced later like that's where it is right now but where it is technically doesn't really matter so you're going to click on your composite hit create or add layer go to media plate twirl this down click on your media plate and you're going to need to go to your inputs scroll down input go to media source and you should see your blackmagic video input right here and that's a live feed from that right now so that's good and then what you're going to want to do is go to your composure hit add new l or add layer element again hit cg layer and it automatically pulls the viewport from the only camera that's in the scene if that is not the case for you you can go to auto receive input and router but you can go to see where is it oh yeah uh composure input inputs right now it's inherited but if you wanted to override it you'll you can hit override and then you'll find any of your camera actors so there it is right there it's the exact same way to do it so if you go back to your media plate we're going to want to key out this green screen and this is just going to be a really sloppy key for the time being you're going to go to composure transform compositing passes twirl down transform passes go to chroma keying and then if you click on key colors hit this plus um and you can't actually click on this so what you're going to want to do is click on your media plate press p not p click on your media plate here press p and it'll pull up this window that'll stay in your viewport so going back to chroma keying if you click on the color hit the eyedropper tool and then click anywhere inside of this all this black that's uh where your key is going to be active so you can exit out of that hit okay and obviously you can dial that in more but for the sake of actually doing what we're doing right now this is what we're going to have to do i'm just gonna save that so now you don't actually see anything inside of your composure composite so what you have to do is you have to write a little bit of script to get these two elements to talk to each other so you're gonna open up your content drawer go to material i'm going to call this vp composure material double click on that to open up the material editing window and you're going to want a 2d uh texture sampler parameter 2d you want two of these you're going to want to name them exactly what your elements are called inside of your composite so this one's going to be media plate 1 hit ctrl d to duplicate we'll call this one cg underscore element one hit enter and then you're going to want to use a function that comes with composure called over and it's taking essentially the rgba of both of these plates and uh putting one over the other essentially putting a over b so this only works with uh like constant four vectors so you have to use an rgba and if you have just an rgb it's not going to work so wire those up this is getting processed as a post process volume so you don't really need to worry about a lot of these if you click on this node go over to material domain go to post process the only option you have is a missive color so you're going to take your rgba and plug it into a missive color hit save exit out of that so now in your compositing you're going to go to your vp composure go down to input wherever that is yeah click the plus button media texture twirl this down media source this is going to be the texture that you just actually know it's a material it's going to be the material you just made so you're going to find your vp composure material plug it in and it's not working oh god so let's figure out what the deal is one going into a missive color so let's see is our video still active okay you're still getting video which is good should i oh i know what i did there we go you don't want to use input you want to use transform passes and then the button for material pops up put in there obviously you can see i still have some transparency coming through my key because i literally did it in one click but you can go in and dial that in if you wanted to by going to your media plate going down to your key and starting to adjust some of the parameters click on your composure you can press p to pull up your preview of this you can see what you're outputting so go down to material parameters and adjusting your black and white clip usually helps dial it in just a little bit there we go it's a little less opaque now or a little more opaque less i don't know but it's working so now you have your video inside of your scene um now let's talk about motion tracking all right so to get motion tracking to work inside of unreal there are a couple things you have to do so we're using the htc vive pro 2 it has two controllers and then we also have a camera or just a tracker that you can use for essentially anything that you want to have tracked and that's really nice because it has a quarter inch thread at the bottom so it's really easy to screw onto the top of the camera to set that up first you have to have steamvr installed which we already do so you can see that right now the base stations are active if i go and turn on the camera tracker you'll see that pop up here too cool so it's right there it's active sometimes things like to fall asleep so if you're wondering why you're not getting any response from any of your tracked items go to your steamvr panel and just make sure that they're online so we can close that so what you're going to do is you're going to open up live link and what this is it's the panel where you can add in tracked elements so we're going to go to source live xr source controllers and then you have hmds which is head mounted displays so check both of those you really don't have to check this one but for the sake of this we will so you'll see the head mounted display right there which is always turned on when you're when this headset is plugged in and the steamvr tracker so we uh unreal can see both of those so now we have to pair that up to the actual camera so down here under the details panel uh if you go to your cinema camera actor make sure you have this selected click add and then you're going to want to go to live link controller so now that is a part of that and then subject representation go to vr tracker so that just moved the camera in real space probably about like three meters or i guess digitally about three meters so we want to make sure that we have a way to control this that's not necessarily just physically moving the camera around the scene in real life because that is a little impractical you can see right now that this is paired because if i go to move the camera see right now i'm just panning left and right on tripod you can see that that same movement is reflected inside of the engine you can do the same thing with tilt too so let me go to perspective left you can see the camera right there and you can see it tilting on its axis so in order to be able to move this camera around in the 3d environment you need to parent it to something because right now if you tried to move it oh wait that was the wrong one i had selected um right now if you tried to move the virtual camera it would not want to move with you see it just keeps snapping back into place so what you're going to want to do is you're going to want to go into your scene outlier or scene outliner you're going to want to create a new shape you can this can do whatever shape i'll just do a sphere and i'm just going to name this vp camera underscore anchor and the way this is going to work is pretty similar to how null objects work inside of after effects this is going to be invisible but it's going to control the superposition of all of the assets you parent to it which in this case are the camera tracker and the camera itself so what you're going to do is you're going to take this camera where is it where did that sit in the camera actor yeah here's your cinema camera actor you're going to take that and you're going to parent it to or make it a child of the anchor so that's going to move it around a lot but basically now whenever you move this sphere that's going to be reflected in the camera movement in your 3d world so since we don't want a big sphere moving around and casting shadows everywhere you can turn off the visibility so go to camera actor search for visibility turn that off and now basically you're just going to use this control right here to move around the position of your camera in 3d space all right so now to get your mixed composite from unreal back into your camera you are going to want to go to your composure settings go to output add an array element capture output oh and that's actually because we need to make an asset that is your black magic output so you're going to press control spacebar go into here make see media black magic media output we'll call this bmd video output yes double click on that i'm going to configure this now we're using the output not the input so we want to make this decklink 8k pro 2 and this is coming out of hd 1080p 30 frames we're using external time code because it's running off of the card not the engine hit apply i'm going to change this back to actually it's coming out as 8-bit but it is okay yeah 8-bit the itc save exit out of that and now this will pop up inside of here we go to search for it so if you look at your camera in real life you'll be able to see this window reflected inside of whatever external monitor you're using with motion tracking and any other things you have applied to that let's see yeah okay that works

Channel: Bennett Win

Views: 16,595

Keywords:

Id: NvcFtDYupfc

Length: 25min 38sec (1538 seconds)

Published: Mon Jun 20 2022