Introduction to Creating Virtual Sets

you hello everyone and welcome back to streaming alchemy on today's show we're going to be talking about how to build virtual sets how you can actually take and look at the different components that go into it assemble them and present that as a single composited view for a as a basically the set of your imagination anything you'd like to put together in the show we're going to cover and I'm gonna start with sort of some of the basics basically looking at multi box effects and how you'd put those together using V mix we're also going to look at key and sort of that that single layer keyed talent over a backdrop we're going to look at something that's really critical to understanding how to do complex virtual sets which are called UV maps or UV gradients so we'll go into detail about how that plays a role and then we're going to look at building a complete virtual set in side of the mix and then also look at building that a similar set inside of the TriCaster and there will be some interesting things we can show to end up that add a lot of realism to the set you crate and one of these things is specifically on the TriCaster so I'm excited to to show you that so I want to remind everybody that as usual we will be taking video colons so if you'd like to join us on Aer ask a question share some of your experiences with virtual sets or your feelings about using virtual sets you know we'd love to have you all so anybody that you know wants to share in the comments we're always looking forward to that so please join the show whatever way is comfortable for you so I started this last week and we're gonna see how it plays out but I'd like each show to be asking a question that will help me understand more about where everyone's interests are in the subject that we're covering so for this week I really wanted to focus on virtual sets so what kind of said would would work best for you in your productions would it be a physical set or would it be a virtual set so if you just type the word physical a word virtual in sort of the community section of whatever channel you're viewing this over YouTube or Facebook that would be that would be great so let's get started and I'm gonna take a look now at at vmix here and show you some of the elements that would go into doing a multi box shot now multi boxes if you you know watch the news they're the most common kind of shot that you see on the air today so typically it's a shot where you have a host and one or several guests and they're all interacting with each other so for this type of shot you're basically dealing with video inputs that are coming in and then a framework that you're compositing these inputs into so in this case if you look we have video coming in to the host we have video of guest 1 and video of guest 2 over here so these are all the elements that are going into creating this the video elements and then we also have a framework that we're putting them in so this allows us to this is where you see any sort of branding or station IDs or any of those bugs and this gives you a nice visual representation for your viewers to understand you know who's involved in the flow of the conversation so the way this is constructed in vmix but really in almost anything is through layering we put these together as a composited set of assets so if I go here where I have the multi box shot already assembled let me click in here the way we do this is through something in vmix called multi-view so in this case we have four layers in this multi view and that's what multiviewer should do stack up layers one on top of the other so at the base we have our framework and that's how we you know sort of set the backdrop and then we have our guests a B and the host now the way this all works is that in this multi box framework in the multi view framework that vmix provides we can actually take and position each one of these assets specifically laid out on the screen so we can control the zoom so we have it zoomed way down in this case we can control the cropping on the sides we can control the panning and positioning so we have the ability to position this whatever way we want in the set and that's true for all of these elements so when we have these together now each of these are just cropped elements of the original assets that are zoomed in scale that are scaled and positioned in the multi box framework but one of the things that you get from this and it's it's it's not specific to virtual sets but something that can be very useful when you use multi boxes there is a transition and V mix called the merge transition and what this merge transition will let you do is take an asset like in this case the host that's already in the on screen element and then merge with that same element that would be coming in as a new video source and you'd see something like this where now that element as opposed to just flipping over or fading in it zooms up and scales out from the position it was in to take on the full screen position it has here and you can again do the same thing to go back we would shrink in and this would work very easily by just having any one of the guests that may be on air or the host that you'd want to bring out and do the same thing now so it's a it's a it's a very simple way to create what is effectively a virtual set a virtual space where you have guests from multiple locations interacting with a host so this is really a good start point for thinking about virtual sets because that's kind of what we're doing here creating that space that you want to interact in so let's take the next look up which is probably the place that most people go when they think about a virtual set nets chroma key so if I come over here let me let me just cut this in and I'll stop that I want it plain and let me show you here so what I have here are two layers that we'd like to composite but now as a chroma key where there's talent in the foreground this gentleman here with the VR headset on and there is imagery in the background that we're going to be using so the first thing that we need to do though is we need to actually key out the talent so if I open this up inside a vmix on this input there's something called color key now color key will let me pick a color that I want to what's called Qiao and Keene effectively is telling your system to take anything that is this specific color that your teen and make it transparent create an alpha channel for that video image in that location so if I come here the way you do this is you pick the dropper and you can go them it'll open up a window and you can go and pick the area that you want you can do any additional dialing you want for the strength of it or you know filtering the edges or how you want to do aliasing all those things are in here too but now what I've done in in this key is make this background transparent and because it's transparent I can actually if I just cut this out here I can actually just layer this on here so the all the elements that we had in the multi box effect we did apply here as well so if you look and run I'm really doing is I'm doing a multi view again I have a foreground in a background object but in this case as opposed to scaling the foreground objects to make them fit I'm actually keying it out but I also had that same ability so if I wanted to come here and let me just reset this he could now actually and I could reset positioning so now what I've actually done is taken the hosts and repositioned them set them all up to to fit on the screen but it's the same basic to what we were using before the multi view component of vmix now combined with chroma key but there's two things to keep in mind when you when you look at chroma keying talent the first is that your backdrop can be anything so it can be in it like we have in this case video which is supposed to represent what he's seen in his VR headset but it could also be a still or it could also be a camera that you have pointed at something so if you wanted to do something where talent is talking in front of you know a video of something that's going on or for video gamers this could be something that they use all the time with they're taking a screen coming in and using that as a way to sort of key themselves over something going on and so this really you know introduces the next new concept which is keen and that's sort of critical to everything that you do sort of with anything that's more sophisticated on the Virtual Set front so let me just stop this and I want to talk about the next thing which is probably one of the critical things that would be a take away from what we talked about today and that is UV gradients so let me see if I can get this thing all set up here and do this here so I'm going to put this up this in and then set this here so this is a UV gradient but what is it it's a it's a composite of different colors that are put into a space but to understand what it does let me take a step back and talk a bit about the types of things you need to do inside a virtual set in virtual sets what we're trying to do is we're trying to simulate in a sophisticated virtual set a 3d environment so this is a space with depth space where you have objects that have shape you know it's three-dimensional so you want to have some way where you can take video or images that will be part of your show part of your set and map them onto specific things that are in that set and that's what gradients do what a gradient is you know for for those who want to sort of dig it that level down it is a basically a grayscale gradient that exists in two of the three color channels it exists in the red Channel and the green Channel but it doesn't exist in the blue Channel the reason you do that is that it gives you a luminance value then for your red and your green in a way now that each pixel in this map has its own coordinate which is basically the value of the red gradient and the value of the green gradient so you have a two-dimensional coordinate system now built into color that you can take an image or a camera feed or any other visual asset and map it so let me just give you a better concept here this is the same gradient on the left though what you have is the gradient and it's sort of rectilinear fashion is just they cut out the same shape sixteen by nine as you'd have from video coming in on a camera on the right though what I've simulated is I simulated taking that flat sixteen by nine image and now curving it so if I had a big wall or something that would be behind a stage this would represent how an image would actually look in that space so let me just take I have a camera right here and I'm just going to take that and I'm gonna map it to these gradients what you see here now are identical video images but in the first case it's just the square mapping pixel for pixel and on the right hand side now it's that curve distorted map so if you if you notice as I go up here you can see my hand is going to the same position you're basically seeing pretty much identical images here it's just perspective wise change to create the illusion of of depth where the part in the middle is further back and the parts on the sides are curved out that's head a little bit of slant sort of coming out from a corner so this is the sort of thing where you could have video plane while a host was in front of this talking so this gives you now not only the ability to take any type of image you want you can take that image and map it to any type of shape you want so gradients effectively give us three dimensions to work with when we bring images in so that's sort of the third leg that we talked about where it's it's compositing and layout it's chroma keying and it's gradients to do 3d positioning so let's put all these things together and start to look at this in constructing a virtual set so let's go over here all right so I'll jump in by by just sort of showing you what it is that we're looking to create and what what's what's available on here so this is a virtual set I put together which combines all of these different things that we talked about so we have a backdrop here which are all these boxes with a spray-painted video game central we have talent which is being keyed in over the backdrop we have which is the first time we've shown this we actually have a foreground object now the desk in this case is positioned in front of the talent so now we've created another step in this compositing process with layering but that's all part of what we saw in the multi view capabilities and V mix in the corner over here we have a video monitor and this video monitor is actually just a scale downside so we just took the video image that we have this one and I think right over here which is plane and we just shrunk it down and fit it into the monitor on the side but in front now or we actually did is we use a gradient to map another image into the front panels of these two boxes that fall in the front of the desk so you can see these are at different angles to each other slightly and it's cut up and split but it's it is just a continuous video it's just split across these two boxes so that is what you see if you look here so these would be the UV maps that we use to map the talent in here and this would be the UV map that we're using to map the desk and so in each of these cases these things are all separate elements and if I come over here this view that you have now is an actual virtual set that we build for 4 V mix so if here in the virtual set there's a setup capability where I can go in and replace every one of the objects with my own objects that I want to put in here so in this case we have the display which is the video that's showing here we have the set background which is these we have the talent the foreground video and the desk the foreground desk and the video on the desk so I can if I want to just through the mixer zone interface change any of these so if I wanted to change what's in the display here I could go and pick another video game clip and so now I have something different or if I wanted to change what's in the desk again very very simply now I could go in here and let's see this video we'll go to one which is a little spooky you're looking but gives you a different video so here I had the ability to just do this type of mapping and the reason I can do that is because I've described the relationship of all these objects to vmix and I did that using let me get over here using an XML file so if let's see okay so this is how you create a virtual set in vmix so in this case now I have an XML that identifies each of these objects names them so I have some way of identifying what it is when you saw down here I said okay what are the different elements that I have these are all the different layers that you see named here so that's how it comes in the other thing is you have the ability inside of the mix to say I want to use a UV map for this image so when you select an image for it what you're telling V makes to do is say take the image I select and map that to the UV map that I told you is part of this set so in this case the display I actually used the UV map but it's basically just zoomed almond positioned on screen so there's nothing you know special about that that would have been easily done with sort of prop in position the set background is a standard image and you can see here now I just have it in there it's god--not zooms got no positioning no rotation but it's using the set background dot PNG file which is something we set up in this virtual set as being an asset that's available for the background for the talent again we're using a UV map and you know this is positioned and rotated slightly to become you know to blend with the angle of the set and this also has the UV map so what I'm telling be makes to do is to take whatever the asset is that is mapped in here in this case this is our little game geek here and map that to the the UV map we have set here and so again I don't have to go through this in complete detail but I think you're understanding the basic framework that's used for creating these types of virtual sets and vmix you describe you all the elements and how they relate you have mapped the UV maps and then you can assign to the UV maps the images you want to put in place so this is really very very powerful now because not only can you describe a set and have all those elements bundled but using UV maps you can also have things that are easy to trip to swap out and blend nicely into your set so if I go through this let me sort of jump in here these were the two components that were really part of this set where I have a foreground component and a background component and when I looked at these things as elements they are they're all flat but what we try to do is in in video so it's often called 2.5 D it's not quite 3d but by doing this type of stacking we have inside the system so let me you know this sort of thing here where I can cut that in and we'll just put here so these are just lay it on top so the desk now has more depth to it because I put something behind it so I just wanted to give you a sense of some of the other elements that we didn't see on the first page but now I reuse TriCaster here for running our productions so I want to spend a little time showing you how you would do a very similar set but using a TriCaster so on the TriCaster what they provide is a framework where you can use a Photoshop file with different layers to describe everything that you're going to be doing inside your set and then they have a utility called virtual set editor that takes that Photoshop file and compiles it into a virtual set that can be installed on the TriCaster so let me show you a couple of things and then I guess to get will just jump to the the end here with this is the set that we put together so slightly different coloration same basic elements though a monitor a split set in the front with a on the desk where we have multiple elements side to side and we have talent sitting over here so let me show you how that would be done inside of Photoshop so we just put this over here okay so if you look at this this is the photo oven actually let me just zoom this in a little bit more here so what I have in this Photoshop file are all the gradients that are now positioned and laid out and I have all the framework elements that we have for the set pieces the foreground desk in the background and I'm trying to make sure you all can see this here but these are the elements inside Photoshop as we have it laid out so I have just a white background the next thing I have is something that I have a special name called input B which is saying in the TriCaster take whatever is on the mix effects bus as input B and map it to this gradient and so that's the gradient we're actually using for the monitor then in front of that I have the set itself with cutout layer for where the monitor shows through so you have that sort of stacking then above that I have the talent which is positioned in the corner and slightly you know tilted and what you'll notice here is that again with this Eileen this now is input a because this is taking what will be on the a channel in the mixer fat and again it's a very logical process of just layered elements that are sitting here so then I put in the desk in front of the towel again trying to create that layout that sense of depth the the sense of 3d depth in it then I actually have the video that gets mapped to the desk and again this is another gradient and we label this as input C so in this case now we have three inputs in this mix effect bus on the TriCaster for the television display talent and the video that's displayed on the front of the desk but your notice here I actually have one other thing up here which is called talent reflection Jessie good to see you thank you for joining us so the so one of the things you can do with gradients that we did really didn't look at on the vmix side as you can take you can make them semi-transparent you can you sort of distort them and position them you can cut them up so what I've done here and I don't know if you can see it in the stream at home but there is a gradiant sitting on the desk that's basically just the 15 percent transparency so very very you know translucent and that allows when the host is moving the talent is moving you now see a reflection of them on the desk using this gradient so you'll notice that I have input a as being the talent reflection as well as input a being the talent so I'm taking the same video source that's coming in and just laying them out now at a at an angle that make gives you the illusion that we have a reflection on the surface and again this adds that next layer of dimensionality to what we're trying to create in the set you can use this with all types of adjustments now to create different types of reflections floor reflections you can darken it to create more shadows so if you wanted to have something to add more of a shadow coming out across the floor you can use gratings to do all of these things and that becomes really really powerful now it creates a true illusion of depth in a space which is essentially just stack flat layers so the other thing though let me switch back to the output that we were using here which is this here top this so if you look at this and I think if if you can start the talent plane just so we have this running live so yep good so now if you look at the desk you should be able to to see things where some of that reflection is showing through and again the goal is to make it subtle because what you don't want to do is something that looks garish the biggest way to give away that you're doing something all virtual is by making it look too shiny - metallic - reflective that's not real life so you want to do things subtly and in that subtle thing you can really get get a lot of benefit and you don't have to hit people over the face with it but there's one other thing I want to show you so there is something that the TriCaster has it's called parallax now parallax is something that we have in real life all the time if I have objects that are at different distances and I'm moving across in front of those objects to my perception the object in the front moves more slowly left-to-right than the object in the back so when I'm on this side of an object I can see behind it and you sort of see the back moving and then when I'm on the other side I can see behind the object from that other angle so creating that that's that's something that you get gives you a real experience of depth and it's used a lot in cinematic shots so if you look at camerawork I mean there are there are things called trucks and dollies which leverage this parallax effects the sense of things moving past each other at different rates now that can be a very effective way of sort of selling a shot and the TriCaster actually lets you do that so if I switch back here and let's call up let's put a camera now in the front of this set so we want to do this illusion here that we have a camera now if I actually taken just pan across here with the virtual camera that's controlling the set if you look in the lower left hand corner look at how that camera is now let's just pick something else will resume side to side somehow so you'll see now that there is a shift in perspective and what I've asked the TriCaster to do is take and create what they call in the TriCaster augmented reality which is just a parallax effect with a certain percentage so I think this is 30% which gives you that sense of gee I can sort of see behind this camera in what's going on so this is an incredibly powerful way to add depth to something in addition to the things we talked about with gradients and reflections so this gives you sort of a complete picture of the technical elements that go into sort of making up a 3d set and some of the tricks you can use to sell it as a realistic set but there are a few things when you look at virtual sets that I think are really important the virtual set that you're using has to tie in with the real-world framing you have in your camera and your lighting so when you create a virtual set I think it's really important to understand the direction of the light that you're gonna have on the talent and a green-screen you have to understand the lens and you know the properties of the lens that you're using to shoot that person and make sure that it lines up fairly well with the properties of the lens used to capture the video or the still that's taken to be used as a backdrop or a for drop we are able to perceive a lot of things as not being right if visually they don't seem when i zoom in or pan across or anything if they don't seem to act like I'd expect it to in real life and it's not that somebody will sit there and go ooh you didn't use the same lens or use different focal I know it's that people just say something isn't right so I think as we sort of create and build these sets we have to think beyond just does this look cool ah - does this look real and so it really comes down to subtlety understanding that the world is imperfect but real consistency when it comes to the visuals that you use from cameras and lighting to make sure the virtual elements and the physical elements all line up in a way that people perceive is natural and normal so I hope this worked for everybody that it was a good run down of you know virtual sets and and how you can use them there's a lot more in this space obviously you know where you talk about them from the creative side I mean we done let me see if I can just show you I mean we've done a lot of work with so I just put something up here on the on the screen here which shows a 3d modelling we did this in Lightwave and we brought it into photoshop so you have the ability not just to limit yourself to real-world things but you can actually build very specific detailed sets using 3d modeling as well so again all these rules apply but these are the types of things as you start to think of it they're a lot more that you can start to to do to integrate virtual reality into your production virtual sets so you know a couple of things though that I want to touch on before we end the show today the first is that I'm gonna be at a conference coming up in New York in November so it's called a stream geek summit and it's gonna be in New York City I believe it's at the the dream dream downtown hotel I believe it's the name of it so it's November 8th in New York City so you can go online to find out more about it I believe they have for a limited time free virtual tickets if you wanted to join virtually and you're not in the area but they also have tickets available but I'm gonna be giving a talk there on how to actually start your own live show some of the things you need to think about and you know plan for to build out your own program so if you love to join I'd love to have you you know come around and join us there if this is something that you'd be be up for doing if nothing else I'd love the chance to meet everybody that's been joining me virtually effectively on on the live stream so that'd be great if if you'd like please find out more there so what we're going to talk about on the next show it's actually a little interesting we're gonna talk a bit about video resolution and I think the key thing especially as streamers to understand is that resolution really isn't just about pixels resolution is about a lot of other elements that composite a video or the composites probably retirement that that make up all the technical elements that go into pulling a video together so we'll go into this in a lot more detail and you know I hope to see all of you next week as well thanks for joining me have a great week see you later

Channel: The Streaming Alchemy Show

Views: 10,245

Rating: 4.7066665 out of 5

Keywords:

Id: uTCehkEdAwo

Length: 34min 0sec (2040 seconds)

Published: Fri Aug 23 2019