Mixed Reality Masterclass | Episode 1 - LIV Basics

[Music] hi my name is david also known as togayvr and welcome to episode 1 of the mixed reality masterclass this is the first part of a tutorial series that will take an in-depth look at the advanced techniques you need for professional mixed reality production even from home we will cover everything from building a studio and camera rig to shooting with a moving camera to manually compositing mixed reality footage and then how to do all of it with an oculus quest these are all techniques that i have developed and learnt from the mr community over the past couple of years producing mixed reality content on youtube and twitch as well as in mixed reality trailers on the steam and oculus stores for games such as synth riders oh shape and shooty skies but before we dive into all of that we're going to take a look at the basics of high quality mixed reality capture using live this will have a focus on a more lower budget more entry-level approach and lay the groundwork for future episodes if you're experienced with mixed reality this will cover some familiar territory but do stick around as you may learn something new from my approach if you're new to mixed reality then this video is definitely for you welcome to mixed reality masterclass episode 1 the basics [Music] to get started with mixed reality you're going to need a green screen the easiest way to achieve this is using green muslin sheet this can be bought quite cheaply somewhere like ebay or amazon a common size is three by six meters which will be enough to cover a wall and the floor for a better result you can combine two of these to cover a corner of your room most of these sheets can be bought with a standing frame to hang them from but you can also use curtain rails thumbtacks or tape to mount it just be sure to secure the sheet in such a way to minimize any wrinkles as this can create keying problems for you later depending on the floor underneath your green screen you may need to consider a different solution while the muslin sheet works great as a floor over carpet if you're on a slippery surface like tiles or wood you may need to consider something like these green foam tiles which are less likely to slide under your feet if you're a little more serious you can also paint your play space in chroma key green though this is something i'll go into detail on in the next episode so for now we'll just focus on a more basic setup next you'll need to consider your lighting you need to ensure you have bright and consistent lighting across both the green screen and the subject you the player it needs to be arranged in such a way to minimize the chance of shadows which can create keying problems for a long time i just used a pair of soft boxes which did a more than decent job instead of soft boxes you could try using a couple of large led panels or even have dedicated green screen lighting using led strips or tubes there's many different ways to approach this it's just a matter of finding what works for your setup and budget onto your camera you can use anything from a webcam to action cameras mirrorless cameras or even a cinema camera but in most cases something with a wide angle lens will give you the best result for mixed reality longer focal lengths may not capture all of the action and be quite difficult to watch looking at you linus a good entry point that i often recommend would be a gopro which has a wide field of view and a linear mode which effectively cancels the distortion often associated with wide-angle lenses certain gopro models such as the gopro hero5 black can output 1080p at 60fps over hdmi which is ideal for mixed reality it can also be powered over usb and run indefinitely combine the gopro with an elgato capture card such as the hd60 or cam link for a lower budget option you can also use conventional webcams such as the logitech c920 and still get an ok result a step up from the gopro would be to use a mirrorless camera for this tutorial i will be demonstrating with a panasonic gh5 with leica 8 to 18 millimeter f 2.8 lens i won't go into too much detail on that in this video as the next master class episode will feature a complete breakdown of my camera rig build so subscribe to this channel if you don't want to miss that to mount the camera you'll need something stable enough to never move unless you really want it to i used to wall mount my gopro using an arm from ram mount which was very solid you can also use the gopro adhesive mounts which work pretty well or a tripod whatever you use keep in mind that anytime the camera moves it will require recalibration in live when mounting your camera you need to think ahead about how you want to frame your shot to assist with this we're going to have a look at the camera output in obs we'll be using obs in every episode of the master class series for streaming and recording our video and if you're not already using it i'll leave a link in the description once you have created a scene in obs we're going to go to settings and then video to set our resolution and frame rate this should match the output settings of your camera typically 1920x1080 at 60fps add a new video capture device source to capture your camera and hit ok you want to make best use of the available frame to show the game and you in it and the best approach to this will vary between games in some cases simply pointing the camera at the center of the playspace will work great but for titles such as beatsaber or synthriders this can often obscure the view and make it hard to follow the gameplay to assist with framing we can use the rule of thirds as a rough guide in the description of this video i've left a link to a rule of thirds image overlay you can use as an image source in obs for now if you position the center of the play space on one third then later on we can adjust the play space itself to place the incoming notes on the other third in terms of height positioning the camera at or above your eye level and down slightly will often produce the best result for titles like pistol whip you need a different angle as the action is happening across a much wider area and much higher in general for pistol whip you want the camera to be more central to avoid walls and to be aimed upwards to catch all the action it's also best with pistol whip to avoid showing your feet sliding across the ground which can be immersion breaking once you're satisfied with your camera position and framing be sure to disable the image overlay and video capture device source in obs as it will be needed elsewhere in a moment try to remember that all of this advice on framing is just a guide to what i consider to be best practice for showing pure gameplay not anything creative if you have something more creative in mind then i encourage you to experiment with framing and try different things in a future masterclass episode we'll explore this more artistic approach in a lot more detail now that we've got our green screen lighting and camera sorted it's time to move on to live the software we'll be using for mixed reality capture live is available for free from the steam store and once downloaded and installed the first thing you want to do is install the live driver once this is done you're ready to launch steamvr and then the live compositor the first thing we're going to do is set our output settings this will give us the window that obs will capture to stream or record and also the window will be looking at during setup if you have multiple monitors choose the monitor you want to display it on lock the position then set the resolution and frame rate in most cases the output resolution should be 1920x1080 at 60 frames per second now from here we're going to jump right to setting up obs to capture this window this may seem out of order but for me this is a necessary step as my live output monitor is attached to my camera rig and is not always visible during live setup so we need to look at the obs monitor firstly we're going to go back to settings and then video to check out resolution and frame rate matches the output settings set earlier in live typically 1080p 60. this is likely the same as it was earlier while checking the camera output we'll also set our obs output settings while we're here this will vary on the capability of your hardware but i'll show you the settings that i use and you can adjust this to suit for streaming i use simple output mode with the nvidia encoder at 7500k bitrate and the quality encoder preset for recording i prefer to use cpu encoding which gives a higher quality result than the nvidia encoder in the advanced output mode set a recording path then recording format to mp4 and encoder to x264 for best performance and quality i use crf rate control with cpu usage set to ultra fast and profile to high these settings will produce some very large file sizes but will have minimal impact on your system performance while recording and be very high quality to capture the live output window we're going to add a new game capture source give it a name and hit ok set mode to capture specific window then choose the live app output window and hit ok to capture audio add a new audio output capture source and then select the audio output device you use for vr back to live now and we're going to create our camera profile in the camera tab hit the add camera icon change type to video camera and then choose your camera or capture card from the device drop down in mode we'll choose the resolution and frame rate from our camera which again should be 1080p60 if possible use a yui 2 mode as this will give you the best result but if that's not available with your hardware any of the other modes will be fine lastly enter a name for your profile and hit save by now you should be able to see your camera in the live output window and in obs before we move on to keying we're going to turn the live viewfinder on set a fairly large size like 2 meters and also opacity all the way to one then in the steamvr play area settings we'll choose a white background later this will help us set an accurate key from within the headset now back into the camera profile and to the keying tab we'll start by setting the threshold to zero so nothing is being keyed out depending on your green screen lighting and camera setup you may have fifty shades of green so to accurately key this you need to choose a middle green in order to have the tightest tolerance possible in a perfect world your camera would see a perfect 255 green with little tolerance required however in the real world this is never the case if you do leave the color at the default 255 green it will require a much higher tolerance value which will lead to keying issues especially around shadows or in areas of heavy spill this is usually most prevalent around your feet where there are often shadows and green reflecting onto your legs in the color picker we're going to use this tool over the camera image in the live output or obs window to pick a middle green or whatever you think the most common green color is in your camera image we're then going to remove all of the red and blue from this color by setting these to zero now we'll start to increase the tolerance and you should see the green start to disappear once you've got this roughly okay put the headset on and we'll continue the keying process from the middle of your playspace with the live viewfinder against the white background we said earlier we'll be able to easily see any shadows or rough edges open the steamvr desktop view to the side of the viewfinder and tweak the threshold and green color until you have clean edges with nothing disappearing that shouldn't be i find the best indication of an accurate key will be to look around your feet as it's usually the area most prone to shadows and spill moving your legs in different poses will highlight any issues you may have if you have any really troublesome shadows you can try adding a tiny amount of blue into your key color maybe five or ten which may help key these out though use this sparingly lastly we'll set the smoothness value to give us the cleanest edges possible i find the default smoothness value makes the edges too soft so decrease this value to your liking but not so low that you have jagged edges in most green screen applications we would now be talking about d-spill or removing the green border or reflections that spill onto the subject this is an important consideration when using a green screen as that green tinge you often get just looks bad especially if you have a lot of green like the room behind me unfortunately d-spill is not a function that is currently built into live or is it in the advanced tab there is an option to use a lut a lut or lookup table is essentially a way of replacing one colour with another it is commonly used in colour correction or grading but in this context we'll be using it to replace all of those green tones with more natural colours in the description of this video i have left a download link to a dspell lut that i use feel free to try this lut with your own setup once you have this lut enabled in live set the amount to 1 and check to see how it affects the color of your image you should find that the green spill is now gone but also check that it has not adversely affected other colors especially yellow if so decrease the lut value until you're satisfied you have natural looking color next we'll look at cropping or masking unwanted areas of the camera image live has a crop function which can bring in the edges of the camera image and while this is quick and easy to use the result is usually imprecise a better method is to use the static mask function in the advanced tab this is a much more precise method and will ensure that none of your green screen area is wasted a static mask is basically an image file that tells liv what parts of the camera image to keep and the parts to discard the white areas live will keep and the black areas will be discarded to create a static mask we'll start by selecting the live app output window and pressing alt print screen this will take a screenshot of just this window paste this screenshot into the image editing software of your choice for this video i'll be demonstrating with as it is free software and more than capable of this task using the free select tool make a border inside the keyed area allowing a little tolerance in case of slight camera or green screen movement then we will bucket fill this area with white to indicate the part of the camera image we want to keep next go to select then invert we will bucket fill the outside area with black now the static mask is complete we can export the image as a png then back in the live camera profile go to the advanced tab and open the image as your static mask you should find all of the unwanted areas of the camera image have now been masked out leaving only the green screen area the next step is to calibrate your camera with live in the calibration tab start by selecting your camera tracker if you have one having a tracked camera can allow you to move the camera without the need to recalibrate we'll be covering this topic in great detail in a future masterclass episode so definitely subscribe if you don't want to miss that for the purpose of this video though we'll stick to having a static camera now hit the begin calibration button and put your headset on you should now see a large mirrored viewfinder that follows your head when you move underneath this hit the begin calibration button to continue the first calibration point is in front of the camera lens i'll be demonstrating this process with the htc vive ones which use the center of the tracking ring as the calibration point most other controllers will use the trigger as the calibration point so just follow the on-screen instructions that liv gives you i find the camera lens point is easier to calibrate if you pop the headset off so you can get the controller as close to the lens as possible while also looking at your monitor to make sure the controller is centered before pulling the trigger for the next two calibration points pop the headset back on and go about two or three meters away from your camera just pay attention to the measurement at the bottom of the display keep this in the green when calibrating these points be sure to position yourself and the controller so that there are no tracking issues so make sure the control has clear line of sight to the lighthouses or in the case of quest or rift s make sure you are looking at the controller with the second point calibrated you should now see the virtual controller lining up with the real world controller at this point you may notice a timing difference between the virtual and real this can be corrected in latency however setting the latency here during calibration and not while in game is an unreliable method and usually results in an inaccurate setting so we'll be setting the latency through a different method a little bit later after the calibration process you may find that you need to tweak the position slightly to get the controllers to line up you can do this in the position tab by pulling the x y and z sliders until everything lines up i would recommend checking this with the controllers in various positions and distances from the camera to get it as close as possible throughout the play space once you're done hit save and we'll continue with calibration complete we're now ready to see how this all looks in game over to the capture tab you'll see there are two methods auto and manual under auto you should have a list of installed games that have live support check the target resolution matches your output which is probably 1080p it is possible to increase the target resolution to 1440p or even 4k and while the output is still 1080p it will be rendered at the higher resolution which will have an anti-aliasing effect for higher quality this does impact performance though so only do this if your hardware is up to the task once you have chosen the game and resolution hit sync and launch and the game will start you should now also see the mixed reality view in the live output if your game did not appear in the list under auto but you know that it has live support you can use manual mode firstly hit export calibration and locate the game directory this should be the same location as the game executable then save the config file for the effect you should set this to sdk normal for most games we'll go over some of these other modes in a future video now you can launch the game separately through steam and then in live hit the target drop down and choose the game executable and that's manual mode done just keep in mind that anytime you change the calibration you will need to perform the export calibration step again to update your game remember earlier when we skipped setting the latency during calibration well now that we're in game we can set the latency more accurately open the steamvr desktop view next to the live viewfinder then go back into your camera profile and so calibration by moving the controllers up and down you can see that our real world camera is behind the virtual camera this can be corrected by adding frames of latency to the virtual camera hitting the plus and minus icons will adjust the latency by an entire frame which at 60fps is approximately 17 milliseconds for my camera and capture card the required latency is 7 frames or around 117 milliseconds but this will likely be different for you once you can see in the viewfinder that everything is in sync you can save and close the camera profile because we've now added latency to the video we also need to add the same latency to the audio so over in obs hit the cog on your audio output capture and go to advanced audio properties set the sync offset to match the latency that was set earlier in live if you have a microphone you will want to set the latency here also the last thing i want to talk about is a few finishing touches to get your shot just right earlier when we were mounting the camera i talked about using the rule of thirds as a guide for framing this is useful for titles like beatsaber to give you the best view of the action without wasting too much of the available frame if you framed the center of your play space around one of the thirds the playspace can be rotated to place the incoming notes around the other third this is done using a tool called openvr advanced settings i'll leave a link to this tool in the description once downloaded and installed advanced settings will appear in the steamvr dashboard in the offsets tab the rotation and position can be adjusted i like to do this while standing in the center of the play space with the map running in no fail mode adjust the rotation and position to your liking and hit apply though be aware that advanced settings will automatically adjust the position whenever you apply the rotation so you may need to set these back to zero it is also possible with lib to increase the field of view beyond what your camera is capable of in the camera profile under calibration there is a field of view override function which can be used to achieve a wider field of view if you're not sure what your camera's field of view is you can find it if you click begin calibration and then look at the field of view tab i would recommend that the field of view override function be used sparingly maybe only 10 or 20 degrees beyond your camera field of view so maybe 70 or 80 degrees max too high of a setting will result in a distorted and unnatural image as i said earlier in this video all of this is just a guide to what i consider to be best practice for framing pure gameplay there are no rules here in fact one of the aims of the mixed reality master class is to encourage you to think of mixed reality as something beyond showing just gameplay but rather as just another tool in your filmmaking arsenal over the coming episodes i'll be showing how you can take all of this to the next level i'll show you how to build a tracked camera rig how to shoot with it and how to manually composite mr footage everything you need to unlock the capabilities of mixed reality as a filmmaking tool subscribe to this channel if you haven't already and consider turning notifications on to get notified when the next episode is live if you have any questions about mixed reality you can leave them in the comments below or hit me up on discord by following the link in the description thank you so much for watching i'll see you next time [Music] oh yeah

Channel: TougeVR

Views: 49,594

Keywords: mixed reality, liv mixed reality setup, liv, vr, virtual reality, green screen, oculus quest, oculus quest 2, mixed reality masterclass, liv tutorial, liv mixed reality, liv mixed reality no green screen, liv mixed reality oculus quest, liv mixed reality android, beat saber, synth riders, ohshape, audica, pistol whip, vr gaming, liv v2, oculus mrc, beat saber mixed reality, tougevr, togue, touge, how to do mixed reality, calibration, camera, filmmaker, filmmaking, panasonic gh5, gopro

Id: jBLl6n7tRQE

Length: 25min 5sec (1505 seconds)

Published: Sun Oct 11 2020