Getting Started with Virtual Production | Everything You Need for Machinima, Mocap and Mixed Reality

[Music] hey everyone sunny here for reality redefined and today i'm coming at you in person i think this video is gonna end up being quite long we've got quite a bit to get through so i just thought i might as well record it to camera so today we'll be going through everything you need to get started with virtual production at home we'll be talking about the hardware used for virtual production the software we'll be covering everything from machinimas to motion capture to mixed reality using green screen i'll also be introducing you to some online resources that will allow you to get hold of assets such as 3d models characters environments and animations for your projects now all the products i recommend this video will be linked down in the description below and just so you know i am an amazon associate so if you follow my links to purchase anything on amazon i'll make a little bit of money from it even if you end up buying something completely unrelated to virtual production in the end it won't cost you any extra money and you'll be supporting the channel so i can bring you guys more content so really it's a win-win so this video will primarily focus on getting into virtual production using unreal engine which is the most popular software in the virtual production community and is completely free to use but if you want to use another platform such as unity much of this information will still apply i'm not going to go through how unreal works and assume that if you're watching this you have at least some level of familiarity with the software and its blueprint system but if not there's tons of resources online to help get you started the learn tab in the unreal launcher is the best way to begin as it links to official tutorials and documentation by epic games also we will be going through step by step on how to set up a virtual camera motion capture animations or key green screens for example as it would end up being hours long instead i'm planning to make individual tutorials on these in the future so make sure you subscribe and hit the notification bell so you know when i release them also be sure to check out the timestamps in the description if you want to skip ahead to any specific section of this video it should also be displayed somewhere on the timeline but with that said i'm finally going to stop rambling so we can actually get into it [Music] the primary concept behind virtual production is tracking a physical camera rig and then translating its movement into unreal engine to produce realistic camera moves in your virtual movies there are tons of professional tracking solutions out there but the best consumer level option by far is to track your rig with a vr system the same thing you'd use to play vr games there are two types of vr systems those that have built-in inside out tracking and those that utilize external base stations for tracking inside out tracking uses cameras built into the headset paired with accelerometers and ai to track the position of the head-mounted displaying controllers this means that the headset is required you can't just use a controller alone to provide tracking information i haven't personally used an inside out vr system but i have heard that they tend to be a little less reliable than base station track systems and they also struggle with finer movements but despite these flaws it's still a good option if you operate in a particularly small space or unable or simply unwilling to set up external base stations the most popular inside out track vr system at the moment is the oculus rift s pretty affordable at 400 and it contains everything you need to get started with virtual production straight out the box on the other hand we have base station tracking this vr system utilizes external sensors that send out infrared pulses which are detected by the headset and controllers which provides more accurate tracking than inside out vr systems that utilize this base station method of tracking include the original oculus rift the valve index the htc vive the vive is the most widely used for virtual production particularly the pro version one big selling point of the vive is that you can actually buy an accessory for them known as a tracker which i actually have on my shelf up there it's basically a vr controller but without any buttons and includes a quarter inch thread in the bottom of it the tracker is extremely useful for virtual production as it's more compact than the controller and the thread in the bottom of it means it can be easily mounted to a camera rig mounting the vive controller to a rig is possible but it requires a clamp or some sort of specialized possibly 3d printed adapter but unlike the tracker the fact that controller has buttons means you can easily map functions such as zoom and focus to the buttons which can be very useful if you don't plan on using any other type of interface on your rig i'd personally say if you want to get into virtual production using a base station track vr system and cost doesn't particularly matter to you too much then simply invest in the full vive pro setup everything will work straight out of the box but because the vive utilizes external tracking it is also entirely possible to buy only base stations in a controller or tracker to achieve virtual production the headset isn't 100 necessary like with inside out this is a great option if you're on a tighter budget as it'll save you around a thousand dollars just keep in mind that because the vive is in the end intended for vr and therefore programmed to work with the headset this method may end up presenting some unknown issues down the line and it does require a more complex initial setup process that includes some modification of the steamvr files basically you can get quite technical so i'd recommend doing some research first to see if this is something you think you can do at home before cutting corners [Music] when it comes to constructing the physical camera rig there's a wide range of options out there however i recommend using a modular dslr rig this way you can customize your rig with a wide range of accessories and configure it to your personal liking for example going from handheld to shoulder the modular nature of these rigs also allows you to mount your controller or tracker in a variety of ways using clamps adapters spigots screws and more i just use a newer dslr i got off amazon for around 50 it's fairly sturdy lightweight and it does the job just fine but if you want to spend a little more on a more premium quality setup that will stand the test of time a little bit better i'd say check out smallrig instead on top of your basic rig it's also nice to have an external monitor and while this isn't 100 necessary it is nice to be able to view your footage without having to break your neck to look at your computer's monitor instead if you're using a vive tracker without any mappable buttons another useful accessory is some sort of control interface which connects to unreal engine and allows you to control your virtual camera settings remotely i've seen some people use an xbox controller which works but you do have to kind of faff about with taking the camera rig down picking up the control and moving the camera about taking the rig back on i've seen some people hook up an arduino micro controller to their tracker to give it physical mappable buttons on the rig i personally just use my smartphone as an interface using an app called touch osc oh by the way thanks to doghook for providing the touch osc layout files i personally use at home i'll link to their website below they're basically an open source virtual camera site really good stuff and they should definitely check it out a couple more nice to haves on a virtual camera rig are some sort of battery pack to keep your tracker or controller charged and a wireless hdmi system so you're not constantly wrestling with hdmi cables going from your monitor to your pc i actually really need to get one of these [Music] if you want to get into mixed reality virtual production filming physical props and actors against a real-time rendered virtual environment kind of like on the mandalorian obviously you'll need to add a camera to your setup this can be anything from a webcam to a dslr or even a full-on digital film camera like the blackmagic ursa mini pro but obviously it goes without saying this all depends on your budget the camera i personally use is a sony a7 dslr i achieved camera tracking by simply attaching my vive tracker to the top using a screw however if your tracker doesn't have a screw you could invest in a cage which is a good option anyway as it provides a much sturdier connection or you could even just try mounting it elsewhere on your camera rig just ensure you don't obscure the tracker too much try to keep it elevated above everything else on the rig so that the base stations can see it so you don't lose tracking most dslrs and digital cameras will require a capture card to stream their footage to your computer live some companies like sony actually have the software available that negates the need for a capture card allowing you to hook your camera straight to your pc via usb so check if this is an available option before investing in a capture device unfortunately because my sony a7r is the first generation it doesn't support this feature so i just use an elgato cam link to feed footage to my pc and set coming in at around 120 dollars it's a pretty good budget option it basically turns your dslr into a high quality webcam converting the hdmi feed to usb just make sure your camera is compatible with the cam link before you buy one you can find a list of supported cameras on our ghetto's website if you're looking for more high-end option the blackmagic decklink series of capture cards allow you to input up to 8k resolution footage depending on the model you choose into your computer via sdi instead of usb which is essential if you're using professional video equipment that utilizes this type of connection these cards also allow you to output footage from unreal engine a feature you'll need when trying to achieve in-camera vfx using an led wall or projectors on the subject of led walls and projectors while these are great for compositing the virtual environment into your scene and providing in-camera vfx they aren't feasible options for most of us at home due to the sheer cost time and space it required to set up for indie virtual production live compositing using a green screen is usually the best way to go although it's not always as believable as in camera vfx green screen is massively cheaper and much easier to set up now i was able to borrow a green screen from a local company for my tests but if you want to purchase one there's plenty of options on amazon suitable for different budgets again links in the description to ensure a good key you also want to make sure that your green screen has as little wrinkles and creases as possible and is evenly lit without any harsh shadows and while you can achieve sufficient lighting from just your room light your ambient light in most cases you'll have to invest in some extra lights to ensure the lighting is really smooth once again there's a huge range of lighting setups to choose from out there i won't be going through them all just take a look on amazon do a little research you should be able to find something suited to your budget needs and because most light tends to be quite harsh you may also need to invest in a soft box to diffuse the light and reduce the hardness of shadows and as a final note on green screens all unreal engine alone works just fine for green screen virtual production it's what i've been using so far there are specialized software and unreal plugins out there such as xsymmetry and reality engine which was recently made completely free by the way they're specifically made for this purpose so definitely check them out too they're worth having a look at [Music] obviously you're going to need assets for your projects and if you don't have the skills necessary or a team to make your own in software like blender and maya or if you just can't be bothered painstakingly making hundreds of 3d models and animations and environments there's plenty of online libraries full of content that can be used in unreal one of the best places to get assets for your projects is the unreal marketplace it features a wide range of content including blueprints plugins tools characters animations props environments there's a lot there and while a lot of these do cost money there's an extensive library of free content too epic also chooses a set of paid assets from the marketplace to give away for free each and every month and although they're only available for a limited amount of time once you download them they're free to keep forever so keep an eye on the marketplace and grab them whenever they're available i've personally been downloading these free assets for months and some of them have been really useful and great so far unreal engine users can also get unlimited access to qixel's library of mega scans completely free this is basically just a collection of thousands of aaa quality photo real textures and 3d assets with everything from props to entire pre-built environments all their stuff is just astounding by the way and as long as you only use these assets in unreal you have to pay a penny for them another great place to get free animations and characters is miximo anyone with an adobe account can sign in and instantly access an online library of professionally motion captured animations and high quality rigged 3d character models to download and use on your projects if you want to make your own unique characters rather than using someone else's but you don't know the ins and outs of 3d modelling then i suggest you check out character creator 3. it uses an extremely user-friendly content creation pipeline that basically allows you to make characters the same way you would customize them in a video game more experienced 3d modelers can also go in and make more complex fine adjustments to their characters using a wide range of tools all of the characters come with complete body rigs and facial morphs so they're ready for animating motion capture and lip syncing and while character creator 3 isn't free with the bass software costing 199 it is a perpetual license meaning there's no recurring subscription fees plus the base software includes a library of over 500 assets including characters hair clothing props and more which i'd argue is worth the price of admission alone you can also download a free 30-day trial to test it out and see if you think it's worth the price tag [Music] now if you don't want to limit your character's animations to those available in online libraries or spend ages creating your own then performance capture is a great solution and when it comes to mocap on an indie level budget there's a wide range of options out there for us hobbyists one of the best options is to invest in a mocap suit which utilizes inertial measurement units or imu sensors now unlike the traditional motion capture systems used on big budget hollywood movies imu based suits don't require external tracking cameras which means they're a lot more portable and easy to set up making them suited to independent filmmakers a lot more they're also a lot cheaper than traditional mocap systems on the whole popular brands of imu suits include rococo perception euron and xsense but there's tons more out there and a quick google search for budget mocap suits will bring up a variety of options at different price points ranging anywhere from five hundred dollars to fifteen thousand or even twenty thousand plus although that's not really budget at that point but there's tons out there if you're looking for finger tracking as well some great solutions include rococo smart gloves perception urine studio gloves stretch senses super splays and manus vr gloves also keep in mind that some mocap suits already come with finger tracking straight out the box like for example the perception euro 32 v2 so just make sure this isn't the case before you go and buy a pair of gloves only to realize you already have some now if you own the htc vive it's also possible to use it as an alternative to a mocap suit by attaching vive trackers to your body at key tracking points using specialized straps such as those by rebuff reality this is a great solution for existing five users as because you've already invested in the core vive hardware all you have to do is go out and buy a few more trackers which is a lot cheaper needless to say than purchasing a dedicated mocap suit and from what i've seen vive mocap provides some surprisingly good results oftentimes even better than those captured using imu suits you can even use val's index controllers which are completely compatible with the vive alongside the trackers to produce basic finger tracking too while this is a great solution unfortunately the vive won't work for performance capture straight out the box specialized software is required to set it up the solutions i've come across are the vive mocap kit plug-in on the unreal marketplace ikinamo orion manus polygon and glycon vr i'll provide links to all four down below as with everything so you can take a look at the different features and price points to see which best suits you another alternative to using a mocap suit is using software such as connect animation studio or ipi motion capture to mocap using your xbox kinect or even multiple webcams obviously your animations aren't going to look quite as refined as if you used a vive or a dedicated mocap suit but you'd be surprised at the results this relatively cheap solution can produce in terms of budget facial tracking solutions i'd say your best bet is using your ios device if you already have one apple's ar kit facial tracking system is now compatible with ue4 it allows you to stream facial mocap data basically directly into the unreal engine using the livelink plug-in just be aware this feature is only available on ios devices with a front-facing true depth camera such as the iphone x and ipad pro if you actually want to export your facial motion captures for using a larger variety of software instead of just streaming it directly into unreal engine then facecap is likely a better solution for you this app allows you to export to both the fbx and txt formats as well as import your own custom avatars from blender and maya while significantly more expensive than the latter solutions at 39 a month rococo's face capture add-on available for their studio software is another great facial tracking solution especially if you've already invested in one of their smart suits it basically allows you to provide facial mocap using an iphone x or 11 with plans to expand compatibility to android if you're a vive user know that htc are currently working on a lip tracker attachment for the vive headset currently only prototypes are available to vr developers to preview but it looks like htc plan to bring it to the market soon this lip tracker combined with their new pro eye headset which tracks eye movement will allow users to achieve full facial performance capture using the vive because this solution utilizes a vr headset it also has the benefit of immersing the actor in the virtual environment allowing them to react to it more believably it also means in the future the vive could be used for your entire virtual production pipeline from the virtual camera to both body and facial mocap which makes it a pretty great deal for beginners and those wanting to stick to a single virtual production ecosystem and while we're in the realm of mocap i just thought i'd mention nvidia's upcoming omniverse machinima toolset which will be available exclusively for rtx graphics card users it essentially allows you to import assets from libraries and supported games to create machinimas with users can automatically animate characters using footage from their webcam aided by an ai-based pose estimator and animate faces using only audio files thanks to nvidia's new audio to face technology needless to say i'm pretty excited about the potential of this new software the ability to essentially mocap with a single webcam and animate faces using only audio is going to be game-changing for indie virtual production and although the nvidia have said it isn't intended to replace traditional mocap and we haven't yet seen it applied in the real world outside of nvidia's own test footage i'd say if you already have an rtx graphics card or plan to upgrade to the new 3000 series anyway if you can somehow find one without paying those crazy scalper prices on ebay then i'd say it's worth giving unreverse macinera a try especially if you're finding it hard to justify spending thousands in motion capture software and equipment so that about covers it for this overview of getting started with virtual production at home i really hope you guys found this video helpful and if you did be sure to give it a like so i know and subscribe for more content i've got quite a few more in-depth virtual production tutorials planned and i've got an idea for a series that i'm pretty excited about and i think you guys will hopefully like it too especially if you're just starting out in your vfx or virtual production or even filmmaking journey so make sure you click the notification bell so you know when the latest videos drop remember that all the equipment software and resources i mentioned in this video are linked in the description and if you follow my amazon links you'll be supporting the channel as always be sure to check out some of my other videos follow me on instagram and twitter as well for the latest updates and just more geeky goofy goodness in general but anyway thanks for watching and until next time i'll see you all later

Channel: Reality Redefined

Views: 22,292

Rating: 4.9309645 out of 5

Keywords: virtual production, vfx, unreal engine, ue4, real-time rendering, machinima, motion capture, mocap, mixed reality, HTC Vive, Vive Tracker, Virtual Camera, the mandalorian, virtual set, greenscreen, in-camera vfx, Nvidia Omniverse, rokoko, perception neuron, xsens, aximmetry, reality engine

Id: TCKq502Dc-U

Length: 18min 23sec (1103 seconds)

Published: Mon Sep 28 2020

Reddit Comments

[u/anothermeadow]

Thanks, really enjoyed this video. I'm looking at some lower budget virtual production setups and this helped a lot. Do you have any examples of what you have shot with your rig? Just wondering what this kind of setup can achieve. Thanks!