Virtual Production with Blender using the Jetset app

hello it's Alden here's how to live preview your 3D scene when shooting on a green screen and how to bring the camera data from your take right into blender allowing you to compose handheld green screen shots like this without any tracking all for free the app is called jetset by lcraft and it brings virtual production capability to your Indie budget projects the app works with a free piece of software they developed called autoshot and together you can put your 3D scenes in the app for Real Time preview record takes and bring that camera data back into your project the free version uses the iPhone camera but there's also a pro version that calibrates to a cinema lens for this test I use the iPhone version this is sort of an overview tutorial of how to use Jet Set and then at the end I also share my thoughts on all of the tech because it's new and still in development but this is not a sponsored post I just saw the first demo of this app thought it was really cool and decided to try it out myself so let's get started I'm using this 3D scene which I used for my sci-fi film Friends of sopia the Jet Set app uses usdz files which need to be fairly small to run smoothly autoshot has a function to help create your usdz file and incorporate textures but because my scene involved a ton of complex 3D models and polycam scans I decided to just make a blocked out version of the scene with simple geometry I also included a model of a person just so I can gauge scale one thing you want to include in your project is an empty with the name scen loock uncore then whatever name you'd like this is where your camera will be when you first open the app you can export your blender project as a USD file and then just manually add the Z in the file name this creates like a zip file version of a USD file you have to use your iCloud account to bring the usdz file into the jetset app I use a PC so I have to use the web portal on my iCloud account jetset creates a folder in iCloud once you use the app so I just made a new project called test project and saved everything under that once you put your usdz file in that folder you can open the app choose a scene to get started and then you need to calibrate the 3D space the app is going to figure out where the floor is and then you're going to select where your origin point is now you can use your phone to view the scene in real time you can also use this ghost slider to adjust the transparency now to download your model you're going to choose this hamburger menu go to model and then open that's going to navigate to your iCloud folder here you can choose your scene if your usdz file project is too big this is where the app will crash and now you can see your scene using AR within the app not only that but it'll also show your subject in front of the background essentially a virtual production on your phone this means you can compose specific shots in your 3D scene instead of shooting flat plates and creating simple compositions and blender for this scene I had the subject sit down and swipe through a hologram emitter I kind of wrap around him and then I push in on a sign on the table this is a dynamic shot that I don't think I could have planned for if I didn't have the scene right in front of me when you record your takes and Jet Set the app saves the camera motion the footage from the camera so in this case your green screenshot a version of your subject roughly comped in that you're viewing as you're watching and also a depth pass should have set up my green screen properly and I also made a bunch of rotoscoping work for myself because I panned the camera over but that type of big camera move I tend not to do when I'm making green screenshots so I really wanted to try out something a bit more Dynamic now let's get all of this back into your computer to do that you have to open the autoshot app which you can download from their website it's in sort of a functionality phase at the moment not a pretty UI but it gets the job done you set your folder where you want to save everything and then you put in your iPhone's IP address into the app and that's how it's going to transfer everything onto your computer to find your iPhone's IP address you go into Wi-Fi and then scroll down and you can find that IP address I had some trouble connecting a few times during this step and maybe it's because I'm going from an iPhone to a PC or something I'm not really sure eventually I did get it to work but I have no idea why it wasn't working and what the fix was I just kept kept trying repeatedly then you wait as all of the clips from your takes and all the camera data saves onto your computer so now let's bring that back into blender down here you can choose a take and then navigate to the blender project you used for your 3D scene select save and run and autoshot will do its thing then open a blender project with your camera inside and it looks exactly as it was previewed in your iPhone which is awesome it also brings in your green screen clip as an image as ples it attempts to key it using blender's keying tools but because my green screen setup was awful it didn't work very well but what's really cool is that the plane that has your footage is kind of moving through the space somewhat accurately because all of your camera is tracked accurately you can just render out your background without your subject and then composite your key and rotoscope footage on top of it later what you won't get is the kind of dynamic Shadows that you might get from having a plane within your scene so if you need that kind of interaction you'll want to keep this plane in your blender project I opted for comping them together later in After Effects in blender I replaced the blocked out Geometry with the full res version and adjusted all the lighting to match what we had in the scene what's really going to sell this is having practical light on your subject and also in your scene so in this case it's the Hologram uh which has this kind of pink magenta light casting on the subject I animated did a cube rotating with each of his swipes I was going to do something more detailed having the Hologram kind of change and transition but I just left it as the cube for this test one thing I noticed is that there are a lot of micro movements of the camera it's probably because I was just holding the iPhone by hand maybe with a gimbal it could have been smoother I tried going into the graph editor and smoothing out the camera movement by selecting all the key frames and hitting alt o which you can use to smooth out any key frames in the graph editor but it looks like some of the key frames rotated 360° between frames so when smoothing out the graph I got some really weird camera movements this also happened in my first render when I turned on motion blur causing some random motion blurring during a few frames throughout the render it's possible that I can manually adjust those key frames and perhaps just deleting the single you know spinning key frame would have done the trick uh but instead I just rendered it out without motion blur and then used a warp stabilizer on the final export which isn't an ideal solution but something fine for this test and then something I'll definitely troubleshoot in my next goar around also I could only get the app to record in 30 frames per second I would have loved 24 and maybe that's something available in the paid cine version I'm not sure so here's the final render with all of the compositing done and after effect and uh this wasn't a keying and rotoscoping tutorial and there's definitely some work that could be done there and the Hologram could be more advanced than a cube but overall as far as the test goes with having this camera move brought into blender and incorporating green screen footage and a 3D scene I think this workflow is incredible and opens so many doors the fact that I created this with just an iPhone and free software is kind of mindboggling to me if you try this out let me know how it goes for you lightcraft also has a ton of step-by-step technical tutorials on their website that can help you along the way so now get to work on your 3D scene [Music]

Channel: Alden Peters

Views: 6,008

Keywords: alden peters, filmmaking, bts, production diary, how to make a film, film youtube, indie filmmaking, vlog, short film, sci-fi, science fiction, lgbt, lgbtq

Id: b7MHSYiDZ5U

Length: 8min 22sec (502 seconds)

Published: Tue Jun 11 2024