Blender tracking tutorial: Add 3d buildings to Live action Footage

what's up guys it's brad from lot architect here in this video i'm going to be showing you how you can add 3d buildings and other objects to your live action footage using motion tracking inside of blender i'll be going through a simple motion tracking process and showing some tips on how you can light and create the environment inside of the computer to match your live action shot a little bit more effectively for this specific video i'll be using some 3d buildings from our updated city builder 3d asset-based add-on released on blender market but of course you can use any buildings or 3d models of your choice in the same process anyways guys let's get started here we are inside a blender the first thing we're going to do is of course just delete everything in our scene here so we can get a clean start so go ahead and just select everything and delete it and the first thing we want to do is import the live action footage inside of blender so that we can motion track it so i'll go ahead and go to this plus tab here and then i'll go to visual effects and then we'll go to motion tracking here and select it and now our motion tracking interface will come up here and what we want to do first is just open up our video footage so i'll go ahead and press open and now i'm just going to navigate to uh some footage i've downloaded from pixabay.com it's just a nice little drone shot that would be cool to add our building to as you can see here and before we go through our motion tracking process i think it's nice to set the blender scene to have the same frames as your live action footage so that we can load our whole clip and render out the 3d environment to last the same amount of time as our movie clip so i'll go ahead and select set scene frames here and then i'll go ahead and also select the prefetch option here which is just going to go ahead and go through this video clip here and preload all of our frames to make the process a little bit faster for us and we'll just wait for it to finish here and now as you can see here we can scroll through the timeline pretty nicely here and it's a pretty cool shot here from a drone again i got this from pixabay you can download it and use it for free i'll go ahead and put the link to this specific clip in the description if you want to follow along with the exact clip but of course this process will also work for other live action footage that you might want to use all right so let's go ahead and track this footage inside of blender and if you don't know what 3d tracking is essentially 3d tracking is the process of telling the 3d camera inside of the computer how your live action camera was moving so that we can overlay 3d objects onto our scene and have it move around like it were actually there then we can use all this tracking data to add different elements to the scene inside of a 3d environment that matches up to the scene as the camera is moving so that's what 3d tracking is and to do that what we want the computer to do is track a bunch of objects in both the foreground and the background of the scene here so that it knows kind of the parallax and the movement of the camera so it's important whenever you're tracking something to choose kind of points in the foreground as well as points in the background and whenever you're choosing tracking points specifically you want to choose points of contrast so there are several different ways to do this the most tedious and if it's a really complex shot the way you might have to do it is by just kind of scrolling in here you want to choose the pattern size which is kind of uh how big of an area you want to track and then you want to choose the search size which is the amount of pixels blender is going to search for into the next frame of that tracking point but essentially what we want to do here is we want to maybe increase the pattern size to 30 and the search size a little bit as well and if your camera is moving really fast you want to increase these a lot more but as you can see here when we scroll through our footage it's a pretty subtle camera move so we shouldn't have any issues tracking 30 pixels with a search size of 80 as we go from frame to frame so uh what we can do here is to add tracking points individually we can just find points of contrast for example this nice car here we can go ahead and press command and click and now we just added a tracking point on this car here and what we can do is if we want to just track through the whole scene with just this one tracking point what we can do is we can go to the track tab here and just click on uh our track markers button and we can just play through it and as you can see blender is going to go through the entire scene and track that point of contrast and since we started halfway through the clip we also want to track backwards with the same point now as you can see throughout our whole clip we have tracked the front of the car here which gives some good tracking data for blender to calculate how the camera is moving for the live action shot so that it can recreate it inside the computer and blender needs a total of at least eight points throughout your clip to track the scene effectively so as you can see here we've just added one but if we wanted to continue this process we could just select eight more uh points of contrast throughout our scene here making sure to choose some in the foreground here as well as some of the background and then let blender track through all those points and then we could just go to the solve panel here and go through the rest of our process that'll show here in a minute if you don't want to add points manually and you want to just have blender detect the features of your scene so that you can kind of move a little bit quicker in your scene if the shot is something simple like this what you can do is you can actually go ahead and start from scratch here what you can do is you can actually just go to the beginning of your scene here go to frame 0 and we can just go ahead and select the detect features button and now blender is going to just kind of select some points of contrast in your scene for you so that you can now just go ahead and track all these points at once so now that we've used the detect features option we can just go ahead and again go through our scene with our tracking button here and blender will go through and track all these points for us and as you can see since the camera move here is uh fairly controlled blender is tracking these points pretty effectively but if the camera move was going pretty crazy sometimes you have to more carefully choose the uh points of contrast to track inside of your scene but uh this is looking pretty good as you can see here we have some pretty good data to work with i'm going to go ahead and go to the end of our scene here and also go ahead and select detect features again and then i'm just going to go backwards so that we have a few more points to work with and again you only need eight points but sometimes having more data to work with is helpful when you go through uh cleaning your track and getting a little bit more accurate results so it's nice to just have a little bit more uh inputs for blender to work off of and as you can see on the bottom here you can see the data on the graph here and one thing i'm always looking out for when i'm looking at this graph after i've tracked something is points that are going crazy off of the main line here so as you can see we have some of these points down here that are going way off the majority of our points here in the center of our graph so that's just kind of a red flag that suggests that some of the points are slipping and aren't very accurate so what we're going to do once blender goes through all these tracking points and finishes up here it's at 96 percent right now is we're going to clean the data so that we don't have uh any skewed points here all right so blender has tracked all of our different markers here now what we can do first is we can just scroll through it here and see if we can notice any slipping tracks so as you can see here this one here in the center of our scene is uh slipping off of its point for a few frames so we can go ahead and select that individually and i'll go ahead and press x to delete it and we can just kind of keep scrolling through it here and we want to just make sure that all of our tracking points are staying on their points of contrast in our scene all right so another blender has some tracking data to work with what we're going to do is we're just going to go to the solve tab here we'll just go ahead and select a few keyframes here where the parallax is more pronounced but as you can see our scene has pretty constant movement so we probably don't really need to change that much however i will just change it from 10 to frame 40 just so it's a little bit further in the clip there and we can also click on the refine characteristics button here and go to uh focal length k1k2 and uh if you don't have the camera data or for example in this case i've just pulled this clip from online i don't have the camera sensor data or the lens data what you can do is you can use the refine tool here so blender can kind of guess the focal length for you but if you do have the camera data and you shot the image yourself what you can do is you can go to the tracking settings here and then you can add your camera data as well as the focal length of the lens that you shot with as well as all of the lens distortion data and all of that but again for this specific case we're going to kind of let blender figure that out for us and actually instead of selecting our own key frames here i'll go ahead and select the keyframe option here which is just going to uh as you can see allow blender to automatically select the keyframe when solving the camera object motion so just kind of a preset there that i found works pretty well and now we'll just go ahead and make sure all of our tracking points are selected and we'll just go ahead and press solve camera motion and now blender is going to go through our tracking data and solve our camera and as you can see here we have a solved error of 1.2 pixels which is super good we could actually use this pretty well in my mind based on my experience anything under 2 is pretty good but i try to get it under one so let's go ahead and clean the data so that we can get a little bit better solver here so what we can do here as well as kind of scrolling through and finding the points that are slipping off of their points of contrast in our scene here and just deleting those what we can do to get blender to help us is we can just use our cleanup tab here so let's go ahead and first kind of click on the side of our scene here so that everything is deselected and then we'll just go ahead and under the cleanup tab we'll just click on reprojection error and change it to 1 and then press enter and then we'll go ahead and press clean tracks and now blender is going to select all of those tracking points in your scene with an error of one or more and it's going to select them and now we can go ahead and delete those pressing x and delete track and now we can go ahead and select everything again and press solve camera motion and see if we get a better result all right so that actually made our solve error worse let's go ahead and see what's going on here i might just scroll through here and see if there's uh any tracking data that i can manually delete i'm seeing uh one back here that's slipping a little bit see what that one's doing here yeah you can see that one's slipping quite a bit i'll go ahead and delete it and let's go ahead and scroll through and see if anything else is slipping this one looks like it might have slipped a little bit as well yep that one's slipping a bit go ahead and delete that one the rest of these up top look pretty good i don't see any noticeable slipping right now and another thing we can do is we can go ahead and select some of these points that are way off our graph and see if they're slipping so as you can see here this is a point that's way off of our graph here compared to everything else and this one's slipping as well we'll go ahead and select that delete that and here's another one over here let's go ahead and select that wherever it is there it is see it slips right there so let's go ahead and just delete that one as well and now that i'm looking at the rest of the tracking data it's looking much cleaner here i don't know what these are let's go ahead and uh try to solve this one more time go ahead and press solve camera motion and we're still getting a pretty high solve error here oh i see a few uh that are a little bit off here this one right here there's one slipping there as well i'm going to delete that one all right so let's go ahead and select everything one more time and this time i might just deselect the keyframe selection here just to see if we could just manually select our keyframes and try this one more time and as you can see here now we have a solve error of 0.34 pixels which is uh you know less than a pixel so super good and now we can go ahead and add our camera and our 3d camera inside of the scene should move the same way as our live action shot has been filmed here so before we get into uh adding our building and all that stuff into our scene here we need to do a few more things in the tracking process let's go ahead and first we'll go ahead and click on set as background and what that's going to do is it's going to make our movie clip here automatically be in the background of our scene inside a blender once we add a camera to it and then we'll also go ahead and go ahead and choose a point for the origin of our scene so since i i want to add a little building to this kind of ridge here so i'll go ahead and make this the origin of our scene here so i'll go ahead and select set origin and actually first we need to add a camera to our scene here and now as you can see here that origin data has been applied to our camera so it's kind of pointing as if this point here in the center of our 3d world is the is the origin point of this tracking point right here so if we go to view camera you can see uh the origin point right here so that's looking pretty good now let's go ahead and click three points in our scene that we want to set up as the floor of our scene then i'll go ahead and select three points on our ridge here and we'll go ahead and make that the floor of our scene and now blender has used these three points to create the floor of our 3d camera here and now what we can do finally is we can go ahead and click on setup tracking scene and now automatically we have a nice ground plane where we'll have our shadow catcher as well as a cube which will be kind of your base uh foreground object to start with to kind of add to your scene from there so now that we've done this let's go ahead and go to layout mode here and we'll go to view viewpoint camera and now as you can see here when we scroll through our camera we have a nice uh 3d track and everything in our 3d world is acting as if it's in the live action shot and by clicking that setup tracking scene button automatically we have a foreground collection here with uh this cube on it which will be rendered right now as our main object in the live action shot but of course we're going to add a building there instead but we just have a default cube there and then in our background collection here is just our ground which will act as our shadow catcher and then in the compositing tab as well blender has automatically set up these two collections to act as both your foreground layer and your background shadow catcher so that you can kind of have a general compositing setup to start from for your compositing process all right so now that we've tracked our scene let's get to the fun part and add the 3d building and create the environment to light the 3d building to match it to our live action shot so let's go ahead and first i'll go ahead and go ahead and select our camera here and i'm just going to go ahead and select the motion tracking button here so we can see our motion tracking data as we scroll through the scenes because it's a little bit nicer to reconstruct your environment as you can see that motion tracking data as you can see here all of these uh points that were once tracking points in the motion tracking tab here that we added are now points inside of your 3d world which are recreating that geometry inside of the 3d world but anyways let's get to adding our 3d building so i'll go ahead and go to viewpoint camera and now i'll go ahead and select our cube here and go ahead and delete that and i just want to add one of the city builder 3d assets here so we'll go ahead and use one of the new soviet assets so i'll go ahead and click the center of our scene here where we want to add it and then just click on soviet small five and now we have a fully textured 3d asset inside of our scene here ready to go and now we can create that environment to light this as if it were in the live action shot so first we'll go ahead and position this kind of near the center of the ridge here maybe scale it down a little bit so it's a little bit more realistic here also scaled down the ground plane a little bit all right that's pretty good and now what we want to do is we want to first go ahead and switch to cycles rendering engine so go ahead and select the render properties switch the render engine to cycle since uh that's going to give us a little more realistic lighting and now that we've kind of generally placed our building here we want to recreate the lighting and the environment around our building here so that when we render it it will match the live action footage a lot better and then we can add all the other compositing layers to blend it in even more in our post-production process but let's go ahead and select our ground plane here and our ground plane by default is going to just be a shadow catcher but we also want the color of this ground plane to be similar to the color of the ground in our live action shot so that our building has relatively realistic environment reflections on it and it isn't just bouncing up a white ground onto our building because that looks a little bit uncanny so what we can do here is while it's selected is we can just go to the material tab add a new material and then we'll just go to diffuse material here and we'll just go to the color tab and then just use the eyedropper here to select maybe just a kind of a darker shade of our tan color here of our live-action shot and now we have this ground plane kind of reflecting the general color of the ground in our live action shot and uh of course you can also reproject this exact background onto this ground plane as well if you want it to be even more accurate but in this specific shot i'm just going to go with a general color because i think that'll look good enough for this final shot all right so now that we've created the ground for our scene let's go ahead and go to the world properties here and one thing that we could do to recreate the world around our 3d asset here is we could kind of use an hdri of a similar environment and just kind of import this into the world settings and kind of create some realistic reflections on our 3d building here but a lot of the times you don't have a 3d hcri of the same environment that you shot in so what you can do instead is you can use the uh basic sky inside of blender to kind of create that ambience from the ground up so what we can do here is under the world settings here we'll go to color and we'll go to sky texture and in blender 2.9 the default uh sky model is the nashita sky model and it's super cool actually before i've used the hozek wilkie which is also really nice but nishita as you can see here you have a lot of different settings that you can play around with you can play around with the density of the air molecules the density of the dust the ozone layer and then of course the strength and then the sun intensity sun side sun elevation sun rotation and then the altitude as well so it's super cool i'll go ahead and go to rendered view here and you can actually see our sky environment lighting up our scene and the reason our 3d building is so dark like this is because it's on our background layer which is our shadow catcher so i'll go ahead and move this to our foreground area and uh now we are seeing our fully textured 3d asset ready to go here so we'll go back to camera view here and now what we want to do is we want to kind of adjust our sky settings to match the sky of our live action shot so i'll go ahead and turn off our background for now and as you can see in our live action video here the sun is coming off from the right side here at a pretty low angle and if we zoom in we can actually see that the car has a pretty hard shadow just directly off to the side here so we want to try to match the sun's direction to that and it's also pretty close to sunset so we want the sun elevation to be probably pretty low so probably 15 degrees is pretty good but we want to change the rotation a little bit because right now the sun is coming from the opposite direction in our sky setting so we can just change this maybe to something like 180 and now as you can see here the sun is lighting our 3d element from the other side which is giving it a lot more realistic result here and again when we render that shadow as well it's going to be even more realistic and again as i mentioned you can also change the altitude of these uh sky models as well so i think it's pretty cool when you go a little bit higher it gets a little more blue and i think it might match our scene a little bit better so maybe something like two so kind of experiment around with these settings to match your 3d sky model to the sky in your live action shot so go ahead and increase the altitude a little bit and see what that does for us that's looking pretty good around uh four kind of blending into our environment pretty nicely and it's also giving our building a nice little side light which is uh matching our 3d environment as well and i might just adjust the rotation a little bit more here try to match it a little bit better i might also just decrease the strength of our sky a little bit maybe to something like 0.8 that's looking a little bit better and now that we've created our environment and adjusted it to the live action shot what i want to do is i want to go to the render properties here and then under film i just want to make the background transparent so now we can see our 3d environment directly on top of our live action shot and we don't have to see the sky when we don't want to so at this point our sky environment is still lighting our scene but we can just uh see what it's doing to our 3d world instead of dealing with it all the time and we can see it with a little bit more clarity here so go ahead and enable that transparent option now let's go ahead i'll go ahead and re-enable our background layer and i'll go ahead and reselect our ground and one thing we want to do is we want to make sure that the shadow catcher of our ground plane is working effectively so what we want to do is we want to go ahead and add a sun to our scene and we'll just put it off to the side here and just side light our building real quick and of course our sky environment is also including a sun in that output especially with the new nishida sky model however it's nice to also have a little bit more control of your uh shadows with a actual sun lamp inside a blender so let's go ahead and try this one first and see what we can do i'll go ahead and reselect the ground plane here make sure that the sun is just in the general scene collection here and select your ground plane and we want to go ahead and disable some of these restriction toggles and now as you can see under the background layer we can disable this hold out and we can see our shadow a little bit better all right so we have a nice little shadow on our ground plane here and what's creating the shadow right now is not actually the sun that we've added here as you can see if we just delete it what's actually creating the shadow is in our world settings again we've added our nishida sky model which includes a sun disc and that sun disc is going to create that hard shadow that you see here as you can see if we deselect it that sun disc goes away and we just have the ambient lighting from the nishida environment without the hard source from the sun itself it's just kind of the ambient light from the sky so there are two main ways that i like to control the hard shadows that are interacting with our shadow catcher ground plane here one way you can do this is of course just enabling the sun disc in the nishida world settings and to soften up the shadows to make them not quite as hard you can just increase the sun size until you get the right kind of shadow quality that you want to match your environment as you can see here we have some pretty uh hard shadows on our live action shot here if we zoom in you can see the shadow from the car here is fairly hard but it's slightly diffused from the sunset that's in the sky for the live action shot so we might just try to do something similar with the sun disc probably something like this is pretty good maybe a little bit smaller sun size something like that and we can also change the sun intensity to something like something a little bit less maybe point seven so now it's not quite as harsh here and our 3d building lighting is matching our live action lighting a little bit better here so that's one way you can do it you can play around with the sun size and the sun intensity inside of the nishida sky model that's probably the simplest way to do it another way you could do it so that you can have some separate control between the sky model in your world setting and the sun itself is you can just disable the sun disk setting here and then you can add a plane here make a new material turn the plane into an emission material make the mission material a little bit warmer like it's the sun kind of rotate it like it's off in the direction of the sun here and since our plane is pretty big it's going to create a nice soft shadow as well and then we can increase the strength of it to say maybe 10. we also need to make sure that this is on the foreground layer here or not the foreground but in the just the general scene collection and now we can just increase the intensity of this emission plane read a 100 and we can create nice soft shadows on our environment here with very specific control over where we place this without changing the entirety of the sky environment in the world settings itself so we can also increase this maybe we'll try 1000 to give an extreme example as you can see getting a nice soft shadow here to make the shadows harder we can of course scale it down and then increase the strength to create a nice hard shadow but i think that soft shadow was much nicer and again we need to need to match the shadow direction now to where that car shadow is and now as you can see we have a little bit more control over the specific placement of the sun and the sky environment a little bit separately but that's just one way to do it of course you can also add you know basic uh lights to your scene as well and they'll still work with that shadow catcher and in lighting your 3d building here just make sure that any lights that you add are on your scene collection layer in your foreground but not in your background because your background is just going to be for your shadow catcher but anyways this video is getting a little bit long so let's go ahead and render out a few frames here i'm going to go ahead and use the nashida sun disk just for this one since this is my first time playing around with it i'll go ahead and just kind of more specifically place my ground plane here and i'm not really worried about this shadow edge here because i can mask this out in our compositing process so we just want to make sure we have enough shadow to mask out and this looking pretty good of course you can add all kinds of different environment elements you can edit your ground plane here so that the shadows are more interactive with your live action shot however this is looking pretty good and the tutorial is getting a little bit long already so i'm going to go ahead and get to our export settings and show you guys how i render it out for the compositing process so in our exports from blender i'll generally export three passes i'll export our beauty paths our shadow pass and an ambient occlusion path so i'll go ahead and go to the layer properties tab here and what we want to do is we want to make sure that we have all the passes that we want here selected so as you can see here we have our combined path we have a z-pass which i don't think we'll use so i'll go ahead and deselect it also go ahead and export a missed pass as well in case i want to add a little bit of atmospheric falloff in the compositing process um and i'll also go ahead and scroll down here to ambient occlusion go ahead and select that and that should be pretty good for our render settings i'm also going to go to the sampling tab here i'll render this at maybe uh try something like 40 samples maybe and then under advanced i'll select our seed stopwatch for some noise variation and then under the output tab i'll go ahead and change the file format to open exr and that's just a good visual effects file export for compositing and also just choose where i want to save the project here and do live action building tutorial output and then we'll just do this one as our beauty pass [Music] so uh the output here will be our beauty pass composite and this is looking pretty good 24 frames per second we have all of our keyframes here 1920 by 1080p resolution then we'll go ahead and go to the compositing tab here all right so in the compositing tab here what we want to do is we want to export all of our different passes to different file outputs so that we can composite them together more effectively right now as the default inside of blender it's going to take both our background shadow paths in our basic beauty pass and it's just going to overlay them on top of each other in a composite here which will be exported in the standard output here but what we want to do is we want to export the beauty pass the missed pass and the ambient occlusion pass and the shadow path separately so that we can have a little bit more freedom in that compositing process so of course feel free to just export the composite as is with the shadow path built in to your live action shot but for a little bit more control i like to export them separately so i'm going to do that here first i'll just take the composite node here where our main file output will be and i'll just export our main beauty patch in the foreground layer and then i'll just take the main beauty pass in the foreground layer and connect it to it and now our beauty pass will go to our main file output here then i'll go ahead and press shift a i'll add a uh another file output for the missed pass and then i'll select the file output here and create a new folder for it we'll do missed pass [Music] and we'll go ahead and create another file output for our ambient occlusion paths create a new folder and finally we want to export our shadow pass which is in our background collection layer here so i'll go ahead and press shift a again go to file output and we'll export the image in our background shadow pass layer here create a new folder and accept and now we have our four different file outputs so that we can composite inside of after effects or back in blender if you'd like and have a lot more control in that whole process and finally let's just go ahead and do a quick test render see how it's generally looking with the default composite inside of blender before we go into after effects for that final composite first i'll go ahead and save the project one more time then i'll go to render and render image and then give blender some time to render out your test image all right guys so here is our test render here as you can see if we zoom in it's looking pretty nice it's a little bit too bright but we can bring that down in the compositing process the general shadow direction is looking pretty nice and i think it's going to work pretty well one thing to note if you just wanted to do all of your compositing inside of after effects what you could do is you could just change your compositing output to the view output here and whatever you are seeing in this viewport here would be your output for your final composite however again like i mentioned before i'm just going to be compositing this inside of after effects in that second part of this video so i'm going to go ahead and keep my composite just as my beauty pass for that building and then we have all of our different passes to work with as well in their various file outputs and since our building is a little bit bright here if i were compositing inside a blender all i would do is i would just press shift a go to color maybe add a rgb curves node here and then connect it to our foreground building layer and then just bring it down a little bit and start matching our environment a little bit better and of course there are so many different things you can play around with in the compositing process but the video is already getting pretty long so i'm going to go ahead and cut it off here until i go through the compositing process in the next video very soon of course to do your final export of all these different passes in animation form you just go to render and render animation and blender will go through all the different frames in your timeline and export those passes accordingly anyways guys that's it for this video i hope it was helpful as always feel free to leave any comments if you have any questions or suggestions in the comment section below i hope this video wasn't confusing in any way i know it's a lot of different steps going on here and all the different render passes can get confusing but if there is any confusion feel free to drop a comment in the comment section below and i'll get back to you as soon as i can i'll see you guys next time you

Channel: LightArchitect

Views: 64,785

Rating: 4.9737129 out of 5

Keywords: 3d tracking in Blender 3d, 3d tracking in blender 2.9, matchmoving in blender, how to camera track in blender 3d, blender 3d camera tracking tutorial, blender motion tracking, motion tracking in blender full tutorial, how to add 3d to live action in blender, live action blender tutorial, blender 3d live action vfx, virtual sets in blender, how to make cities in blender, cityscape blender tutorial, blender 3d city tutorial, citybuilder3d tutorial, citybuilder3d assets for blender

Id: 7FltiqN8R9g

Length: 30min 59sec (1859 seconds)

Published: Sat Oct 03 2020

Reddit Comments

[u/dphamiltonbradford]

Hey Redditors!

In this video I show how to 3d track your live action footage in Blender and then recreate the lighting in Blender so that you can add CGI to over your footage in compositing! Part two in After Effects coming soon! Let us know what you would like to see next!

​

Our Blender Add-ons:

Ultimate Blender add-on value pack: https://www.blendermarket.com/products/add-on-value-pack-citybuilder3dkhaoslightarchitectcablecam-bundle

KHAOS (Ultimate Explosion Add-on): https://www.blendermarket.com/products/khaos-ultimate-explosion-simulator

CityBuilder3D: https://www.blendermarket.com/products/citybuilder-3d

Cablecam (Cinematic movement rig): https://www.blendermarket.com/products/cablecam-cinematic-camera-movement-rig

LightArchitect (Film setup previsualization): https://www.blendermarket.com/products/lightarchitect---filmmaking-add-on

KHAOS Fire Shader: https://blendermarket.com/products/khaos-fire-shader-fire-shading-simplified

​

_

[u/Leikaderp]

actually a great tutorial