Blender 2.8 Motion tracking #1: Everything you need to know (tutorial)

so the masses have spoken and I guess we're about to do the longest motion tracking tutorial that's ever been made in general and specifically we're going to be doing it with blenders motion tracker which is a part of blender that doesn't seem to get enough love even though it's probably the best part of blender it is the best free motion tracking solution out there and often times it's better than paid solutions like cynthaiz 3d equalizer Buju things like this so if you ever wanted to know anything about blenders motion tracker or you're just learning about it for the first time this is the video you should watch I have my hair up to Zuko style because I'm expecting to be here for a couple hours we have a lot to cover I wrote it all down on a sheet of paper here we're gonna be covering the beginning stuff the intermediate stuff and then we're gonna get into what I consider a you know advanced topic so let's begin I guess opening up blender this is version 2.8 if you're watching this in the future like two point eight one two point eight two unless the motion tracker gets a major overhaul which I would love but I don't think it's gonna be the case everything is going to be identical or very similar with maybe a couple interface changes so the first thing on my list the first thing we need to talk about is what is the purpose of this tutorial what are what is it that we're going to be learning about so we are gonna be learning about motion tracking and this is the idea of taking footage you film something with your phone or with the camera and you want to extract data from this and we're gonna be talking about ways we can do that and that's pretty much the idea of motion tracking you take video extract data now the creative process usually comes after that what do you do with this data what are you putting into your scene what are you attaching on to your data and well I guess we'll talk about that a little but here we're just gonna talk about raw motion tracking techniques so well so we'll see what ends up coming up okay so first of all you took some footage on your phone all the footage that I used that I'm going to use for this tutorial was recorded on a phone you cannot just use this video in blenders motion tracker technically you can but you don't want to so the issue here is we're using video files and blender tends to have some syncing issues if you do a video file and track some stuff with it and then do some stuff later and then you close down blender next time you open it nine times out of ten things will be out of sync and you would have wasted a lot of time so what we need to do is take this video and convert it to something that is fail-proof it doesn't failsafe fail-proof either way you're gonna convert it into something that has no issues whatsoever and that is an image sequence this is the most important thing to start off with we do not track videos we track image sequences and you can convert this very easily but we want to convert it with blender we want to do everything with blender so suppose we have some footage let me just open up my folder here's pretty much the list of topics we're going over so let's say we have some footage here that we're going to open up with VLC and we want to track this right what we need to do is import this into blender find a way to convert this into an image sequence which is something I've already done but I'm still going to show you and then we can actually track it so the way we do this is inside a blender you want to open up the video editing workspace and this is where you can take a video and convert it so first of all we need to add in our footage so add movie and then navigate over to where your footage is so for me add this giant tracking folder and we're gonna load in this footage okay so we have our footage in here and there are a couple things we need to do before we can export this as an image sequence the first thing is if you have a very you know close careful eye you're gonna notice that something is looking very different here and that is the color space that is pretty much the view transform that we're seeing the colors here have been altered and we need to bring them back to standard the way we do this is you go to the render tab you're going to scroll down to color management and here's where we control all our color stuff so in view transform you're going to be seeing that it's set to filmic by default and this is good when you're rendering stuff you're modeling blah blah blah not good for this here you want to go back to standard and you see there was a very subtle change I'm gonna just flip between them filmic standard it's very important to go back to standard because there's a lot of detail we don't want lost because we're doing filmic mode so that is the first thing we need to do and then the second thing is we need to trim our project in endpoint and out points are here it's starting at frame 1 and ending at 250 so it matches with our video so in this case I'm just going to delete the audio so select the audio strip X 4 delete we're playing through it okay everything's fine we can skip towards the end okay so it looks like we want to end pretty much on the last frame and if you have blurry footage at the very end like it goes out of focus you want to trim it a bit before that okay so last frame is 234 hover over this hit ctrl C it will copy by just hovering over it and hitting ctrl C and then ctrl V so now our endpoint is 234 and you can see this is now matching and if we go to the first frame which this is a short coat we're gonna be using all the time shift left arrow for the endpoint shift right arrow these are going to be very useful for tracking so now when we play through this we have our whole footage it's gonna you know cache through this whatever but everything is looking fine so now we just need to export this output tab is where we're gonna do this you're gonna see we have this very weird frame rate and this is because my foot my phone is not the best at recording things it will try I say do 30 frames per second what does it give me 29.95 but whatever this is one of the issues that we want to fix so we have this frame rate that was automatically synced to this one and we also have this resolution synched up post 1080p and we want to export this as an image sequence I recommend PNG you can do JPEG which will be a smaller file size but you're losing detail and we're tracking you want to maintain as much detail as possible because that is what you're gonna be tracking if you have very if you're like degrading your footage you're gonna have a less to grab on to which is not what you want so PNG or you can get crazy with EXR but PNG works zero compression and then hopefully you don't hear that there's some stuff outside hopefully that's not a big deal and then you want to set a output directory and I'm just gonna put this onto desktop and a new folder test sequence we're not actually going to be using this one because I made sequences for all of these but so inside test sequence we can just call this test sequence underscore so it's going to be test sequence 0 0 0 and then 0 0 1 or we'll start with 0 0 1 etc that's just the indexing click accept and now we are all good we have changed our color space all our frame rates are matched up etc so just hit ctrl f12 to render out the animation and you see it's gonna go frame by frame it's a very quick process because it's just taking the frame of the video and then saving it as an image there's nothing really to render out that's super crazy so you just want to fit wait for this to finish it might take a second I brought a lot of water because I'm assuming my throat is gonna get sore I think that's gonna be the case so wait for this to finish should be done in a second and you're gonna need to do this every time you're tracking something in blunder do not use video use an image sequence believe me okay just do it okay so it should go to 250 I think Oh 234 because we trimmed it okay perfect and now when we go to this folder you're gonna see we have all these images which are all our frames and this is what we want again I've already made this I'm just going to delete it but that is what you want to create so now we have our image sequence okay perfect so now we've taken the video converted it into something we can use and we want to begin talking about tracking which is the point of this video so we're gonna do eat everything in this sequencer cuz we don't need it anymore X to eat and I like to go back to the layout workspace you don't need to do this but just do it layout workspace and then in the output tab since we're now using an image sequence which is just a list of photos it can be in any frame brain doesn't need to be 30 frames per second doesn't need to be 60 it can be whatever you want so that's another advantage of course we want to match this as close as possible because when we play this at speed we don't want it to look like it's twice the speed 30 frames per second that's close to 29.95 okay so now we have this nice 30 frames per second and we want to start tracking so here's the 3d viewport this is not where we do it all our work 99% of our work is going to be done in this movie clip editor this is where we're gonna live this is where we're going to do our tracking there are a couple exceptions like we're gonna do some compositing is it going to be pretty useful a tiny bit of stuff into 3d viewport this is where we're living and you're gonna see that we have nothing we can zoom in and out there's nothing here so we want to click open which is where we import in our not footage but image sequence navigate to your image sequence so for me it's this folder sequence and to import it you can either click this first one and it will automatically recognize this as a seat what I like to do again this works what I like to do is hit a that's gonna select everything open both of these work okay so now we have the cell loaded and when we scrub through the timeline you see this is indeed video and it's very slow not too slow but very slow probably a bit slower because I'm recording right now so there's a couple things we want to do to speed this up the first thing we want to do is make sure our end point and out point start point and end point I don't know what you call it in blender I guess start an end point you want to make sure those are matched up and of course they are because we did the video sequencer stuff in this thing but assume that let's say we have 300 here and we need it synched up well you can do a couple things you can either find where the last frame is and then say Oh change your end point or what I like to do what I like to do is just hit set scene frames you hit this button and it will automatically recognize where the last frame is so look at that so in this case it goes to 200 of course we had 234 before but I already rendered out the sequence before and I trimmed it a bit earlier so set scene frames will automatically fix this for us so if we have something like 50 and 60 what we need it to be is 1 and 200 hit this button it will automatically do it the next thing we want to do is load this video into memory so it doesn't play so slowly for that you hit prefetch and you see this Purple Line becomes a solid purple line and that means this frame is cached when we play this with spacebar everything is playing at real time full speed you can see the frame number here and you can see the resolution is also matching up and everything is good to go ok so we have our footage imported how do we begin tracking well this is where it gets really it's simple but this is where there's a lot to talk about we're gonna start off very simple so I'm gonna track this dot right here that's why I filmed it and we're not gonna worry about any of these settings yet we're just going to use the default settings to add a tracker hold control and click so that's control click and you're going to get this little tracker we can delete it with X add one here doesn't matter so I'm gonna add one here we can move it around with the G which is I don't know if I said that word with G which is the same as the way you do it I mean anytime you move something in blender you're hitting gene applies here too we can also scale with s and rotate with R so GS and are pretty much like any other part of blender so we want to make sure that this tracker is pretty much covering our dot we don't want it inside we want the whole thing perfect and there's a lot going on here that we're not going to talk about yet but we have this pattern area that's what this is called that's called the pattern area and this is what we're gonna be tracking basically this little texture or this little like chunk of our footage we're gonna be tracking this little square and to track you can open up this menu right here and you're gonna have all these controls and we can track all on all the frames at once or one frame at a time to do one at a time you hit this button pretty sure yes and you see now we're on frame two and you can just keep hitting this and it's going to track along and you can see it's leaving behind this little trail and when we go back you see that's pretty much defining the motion path so let's just track a bit more and you just want to make sure that it's really staying centered and one thing you can do if you don't like this motion path or you want it shorter or longer in this submenu right here we can change the path to be disabled or enabled we can change the length I'm going to keep it at 20 and to track the rest of this without going frame-by-frame and the shortcut for this as you can see is alt right arrow so alt right arrow frame-by-frame to do the rest of this I think it's this button yes this button or you can use the shortcut ctrl T and you want to be using shortcuts so control team and it's going to track all the way to the end all the way to frame 200 and this was a very easy track if it has issues it will stop halfway through okay so we have this track if we want to Center it so we can actually see if it's good you can hit this button right here or you can just hit L for a lock is it for lock I think that's what it's for yeah it's for a lock to selection and when we play it it stays centered okay cool and the reason it's staying perfectly centered is because it's such a good track okay so this is tracking on perfectly but let's say that we want to clear all this data so for some reason we don't like this or we want to redo it you can hit this Clear button so this is clear to the right so everything beyond this frame is going to get cleared hit that and now let's do the same process backwards so I'm going to go to the last frame that's shift right arrow to go to frame 200 I'm going to ctrl click to add in a tracker s for scale this time to track backwards which is also something you can do alt left arrow and you can hold or you can click L to Center this to speed this up you can hit this button right here or shift control T so it's ctrl T but with the shift shift ctrl T look at that that is tracking beautifully and something about blender something that's a little bit weird if you know about how it's actually programmed I'm sure you know why this is the case backwards tracking is a bit slower than forwards tracking I don't know if it's like trying to access the frames of the video in a weird way but it's slower even though it's the same thing I don't know why but you should get pretty identical results we can stop it at whoops you can click this to stop it at any point so we have this tract and then once we go about beyond this point it's not gonna be tracked anymore we can then on this frame we can move it so it's now tracking almost the same thing but really it's tracking like the the perimeter of this and then we can keep tracking and now we're tracking this area so you can move it mid track so shift ctrl T to finish that up of course you wouldn't want to do that you want to keep it at the same point but look at that that is looking good okay so some other things we can do just with this very simple setup go to this tracking tab on the right over here and you can see a whole bunch of stuff first of all we see this kind of zoomed in view that updates as we play and this is a good area to adjust you can just move it like this you can hold down shift while moving it to get fine control this is a good area to adjust it and also a good area to see if everything is lining up like that some other things we can do I'm just going to delete this tracker and go to the first frame so I'm just gonna add a tracker control click s4 scale so some other things we can do is we can really dive into the settings here so in the tracking settings tracking settings extra you see a whole bunch of stuff first of all you're gonna see the speed which is going to be you know how fast is it tracking right now it's going as fast as possible we can turn it to quarter speed ctrl T to track forwards and now we're very limited here because blender can go much much faster than this it has great algorithms but now we're seeing it at a very manageable speed we can stop it at any point make a tiny correction G for grab and then ctrl T to continue and if we're very confident with this we can bring it up to fastest and ctrl T and you can see that's much faster so that's the first thing that's kind of like playback speed but it's really it's tracking speed the other thing that's very important is this frames limit right now what we're doing is we're saying track forwards and we'll keep going into it reaches the end or it has an issue so here's what we can do instead we can add a tracker scale it up and take our frames limit to let's say 15 so now we have 15 inputted and when we hit ctrl T it's going to track to frame 16 1 plus 15 is 16 hit ctrl T again it should be 31 let's see yes 31 and then 46 I'm doing math it's tracking 15 frames at a time why is this useful you may think oh track all the way to the end this is a very easy shot in hard shots there's gonna be motion blur it's gonna get occluded by things you can't track the whole thing you want to go a couple frames at a time and then do a little adjustment make sure it stays centered so maybe it goes a little off-center do it again do it again etc so that's why you might want to use that okay so that is the very basics of you know just tracking a point but of course there's a lot more nuance there's actually things hiding here if you hit if you hit all 2's what do you get you get a search box so now we have our pattern area which is you know what we have before and we have our search area that is a hard word to say area so here these two boxes are doing very different things and it's important to understand this so this a pattern area what it's doing is saying this is what we want to track so each frame it will take this whole pattern and try to match it will try to match it and that defines the motion path but the question is where should it search is it searching for this dot and the whole video no no no that's crazy you want to limit the amount of area you're searching for this pattern and that is what the search area is right so if we make this very big very very big and that let's let's actually make a new tracker for this so I'm gonna make a tracker scale it up and then scale up our search area so it's very big so remember how fast is tracking before now when we hit ctrl T it's so much slower I've already hit it it's not even updating its because it's searching this giant area for this tiny little pattern so of course the advantage of this is you're probably not gonna have your pattern leave the search area the downside is it's taking forever to calculate so you need to find this balance it's actually not that slow but sometimes you track 20 trackers at a time here we're just doing one at a time which is not what you usually do so really once you got into complicated shots you want to consider how fast you're tracking this can actually become an issue so the bigger the search area is the longer it's gonna take because you're searching a larger area and I identically similarly the bigger the pattern area is the more time it will take as well because now it's trying to match a bigger pattern so you really want to keep both of these as small as possible without losing any accuracy so let's try to make this as small as possible and see how fast we can get this to track so I'm just gonna scale this so it's barely bigger we can see it right here and we can scale this down it might be too small but we'll try it ctrl T look at that it's already done so what might happen sometimes is if you're moving the camera very quickly this search are sorry this pattern area will be like over here on the next frame and it will have exited sorry it will have exited the search area and then the tracks gonna stop so you need to make sure this is big enough to account for how much emotion you have going on and I think that all I want to talk about for this very simple example let's go over a couple other other things so right now when we control click you see it always the same things happening that it's using these settings over here if we make the pattern size bigger when we control click it's gonna start off with this giant pattern we can make the search area big so this is pretty much the loading settings and then once we have a specific tracker let me add another once we have a specific tracker we can then modify the settings of this so we can go to the marker settings and here we have a whole bunch of stuff like the position the I don't want to talk about about offset yet this is very important we're not going to talk about it yet we have pattern area and search area and this actually brings up a good point we can use this old handle to rotate and scale nicely we can also move each point in our pattern individually and just will become important to one we do perspective tracking what else can we do we can scale ask for scale and then why only on the y-axis or only on the x axis and of course we can rotate so we really have full control over this and something I want to make very clear let me just get rid of these when we add in a tracker and let's say we're doing I mean this is a perfect perfectly set up tracker if we take some points and kind of distort it as you can see over here we're getting a lot of weird Distortion none of this matters this is just the pattern we are looking for the data are really kind of like where the tracking is happening like where the center of the track is this dot right here this is the only thing that matters we can even get crazy with this bring it all the way over here all that's fine when we are looking at the quality of a tracker we're just looking at this point right here so let's try to track this forwards okay so we got a perfect track and yes it kept this weird deformed pattern area but this Center remained where it's supposed to be let's head l to center L to Center so yeah so what's happening here here we are tracking a piece of footage and extracting X&Y data horizontal data vertical data where do we see this in here we can go to the graph option let's zoom that out okay so here we are seeing this red and green graph data I'm pretty sure red represents X data y represents Y data and this is pretty much the numerical representation of what our tracker is doing so here it's gonna have some initial X and y position and then this means and in terms of the red it's going one way and then it's going to go another and then back and forth and this wave pattern and the reason it's it's so like the reason it's oscillating so sharply that there's a lot of noise in this is because the camera was very shaky and you can see that here if we add very smooth camera movement like we filmed this on a dolly or something both these graphs would be very smooth and let me split this window and bring this back to clip so let's go to the first frame so now let's focus only on this red X graph so here it's moving let's turn off lock so we can actually see it moving so here it's moving to the left which means this red is going down and then it should start swinging back over to the right and then back to the left and to the right and the same kind of thing applies for the green Y data and I don't want to really talk about extraction yet but one thing we can do is in our 3d viewport I'm gonna do eat everything except for the camera you can take this tracker and once we're done with it hit ctrl L to lock it or you can just hit this button right here to lock and this will mean we can't overwrite this data it's locked in we're done with this track so with this track selected we can hit reconstruction link empty to track okay what did this do you see relative to our camera we now have this data represented in an empty so if we play this you can almost see you know this dot moving around in our camera properties we can enable a background image and then in movieclip I'm pretty sure we can load in our sequence and you can see that this empty is now over the dot so when we play this you see it's tracking on perfectly and it's relative to the camera so we can move it let's reset it to zero and bring it up and you can see this is tracking on perfectly and this is actually an empty object so we can scale this down and it will still be in the correct position so we can add in something like a cube a smaller cube and we can take this let's make sure it's centered we can take this shift-click the empty ctrl P for parent and then object so now our cube is parented to the object so this is kind of like a very first lesson in 3d integration so what did we do we track some data we put it inside our 3d scene even though it's only X&Y data which is why it only exists on this floor plane and then we just parented a 3d object to it and this is just the very basic so that was just location-based tracking if we go back here you see we were using the location model and the next thing I want to talk about is you know going up this list so we're gonna talk about a location rotation I think we're gonna skip this one and then go to location rotation scale f-fine and then perspective so in the next part let's talk about location rotation so we finished doing location-based tracking which is really the most simple case we can handle we want to be able to add layers of complexity and be able to track more difficult footage and just be more flexible with how we can track so what we basically need to do is just add layers of information new channels of information that we can track and make use of so so far we've been doing location-based tracking which means we have a pattern and we are seeing if we can match it by just translating it around from frame to frame so if we add rotation for example we have we're moving it around and we're also rotating it which is going to be much more powerful for a whole different group of trackers so for example I have this footage right here let me just play that and you can see we have this twig or leaf or whatever it is on the floor and we want to track it but of course there's rotation going on so location motion models not really going to cut it we can test it and see where it fails so in the movie clip editor we want to import in the new sequence and let's just find where that is so tracking location rotation and you want to either click the first one or a for all of them and open clip just like last time we want to hit set scene frames to make sure this 250 turns into whatever it's supposed to be which is 200 and then prefetch to load this whole sequence into memory so we can see we can play this in here everything looks exactly the same our color space is still in standard and our frame rate is 30 frames per second because it's an image sequence we can do whatever we want if we use the location based model let's actually see what happens we're going to add in a tracker by ctrl-clicking and scale this up we can hit alt asks for the search box and I'm just going to get some water here and we want to make sure our search box is sufficiently large and also our tracker area or our pattern area so we do not need to track the entire twig even though this has the most amount of data even tracking a small section of it should be sufficient as long as blender can grasp on to this plot this pattern and see where it's actually moving to so again we're using the location model and if we hit ctrl T let's see actually it makes it to frame 12 which isn't very good it seems to really need rotation if we were very stubborn about this and wanted to just continue using the location model for some reason we can go to the last active frame which is frame 11 and just update this little by hitting G for grab and I'm just moving it a little bit effectively what this does is it adds another keyframe so from this point on blender is gonna be referencing frame 11 and trying to match that to future frames so initially we add frame 1 we add a keyframe frame 2 references frame 1 so does 3 4 5 etc and now on frame 11 we've added a new keyframe a new reference point if you want to think about it like that ctrl T is gonna let us get a bit further we run into an issue we can scale it that's another way to add a keyframe try to make it a bit further and basically every time it's rotating a bit too much it's gonna fail so what we can do to fix this is in two-track settings and then in here what we can do is we can change the model to location rotation and from this point onwards it would track that way so for example location rotation so we've we've now changed the model that this tracker is using so if we all try taro you see now it's actually tracking on fairly well of course I'm just gonna restart this from the beginning so we're gonna start off with on frame 1 and then we're gonna use a location rotation for our default settings add this in scale it up and then hit control T this is a lot nicer so you can see that really blender is accounting for what motion model we're using and there really is the best choice in most cases sometimes there's a bit of gray area like is it location rotation or a location rotation scale sometimes you can play around with it a little but in general you want to pick the motion model that works best for what you're doing so you see this is tracking on very well we can see this in our little zoomed in area right here you can see that this twig is perfectly vertical in the search area not the search area this pattern area which means that the rotate rotation is very accurate or otherwise this would have a bit of rotation in here and something that's very important to mention about this is that even though and this is like this is important so listen up even though we're using the rotation or location rotation motion model and it's accounting for rotation by met wit when it's matching the pattern it's not gonna be storing rotation data again the only thing the tracker stores is the location of the center point even though we're using rotation to find it it's not storing it if we go to the graph data over here you're gonna see it only has X&Y data nothing to do with rotation well why is this a big deal well maybe you would have thought that we can use this to attach an object to it and it will rotate with it that is not the case for rotation and for scaling data for that matter you're gonna need two trackers so location is obviously only one tracker you just follow the track for rotation it's gonna reference two trackers and see the angle between them and for scale the distance between them so let's say we actually wanted to get rotation data out of this what we would do again is we would use two trackers so I'm just gonna add a tracker on this end right here again it's using the location rotation model and then add another one over here and there's no reason that two different trackers can't use different models they're separate you can do whatever you want but this example is very important because most of the time you're tracking multiple things at once so hit a or just click one and then shift-click to select both and then ctrl T and it's gonna track both of them forwards a bit slower probably choices slow because it's doing twice the amount of computations but you see this is very good we can hit ctrl L to lock both of these trackers and relative to both these trackers we can actually get rotation data so they start off vertical pretty much a vertical to each other and then the angle between them kind of you see this pivoting I'll show you what I mean in a second but this is enough to get rotation data so for example last time we did the cube example I just want to show you how we might extract something like this so let's delete both of these and then take our camera and four properties and Center it just like last time and just bring it up this is just a nice and tidy way to have everything set up we're going to select all our trackers and they're both locked that's very important and then in reconstruction link empty to track just like we did last time and with these empties we can just scale these down they're not too big and with our camera properties we want to add a background image movie clip and then just choose this sequence so now we can see what what these trackers are attached to essentially and to make this brighter because it's not matching this this is not a 3d viewport issue we can just take this and bring the Alpha to one so it becomes completely opaque I don't know if I've mentioned this yet but some other things we can do with this by the way for the depth we can set this to front and then the video is essentially above our trackers it's in the foreground or you want to keep it in back is what you usually do you can make this also completely transparent if you want to but if we play this you see that everything is attaching nicely so how do we actually extract something similar to rotation data this is not going to be the perfect example but it's kind of it's gonna kind of illustrate a point so what I'm gonna do is I'm gonna add in this time let's do a plane scale that down maybe make that kind of the shape of our twig roughly it's smaller and what we're gonna do is we're gonna pin some of these vertices to one tracker and then the other vertices to another and that's gonna kind of be like rotation but really it's just going to be a deformation so to do this last time we use parenting and if we parent so ctrl P we can only pan to one object at a time this doesn't have any of their rotation as you can see it's just staying vertical so we're gonna want to undo that okay so what we want to do is with this plane go into edit mode and then with the vertices select the bottom two we want both of these to be connected to this empty so with these two vertices control yes control click the empty and then ctrl H for the hook menu and then hook to selected object this is gonna pin our vertices to thus empty and now when we go to object mode and play it's kind of hard to tell but these vertices seem to be pinned let's do it for the other pair so plane edit mode select both of these control click control H hooks to selected object and now when we play this it's not really rotation it's kind of like a skew kind of and you see this isn't very very good once you get to the extreme of horizontal but this kind of has rotation data as you can see there are ways to do this properly by using an armature you can hook the tail to one and the head to the other we're not going to worry about this for now I just want to illustrate that two empties have more data than a single empty and they in some sense have scaling and rotation data so if these two trackers were to go further apart our plane would deform by stretching okay so that is that and actually we can just delete that and one thing that I thought about doing is for each layer that we go up so now we're on location rotation I just reveal one more setting that we could play with right so for example I want to talk about normalized for example it's not going to be very relevant for this piece of footage but it is something so it's just delete these trackers so what normalized does as you can see in the description normalizes light intensities while tracking what this means is that when it's tracking right now it's already counting for translation for rotation it's trying to match the pattern what happens if a shadow gets cast over the twig well from our eyes we have human intelligence we can see oh it's the same twig it's just a bit darker because it's in shadow blender sees this as a different pattern you know all the pixels are darker so normal eyes essentially what it does is it averages out to light intensity and in most cases it can actually ignore these lighting changes so it makes it invariant to lighting changes this is going to be slower to process because this is just another thing it has to go through but if I was to know cast a shadow over it generally would work also the other way around if somebody was to shine a flashlight if it wasn't too drastic of a change normal eyes would actually fix this so let's actually see if we can see any speed differences there shouldn't be shouldn't be too bad so this is a normal tracker that's about this is how long it takes and then we enable normal eyes shouldn't be too different control-t it's a bit slower nothing crazy but if you only have like you know in this case only two trackers enable normal eyes you're not he's ten seconds that you're losing they don't matter you know you're losing ten seconds you're getting something that's invariant to lighting and who knows maybe when I film this you know I was standing above this may be my shadow slightly cast over it you know anything is possible so so again with this tracker I just or with this motion model I just wanted to reveal another thing in the tracker settings normalized and now what we want to do is move up just another notch just one more layer of complexity and we're gonna be going to location rotation and scale which is going to be pretty much what we've been doing but also count for scale which is it's not the tracker getting bigger smaller it's the camera getting further and closer to the tracker making it seem like it's bigger and smaller again we have the human intelligence we know most of us we know that the tracker isn't changing size in most cases what's really happening is we're looking it's a machine it's blender it's looking at video it's looking at data it doesn't know any of this context it just sees this getting closer and further away or this thing rotating it's not really rotating it's the camera that's rotating whatever we know this so now let's just add another layer of complexity and the next motion model so we've gone over tracking location and even rotation and at this point you may be thinking there's no way we can up it from here there's nothing else we can add to this well you would be wrong because we have location rotation and scale and as we've just been talking about scale isn't really the idea of anchors themselves getting bigger and smaller in most cases but really is just the camera getting closer and further away from a tracker and if you're just viewing this as 2d footage that is the same idea as scaling so I actually have some footage that illustrates this idea perfectly so here we have this thing on the sidewalk it says no dumping drains to stream and we're using this to go closer and further away and it's also rotating and we're moving the position so this is really the perfect example for a location rotation scale and of course I've already turned this into an image sequence so we can import this in by finding the folder and then either wait and then either clicking the first one or clicking a and then open clip and this is imported except we don't see anything and you need to think why is that the case well first of all let's go to the first frame with shift left arrow that will go to the first frame and then if I open up our timeline we can see we have an endpoint of 200 frames which isn't necessarily correct for this footage so we need to hit set scene frames gives us 190 which is where I trimmed it apparently and we need a hit prefetch to load this into memory so just get used to hitting these two buttons one after another it's pretty much how you set things up and you see we have this footage imported okay so just like all the other times we're gonna see the limitations of the methods we've been using so far before using in this case location rotation and skill so the last motion model we used is location rotation so let's try to add a tracker with this and scale it up of course we don't need this whole emblem sign thing we can just get this fish part of it and let's try tracking this with ctrl T and it will probably make it pretty far up so it stopped on frame 27 and what's happening here is it's capturing the position of the pattern the rotation of the pattern it's matching from frame to frame but it's not getting the scale and this thing is getting closer and closer so on frame 26 we can just add a keyframe by scaling this up ctrl T how far does this make us is it gonna stop is it just gonna keep going there we go so again this is a brute-force approach you don't want to do this but it does work so instead we're gonna close this down and change the motion model to location rotation and scale and we're almost all the way up the ladder pretty much everyone is better than the one under it that's not always the case but it's generally true location rotation scale we're gonna add this in just like that and again we need to remember what's happening here we're tracking three channels again we're not holding this data but we're using three channels of information to match from frame to frame matching a pattern by translating rotating and now scaling so ctrl T it will start tracking soon enough and of course this should be slower than the other two because now we're handling even more possibilities for blender to check we can make it even worse by enabling normal eyes so it made it to frame 34 immediately which isn't too bad we're just gonna scale this up and also make our search area smaller since I don't think it's going to escape outside of it ctrl T and of course some other reasons that this might not be holding on is motion blur so let's just do one more round of this okay so at this point I want to introduce a an idea that will probably help us not add a new reference point every 15 frames or so and essentially the way we've been tracking is we have a reference point and then it's trying to match this pattern until it can't do it anymore and then we add another reference point and so on this isn't for this case apparently it's not the way to do it so there is another setting that I want to talk about for this and that is where is it it's right here it's match for this tracker the match is set to keyframe and that is essentially what we've been talking about so keyframe what does it say track pattern from keyframe to next frame exactly what we've been talking about reference point and then matching instead we can try to do this one right here missed it this one right here previous frame and what this does is every frame turns into a reference point so that each new frame is just referencing the one before it so it's not referencing 15 frames ago when we add our last keyframe but instead it's continuously updating the keyframe this would be great for something like this as long as it doesn't make a small error say that's fine and then keep amplifying it that's the danger of using previous frame but for this case I think will work so it's a new ctrl T and I think it is holding on a lot longer so again this is the power previous frame you're gonna get more drift in that's what's gonna happen when you use previous frame but you can always go back and refine it so you know not the worst thing in the world you could use this as an initial pass and then you correct some stuff in the graph editor that's something you could do but you see this time it made it all the way down and really this is the power of scaling and also the power of previous frame so that's not too bad and just like all the other times it's so important to note this tracker is using rotation and scale data it is not storing it again graph you see only x and y data nothing else and it's all staying very centered doesn't look like it here but if we trying to zoom out whatever if we zoom out and kind of squish this down I don't know what I'm doing wrong with this whatever if you squish this down it's all staying a fairly centered he seeds bounded between 10 and minus 10 which means that our point which is pretty much in the center of the footage is remaining very centered the X&Y locations aren't changing that much so one more thing I want to talk about before we move on again this doesn't hold scale data very important one more thing I want to talk about though is we've already talked about normalized but the most probably one of the most important things you could play around with is this correlation value so I'm just gonna read out what this says and then explain it so minimal value of correlation between match pattern and reference that is still treated as successful tracking so if you understood that that's great most of you probably just heard a bunch of nonsense so here is what's going on with the correlation value so the tracker has a reference point it goes down a frame and tries to match the pattern in this case using location rotation and scale now unless you have like a perfect camera and this algorithm does it perfectly the the pixels that it's gonna match this onto you aren't gonna be identically the same there's gonna be some noise in your footage or maybe the lighting changes a little so it matches it with some degree of tolerance it's not a hundred percent sure that it's a match so with every every frame that's tracked it also has this value of confidence you can think of it as so a confidence of one which is a hundred percent means this is definitely where the tracking should go next this is where the motion path should lead correlation or confidence of zero means it has no idea what it's doing and basically this correlation value is setting up a tolerance so if we said point seven five for example it's saying keep tracking until you go one frame and you have a confidence of under seventy-five percent and then just stop so here we're saying stop when you are under a certain degree of confidence and I could talk about this all day or I can show you what it means so let me show you what it means I'm gonna add a tracker but before I do that let's set a correlation not of one which is ridiculous but point 9:8 so needs to be 98 percent sure every frame that goes down or else it has to stop and to make this more drastic we need to keep it on keyframe and not previous frame because it makes it easier to match if it's previous frame let me just add one right here so what we expect from this is it's gonna stop very frequently because there's no way it's gonna be above 98 percent sure each time but we'll see about that control-t see if it makes it even one frame okay so it stopped 13 frames down from here to here it wasn't completely sure Oh giant mistake you guys are probably yelling at me I was changing the correlation here which is what we the settings we have when we add a tracker we need to be messing with the correlation values here I'm sorry it was a mistake let's do the same demo correlation of zero we'll see if that just keeps tracking no matter what I think that should be the case is it yep it's still going it's going very slowly you see here it started going off but it should just keep going because it has a correlation of zero so even if it's zero percent confident it will just keep chugging along yeah you see this exact pattern this is why you need a high correlation to avoid issues like this so we're just gonna close that down so let's position our tracker back here bring up the correlation to 0.75 so it needs to be 75% confident and now we're getting something that's sticking on much more in the center so sorry about that mistake I made earlier but pretty much everything I said is correct I just changed the value in the wrong area so again we're getting it to stop here because we only have a correlation of 0.75 and it must be under 75 per son confident that it can continue so that is the location rotation scale with information about correlation and match settings for keyframe and previous keyframe okay well what's next up the totem pole what's the next thing we need to learn about well that is gonna be what's it gonna be it's gonna be half fine ah fine okay so let's do half fine let me get a bit of water and then we'll do a fine okay so you may be wondering we've done movement rotation we've done scaling what other type of motion even is there you want to be thinking of distortion and what do I mean by this well what a fine refers to is stretching and skew so if we had a square and force and it's like on a piece of fabric and we stretched it it would turn into a rectangle now scale scales uniformly so won't pick up on the fact that it's turning into a rectangle that deformation would cause it to stop tracking another thing is if we took a rectangle and skewed it so it becomes kind of this diamond shape so one side goes up one side goes down that's another type of deformation that rotation can't handle location can't handle and scale can't handle so let me just give you an example of something like this so we have a fine let me just bring down the audio had somebody help me record this had a bit of a conversation so here we have here we have a dirty sock I used to use this for running long distance but it's a dirty sock and I put a tracker marker on it and you see here we're doing some deformation this is the stretching and this is the skew I was talking about all the tracking methods we've done so far would not it would just not handle this well so this is where a fine comes in and of course like every step before it it's gonna take more computation time but it's worth it f-fine generally when we're tracking things you want to use a fine you can sometimes use the thing above it which is perspective but generally we're just using a fine of course you may be thinking oh if everything beneath a fine is covered by a fine why not just use it well if you're sure that something only has a location data you can use that because it will save you so much time and you might avoid some inaccuracies that you might gather if you're also checking for other things that wouldn't necessarily be there so let's do this a fine we're going to import in the sequence did I just important the same thing I think I just did let me try that again where is it tracking a fine sequence import there we go okay so first thing we need to do set scene frames 1 to 111 and then prefetch so I cut out the beginning of the footage that had all the setup in the end where I you know move my hands off of it and what we are left with is just the stretching and the skewing which is all we needed so I'm gonna try location rotation scale it might work I highly doubt it though because of the reasons we talked about location rotation scale we can even enable normalize you know lighting conditions whatever and in this case there might actually be some changing in lighting conditions because light is gathering in the crevices of the wool that will change when there's bigger gaps in between it's kind of minut but it exists and we want the correlation 0.75 is fine okay let's turn off lock so we can actually move add a tracker scale it up and in this case we want a scale on the x-axis to cover this whole tracker so this is location rotation scale take one let's see how it does I'm thinking it will stay on for the beginning of it and then it will lose it once we do too much stretching I'll guess so it's doing well so far how embarrassing would it be if he made it all the way through I guess it is making it all the way through I don't know what to tell you well it made it all the way through so I guess what that says is that this is not a great example of a fine but I guess because it's such a simple design it really does just look like scaling and then a rotation but either way we're still going to go through the a fine example because it is much better at stuff like this and this is this is really as the perfect example for a fine this is really an affine transformation so same setup as before and now control T and see how this does now if this um let go of the tracker at some point I would be shocked because affine is destined for something like this so let's see how does and if anything this might actually be more accurate so that's one reason to not use location rotation scale it might have held on for the whole thing but it was it centered probably not as well because again it's approximating these things with scaling and rotation it's not the same thing as a fine okay so here we have a very nice affine deformation look at that that's really nice and we can lock that in and something else we could do is if we wanted to this is a very weird tracker so for example we could put a location tracker here here here and here so we have four trackers and we can have a little plane that deforms with this cross but that's just something we can do let me see if we can experiment a bit more with the affine this might not work but it's just something I want to try out let's say we're trying to track something that isn't really a tracker marker because so far we've just been dealing with you know perfectly placed tracker markers let's try tracking part of this bracelet I don't know if it's gonna work we're using the affine model rotate this a little we'll see how it does I wouldn't be shocked if it if it works perfectly yeah that worked pretty well you see at the end we're getting a bit of pinching that isn't necessarily there but this is staying on fairly well and again this is the power of the affine motion model I pretty much dropped this on anything if we instead try location rotation scale I want to see what happens so we can in these tracking options we can clear everything past this point so we have a fresh tracker and let's try changing this to location rotation scale and see if this does the same thing in light I don't know ctrl T now something you're seeing here is this bounding box this pattern area is always staying a square well why is that well because it's not deforming at all and you may want that you may not want that for the socket sample the actual sock you definitely don't want that you actually want the deformation here so it's not always staying the same aspect ratio but that is the affine settings and I think we are now ready to move on to the Big Kahuna this is the one that is the most powerful but you don't want to use it all the time and there are reasons for this and that is the perspective dun-dun-dun perspectives very good it does significantly better than every other motion model but it can be abused and it can be broken so first of all let's get some footage that we want to track with perspective so we have these tiles here specifically I want to focus on this tile and notice that we're revolving around it but not in a rotation kind of it's not just rotating we're not looking at it and then it's just turning this is a perspective change we're looking at it from a different angle is the way you want to think about it especially when we dip the camera down here and this is the idea of perspective so perspective tracks planar surfaces it's very important tracks planar surfaces if you did this on a curved ball it would sometimes work not the greatest idea so tracks the perspective deformation of planar surfaces and then this sense this is blenders real planar tracker there are other things in blender called the plane track this is blenders real planar tracker that accounts for a planar geometry in a very accurate way and the perspective things actually much faster than you think so let's close this out and open our perspective sequence set scene frames 1 to 200 and then prefetch okay so let's play through this footage make sure we have what we want okay perfect so there are a lot of things we can do here with perspective and here I want to introduce a new kind of idea which is normally you think of trackers as tracking these tiny little markers may be the size of a twig like we saw before but never something huge and the reason for that is because it's too much data and it can slow things down with the perspective tracker I highly recommend taking the extra computation time to take big surfaces there's no reason not to just because it's a tracker doesn't mean it needs to be a small point it can be a giant planar surface so let's enable perspective even enable normalize and I'm gonna bring the correlation up pretty high so not only is this checking for warping perspective warping true planar tracking not only is it averaging the light intensities with this normalize and not only well and the last thing is that it needs 90% confidence each frame or it's gonna stop and we're gonna use the keyframe model which is even more ridiculous I wonder how far this will go so we're gonna add in a tracker and like I said this is going to be huge so we're going to take each of these corners and line them up to the cracks in this floor and you can see in this view right here what we're doing this is actually a very good example for this I'm gonna make this a bit bigger so we can actually see the cracks on the edges make sure that's there and then we need our search box to be not too big but a bit bigger we'll see if we need to make it even bigger than this the motion isn't too drastic sigh doubt it so again we expect this to be extremely slow and just in case we have a crash I'm gonna save so ctrl T see how it does I expect this to be very accurate and you can think of each one of these tiles as its own perspective track but of course there is a smarter way to do this where you put the trackers on the cracks in between where there's these crosses but I want to try one whole tile now if this is too slow we might have an issue but we'll see how this does and later on when we apply all the again this is just the fundamentals here when we apply all these concepts to things like object tracking camera tracking we're gonna need a pick and choose what kind of tracker we're using and oftentimes I use the perspective tracker whenever we have planar surfaces even if it takes a long time to calculate an incredibly long time okay if this is not gonna do anything at all within the next 30 seconds I'm gonna you know shrink this down but same idea but Papa yeah you you think that a tracking is something that doesn't take a lot of time you know you're just matching small bundles of pixels with in pretty small search areas but it can get pretty ridiculous especially when you automate the whole thing you can do what is essentially automatic tracking in blender and I think we'll cover it we'll see yeah I don't think this is gonna fly ten more seconds and then I think we're gonna quit this one by ten more seconds I mean now so let's cancel this we the canceling doesn't take too long so again we have a bunch of different approaches we can either make this smaller so we only get this small piece of the tile or do the crosses and I say we do both so let's scale this down and look at this pattern it still has a lot of detail it should work of course it might be a little whoops it might be a little too similar to everything around it but we'll see and it turns out that I think it did actually end up tracking one frame but I'm just gonna clear everything behind it so now we only have frame two and let's try tracking backwards a frame I did alt left arrow we're gonna see if that processes Oh No if we get a freeze I'm gonna be disappointed there we go now something I want you to know is that it's not really this slow usually it's that I'm also recording with OBS which makes everything maybe twice as slow but whatever ctrl T to track forward see if it actually works this time there we go so again we don't really need to focus on this on these four corners I mean they'll be pretty accurate but expect a bit of shaking what we care about is the center point which we can view right here in this in this I keep calling it different things I want to call it search area but that's already taken up by this terminology over here but you can see that this center point is staying exactly where it's supposed to so let's have it do a couple more frames and you can see how this really is pointer tracking we're essentially tracking a chunk of a planar surface pretty much the same thing you do in mocha or with the planar track or in fusion I just wish there was a spline based solution here but this is what we got and you see it's starting to speed up a little what we can do is cancel this so it stops at whatever frame it's gonna be at and we'll find out what frame that is in a moment okay so here we can bring down the search area make that a bit small or even that should speed things up just a little more yeah there we go now we're now we're zooming if you thought modeling was thrilling if he thought lighting was trying mm-hmm this is where it's at motion tracking but seriously I've seen a lot more motion tracking type things on the blender subreddit so a lot of people are taking interest to it I think that's great because nobody's really integrating real world footage with blender which is kind of some of the core stuff in my opinion you can model all day and make these Pixar type things I'm not that into it whatever okay so I think we get the point this is very accurate if we play through this especially if we lock with L so it stays centered this is very accurate again that zoom out was because we changed our scale but again only focus on the center point even if it's oom Zin and of course at this point it stopped tracking so it starts vibrating but this is a very accurate track so let's try doing the cross solution with perspective tracks which is great because we have this cross that's going to be deforming as we move around so I'm just gonna make these pretty small Center it here and then we're going to add for them and see if we can track four at the same time like that smaller and this is going to be the basis for something very soon which is what blender calls plane tracking which requires four corners like we're doing right now well we'll see if this works again the fast version of this the fast approximation of this is a fine which is kind of like a pretty good but weak approximation of what we're doing here with perspective so a to select everything I'm going to save and then ctrl T and you see this is what I'm normally used to it's pretty fast you see a lot of them got detached so let's pick this one for example so it ends let's see where it ends and again it's not unexpected that it stopped tracking we had a 90 percent tolerance pretty high so let's add a keyframe I would keep going all the way to the end perfect and by the way the reason the reason it stopped tracking is as we move around I want you to focus on this area here we're kind of seeing even though it's very shallow the depths of the depth of the crack as we move around it's actually changing more than just planar perspective it's a 3d surface but we approximate it with this 2d planar surface but you can see what I mean it's kind of slightly changing here so let's scale this up a little and have it finish it does really does not like that frame let's see if we can just expand our search box there we go now it's working through that sometimes you don't even need to do the manual correction and all you have to do it's just a bit of editing on your search box okay so this one ends around here make that bigger and this time it makes it all the way to the end and then finally what do we have did we get all of them we didn't we need this one and keep tracking from there perfect so now we have these four trackers we can play through this turn this off and if we really want to see what this is looking like this isn't the best you know way to view what's going on but what we can do is first of all disable the paths so you can just see these tiny squares and how they're moving around looks very accurate and we can also actually hide the footage so we just see these squares and if you can kind of see this tile if you can imagine it then it's a pretty good track and in this case I think we got it in this case especially you can really see in this area right here how this is changing as a 3d surface and the beginning of this you can see this center of the cross and then as we change angle you can't even see it anymore which is why it's somewhat of a difficult track okay so let's enable this and then I said with each track we're gonna introduce a new idea with this one I want to talk about again it's not relevant to the shot but I need to talk about it at some point and that is this RGB black and white what's that all about so again in this case we're doing perspective tracks so blender will take a pattern and do all these perspective deformations and see which one will fit with the next keyframe and if it can do it it will keep tracking keep tracking I'm really losing my voice here it's okay I'm almost done recording for this session you saw you saw a change I change shirts right as getting like there's too much for moment but so it's doing all this tracking and it's taking this data of the pixels in our G and B that's red green and blue you know what we can do to actually speed things up we can track only one Channel so we can track only the blue Channel or only the green Channel or two channels at a time and in some cases this is actually a very good idea so for example let's say we have trying to think let's say we have a green screen and on that green screen we have like an orange marker or something like that which green and orange are pretty similar especially when it's just a dot on a green screen so in that case you might want to ignore like green for example and then if you're left over with red and blue you might see a lot more difference there or maybe the opposite maybe you only you only want to track green because that's where the real difference is the whole green screen versus this thing that's slightly off green who this is too much and then another thing we can do is just track black and white but what I recommend is keeping our G and B unless you have some reason to only track one color channel or you're like you know what I don't care I'm gonna take away like what is essentially a third of the computation time you're tracking a third less data because you're not checking blue Channel or maybe you're checking two-thirds less so that's the idea with that so my throat I just cannot talk anymore but this has been the very first part of I don't know what I want to call this the most extreme motion tracking tutorial out there in the next part which is gonna be long this is a long tutorial series we're gonna be talking about applications mainly and then some things that I deliberately left out like join tracks what's that about and then most importantly the feature nobody knows I don't know nobody knows about it is offset tracking which is right here we have offset tracking and this is the I do just give you a taste right now we're tracking this area but what we can do is with some magic we can keep our point right here but track a different area so we can get data from here but put the information over here so that's why it's offset track and you're offsetting your tracker but keeping the center point where it's supposed to be just a taste so hopefully you guys enjoyed this very first video this was just the core fundamentals I know I took it very slow but that was the point we just wanted to go over the theory the image sequences the motion models what each of these do except for a location scale but I think you can infer that's a good way to tell if you understood this tutorial do you know what location scale does what scenario would you use it in you know but we've gone over the motion models and pretty much most of these important settings we left over a couple things like detect features and we'll get into that eventually but thank you guys for watching if you want to support more videos like this I have a patreon I have 24 25 people who are donating on patreon I want to say thank you if you guys are watching this I really appreciate you guys donating money to free content on YouTube you don't need to but it's very nice that you do if you are a viewer and aren't on patreon on my patreon consider it but thank you guys for watching I'll see you guys on the next one and for the people waiting for the tracker marker removal tutorial it's coming probably after these videos but I make them somewhat quicker because I'm not editing this giant it's not a masterpiece this giant script scripted video but it's coming soon probably within a week thank you guys for I'm just I need to stop talking thank you guys for watching I tried to keep this as interesting as possible even though it's a very slow format I'll see you in the next one

Channel: CGMatter

Views: 114,374

Rating: 4.9666228 out of 5

Keywords: blender, 2.8, motion, tracking, matchmove, track, marker, tutorial, camera, object, solve, everything you need to know, location, rotation, scale, affine, perspective, cgmatter

Id: WLSGG7sDEac

Length: 77min 50sec (4670 seconds)

Published: Wed Jul 10 2019