Vector Projections : Data Science Basics

[Music] hello everybody in this video we're gonna be talking about what it means to project one vector onto another vector so we're gonna be talking about vector projections now vector projections is one of those concepts in linear algebra that I think the geometry behind it can be pretty intuitive the algebra gets a little bit complicated it's hard to see how the algebra relates to the geometry that we looked at and also beyond that it's kind of hard for a lot of students including myself at first to figure out why vector projections are even important in the context of data science for example I think that's because a lot of times they don't get taught in the concept in the context of data science so it can be difficult for students to see why are we doing all this crazy nonsense so with those ideas in mind well first talk about geometrically what is a vector projection then we'll talk about why it could be useful in a data science context and lastly we'll talk about how to derive the algebra from the more intuitive geometry so you guys can see that they're not two kind of independent entities but one related topic called vector projections cool so without any further ado let's go ahead and talk about what it means to take a vector projection what does it mean for one vector one vector to be projected onto another vector the lucky thing is that it's kind of all in the name so notice that we have these seven data points here all these little X's you see and each of them is a collection of a x1 variable and an x2 variable so they live in the two-dimensional plane here okay now let's say someone says can you draw the best-fit vector through these data points draw the vector which the all the other data points kind of tend to cluster around you would probably produce a vector very similar to this one because it seems that this direction this vector here is kind of the direction that all the data points are going in it's similar to just saying draw the best-fit line right now let's say your next task is to take any one of these data points which by the way can be represented as a vector although I've drawn these data points as X's for example this guy is drawn as an X but it's really also representable as a vector starting at the origin ending at that data point right so let's say someone says I want you to project that red vector which represents this data point here on to that best fit black vector that you drew earlier now without even knowing what a projection is I think a lot of people would find it clear that if we imagine like a light source coming from this direction what does the shadow of that red vector look like on this black vector in other words you're taking this red vector and projecting it onto that black vector you drew the answer would be that it looks like this it's going in the same direction as this original black vector you drew it just has a shorter magnitude right and we can do similar things for any other one of these data points for example here is another data point representable by this red vector right here and let's say we want to project that red vector onto the original black vector now imagine a light source coming from down here and we see that the projection we get is a shorter version of the original black vector we drew okay so that's all it means to project one vector onto another vector we basically geometrically I mean in as many dimensions as we can draw which is up to three dimensions we would take that vector and kind of just project it down imagine there's a light source coming from the direction perpendicular to this vector that we're looking at right here and then you see what the shadow of that vector would be on the target vector okay so that's kind of a fuzzy intuitive idea of what a projection means now of course the question is why would we ever want to do this I mean maybe there's some applications to physics maybe we care about actual literal shadows but in a more abstract context in data science why do we care about projecting one vector onto another vector well let's take a look at what happens if we were to successfully project all seven of these original data points on to that black vector we drew right here let's look at before we did that before we did that how much pieces of how much basically variables did we need to store all seven of these data points remember there are seven data points and each one has an x one variable and an X 2 variable so 7 x 2 means that there was 14 distinct numbers to store for our complete story so you can see that here so we have 1 2 3 4 5 6 7 rows and 2 columns so there are 14 total numbers to store in our computer or database or wherever in order to get the full picture here now let's say we successfully did project each of these seven data points on to this vector here let's say this vector is called you in fact you is not exactly that vector it's going to be a unit vector that goes in the same direction which is a quick tangent quick side note is that when you're projecting some vector like this guy onto a target vector it doesn't matter if you're projecting it on like this target vector or this target vector or this target vector it just has to be going in the same direction so for example I could have chosen to project it on to this vector or on to this vector I would get the same exact answer for my projection in the end so basically when you're projecting one vector on another vector the vector you're projecting onto its magnitude doesn't matter okay because in the end you're just projecting on a direction a particular direction so that's why that we can just be safe assuming that we're projecting always on a unit vector you going back to this story before we needed 14 distinct numbers in our database to store this entire story now what do we need now we basically need the unit vector we're projecting on so U which is given by u 1 and u 2 right because it lives in two dimensions so that's two pieces of information and then the only other pieces of information we need are the multipliers for all the projections remember why because we know after we project each of these seven data points on to the unit vector we already know the direction the direction is the direction of the unit vector Y because that's how projections work they take a vector and they projected onto a very specific other direction in this case the direction being u the only ambiguity the only thing we need to track is the magnitude of the projection itself remember it can be short it can be longer in fact it can be even longer than this if there was a point up here for example so we just care about the magnitude which we're gonna say is K K 1 through K 7 for each of these seven data points so there are seven magnitudes to store and there are two components in the U vector to store so that makes a grand total of nine different nine different variables nine different pieces of information we have the storing database so you're saying okay we went from 14 to 9 no big deal well yeah but that's because this was a very toy example imagine that you have billions and billions of data points and maybe they don't live in just two dimensions they have ten different dimensions right and now you're able to project those ten dimensions onto maybe two or three dimensions that can be a substantial gain in computational time in data storage a basic considerations like that right so hopefully you see Y vector projections can be a really big deal for us to do things in that nature it'll become more apparent when we do principal component analysis video and talk about vector projections in that context okay cool so that is the geometry behind vector projections and Y vector projections are important for data science the last part of this video we'll be using this geometry to derive the algebra behind vector projections alright welcome back so here is our goal we have vector X and we want to project it onto vector V of course if we just drew the solution geometrically that would be the easiest thing ever right we know that it's gonna be kind of like if I drew a perpendicular down that's right angle and then the answer would be this vector starting at the origin ending right here along the direction of vector V that is my answer let's call that answer as p4 now that's the thing we're gonna be trying to derive algebraically find a closed form solution for it before we do that let's go back to a concept we talked about earlier which was that if you project onto a vector it's the same thing as projecting onto a vector that's longer than it or shorter than it in our specific case to make our computations a lot easier we're going to be asking to project on the unit vector you remember a unit vector is a vector with length or magnitude one we're going to be projecting X onto the unit vector U which goes in the same direction as V and you'll see why that makes our computation a little bit easier first though how do we derive that unit vector well that's pretty easy right a unit vector is given by the vector divided by the magnitude of that vector take a second to think about that if I take my vector V right here and I divide it by its magnitude then it's going to have magnitude one the result vector will have magnitude 1 and of course it'll be going in the same direction as V because all I did was take a scalar multiplier of it if you're not familiar with this double bar notation this thing just means the magnitude of the vector inside the double bars okay so that's vector U is equal to V over magnitude of V now what's the next thing I need to know in order to find the projection let's define another vector which secretly lives in this diagram that's going to be vector D vector D is going to be starting right here and going up and ending where vector X tanks now notice something notice that vector P which is the projection vector plus vector D which is this one we just defined is vector X so P Plus D is equal to X okay convince yourself of that because we start here at P with the rules of vector addition then we go up D and that's the same thing as if we had X so P plus D is equal to X let's rearrange that guy a little bit so we have that D is equal to X minus P now we're going to do something very clever we're going to be using this subtle idea of the right angle which I talked about right here and we're gonna be using this to derive what vector P would have to be but before we do even that let's also define P as some multiplier of vector U right because in the end we know that P is going to be some scalar multiple of vector U because it has to be going in the same direction as the original vector V which is the same direction as u it's just a question of what's the scalar multiplier that's going to get attached to it so P is equal to K U so that D is equal to X minus K U okay now what we're gonna do is enforce that P and D are perpendicular to each other they just have to be because if P was any longer than the actual projection if it was this long then D would end up looking like this and P and D would no longer be perpendicular if P were any shorter than it should be D would end up looking like this and they would no longer be perpendicular either so the only way this whole projection business works out is if P and D are exactly perpendicular to each other and we're going to take advantage of that fact how are we going to take advantage of that fact we're going to take advantage of that pact by using the dot product remember the dot product of two vectors is equal to zero if those two vectors are perpendicular to each other so basically we want the dot product of which two vectors equal to zero we're looking at P and D so P being K U and D being X minus K U so what I want to do here is actually remove the diagram so I have some space on this side so we care about the dot product of K U dot remember this is just P the dot product of K U and X minus K u X minus K U which again is just D we care about the dot product of P and D being equal to zero the cool thing about dot products is that they behave just kind of like the way you would expect with distributions and stuff so we have K dot X I mean sorry K X dot u K being a scalar so u dot X is the same thing as X dot u so I just flipped them - that's where this - distributor comes from k squared u dot u has to be equal to zero okay so now we can do a lot of simplifications first of all there's a factor of K here a factor of K here and there's a 0 here so we can safely just remove a factor of K from all sides of the equation and now what is U dot u what is a unit vector dot itself well any vector dot itself just gives us the magnitude of that vector right or magnitude squared of that vector rather but since the magnitude of U is just 1 + 1 squared is just 1 this u dot u is just 1 and we can basically just not write it at all so what we get in the end after we rearrange this equation is equal to K is equal to X dot u and remember K is what we were looking for the entire time because K was the multiplier of vector U which tells us what is the actual projection of vector X onto vector V so now let's plug in K into the formula so we get P is equal to X dot u so we get P is equal to X dot u times u one last substitution to make before we're completely done with this guy remember U is equal to V over magnitude of V so we get this guy is equal to X dot V over magnitude of V times U and we can also expand this other U as the as the V representation as well okay so that's the answer the projection of vector X onto vector V is given by X dot V over magnitude of V times u where u is again V over magnitude of V so the formula looks really really ugly I'll give you that it's not pretty and that's why a lot of students get confused because it ends up being one of those concepts you just memorize it commits a memory instead of thinking about how you got there geometrically in fact it took us all this work just to get from the geometric geometric interpretation to the algebraic interpretation but the fact that we went through it the fact that we took the time to devise it means that we have a better understanding of how these two things are related right if we look at this guy we know that this guy is the multiplier and this guy is the unit vector in the direction of the thing we're projecting on and where does this come from we know it came from the fact that the dot product between P and D has to be 0 ok so that's in a nutshell projecting one vector onto another vector we'll be using this concept in principal component analysis ok so until next time

Channel: ritvikmath

Views: 24,924

Rating: 4.930131 out of 5

Keywords:

Id: X78tLBY3BMk

Length: 14min 58sec (898 seconds)

Published: Fri Sep 20 2019