"Eigenvectors and eigenvalues" is one of those topics that a lot of students find particularly unintuitive. Questions like "why are we doing this" and "what does this actually mean" are too often left just floating away in an unanswered sea of computations. And as I put out the videos of the series, a lot of you have commented about looking forward to visualizing this topic in particular. I suspect that the reason for this is not so much that eigen-things are particularly complicated or poorly explained. In fact, it's comparatively straightforward and I think most books do a fine job explaining it. The issue is that it only really make sense if you have a solid visual understanding for many of the topics that precede it. Most important here is that you know how to think about matrices as linear transformations, but you also need to be comfortable with things like determinants, linear systems of equations and change of basis. Confusion about eigen stuffs usually has more to do with a shaky foundation in one of these topics than it does with eigenvectors and eigenvalues themselves. To start, consider some linear transformation in two dimensions, like the one shown here. It moves the basis vector i-hat to the coordinates (3, 0) and j-hat to (1, 2), so it's represented with a matrix, whose columns are (3, 0) and (1, 2). Focus in on what it does to one particular vector and think about the span of that vector, the line passing through its origin and its tip. Most vectors are going to get knocked off their span during the transformation. I mean, it would seem pretty coincidental if the place where the vector landed also happens to be somewhere on that line. But some special vectors do remain on their own span, meaning the effect that the matrix has on such a vector is just to stretch it or squish it, like a scalar. For this specific example, the basis vector i-hat is one such special vector. The span of i-hat is the x-axis, and from the first column of the matrix, we can see that i-hat moves over to 3 times itself, still on that x axis. What's more, because of the way linear transformations work, any other vector on the x-axis is also just stretched by a factor of 3 and hence remains on its own span. A slightly sneakier vector that remains on its own span during this transformation is (-1, 1), it ends up getting stretched by a factor of 2. And again, linearity is going to imply that any other vector on the diagonal line spanned by this guy is just going to get stretched out by a factor of 2. And for this transformation, those are all the vectors with this special property of staying on their span. Those on the x-axis getting stretched out by a factor of 3 and those on this diagonal line getting stretched by a factor of 2. Any other vector is going to get rotated somewhat during the transformation, knocked off the line that it spans. As you might have guessed by now, these special vectors are called the "eigenvectors" of the transformation, and each eigenvector has associated with it, what's called an "eigenvalue", which is just the factor by which it stretched or squashed during the transformation. Of course, there's nothing special about stretching vs. squishing or the fact that these eigenvalues happen to be positive. In another example, you could have an eigenvector with eigenvalue -1/2, meaning that the vector gets flipped and squished by a factor of 1/2. But the important part here is that it stays on the line that it spans out without getting rotated off of it. For a glimpse of why this might be a useful thing to think about, consider some three-dimensional rotation. If you can find an eigenvector for that rotation, a vector that remains on its own span, what you have found is the axis of rotation. And it's much easier to think about a 3-D rotation in terms of some axis of rotation and an angle by which is rotating, rather than thinking about the full 3-by-3 matrix associated with that transformation. In this case, by the way, the corresponding eigenvalue would have to be 1, since rotations never stretch or squish anything, so the length of the vector would remain the same. This pattern shows up a lot in linear algebra. With any linear transformation described by a matrix, you could understand what it's doing by reading off the columns of this matrix as the landing spots for basis vectors. But often a better way to get at the heart of what the linear transformation actually does, less dependent on your particular coordinate system, is to find the eigenvectors and eigenvalues. I won't cover the full details on methods for computing eigenvectors and eigenvalues here, but I'll try to give an overview of the computational ideas that are most important for a conceptual understanding. Symbolically, here's what the idea of an eigenvector looks like. A is the matrix representing some transformation, with v as the eigenvector, and λ is a number, namely the corresponding eigenvalue. What this expression is saying is that the matrix-vector product - A times v gives the same result as just scaling the eigenvector v by some value λ. So finding the eigenvectors and their eigenvalues of a matrix A comes down to finding the values of v and λ that make this expression true. It's a little awkward to work with at first, because that left hand side represents matrix-vector multiplication, but the right hand side here is scalar-vector multiplication. So let's start by rewriting that right hand side as some kind of matrix-vector multiplication, using a matrix, which has the effect of scaling any vector by a factor of λ. The columns of such a matrix will represent what happens to each basis vector, and each basis vector is simply times λ, so this matrix will have the number λ down the diagonal with 0's everywhere else. The common way to write this guy is to factor that λ out and write it as λ times I, where I is the identity matrix with 1's down the diagonal. With both sides looking like matrix-vector multiplication, we can subtract off that right hand side and factor out the v. So what we now have is a new matrix - A minus λ times the identity, and we're looking for a vector v, such that this new matrix times v gives the zero vector. Now this will always be true if v itself is the zero vector, but that's boring. What we want is a non-zero eigenvector. And if you watched Chapters 5 and 6, you'll know that the only way it's possible for the product of a matrix with a non-zero vector to become zero is if the transformation associated with that matrix squishes space into a lower dimension. And that squishification corresponds to a zero determinant for the matrix. To be concrete, let's say your matrix a has columns (2, 1) and (2, 3), and think about subtracting off a variable amount λ from each diagonal entry. Now imagine tweaking λ, turning a knob to change its value. As that value of λ changes, the matrix itself changes, and so the determinant of the matrix changes. The goal here is to find a value of λ that will make this determinant zero, meaning the tweaked transformation squishes space into a lower dimension. In this case, the sweet spot comes when λ equals 1. Of course, if we have chosen some other matrix, the eigenvalue might not necessarily be 1, the sweet spot might be hit some other value of λ. So this is kind of a lot, but let's unravel what this is saying. When λ equals 1, the matrix A minus λ times the identity squishes space onto a line. That means there's a non-zero vector v, such that A minus λ times the identity times v equals the zero vector. And remember, the reason we care about that is because it means A times v equals λ times v, which you can read off as saying that the vector v is an eigenvector of A, staying on its own span during the transformation A. In this example, the corresponding eigenvalue is 1, so v would actually just a fixed in place. Pause and ponder if you need to make sure that line of reasoning feels good. This is the kind of thing I mentioned in the introduction, if you didn't have a solid grasp of determinants and why they relate to linear systems of equations having non-zero solutions, an expression like this would feel completely out of the blue. To see this in action, let's revisit the example from the start with the matrix whose columns are (3, 0) and (1, 2). To find if a value λ is an eigenvalue, subtracted from the diagonals of this matrix and compute the determinant. Doing this, we get a certain quadratic polynomial in λ, (3-λ)(2-λ). Since λ can only be an eigenvalue if this determinant happens to be zero, you can conclude that the only possible eigenvalues are λ equals 2 and λ equals 3. To figure out what the eigenvectors are that actually have one of these eigenvalues, say λ equals 2, plug in that value of λ to the matrix and then solve for which vectors this diagonally altered matrix sends to 0. If you computed this the way you would any other linear system, you'd see that the solutions are all the vectors on the diagonal line spanned by (-1, 1). This corresponds to the fact that the unaltered matrix [(3, 0), (1, 2)] has the effect of stretching all those vectors by a factor of 2. Now, a 2-D transformation doesn't have to have eigenvectors. For example, consider a rotation by 90 degrees. This doesn't have any eigenvectors, since it rotates every vector off of its own span. If you actually try computing the eigenvalues of a rotation like this, notice what happens. Its matrix has columns (0, 1) and (-1, 0), subtract off λ from the diagonal elements and look for when the determinant is 0. In this case, you get the polynomial λ^2+1, the only roots of that polynomial are the imaginary numbers i and -i. The fact that there are no real number solutions indicates that there are no eigenvectors. Another pretty interesting example worth holding in the back of your mind is a shear. This fixes i-hat in place and moves j-hat one over, so its matrix has columns (1, 0) and (1, 1). All of the vectors on the x-axis are eigenvectors with eigenvalue 1, since they remain fixed in place. In fact, these are the only eigenvectors. When you subtract off λ from the diagonals and compute the determinant, what you get is (1-λ)^2, and the only root of this expression is λ equals 1. This lines up with what we see geometrically that all of the eigenvectors have eigenvalue 1. Keep in mind though, it's also possible to have just one eigenvalue, but with more than just a line full of eigenvectors. A simple example is a matrix that scales everything by 2, the only eigenvalue is 2, but every vector in the plane gets to be an eigenvector with that eigenvalue. Now is another good time to pause and ponder some of this before I move on to the last topic. I want to finish off here with the idea of an eigenbasis, which relies heavily on ideas from the last video. Take a look at what happens if our basis vectors just so happened to be eigenvectors. For example, maybe i-hat is scaled by -1 and j-hat is scaled by 2. Writing their new coordinates as the columns of a matrix, notice that those scalar multiples -1 and 2, which are the eigenvalues of i-hat and j-hat, sit on the diagonal of our matrix and every other entry is a 0. Anytime a matrix has 0's everywhere other than the diagonal, it's called, reasonably enough, a diagonal matrix. And the way to interpret this is that all the basis vectors are eigenvectors, with the diagonal entries of this matrix being their eigenvalues. There are a lot of things that make diagonal matrices much nicer to work with. One big one is that it's easier to compute what will happen if you multiply this matrix by itself a whole bunch of times. Since all one of these matrices does is scale each basis vector by some eigenvalue, applying that matrix many times, say 100 times, is just going to correspond to scaling each basis vector by the 100-th power of the corresponding eigenvalue. In contrast, try computing the 100-th power of a non-diagonal matrix. Really, try it for a moment, it's a nightmare. Of course, you will rarely be so lucky as to have your basis vectors also be eigenvectors, but if your transformation has a lot of eigenvectors, like the one from the start of this video, enough so that you can choose a set that spans the full space, then you could change your coordinate system so that these eigenvectors are your basis vectors. I talked about change of basis last video, but I'll go through a super quick reminder here of how to express a transformation currently written in our coordinate system into a different system. Take the coordinates of the vectors that you want to use as a new basis, which, in this case, means are two eigenvectors, that make those coordinates the columns of a matrix, known as the change of basis matrix. When you sandwich the original transformation putting the change of basis matrix on it's right and the inverse of the change of basis matrix on its left, the result will be a matrix representing that same transformation, but from the perspective of the new basis vectors coordinate system. The whole point of doing this with eigenvectors is that this new matrix is guaranteed to be diagonal with its corresponding eigenvalues down that diagonal. This is because it represents working in a coordinate system where what happens to the basis vectors is that they get scaled during the transformation. A set of basis vectors, which are also eigenvectors, is called, again, reasonably enough, an "eigenbasis". So if, for example, you needed to compute the 100-th power of this matrix, it would be much easier to change to an eigenbasis, compute the 100-th power in that system, then convert back to our standard system. You can't do this with all transformations. A shear, for example, doesn't have enough eigenvectors to span the full space. But if you can find an eigenbasis, it makes matrix operations really lovely. For those of you willing to work through a pretty neat puzzle to see what this looks like in action and how it can be used to produce some surprising results, I'll leave up a prompt here on the screen. It takes a bit of work, but I think you'll enjoy it. The next and final video of this series is going to be on abstract vector spaces. See you then!
I wish this would go further, it's been so great. Generalized eigenspaces and nilpotence (which he hinted at) and inner product spaces and orthogonal bases and Jordan form and adjoint transormations and all that, geometrically. Of course, a lot of that stuff is hard to describe (at least, in a way that adequately encapsulates their full power) without allowing for the field to be C, but /u/3Blue1Brown does such a good job conveying actions on the complex plane visually in his other videos, I feel like so much of the stuff I only ever learned purely algebraically and abstractly could be reexamined with visual intuition.
EDIT: By the way, if any of you are having trouble working out the puzzle posed toward the end of the video, here is a short paper going over it. They use a slightly rearranged matrix, one with its diagonal flipped, and therefore end up with flipped eigenvectors, but conceptually and computationally it's the same problem otherwise.
I was more excited for this than the new season of narcos.
Animating the process of finding the eigenvalue and watching the determinant become zero was spectacular. I laughed with delight to see the connection.
His channel is great
These are so good!
This is be a great way to demonstrate that "multiplication by i is equivalent to rotating by 90 degrees". Just look at the eigenvalues of the rotation transform!
This video was genuinely a mind blow for me.
Eigenvectors with complex eigenvalues (and complex entries) come up a lot in quantum computation.
For example, the 90-degree rotation matrix [[0,-1],[1,0]] has eigenvectors [1,i] and [1,-i]. Those are the basis vectors that define the Y axis of the Bloch sphere, and that indicates that in the Bloch-sphere representation of a 2d complex vector this operation is a rotation around the Y axis.
A rotation around the X axis would be [[0,i],[i,0]]. It has eigenvectors [1,1] and [1,-1]. For the Z axis it's [[1,0],[0,i]] with eigenvectors [1,0] and [0,1]. You can compute the rotation matrices by exponentiating the corresponding Pauli matrix times i times the radians you want to rotate.
I have been thinking lately about eigenvectors and values and their similarity to solving for roots of a polynomial and dimensional analysis of physical units. Are these all some how related to a change or basis? Is there some unifying mathematical theory on this? Is it similar to the concept of an identity element and an inverse element? Sorry i am not mathematically trained if this question seems silly.