Okay. This lecture is mostly about
the idea of similar matrixes. I'm going to tell you what
that word similar means and in what way two
matrixes are called similar. But before I do that,
I have a little more to say about positive
definite matrixes. You can tell this is a subject I
think is really important and I told you what positive
definite meant -- it means that this -- this expression, this
quadratic form, x transpose I x is always positive. But the direct way to test
it was with eigenvalues or pivots or determinants. So I -- we know what it
means, we know how to test it, but I didn't really say where
positive definite matrixes come from. And so one thing I want to say
is that they come from least squares in -- and all sorts of
physical problems start with a rectangular matrix -- well,
you remember in least squares the crucial combination
was A transpose A. So I want to show that that's
a positive definite matrix. Can -- so I -- I'm going to speak a little
more about positive definite matrixes, just recapping -- so let me ask a question. It may be on the homework. Suppose a matrix A
is positive definite. I mean by that it's all -- I'm assuming it's symmetric. That's always built
into the definition. So we have a symmetric
positive definite matrix. What about its inverse? Is the inverse of a symmetric
positive definite matrix also symmetric positive definite? So you quickly think,
okay, what do I know about the pivots
of the inverse matrix? Not much. What do I know about
the eigenvalues of the inverse matrix? Everything, right? The eigenvalues
of the inverse are one over the eigenvalues
of the matrix. So if my matrix starts
out positive definite, then right away I know that its
inverse is positive definite, because those positive
eigenvalues -- then one over the
eigenvalue is also positive. What if I know that A -- a
matrix A and a matrix B are both positive definite? But let me ask you this. Suppose if A and B are positive
definite, what about -- what about A plus B? In some way, you hope
that that would be true. It's -- positive definite for a
matrix is kind of like positive for a real number. But we don't know the
eigenvalues of A plus B. We don't know the
pivots of A plus B. So we just, like, have to go
down this list of, all right, which approach to
positive definite can we get a handle on? And this is a good one. This is a good one. Can we -- how would
we decide that -- if A was like this and
if B was like this, then we would look at
x transpose A plus B x. I'm sure this is
in the homework. Now -- so we have x transpose
A x bigger than zero, x transpose B x positive
for all -- for all x, so now I ask you about this guy. And of course, you
just add that and that and we get what we want. If A and B are positive
definites, so is A plus B. So that's what I've shown. So is A plus B. Just -- be sort of ready for
all the approaches through eigenvalues and through
this expression. And now, finally, one more
thought about positive definite is this combination that
came up in least squares. Can I do that? So now -- now suppose A
is rectangular, m by n. I -- so I'm sorry that
I've used the same letter A for the positive definite
matrixes in the eigenvalue chapter that I used way back
in earlier chapters when the matrix was rectangular. Now, that matrix --
a rectangular matrix, no way its positive definite. It's not symmetric. It's not even square in general. But you remember that the key
for these rectangular ones was A transpose A. That's square. That's symmetric. Those are things we knew -- we knew back when
we met this thing in the least square stuff,
in the projection stuff. But now we know
something more -- we can ask a more important
question, a deeper question -- is it positive definite? And we sort of hope so. Like, we -- we might -- in analogy with
numbers, this is like -- sort of like the square of a
number, and that's positive. So now I want to ask
the matrix question. Is A transpose A
positive definite? Okay, now it's -- so again,
it's a rectangular A that I'm starting with, but it's
the combination A transpose A that's the square, symmetric
and hopefully positive definite matrix. So how -- how do I see that
it is positive definite, or at least positive
semi-definite? You'll see that. Well, I don't know the
eigenvalues of this product. I don't want to work
with the pivots. The right thing -- the right
quantity to look at is this, x transpose Ax -- A -- x transpose times
my matrix times x. I'd like to see
that this thing -- that that expression
is always positive. I'm not doing it with numbers,
I'm doing it with symbols. Do you see -- how do I see
that that expression comes out positive? I'm taking a rectangular
matrix A and an A transpose -- that gives me something
square symmetric, but now I want to see
that if I multiply -- that if I do this -- I form this quadratic
expression that I get this positive thing that
goes upwards when I graph it. How do I see that
that's positive, or absolutely it
isn't negative anyway? We'll have to, like, spend
a minute on the question could it be zero, but
it can't be negative. Why can this never be negative? The argument is -- like the one key idea in so
many steps in linear algebra -- put those parentheses
in a good way. Put the parentheses around
Ax and what's the first part? What's this x
transpose A transpose? That is Ax transpose. So what do we have? We have the length
squared of Ax. We have -- that's the column
vector Ax that's the row vector Ax, its length squared,
certainly greater than or possibly equal to zero. So we have to deal with
this little possibility. Could it be equal? Well, when could the
length squared be zero? Only if the vector
is zero, right? That's the only vector that
has length squared zero. So we have -- we
would like to -- I would like to get that
possibility out of there. So I want to have Ax
never -- never be zero, except of course
for the zero vector. How do I assure that
Ax is never zero? The -- in other words, how do I
show that there's no null space of A? The rank should be -- so now remember -- what's
the rank when there's no null space? By no null space,
you know what I mean. Only the zero vector
in the null space. So if I have a -- if I
have an 11 by 5 matrix -- so it's got 11 rows, 5 columns,
when is there no null space? So the columns should be
independent -- what's the rank? n 5 -- rank n. Independent columns,
when -- so if I -- then I conclude yes,
positive definite. And this was the assumption
-- then A transpose A is invertible -- the least squares
equations all work fine. And more than that -- the matrix
is even positive definite. And I just to say one comment
about numerical things, with a positive definite
matrix, you never have to do row exchanges. You never run into unsuitably
small numbers or zeroes in the pivot position. They're the right -- they're the
great matrixes to compute with, and they're the great
matrixes to study. So that's -- I wanted to take
this first ten minutes of grab the first ten minutes away from
similar matrixes and continue a -- this much more
with positive definite. I'm really at this
point, now, coming close to the end of the heart
of linear algebra. The positive definiteness
brought everything together. Similar matrixes, which is
coming the rest of this hour is a key topic, and
please come on Monday. Monday is about what's called
the SVD, singular values. It's the -- has become
a central fact in -- a central part of
linear algebra. I mean, you can come
after Monday also, but -- Monday is, -- that singular
value thing has made it into this course. Ten years ago, five years
ago it wasn't in the course, now it has to be. Okay. So can I begin today's
lecture proper with this idea of similar matrixes. This is what similar
matrixes mean. So here -- let's start again. I'll write it again. So A and B are similar. A and B are -- now I'm
-- these matrixes -- I'm no longer talking about
symmetric matrixes, in -- at least no longer expecting
symmetric matrixes. I'm talking about two
square matrixes n by n. A and B, they're
n by n matrixes. And I'm introducing
this word similar. So I'm going to say
what does it mean? It means that they're
connected in the way -- well, in the way I've written
here, so let me rewrite it. That means that for some matrix
M, which has to be invertible, because you'll see that -- this one matrix is -- take the other matrix,
multiply on the right by M and on the
left by M inverse. So the question is,
why that combination? But part of the answer
you know already. You remember -- we've done this
-- we've taken a matrix A -- so let's do an
example of similar. Suppose A -- the matrix A
-- suppose it has a full set of eigenvectors. They go in this
eigenvector matrix S. Then what was the main
point of the whole -- the main calculation of the
whole chapter was -- is -- use that eigenvector
matrix S and its inverse comes over there to produce the
nicest possible matrix lambda. Nicest possible
because it's diagonal. So in our new language, this is
saying A is similar to lambda. A is similar to lambda,
because there is a matrix, and this particular -- there is an M and
this particular M is this important guy,
this eigenvector matrix. But if I take a different matrix
M and I look at M inverse A M, the result won't
come out diagonal, but it will come out a
matrix B that's similar to A. Do you see that I'm --
what I'm doing is, like -- I'm putting these
matrixes into families. All the matrixes in one -- in
the family are similar to each other. They're all -- each one in this
family is connected to each other one by some
matrix M and the -- like the outstanding member of
the family is the diagonal guy. I mean, that's the
simplest, neatest matrix in this family of all the
matrixes that are similar to A, the best one is lambda. But there are lots of others,
because I can take different -- instead of S, I can
take any old matrix M, any old invertible
matrix and -- and do it. I'd better do an example. Okay. Suppose I take A as the
matrix two one one two. Okay. Do you know the eigenvalue
matrix for that? The eigenvalues of
that matrix are -- well, three and one. So that -- and the eigenvectors
would be easy to find. So this matrix is
similar to this one. But my point is -- but also, I can also take
my matrix, two one one two, I could multiply it by
-- let's see, what -- I'm just going to cook
up a matrix M here. I'm -- I'll -- let me just
invent -- one four one zero. And over here I'll
put M inverse, and because I happened
to make that triangular, I know that its
inverse is that, right? So there's M inverse A M, that's
going to produce some matrix -- oh, well, I've got to
do the multiplication, so hang on a second, let -- I'll just copy that
one minus four zero one and multiply these guys so I'm
getting two nine one and six, I think. Can you check it as I go,
because you -- see I'm just -- so that's two minus four,
I'm getting a minus two nine minus 24 is a minus 15, my
God, how did I get this? And that's probably one and six. So there's my matrix B. And there's my matrix
lambda, there's my matrix A and my point is these
are all similar matrixes. They all have
something in common, besides being just two by two. They have something in common. And that's -- and what is it? What's the point about two
matrixes that are built out of -- the B is built out
of M inverse A M. What is it that A
and B have in common? That's the main -- now I'm
telling you the main fact about similar matrixes. They have the same eigenvalues. This is -- this chapter
is about eigenvalues, and that's why we're interested
in this family of matrixes that have the same eigenvalues. What are the eigenvalues
in this example? Lambda. The eigenvalues of
that I could compute. The eigenvalues of that I
can compute really fast. So the eigenvalues
are three and one -- for this for sure. Now did we -- do you see why the
eigenvalues are three and one for that one? If I tell you the eigenvalues
are three and one, you prick -- quickly process the trace,
which is -- and four -- agrees with four and you
process the determinant, three times one -- the determinant is three
and you say yes, it's right. Now I'm hoping that the
eigenvalues of this thing are three and one. May I process the trace and
the determinant for that one? What's the trace here? The trace of this matrix
is four minus two and six, and that's what it should be. What's the determinant minus
twelve plus fifteen is three. The determinant is three. The eigenvalues of that
matrix are also three and one. And you see I created
this matrix just like -- I just took any M, like,
one that popped into my head and computed M inverse
A M, got that matrix, it didn't look anything
special but it's -- like A itself, it has those
eigenvalues three and one. So that's the main fact
and let me write it down. Similar matrixes have
the same eigenvalues. So I'll just put that
as an important point. And think why. Why is that? So that's what that
family of matrixes is. The matrixes that
are similar to this A here are all the matrixes with
eigenvalues three and one. Every matrix with
eigenvalues three and one, there's some M that
connects this guy to the one you think of. And then of course, the most
special guy in the whole family is the diagonal one with
eigenvalues three and one sitting there on the diagonal. But also, I could find -- I mean, tell me just a couple
more members of the family. Another -- tell me another
matrix that has eigenvalues three and one. Well, let's see, I -- oh,
I'll just make it triangular. That's in the family. There is some M that --
that connects to this one. And -- and also this. There's some matrix M -- so that
M inverse A M comes out to be that. There's a whole family here. And they all share
the same eigenvalues. So why is that? Okay. I'm going to start -- the only
possibility is to start with Ax equal lambda x. Okay, so suppose A has
the eigenvalue lambda. Now I want to get B into
the picture here somehow. You remember B is M inverse A M. Let's just remember
that over here. B is M inverse A M. And I want to see
its eigenvalues. How I going to get M inverse
A M into this equation? Let me just sort of do it. I'll put an M times an M
inverse in there, right? That was -- I haven't changed
the left-hand side, so I better not change
the right-hand side. So everybody's okay so far,
I just put in there -- see, I want to get a -- so now
I'll multiply on the left by M inverse -- I have to do the
same to this side and that number
lambda's just a number, so it factors out in the front. So what I have here
is this was safe. I did the same
thing to both sides. And now I've got B. There's B. That's B times this
vector M inverse x is equal to lambda times
this vector M inverse x. So what have I learned? I've learned that
B times some vector is lambda times that vector. I've learned that lambda
is an eigenvalue of B also. So this is -- if
-- so this is -- if lambda's an eigenvalue of
A, then I can write it this way and I discover that
lambda's an eigenvalue of B. That's the end of the proof. The eigenvector
didn't stay the same. Of course I don't expect the
eigenvectors to stay the same. If all the eigenvalues are the
same and all the eigenvectors are the same, then probably
the matrix is the same. Here the eigenvector changes,
so the eigenvector -- so the point is then
the eigenvector of B -- of B is M inverse times
the eigenvector of A. Okay. That's all that this says here. The eigenvector of A was
X, and so the M inverse -- similar matrixes, then
have the same eigenvalues and their eigenvectors
are just moved around. Of course, that's what we --
that's what happened way back -- and the most important similar
matrixes are to diagonalize. So what was the point
when we diagonalized? The eigenvalues stayed
the same, of course. Three and one. What about the eigenvectors? The eigenvectors were whatever
they were for the matrix A, but then what were
the eigenvectors for the diagonal matrix? They're just -- what are the
eigenvectors of a diagonal matrix? They're just one
zero and zero one. So this step made the
eigenvectors nice, didn't change the
eigenvalues, and every time we don't change the eigenvalues. Same eigenvalues. Okay. Now -- so I've got all
these matrixes in -- I've got this family of matrixes
with eigenvalues three and one. Fine. That's a nice family. It's nice because those two
eigenvalues are different. I now have to -- to get into that -- the -- into the less happy
possibility that the two eigenvalues could be the same. And then it's a little
trickier, because you remember when two eigenvalues
are the same, what's the bad possibility? That there might
not be enough -- a full set of eigenvectors
and we might not be able to diagonalize. So I need to discuss
the bad case. So the bad -- can
I just say bad? If lambda one equals
lambda two, then the matrix might not be diagonalizable. Suppose lambda one equals
lambda two equals four, say. Now if I look at the family of
matrixes with eigenvalues four and four, well, one
possibility occurs to me. One family with eigenvalues four
and four has this matrix in it, four times the identity. Then another -- but now I want
to ask also about the matrix four four one zero. And my point -- here's
the whole point of this -- of this bad stuff, is that this
guy is not in the same family with that one. The family of a -- of matrixes
that have eigenvalues four and four is two families. There's this total loner
here who's in a family off -- right? Just by himself. And all the others
are in with this guy. So the big family
includes this one. And it includes a whole lot
of other matrixes, all -- in fact, in this two by two
case, it -- you see where -- what do I mean -- so what I
using, this word family -- in a family, I mean
they're similar. So my point is that the only
matrix that's similar to this is itself. The only matrix that's similar
to four times the identity is four times the identity. It's off by itself. Why is that? The -- if this is my matrix,
four times the identity, and I take it, I multiply on
the right by any matrix M, I multiply on the left by
M inverse, what do I get? This is any M, but
what's the result? Well, factoring
out a four, that's just the identity
matrix in there. So then the M inverse
cancels the M, so I've just got this
matrix back again. So whatever the M
is, I'm not getting any more members of the family. So this is one small family,
because it only has one person. One matrix, excuse me. I think of these matrixes
as people by this point, in eighteen oh six. Okay, the other family
includes all the rest -- all other matrixes that have
eigenvalues four and four. This is somehow the
best one in that family. See, I can't make it diagonal. If I -- if it's
diagonal, it's this one. It's in its own, by itself. So I have to think, okay,
what's the nearest I can get to diagonal? But it will not
be diagonalizable. That -- do you know that that
matrix is not diagonalizable? Of course, because if
it was diagonalizable, it would be similar to
that, which it isn't. The eigenvalues of
this are four and four, but what's the catch
with that matrix? It's only got one eigenvector. That's a
non-diagonalizable matrix. Only one eigenvector. And somehow, if I made that
one into a ten or to a million, I could find an M, it's in
the family, it's similar. But the best -- so the best
guy in this family is this one. And this is called the Jordan -- so this guy Jordan picked
out -- so he, like, studied, these families of
matrixes, and each family, he picked out the nicest,
most diagonal one. But not completely diagonal,
because there's nobody -- there isn't a diagonal
matrix in this family, so there's a one up
there in the Jordan form. Okay. I think we've got to see some
more matrixes in that family. So, all right, let me --
let's just think of some other matrixes whose eigenvalues are
four and four but they're not four times the identity. So -- and I believe that -- that this -- that all the
examples we pick up will be similar to each other
and -- do you see why -- in this topic of
similar matrixes, the climax is the Jordan form. So it says that every matrix -- I'll write down what the Jordan
form -- what Jordan discovered. He found the best looking
matrix in each family. And that's -- then we've got --
then we've covered all matrixes including the
non-diagonalizable one. That -- that's the
point, that in some way, Jordan completed the
diagonalization by coming as near as he could,
which is his Jordan form. And therefore, if you want
to cover all matrixes, you've got to get
him in the picture. It used to be -- when
I took eighteen oh six, that was the climax of the
course, this Jordan form stuff. I think it's not the climax
of linear algebra anymore, because -- it's not easy to
find this Jordan form for a general matrix, because
it depends on these eigenvalues being exactly the same. You'd have to know exactly
the eigenvalues and it -- and you'd have to know exactly
the rank and the slightest change in numbers will
change those eigenvalues, change the rank and therefore
the whole thing is numerically not an -- a good thing. But for algebra,
it's the right thing to understand this family. So just tell me another matrix
-- a few more matrixes -- so more members of the family. Let me put down again
what the best one is. Okay. All right. Some more matrixes. Let's see, what I looking for? I'm looking for matrixes
whose trace is what? So if I'm looking for more
matrixes in the family, they'll all have the same
eigenvalues, four and four. So their trace will be eight. So why don't I just take,
like, five and three -- I've got the trace right, now
the determinant should be what? Sixteen. So I just fix this up -- shall I
put maybe a one and a minus one there? Okay. There's a matrix with
eigenvalues four and four, because the trace is eight and
the determinant is sixteen. And I don't think
it's diagonalizable. Do you know why it's
not diagonalizable? Because if it was
diagonalizable, the diagonal form
would have to be this. But I can't get to that
form, because whatever I do with any M inverse and
M I stay with that form. I could never get
-- connect those. So I can put down more
members -- here -- here's another easy one. I could put the four and
the four and a seventeen down there. All these matrixes are similar. If I'm -- I could find an M
that would show that that one is similar to that one. And in -- you can see the
general picture is I can take any a and any 8-a here and
any -- oh, I don't know, whatever you put it'd be
-- anyway, you can see. I can fill this in, fill this in
to make the trace equal eight, the determinant equal 16, I
get all that family of matrixes and they're all similar. So we see what eigenvalues do. They're all similar and they
all have only one eigenvector. So I -- if I'm -- if
you were going to -- allow me to add to this picture,
they have the same lambdas and they also have the
same number of independent eigenvectors. Because if I get an eigenvector
for x I get one for -- for A, I get one for B also. So -- and same number
of eigenvectors. But even more than that -- even more than that -- I mean, it's not enough
just to count eigenvectors. Yes, let me give you an
example why it's not even enough to count eigenvectors. So another example. So here are some matrixes -- oh, let me make
them four by four -- okay, here -- here's a matrix. I mean, like if you
want nightmares, think about matrixes like these. Uh, so a one off the diagonal
-- say a one there, how many -- what are the eigenvalues
of that matrix? Oh, I mean -- okay. What are the eigenvalues
of that matrix? Please. Four 0s, right? So we're really getting
bad matrixes now. So I mean, this is, like -- Jordan was a good guy, but he
had to think about matrixes that all -- that had, like -- an
eigenvalue repeated four times. How many eigenvectors
does that matrix have? Well, I'm --
eigenvectors will be -- since the eigenvalue is
zero, eigenvectors will be in the null space, right? I'm -- eigenvectors have
got to be A x equal zero x. So what's the dimension
of the null space? Two. Somebody said two. And that's right. How -- why? Because you ask what's
the rank of that matrix, the rank is obviously two. The number of
independent rows is two, the number of independent
columns is two, the rank is two so the null --
the dimension of the null space is four minus two, so
it's got two eigenvectors. Two eigenvectors. Two independent eigenvectors. All right. The dimension of the
null space is two. Now, suppose I change
this zero to a seven. The eigenvalues are all still
zero, how -- what about -- how many eigenvectors? What's the dimension of the --
what's the rank of this matrix now? Still two, right? So it's okay. And actually, this would
be similar to the one that had a zero in there. But it's not as beautiful,
Jordan picked this one. He picked -- he put ones -- we have a one on the -- above
the diagonal for every missing eigenvector, and here we're
missing two because we've got two, so we've got two
eigenvectors and two are missing, because it's
a four by four matrix. Okay, now -- but I was going to
give you this second example. 0 1 0 0, let me
just move the one. Oop, not there. Off the diagonal and
zero zero zero zero zero. Okay. So now tell me
about this matrix. Its eigenvalues are
four zeroes again. Its rank is two again. So it has two eigenvectors
and two missing. But the darn thing is
not similar to that one. A -- a count of eigenvectors
looks like these could be similar, but they're not. Jordan -- see, this is like --
a little three by three block and a little one by one block. And this one is like a
two by two block and a two by two block, and those blocks
are called Jordan blocks. So let me say what
is a Jordan block? J block number I has -- so a Jordan block has a repeated
eigenvalue, lambda I, lambda I on the diagonal. Zeroes below and ones above. So there's a block
with this guy repeated, but it only has one eigenvector. So a Jordan block has
one eigenvector only. This one has one eigenvector,
this block has one eigenvector and we get two. This block has one
eigenvector and that block has one eigenvector and we get two. So -- but the blocks
are different sizes. And that -- it turns
out Jordan worked out -- then this is not similar,
not similar to this one. So the -- so I'm, like,
giving you the whole story -- well, not the whole story, but
the main themes of the story -- is here's Jordan's theorem. Every square matrix A is
similar to A Jordan matrix J. And what's a Jordan matrix J? It's a matrix with
these blocks, block -- Jordan block number one, Jordan
block number two and so on. And let's say Jordan
block number d. And those Jordan
blocks look like that, so the eigenvalues are
sitting on the diagonal, but we've got some of these
ones above the diagonal. We've got the number of -- so the number of blocks -- the number of blocks is
the number of eigenvectors, because we get one
eigenvector per block. So what I'm -- so if I
summarize Jordan's idea -- start with any A. If its eigenvalues are
distinct, then what's it similar to? This is the good case. if I start with a matrix A and
it has different eigenvalues -- it's n eigenvalues, none
of them are repeated, then that's a diagonal --
diagonalizable matrix -- the Jordan blocks is -- has --
the Jordan matrix is diagonal. It's lambda. So the good case -- the good case, J is lambda. All -- there are -- d=n. There are n eigenvectors, n
blocks, diagonal, everything great. But Jordan covered
all cases by including these cases of repeated
eigenvalues and missing eigenvectors. Okay. That's a description of Jordan. That -- that's -- I haven't told you how
to compute this thing, and it isn't easy. Whereas the good case is the --
the good case is what 18.06 is about. The -- this case is what
18.06 was about 20 years ago. So you can see you probably
won't have on the final exam the computation of a Jordan
matrix for some horrible thing with four repeated eigenvalues. I'm not that crazy
about the Jordan form. But I'm very positive about
positive definite matrixes and about the idea
that's coming Monday, the singular value
decomposition. So I'll see you on Monday,
and have a great weekend. Bye.
