The following content is
provided under a Creative Commons license. Your support will help
MIT OpenCourseWare continue to offer high quality
educational resources for free. To make a donation or
view additional materials from hundreds of MIT courses,
visit MIT OpenCourseWare at ocw.mit.edu. PROFESSOR: Today's topic
is regression analysis. And this subject
is one that we're going to cover it today
covering the mathematical and statistical foundations
of regression and focus particularly
on linear regression. This methodology is perhaps
the most powerful method in statistical modeling. And the foundations
of it, I think, are very, very important
to understand and master, and they'll help you in any
kind of statistical modeling exercise you might entertain
during or after this course. And its popularity in
finance is very, very high, but it's also a very
popular methodology in all other disciplines
that do applied statistics. So let's begin with setting up
the multiple linear regression problem. So we begin with a data set that
consists of data observations on different cases,
a number of cases. So we have n cases indexed by i. And there's a single variable,
a dependent variable or response variable, which is
the variable of focus. And we'll denote that y sub i. And together with that,
for each of the cases, there are explanatory variables
that we might observe. So the y_i's, the
dependent variables, could be returns on stocks. The explanatory variables could
be underlying characteristics of those stocks
over a given period. The dependent variable
could be the change in value of an index, the S&P
500 index or the yield rate, and the explanatory
variables can be various macroeconomic
factors or other factors that might be used to explain how
the response variable changes and takes on its value. Let's go through various
goals of regression analysis. OK, first it can be
to extract or exploit the relationship between
the dependent variable and the independent variable. And examples of
this are prediction. Indeed, in finance
that's where I've used regression analysis most. We want to predict what's going
to happen and take actions to take advantage of that. One can also use
regression analysis to talk about causal inference. What factors are really
driving a dependent variable? And so one can actually
test hypotheses about what are true
causal factors underlying the relationships
between the variables. Another application is for
just simple approximation. As mathematicians,
you're all very familiar with how
smooth functions can be-- that are smooth
in the sense of being differentiable and bounded. Those can be approximated
well by a Taylor series if you have a function of
a single variable or even a multivariable function. So one can use
regression analysis to actually approximate
functions nicely. And one can also use
regression analysis to uncover functional
relationships and validate functional
relationships amongst the variables. So let's set up the
general linear model from a mathematical
standpoint to begin with. In this lecture, OK,
we're going to start off with discussing ordinary
least squares, which is a purely mathematical
criterion for how you specify regression models. And then we're going to turn to
the Gauss-Markov theorem which incorporates some statistical
modeling principles there. They're essentially
weak principles. And then we will
turn to formal models with normal linear
regression models, and then consider extensions
of those to broader classes. Now we're in the
mathematical context. And a linear model is
basically attempting to model the conditional
distribution of the response variable y_i given the
independent variables x_i. And the conditional distribution
of the response variable is modeled simply
as a linear function of the independent variables. So the x_i's, x_(i,1)
through x_(i,p), are the key explanatory
variables that relate to the response
variables, possibly. And the beta_1, beta_2,
beta_i, or beta_p, are the regression
parameters which would be used in defining
that linear relationship. So this relationship has
residuals, epsilon_i, basically where there's
uncertainty in the data-- whether it's either due to a
measurement error or modeling error or underlying
stochastic processes that are driving the error. This epsilon_i is a
residual error variable that will indicate how this
linear relationship varies across the different n cases. So OK, how broad are the models? Well, the models
really are very broad. First of all,
polynomial approximation is indicated here. It corresponds, essentially,
to a truncated Taylor series approximation
to a functional form. With variables that
exhibit cyclical behavior, Fourier series can be applied
in a linear regression context. How many people in here are
familiar with Fourier series? Almost everybody. So Fourier series
basically provide a set of basis functions
that allow you to closely approximate most functions. And certainly with
bounded functions that possibly have a
cyclical structure to them, it provides a
complete description. So we could apply
Fourier series here. Finally, time series regressions
where the cases i one through n are really indexes of different
time points can be applied. And so the independent
variables can be variables that are
observable at a given time point or known at a given time. So those can include lags
of the response variables. So we'll see actually when
we talk about time series that there's
autoregressive time series models that can be specified. And those are very broadly
applied in finance. All right, so let's go through
what the steps are for fitting a regression model. First, one wants
to propose a model in terms of what
is it that we have to identify or be interested in
a particular response variable. And critical here is
specifying the scale of that response variable. Choongbum was discussing
problems of modeling stock prices. If, say, y is the stock price? Well, it may be that it's
more appropriate to consider modeling it on a logarithmic
scale than on a linear scale. Who can tell me why that
would be a good idea? AUDIENCE: Because
the changes might become more percent
changes in price rather than absolute
changes in price. PROFESSOR: Very good, yeah. So price changes basically
on the percentage scale, which log changes would be,
may be much better predicted by knowing factors than
the absolute price level. OK, and so we have
to have a collection of independent variables,
which to include in the model. And it's important
to think about how general this set up is. I mean, the
independent variables can be functions, lag values
of the response variable. They can be different
functional forms of other independent variables. So the fact that we're talking
about a linear regression model here is it's not so limiting
in terms of the linearity. We can really capture
lot of nonlinear behavior in this framework. So then third, we need to
address the assumptions about the distribution of
the residuals, epsilon, over the cases. So that has to be specified. Once we've set up
the model in terms of identifying the response
of the explanatory variables and the assumptions
underlying the distribution of the residuals, we need to
specify a criterion for judging different estimators. So given a particular
setup, what we want to do is be able to define a
methodology for specifying the regression parameters
so that we can then use this regression
model for prediction or whatever our purpose is. So the second
thing we want to do is define a criterion
for how we might judge different estimators of
the progression parameters. We're going to go
through several of those. And you'll see those-- least
squares is the first one, but there are actually
more general ones. In fact, the last
section of this lecture on generalized estimators
will cover those as well. Third, we need to characterize
the best estimator and apply it to the given data. So once we choose a
criterion for how good an estimate of
regression parameters is, then we have to have
a technology for solving for that. And then fourth, we need
to check our assumptions. Now, it's very often the case
that at this fourth step, where you're checking the
assumptions that you've made, you'll discover features
of your data or the process that it's modeling
that make you want to expand upon your assumptions
or change your assumptions. And so checking the
assumptions is a critical part of any modeling process. And then if necessary, modify
the model and assumptions and repeat this process. What I can tell you
is that this sort of protocol for
how you fit models is what I've applied
many, many times. And if you are lucky in a
particular problem area, the very simple
models will work well with small changes
in assumptions. But when you get
challenging problems, then this item five
of modify the model and/or assumptions is critical. And in statistical
modeling, my philosophy is you really want to,
as much as possible, tailor the model to the
process you're modeling. You don't want to fit a
square peg in a round hole and just apply, say,
simple linear regression to everything. You want to apply it when
the assumptions are valid. If the assumptions
aren't valid, maybe you can change the
specification of the problem so a linear model is still
applicable in a changed framework. But if not, then
you'll want to extend to other kinds of models. But what we'll be
doing-- or what you will be doing if you do
that-- is basically applying all the same principles
that are developed in the linear
modeling framework. OK, now let's see. I wanted to make
some comments here about specifying assumptions
for the residual distribution. What kind of assumptions
might we make? OK, would anyone like to
suggest some assumptions you might make in
a linear regression model for the residuals? Yes? What's your name, by the way? AUDIENCE: My name is Will. PROFESSOR: Will, OK. Will what? [? AUDIENCE: Ossler. ?] PROFESSOR: [? Ossler, ?] great. OK, thank you, Will. AUDIENCE: It might
be-- or we might want to say that the residual
might be normally distributed and it might not depend too
much on what value of the input variable we'd use. PROFESSOR: OK. Anyone else? OK. Well, that certainly
is an excellent place to start in terms of starting
with a distribution that's familiar. Familiar is always good. Although it's not something
that should be necessary, but we know from some of
Choongbum's lecture areas that Gaussian and
normal distributions arise in many
settings where we're taking basically sums of
independent, random variables. And so it may be that these
residuals are like that. Anyway, a slightly simpler
or weaker condition is to use the Gauss-- what
are called in statistics the Gauss-Markov assumptions. And these are assumptions
where we're only concerned with the means
or averages, statistically, and the variances
of the residuals. And so we assume that
there's zero mean. So on average, they're not
adding a bias up or down to the dependent variable. And those have a
constant variance. So the level of
uncertainty in our model doesn't depend on the case. And so indeed, if errors
on the percentage scale are more appropriate, then
one could look at, say, a time series of prices
that you're trying to model. And it may be that
on the log scale, that constant variance
looks much more appropriate than on the original
scale, which would have-- And then a third attribute of
the Gauss-Markov assumptions is that the residuals
are uncorrelated. So now uncorrelated does
not mean independent or statistically independent. So this is a somewhat weak
condition, or weaker condition, than independence
of the residuals. But in the Gauss-Markov
setting, we're just setting up
basically a reduced set of assumptions that we might
apply to fit the model. If we extend upon
that, we can then consider normal linear
regression models, which Will just suggested. And in this case, those could
be assumed to be independent and identically
distributed-- IID is that notation for that-- with
Gaussian or normal with mean 0 and variance sigma squared. We can extend upon
that to consider generalized Gauss-Markov
assumptions where we maintain still the zero mean
for the residuals, but the general-- we might
have a covariance matrix which does not correspond to
independent and identically distributed random variables. Now, let's see. In the discussion of
probability theory, we really haven't talked yet
about matrix-valued random variables, right? But how many people
in the class have covered matrix-value or
vector-valued random variables before? OK, just a handful. Well, a vector-valued
random variable, we think of the
values of these n cases for the dependent variable
to be an n-valued, an n-vector of random variables. And so we can
generalize the variance of individual random variables
to the variance covariance matrix of the collection. And so you have a covariance
matrix characterizing the variance of the n-vector
which gives us the-- the (i, j) element gives us the
value of the covariance. All right, let me put
the screen up and just write that on the board so
that you're familiar with that. All right, so we have
y_1, y_2, down to y_n, our n values of our
response variable. And we can basically talk
about the expectation of that being equal to
mu_1, mu_2, down to mu_n. And the covariance matrix
of y_1, y_2, down to y_n is equal to a matrix
with the variance of y_1 in the upper 1, 1 element, and
the variance of y_2 in the 2, 2 element, and the variance of
y_n in the nth column and nth row. And in the (i,j)-th row, (i, j),
we have the covariance between y_i and y_j. So we're going to use matrices
to represent covariances. And that's something
which I want everyone to get very familiar
with because we're going to assume that we
are comfortable with those, and apply matrix algebra with
these kinds of constructs. So the generalized
Gauss-Markov theorem assumes a general
covariance matrix where you can have
nonzero covariances between the
independent variables or the dependent variables
and the residuals. And those can be correlated. Now, who can come
up with an example of why the residuals might
be correlated in a regression model? Dan? OK. That's a really good example
because it's nonlinear. If you imagine sort of
a simple nonlinear curve and you try to fit a
straight line to it, then the residuals
from that linear fit are going to be consistently
above or below the line depending on where you are
in the nonlinearity, how it might be fitting. So that's one example
where that could arise. Any other possibilities? Well, next week we'll be talking
about some time series models. And there can be time
dependence amongst variables where there are some
underlying factors maybe that are driving the process. And those ongoing
factors can persist in making the
linear relationship over or under gauge
the dependent variable. So that can happen as well. All right, yes? AUDIENCE: The Gauss-Markov
is just the diagonal case? PROFESSOR: Yes, the Gauss-Markov
is simply the diagonal case. And explicitly if we replace
y's here by the residuals, epsilon_1 through
epsilon_n, then that diagonal matrix
with a constant diagonal is the simple Gauss-Markov
assumption, yeah. Now, I'm sure it
comes as no surprise that Gaussian distributions
don't always fit everything. And so one needs to get
clever with extending the models to other cases. And there are-- I know--
Laplace distributions, Pareto distributions, contaminated
normal distributions, which can be used to
fit regression models. And these general cases really
extend the applicability of regression models to
many interesting settings. So let's turn to specifying
the estimator criterion in two. So how do we judge what's a
good estimate of the regression parameters? Well, we're going to cover least
squares, maximum likelihood, robust methods, which are
contamination resistant. And other methods exist
that we will mention but not get into really in
the lectures, are Bayes methods and accommodating
incomplete or missing data. Essentially, as your approach
to modeling a problem gets more and more
realistic, you start adding more and more
complexity as it's needed. And certainly issues
of-- well, robust methods is where you assume
most of the data arrives under normal
conditions, but once in a while there may be some
problem with the data. And you don't want
your methodology just to break down if there happens
to be some outliers in the data or contamination. Bayes methodologies
are the technology for incorporating
subjective beliefs into statistical models. And I think it's fair
to say that probably all statistical modeling
is essentially subjective. And so if you're going to be
good at statistical modeling, you want to be sure that you're
effectively incorporating subjective information in that. And so Bayes methodologies
are very, very useful, and indeed pretty much
required to engage in appropriate modeling. And then finally, accommodate
incomplete or missing data. The world is always sort
of cruel in terms of you often are missing what you
think is critical information to do your analysis. And so how do you
deal with situations where you have some
holes in your data? Statistical models provide
good methods and tools for dealing with that situation. OK. Then let's see. In case analyses for
checking assumptions, let me go through this. Basically when you fit
a regression model, you check assumptions by
looking at the residuals, which are the basically estimates of
the epsilons, the deviations of the dependent variable
from their predictions. And what one wants
to do is analyze these to determine whether our
assumptions are appropriate. OK, but the Gauss-Markov
assumptions would be, do these appear to
have constant variance? And it may be that their
variance depends on time, if the i is indexing time. Residuals might depend on
the other variables as well, and one wants to determine
that that isn't the case. There are also influence
diagnostics identifying cases which are highly influential. It turns out that when you
are building a regression model with data, you
treat all the cases as if they're equally important. Well, it may be
that certain cases are really critical to
estimated certain factors. And it may be that much of the
inference about how important a certain factor
is is determined by very small number of points. So even though you
have a massive data set that you're using
to fit a model, it could be that
some of the structure is driven by a very
small number of cases. So influence diagnostics give
you a way of analyzing that. In the problem set
for this lecture, you'll be deriving some
influence diagnostics for linear regression
models and seeing how they're mathematically defined. And I'll be distributing
a case study which illustrates fitting
linear regression models for asset prices. And you can see
how those play out with some practical examples. OK, finally there's
outlier detection. With outliers, it's interesting. The exceptions in data are
often the most interesting. It's important in
modeling to understand whether certain
cases are unusual. And sometimes their
degree of idiosyncrasy can be explained away
so that one essentially discards those outliers. But other times,
those idiosyncrasies lead to extensions of the model. And so outlier detection can be
very important for validating a model. OK, so with that introduction to
regression, linear regression, let's talk about
ordinary least squares. Ah. OK, the least squares criterion
is for a given a regression parameter, beta,
which is considered to be a column vector-- so I'm
taking the transpose of a row vector. The least squares criterion
is to basically take the sum of square deviations
from the actual value of the response variable
from its linear prediction. So y_i minus y hat i,
we're just plugging in for y hat i the
linear function of the independent variables
and the squaring that. And the ordinary least
squares estimate, beta hat, minimizes this function. So in order to solve for this,
we're going to use matrices. And so we're going to take
the y vector, the vector of n values of the
dependent variable, or the response variable,
and X, the matrix of values of the
independent variable. It's important in this
set up to keep straight that cases go by
rows and columns go by values of the
independent variable. Boy, this thing is
ultra sensitive. Excuse me. Do I turn off the touchpad here? OK. So we can now define
our fitted value, y hat, to be equal to the
matrix x times beta. And with matrix multiplication,
that results in the y hat 1 through y hat n. And Q of beta can basically
be written as y minus X beta transpose y minus X beta. So this term here is an
n-vector minus the product of the X matrix times beta,
which is another n-vector. And we're just taking the
cross product of that. And the ordinary least
squares estimate for beta solves the derivative of
this criterion equaling 0. Now, that's in
fact true, but who can tell me why that's true? Say again? AUDIENCE: Is that minimum? PROFESSOR: OK. So your name? AUDIENCE: Seth. PROFESSOR: Seth? Seth. Very good, Seth. Thanks, Seth. So if we want to
find a minimum of Q, then that minimum will have,
if it's a smooth function, will have a minimum
at slope equals 0. Now, how do we know whether
it's a minimum or not? It could be a maximum. AUDIENCE: [INAUDIBLE]? PROFESSOR: OK, right. So in fact, this
is a-- Q of beta is a convex function of beta. And so its second
derivative is positive. And if you basically think
about the set-- basically, this is the first
derivative of Q with respect to beta equaling 0. If you were to solve for
the second derivative of Q with respect to beta,
well, beta is a p-vector. So the second
derivative is actually a second derivative
matrix, and that matrix, you can solve for it. It will be X
transpose X, which is a positive definite or
semi-definite matrix. So it basically had a
positive derivative there. So anyway, this ordinary
least squares estimates will solve this d Q of
beta by d beta equals 0. What does d Q beta by d beta_j? Well, you just take the
derivative of this sum. So we're taking the sum
of all these elements. And if you take the
derivative-- well, OK, the derivative
is a linear operator. So the derivative of a sum is
the sum of the derivatives. So we take the summation out and
we take the derivative of each term, so we get 2 minus x_(i,j),
then the thing in square brackets, y_i minus that. And what is that? Well, in matrix
notation, if we let this sort of bold X
sub square j denote the j-th column of the
independent variables, then this is minus 2. Basically, the j-th column of X
transpose times y minus X beta. So this j-th equation for
ordinary least squares has that representation in
terms-- in matrix notation. Now if we put that all
together, we basically can define this derivative
of Q with respect to the different
regression parameters as basically the minus twice
the j-th column stacked times y minus X beta, which is simply
minus 2 X transpose, y minus X beta. And this has to equal 0. And if we just simplify,
taking out the two, we get this set of equations. It must be satisfied by
the ordinary least squares estimate, beta. And that's called the
normal equations in books on regression modeling. So let's consider
how we solve that. Well, we can re-express that
by multiplying through the X transpose on each of the terms. And then beta hat basically
solves this equation. And if X transpose
X inverse exists, we get beta hat is equal
to X transpose X inverse X transpose y. So with matrix algebra, we
can actually solve this. And matrix algebra
is going to be very important to this
lecture and other lectures. So if this stuff is-- if
you're a bit rusty on this, do brush up. This particular
solution for beta hat assumes that X transpose
X inverse exists. Who can tell me
what assumptions do we need to make for X
transpose X to have an inverse? I'll call you in a second
if no one else does. Somebody just said something. Someone else. No? All right. OK, Will. AUDIENCE: So X
transpose X inverse needs to have full
rank, which means that each of the submatrices
needs to have [INAUDIBLE] smaller dimension. PROFESSOR: OK, so Will said,
basically, the matrix X needs to have full rank. And so if X has full rank,
then-- well, let's see. If X has full rank, then the
singular value decomposition which was in the very
first class can exist. And you have basically
p singular values that are all non-zero. And X transpose X
can be expressed as sort of a, from the
singular value decomposition, as one of the orthogonal
matrices times the square of the singular values times
that same matrix transpose, if you recall that definition. So that actually
is-- it basically provides a solution for X
transpose X inverse, indeed, from the singular value
decomposition of X. But what's required is that
you have a full rank in X. And what that means
is that you can't have independent variables
that are explained by other independent variables. So different columns of
X have to be linear-- or they can't linearly depend
on any other columns of X. Otherwise, you would
have reduced rank. So now if beta hat
doesn't have full rank, then our least squares estimate
of beta might be non-unique. And in fact, it is
the case that if you are really interested
in just predicting values of a dependent
variable, then having non-unique
least squares estimates isn't as much of a
problem, because you still get estimates out of that. But for now, we want to assume
that there's full column rank in the independent variables. All right. Now, if we plug in the value
of the solution for the least squares estimate,
we get fitted values for the response variable, which
are simply the matrix X times beta hat. And this expression
for the fitted values is basically X times X transpose
X inverse X transpose y, which we can represent as Hy. Basically, this H matrix in
linear models and statistics is called the hat matrix. It's basically a
projection matrix that takes the linear vector,
or the vector of values of the response variable,
into the fitted values. So this hat matrix
is quite important. The problem set's going
to cover some features, go into some properties
of the hat matrix. Does anyone want to make any
comments about this hat matrix? It's actually a very
special type of matrix. Does anyone want to point out
what that special type is? It's a projection matrix, OK. Yeah. And in linear algebra,
projection matrices have some very
special properties. And it's actually an
orthogonal projection matrix. And so if you're
interested in that feature, you should look into that. But it's really a very rich
set of properties associated with this hat matrix. It's an orthogonal projection,
and it's-- let's see. What's it projecting? It's projecting from
n-space into what? Go ahead. What's your name? AUDIENCE: Ethan. PROFESSOR: Ethan, OK. AUDIENCE: Into space [INAUDIBLE] PROFESSOR: Basically, yeah. It's projecting into
the column space of X. So that's what linear
regression is doing. So in focusing and
understanding linear regression, you can think of, how do we
get estimates of this p-vector? That's all very good and useful,
and we'll do a lot of that. But you can also
think of it as, what's happening in the
n-dimensional space? So you basically
are representing this n-dimensional vector
y by its projection onto the column space. Now, the residuals are
basically the difference between the response value
and the fitted value. And this can be expressed
as y minus y hat, or I_n minus H times y. And it turns out that I_n minus
H is also a projection matrix, and it's projecting the data
onto the space orthogonal to the column space of x. And to show that that's
true, if we consider the normal equations, which
are X transpose y minus X beta hat equaling 0, that basically
is X transpose epsilon hat equals 0. And so from the
normal equations, we can see that
what they mean is they mean that the residual
vector epsilon hat is orthogonal to each
of the columns of X. You can take any column
in X, multiply that by the residual vector,
and get 0 coming out. So that's a feature
of the residuals as they relate to the
independent variables. OK, all right. So at this point, we've gone
through really not talking about any statistical
properties to specify the betas. All we've done is talked-- we've
introduced the least squares criterion and said, what
value of the beta vector minimizes that least
squares criterion? Let's turn to the
Gauss-Markov theorem and start introducing some
statistical properties, probability properties. So with our data, y and X-- yes? Yes. AUDIENCE: [INAUDIBLE]? PROFESSOR: That epsilon-- AUDIENCE: [INAUDIBLE]? PROFESSOR: OK. Let me go back to that. It's that X, the columns
of X, and the column vector of the residual are
orthogonal to each other. So we're not doing a
projection onto a null space. This is just a statement that
those values, or those column vectors, are orthogonal
to each other. And just to recap, the
epsilon is a projection of y onto the space orthogonal
to the column space. And y hat is a projection
onto the column space of y. And these projections are
all orthogonal projections, and so they happen to result in
the projected value epsilon hat must be orthogonal to
the column space of X, if you project it out. OK? All right. So the Gauss-Markov theorem,
we have data y and X again. And now we're going to
think of the observed data, little y_1 through
y_n, is actually an observation of the
random vector capital Y, composed of random
variables Y_1 up to Y_n. And the expectation
of this vector conditional on the values
of the independent variables and their regression
parameters given by X, beta-- so the dependent
variable vector has expectation
given by the product of the independent variables
matrix times the regression parameters. And the covariance matrix
of Y given X and beta is sigma squared
times the identity matrix, the n-dimensional
identity matrix. So the identity matrix has
1's along the diagonal, n-dimensional, and
0's off the diagonal. So the variances of the Y's
are the diagonal entries, those are all the
same, sigma squared. And the covariance between
any two are equal to 0 conditionally. OK, now the
Gauss-Markov theorem. This is a terrific result
in linear models theory. And it's terrific in terms of
the mathematical content of it. I think it's-- for a math class,
it's really a nice theorem to introduce you to and
highlight the power of, I guess, results that can arise
from applying the theory. And so to set this
theorem up, we want to think about trying
to estimate some function of the regression parameters. And so OK, our problem is
with ordinary least squares-- it was, how do we specify
the regression parameters beta_1 through beta_p? Let's consider a general
target of interest, which is a linear
combination of the betas. So we want to predict
a parameter theta which is some linear combination
of the regression parameters. And because that linear
combination of the regression parameters corresponds to the
expectation of the response variable corresponding
to a given row of the independent
variables matrix, this is just a
generalization of trying to estimate the means
of the regression model at different points
in the space, or to be estimating other
quantities that might arise. So this is really a very
general kind of thing to want to estimate. It certainly is appropriate
for predictions. And if we consider the
least squares estimate by just plugging in beta hat
one through beta hat p, solved by the least squares,
well, it turns out that those are an unbiased
estimator of the parameter theta. So if we're trying to
estimate this combination of these unknown parameters,
you plug in the least squares estimate, you're going to get
an estimator that's unbiased. Who can tell me
what unbiased is? It's probably going to be a new
concept for some people here. Anyone? OK, well it's a basic
property of estimators in statistics where the
expectation of this statistic is the true parameter. So it doesn't, on average,
probabilistically, it doesn't over- or
underestimate the value. So that's what unbiased means. Now, it's also a
linear estimator of theta in terms
of this theta hat being a particular
linear combination of the dependent variables. So with our original
response variable y, in the case of y_1 through
y_n, this theta hat is simply a linear combination
of all the y's. And now why is that true? Well, we know that beta hat,
from the normal equations, is solved by X transpose
X inverse X transpose y. So it's a linear
transform of the y vector. So if we take a
linear combination of those components, it's also
another linear combination of the y vector. So this is a linear
function of the underlying-- of the response variables. Now, the Gauss-Markov
theorem says that, if the Gauss-Markov
assumptions apply, then the estimator theta
has the smallest variance amongst all linear unbiased
estimators of theta. So it actually is
like the optimal one, as long as this is our criteria. And this is really a
very powerful result. And to prove it, it's very easy. Let's see. Actually, these notes are
going to be distributed. So I'm going to go through
this very, very quickly and come back to it later
if we have more time. But you basically-- the
argument for the proof here is you consider another
linear estimate which is also an unbiased estimate. So let's consider a competitor
to the least squares value and then look at the difference
between that estimator and theta hat. And so that can be characterized
as basically this vector, f transpose y. And this difference
in the estimates must have expectation 0. So basically, if we look at--
if theta tilde is unbiased, then this expression
here is going to be equal to zero,
which means that f-- the difference in
these two estimators, f defines the difference
in the two estimators-- has to be orthogonal to
the column space of x. And with this
result, one then uses this orthogonality of
f and d to evaluate the variance of theta tilde. And in this proof, the
mathematical argument here is really something--
I should put some asterisks on a few lines here. This expression here is
actually very important. We're basically looking
at the decomposition of the variance to
be the variance of b transpose y, which is
the variance of the sum of these two random variables. So the page before
basically defined d and f such that this is true. Now when you consider
the variance of a sum, it's not the sum
of the variances. It's the sum of the
variances plus twice the sum of the covariances. And so when you are
calculating variances of sums of random variables,
you have to really keep track of the covariance terms. In this case, this
argument shows that the covariance
terms are, in fact, 0, and you get the
result popping out. But that's really a-- in
an econometrics class, they'll talk about BLUE
estimates of regression, or the BLUE property of the
least squares estimates. That's where that comes from. All right, so let's now consider
generalizing from Gauss-Markov to allow for unequal variances
and possibly correlated nonzero covariances
between the components. And in this case,
the regression model has the same linear set up. The only difference
is the expectation of the residual
vector is still 0. But the covariance matrix
of the residual vector is sigma squared,
a single parameter, times let's say capital sigma. And we'll assume here
that this capital sigma matrix is a known n by n
positive definite matrix specifying relative
variances and correlations between the observations. OK. Well, in order to solve
for regression estimates under these generalized
Gauss-Markov assumptions, we can transform the
data Y, X to Y star equals sigma to the
minus 1/2 y and X to X star, which is
sigma to the minus 1/2 x. And this model then becomes
a model, a linear regression model, in terms of
Y star and X star. We're basically multiplying
this regression model by sigma to the minus 1/2 across. And epsilon star actually
has a covariance matrix equal to sigma squared
times the identity. So if we just take a
linear transformation of the original data,
we get a representation of the regression
model that satisfies the original
Gauss-Markov assumptions. And what we had to
do was basically do a linear transformation
that makes the response variables all have constant
variance and be uncorrelated. So with that, we then have the
least squares estimate of beta is the least squares, the
ordinary least squares, in terms of Y star and X star. And so plugging that in, we then
have X star transpose X star inverse X star transpose Y star. And if you multiply through,
that's how the formula changes. So this formula characterizing
the least squares estimate under this generalized
set of assumptions highlights what you
need to do to be able to apply that theorem. So with response values that
have very large variances, you basically want to discount
those by the sigma inverse. And that's part of the way in
which these generalized least squares work. All right. So now let's turn to
distribution theory for normal regression models. Let's assume that
the residuals are normals with mean 0 and
variance sigma squared. OK, conditioning on the values
of the independent variable, the Y's, the response
variables, are going to be independent
over the index i. They're not going to be
identically distributed because they have
different means, mu_i for the dependent
variable Y_i, but they will have a constant variance. And what we can do is
basically condition on X, beta, and sigma squared
and then represent this model in terms of the
distribution of the epsilons. So if we're conditioning
on x and beta, this X beta is a constant,
known, we've conditioned on it. And the remaining uncertainty
is in the residual vector, which is assumed to
be all independent and identically distributed
normal random variables. Now, this is the
first time you'll see this notation, capital N sub
little n, for a random vector. It's a multivariate
normal random variable where you consider an n-vector
where each component is normally distributed,
with mean given by some corresponding
mean vector, and a covariance matrix
given by a covariance matrix. In terms of independent and
identically distributed values, the probability structure
here is totally well-defined. Anyone here who's taken a
beginning probability class knows what the
density function is for this multivariate
normal distribution because it's the product
of the independent density functions for the
independent components, because they're all
independent random variables. So this multivariate
normal random vector has a density function
which you can write down, given your first
probability class. OK, here I'm just
highlighting or defining the mu vector for the means
of the cases of the data. And the covariance matrix
sigma is this diagonal matrix. And so basically sigma_(i,j)
is equal to sigma squared times the Kronecker delta
for the (i,j) element. Now what we want to do
is, under the assumptions of normally
distributed residuals, to solve for the distribution
of the least squares estimators. We want to know, basically,
what kind of distribution does it have? Because what we want
to be able to do is to determine
whether estimates are particularly large or not. And maybe there's
no structure at all and the regression
parameters are 0 so that there's no dependence
on a given factor. And we need to be able to
judge how significant that is. So we need to know what
the distribution is of our least squares estimate. So what we're going to do
is apply moment generating functions to derive the
joint distribution of y and the joint
distribution of beta hat. And so Choongbum introduced
the moment generating function for individual random variables
for single-variate random variables. For n-variate
random variables, we can define the moment generating
function of the Y vector to be the expectation of
e to the t transpose Y. So t is an argument of the
moment generating function. It's another n-vector. And it's equal to the
expectation of e to the t_1 Y_1 plus t_2 Y_2 up to t_n Y_n. So this is a very
simple definition. Because of independence,
the expectation of the products, or
this exponential sum is the product of
the exponentials. And so this moment
generating function is simply the product of the moment
generating functions for Y_1 up through Y_n. And I think-- I don't know if
it was in the first problem set or in the first lecture, but e
to the t_i mu_i plus a half t_i squared sigma squared
was the moment generating function for the
single univariate normal random variable,
mean mu_i and variance sigma squared. And so if we have n of
these, we take their product. And the moment
generating function for y is simply e to the
t transpose mu plus 1/2 t transpose sigma t. And so for this multivariate
normal distribution, this is its moment
generating function. And this happens to be--
the distribution of y is a multivariate normal with
mean mu and covariance matrix sigma. So a fact that
we're going to use is that if we're working with
multivariate normal random variables, this is the structure
of their moment generating functions. And so if we solve for
the moment generation function of some
other item of interest and recognize that
it has the same form, we can conclude that it's also
a multivariate normal random variable. So let's do that. Let's solve for the
moment generation function of the least
squares estimate, beta hat. Now rather than dealing
with an n-vector, we're dealing with a p-vector
of the betas, beta hats. And this is simply the
definition of the moment generating function. If we plug in for basically
what the functional form is for the ordinary least
squares estimates and how they depend on
the underlying Y, then we basically-- OK, we have
A equal to, essentially, the linear projection of Y.
That gives us the least squares estimate. And then we can say that
this moment generating function for beta hat is
equal to the expectation of e to the t transpose Y, where
little t is A transpose tau. Well, we know what this is. This is the moment
generating function of X-- sorry, of Y-- evaluated
at the vector little t. So we just need to plug in
little t, that expression A transpose tau. So let's do that. And you do that and it turns
out to be e to the t transpose mu plus that. And we go through a
number of calculations. And at the end of the day, we
get that the moment generating function is just e to the tau
transpose beta plus a 1/2 tau transpose this matrix tau. And that is the moment
generation function of a multivariate normal. So these few lines that you
can go through after class basically solve for
the moment generating function of beta hat. And because we can
recognize this as the MGF of a multivariate normal, we
know that that's-- beta hat is a multivariate normal,
with mean the true beta, and covariance matrix given by
the object in square brackets there. OK, so this is
essentially the conclusion of that previous analysis. The marginal distribution
of each of the beta hats is given by beta hat-- by a
univariate normal distribution with mean beta_j and variance
equal to the diagonal. Now at this point, saying
that is like an assertion. But one can actually
prove that very easily, given this sequence of argument. And can anyone tell
me why this is true? Let me tell you. If you consider plugging in
the moment generating function, the value tau, where only
the j-th entry is non-zero, then you have the moment
generating function of the j-th component
of beta hat. And that's a Gaussian
moment generating function. So the marginal distribution of
the j-th component is normal. So you get that
almost for free from this multivariate analysis. And so there's no hand waving
going on in having that result. This actually follows
directly from the moment generating functions. OK, let's now turn
to another topic. Related, but it's the
QR decomposition of X. So we have-- with our
independent variables X, we want to express
this as a product of an orthonormal matrix
Q which is n by p, and an upper
triangular matrix R. So it turns out that any
matrix, n by p matrix, can be expressed in this form. And we'll quickly show you
how that can be accomplished. We can accomplish
that by conducting a Gram-Schmidt
orthonormalization of the independent
variables matrix X. And let's see. So if we define R, the upper
triangular matrix in the QR decomposition, to have
0's off the diagonal below and then possibly nonzero
value along the diagonal into the right, we're just
going to solve for Q and R through this
Gram-Schmidt process. So the first column of X is
equal to the first column of Q times the first
element, the top left corner of the matrix R. And if we take the cross product
of that vector with itself, then we get this expression
for r_(1,1) squared-- we can basically solve for
r_(1,1) as the square root of this dot product. And Q_Q_[1] is simply the first
column of X divided by that square root. So this first element
of the Q matrix and the first element r, this
can be solved for right away. Then let's solve for
the second column of Q and the second column
of the R matrix. Well, X_X_[2], the second
column of the X matrix, is the first column
of Q times r_(1,2), plus the second column
of Q times r_(2,2). And if we multiply this
expression by Q_Q_[1] transpose, then we basically get this
expression for r_(1,2). So we actually have
just solved for r_(1,2). And Q_Q_[2] is solved for by
the arguments given here. So basically, we successively
are orthogonalizing columns of X to the
previous columns of X through this
Gram-Schmidt process. And it basically can be repeated
through all the columns. Now with this QR
decomposition, what we get is a really nice form for
the least squares estimate. Basically, it simplifies to the
inverse of R times Q transpose y. And this basically
means that you can solve for least squares
estimates by calculating the QR decomposition, which is a
very simple linear algebra operation, and then just do
a couple of matrix products to get the-- well, you do have
to do a matrix inverse with R to get that out. And the covariance
matrix of beta hat is equal to sigma squared
X transpose X inverse. And in terms of the covariance
matrix, what is implicit here but you should make
explicit in your study, is if you consider taking a
matrix, R inverse Q transpose times y, the only thing that's
random there is that y vector, OK? The covariance of a matrix
times a random vector is that matrix
times the covariance of the vector times the
transpose of the matrix. So if you take a
matrix transformation of a random vector,
then the covariance of that transformation
has that form. So that's where this covariance
matrix is coming into play. And from the MGF, the
moment generating function, for the least squares
estimate, this basically comes out of the moment
generating function definition as well. And if we take X
transpose X, plug in the QR decomposition,
only the R's play out, giving you that. Now, this also gives
us a very nice form for the hat matrix,
which turns out to just be Q times Q transpose. So that's a very simple form. So now with the
distribution theory, this next section is
going to actually prove what's really a
fundamental result about normal linear
regression models. And I'm going to go through
this somewhat quickly just so that we cover what the
main ideas are of the theorem. But the details, I think,
are very straightforward. And these course notes
that will be posted online will go through the various
steps of the analysis. OK, so there's an
important theorem here which is for any
matrix A, m by n, you consider transforming
the random vector y by this matrix A. It is
also a random normal vector. And its distribution
is going to have a mean and covariance
matrix given by mu_z and sigma_z, which have
this simple expression in terms of the matrix A and
the underlying means and covariances of y. OK, earlier we actually
applied this theorem with A corresponding to the
matrix that generates the least squares estimates. So with A equal to X
transpose X inverse, we actually previously went
through the solution for what's the distribution of beta hat. And with any other
matrix A, we can go through the same analysis
and get the distribution. So if we do that here,
well, we can actually prove this important
theorem, which says that with least
squares estimates of normal linear regression
models, our least squares estimate beta hat and
our residual vector epsilon hat are independent
random variables. So when we construct
these statistics, they are statistically
independent of each other. And the distribution of beta
hat is multivariate normal. The sum of the squared
residuals is, in fact, a multiple of a chi-squared
random variable. Now who in here can tell me what
a chi-squared random variable is? Anyone? AUDIENCE: [INAUDIBLE]? PROFESSOR: Yes, that's right. So a chi-squared random variable
with one degree of freedom is a squared normal zero
one random variable. A chi-squared with
two degrees of freedom is the sum of two independent
normals, zero one, squared. And so the sum of n squared
residuals is, in fact, an n minus p chi-squared random
variable scale it by sigma squared. And for each component
j, if we take the difference between the least
squares estimate beta hat j and beta_j and divide
through by this estimate of the standard
deviation of that, then that will, in fact, have a
t distribution on n minus p degrees of freedom. And let's see, a t distribution
in probability theory is the ratio of a normal random
variable to an independent chi squared random variable, or
the root of an independent chi squared random variable. So basically these
properties characterize our regression parameter
estimates and t statistics for those estimates. Now, OK, in the
course notes, there's a moderately long proof. But all the details
are given, and I'll be happy to go through any
of those details with people during office hours. Let me just push
on to-- let's see. We have maybe two minutes
left in the class. Let me just talk about
maximum likelihood estimation. And in fitting models
and statistics, maximum likelihood estimation
comes up again and again. And with normal linear
regression models, it turns out that ordinary
least squares estimate are, in fact, our maximum
likelihood estimates. And what we want to do
with a maximum likelihood is to maximize. We want to define the
likelihood function, which is the density function
for the data given the unknown parameters. And this density
function is simply the density function for a
multivariate normal random variable. And the maximum
likelihood estimates are the estimates of the
underlying parameters that basically maximize
the density function. So it's the values of
the underlying parameters that make the data that was
observed the most likely. And if you plug in the values
of the density function, basically we have these
independent random variables, Y_i, whose product
is the joint density. The likelihood
function turns out to be basically a function of
the least squares criterion. So if you fit models
by least squares, you're consistent with doing
something decent in at least applying the maximum
likelihood principle if you had a normal
linear regression model. And it's useful to know when
your statistical estimation algorithms are consistent
with certain principles like maximum likelihood
estimation or others. So let me, I guess,
finish there. And next time, I will
just talk a little bit about generalized M estimators. Those provide a
class of estimators that can be used for
finding robust estimates and also quantile estimates
of regression parameters which are very interesting.
