The following content is
provided under a Creative Commons license. Your support will help
MIT OpenCourseWare continue to offer high quality
educational resources for free. To make a donation or
view additional materials from hundreds of MIT courses,
visit MIT OpenCourseWare at ocw.mit.edu. PROFESSOR: OK. Well, last time I
was lecturing, we were talking about
regression analysis. And we finished up talking
about estimation methods for fitting regression models. I want to recap the method
of maximum likelihood, because this is really
the primary estimation method in statistical
modeling that you start with. And so let me just
review where we were. We have a normal linear
regression model. A dependent variable
y is explained by a linear combination
of independent variables given by a regression
parameter beta. And we assume that there are
errors about all the cases which are independent
identically distributed normal random variables. So because of that relationship,
the dependent variable vector y, which is an
n-vector, for n cases, is a multivariate
normal random variable. Now, the likelihood function is
equal to the density function for the data. And there's some
ambiguity really about how one manipulates
the likelihood function. The likelihood function
becomes defined once we've observed a sample of data. So in this expression for
the likelihood function as a function of beta
and sigma squared, we're considering evaluating
the probability density function for the
data conditional on the unknown parameters. So if this were simply a
univariate normal distribution with some unknown mean
and variance, then what we would have is
just a bell curve for mu centered around a
single observation y, if you look at the
likelihood function and how it varies with
the underlying mean of the normal distribution. So this likelihood
function is-- well, the challenge really
in maximum estimation is really calculating
and computing the likelihood function. And with normal linear
regression models, it's very easy. Now, the maximum
likelihood estimates are those values that
maximize this function. And the question is, why
are those good estimates of the underlying parameters? Well, what those
estimates do is they are the parameter values for
which the observed data is most likely. So we're able to scale
the unknown parameters by how likely those parameters
could have generated these data values. So let's look at the
likelihood function for this normal linear
regression model. These first two lines here are
highlighting-- the first line is highlighting that
our response variable values are independent. They're conditionally
independent given the unknown parameters. And so the density of the
full vector of y's is simply the product of the density
functions for those components. And because this is a normal
linear regression model, each of the y_i's is
normally distributed. So what's in there
is simply the density function of a normal random
variable with mean given by the beta sum of independent
variables for each i, case i, given by the
regression parameters. And that expression
basically can be expressed in matrix form this way. And what we have is
the likelihood function ends up being a function
of our Q of beta, which was our least squares criteria. So the least squares
estimation is equivalent to maximum likelihood
estimation for the regression parameters if we have a normal
linear regression model. And there's this
extra term, minus n. Well, actually, if we're going
to maximize the likelihood function, we can also maximize
the log of the likelihood function, because that's
just a monotone function of the likelihood. And it's easier to maximize the
log of the likelihood function which is expressed here. And so we're able to
maximize over beta by minimizing Q of beta. And then we can maximize
over sigma squared given our estimate for beta. And that's achieved by
taking the derivative of the log-likelihood with
respect to sigma squared. So we basically have this
first order condition that finds the
maximum because things are appropriately convex. And taking that derivative
and solving for zero, we basically get
this expression. So this is just
taking the derivative of the log-likelihood with
respect to sigma squared. And you'll notice
here I'm taking the derivative with
respect to sigma squared as a parameter, not sigma. And that gives us that
the maximum likelihood estimate of the error variance
is Q of beta hat over n. So this is the sum of the
squared residuals divided by n. Now, I emphasize here
that that's biased. Who can tell me
why that's biased or why it ought to be biased? AUDIENCE: [INAUDIBLE]. PROFESSOR: OK. Well, it should be n
minus 1 if we're actually estimating one parameter. So if the independent variables
were, say, a constant, 1, so we're just estimating a
sample from a normal with mean beta 1 corresponding to
the units vector of the X, then we would have a one
degree of freedom correction to the residuals to get
an unbiased estimator. But what if we
have p parameters? Well, let me ask you this. What if we had n parameters
in our regression model? What would happen if
we had a full rank n independent variable matrix
and n independent observations? AUDIENCE: [INAUDIBLE]. PROFESSOR: Yes, you'd have
an exact fit to the data. So this estimate would be 0. And so clearly, if
the data do arise from a normal linear regression
model, 0 is not unbiased. And you need to have
some correction. Turns out you need
to divide by n minus the rank of the X
matrix, the degrees of freedom in the model, to get
a biased estimate. So this is an important
issue, highlights how the more parameters you add
in the model, the more precise your fitted values are. In a sense, there's
dangers of curve fitting which you want to avoid. But the maximum likelihood
estimates, in fact, are biased. You just have to
be aware of that. And when you're using
different software, fitting different
models, you need to know whether there are
various corrections be made for biasedness or not. So this solves the
estimation problem for normal linear
regression models. And when we have normal
linear regression models, the theorem we
went through last time-- this is very important. Let me just go back and
highlight that for you. This theorem right here. This is really a very
important theorem indicating what is the
distribution of the least squares, now the maximum
likelihood estimates of our regression model? They are normally distributed. And the residuals, sum
of squares, have a chi squared distribution
with degrees of freedom given by n minus p. And we can look at how
much signal to noise there is in estimating
our regression parameters by calculating a t
statistic, which is take away from an estimate its
expected value, its mean, and divide through by an
estimate of the variability in standard deviation units. And that will have
a t distribution. So that's a critical
way to assess the relevance of different
explanatory variables in our model. And this approach will apply
with maximum likelihood estimation in all
kinds of models apart from normal linear
regression models. It turns out maximum
likelihood estimates generally are asymptotically
normally distributed. And so these properties here
will apply for those models as well. So let's finish up these
notes on estimation by talking about
generalized M estimation. So what we want to consider is
estimating unknown parameters by minimizing some
function, Q of beta, which is a sum of evaluations
of another function h, evaluated for each of
the individual cases. And choosing h to take on
different functional forms will define different
kinds of estimators. We've seen how when h
is simply the square of the case minus its
regression prediction, that leads to least squares,
and in fact, maximum likelihood estimation, as we saw before. Rather than taking the
square of the residual, the fitted residual,
we could take simply the modulus of that. And so that would be the
mean absolute deviation. So rather than summing
the squared deviations from the mean, we could
sum the absolute deviations from the mean. Now, from a
mathematical standpoint, if we want to solve
for those estimates, how would you go
about doing that? What methodology would you
use to maximize this function? Well, we try and apply
basically the same principles of if this is a
convex function, then we just want to take derivatives
of that and solve for that being equal to 0. So what happens when
you take the derivative of the modulus of y minus xi
beta with respect to beta? AUDIENCE: [INAUDIBLE]. PROFESSOR: What did you say? What did you say? AUDIENCE: Yeah, it's
not [INAUDIBLE]. The first [INAUDIBLE]
derivative is not continuous. PROFESSOR: OK. Well, this is not
a smooth function. But let me just plot x_i beta
here, and y_i minus that. Basically, this is going
to be a function that has slope 1 when it's positive
and slope minus 1 when it's negative. And so that will be true,
component-wise, or for the y. So what we end up
wanting to do is find the value of the
regression estimate that minimizes the
sum of predictions that are below the estimate plus
the sum of the predictions that are above the estimate given
by the regression line. And that solves the problem. Now, with the maximum
likelihood estimation, one can plug in minus log the
density of y_i given beta, x and sigma_i squared. And that function simply sums
to the log of the joint density for all the data. So that works as well. With robust M estimators, we can
consider another function chi which can be defined to have
good properties with estimates. And there's a whole theory
of robust estimation-- it's very rich-- which
talks about how best to specify this chi function. Now, one of the problems
with least squares estimation is that the squares
of very large values are very, very
large in magnitude. So there's perhaps
an undue influence of very large values, very large
residuals under least squares estimation and maximum
[INAUDIBLE] estimation. So robust estimators
allow you to control that by defining the
function differently. Finally, there are
quantile estimators, which extend the mean
absolute deviation criterion. And so if we consider
the h function to be basically a
multiple of the deviation if the residual is positive
and a different multiple, a complementary multiple if
the derivation, the residual, is less than 0,
then by varying tau, you end up getting
quantile estimators, where what you're doing is minimizing
the estimate of the tau quantile. So this general
class of M estimators encompasses most
estimators that we will encounter in fitting models. So that finishes the technical
or the mathematical discussion of regression analysis. Let me highlight for you--
there's a case study that I dragged to the desktop here. And I wanted to find that. Let me find that. There's a case study that's been
added to the course website. And this first one is on
linear regression models for asset pricing. And I want you to
read through that just to see how it applies to
fitting various simple linear regression models. And enter full screen. This case study begins by
introducing the capital asset pricing model, which
basically suggests that if you look at the
returns on any stocks in an efficient
market, then those should depend on the return
of the overall market but scaled by how
risky the stock is. And so if one looks
at basically what the return is on the
stock on the right scale, you should have a simple
linear regression model. So here, we just look at
a time series for GE stock in the S&P 500. And the case study guide
through how you can actually collect this data
on the web using R. And so the case notes
provide those details. There's also the
three-month treasury rate which is collected. And so if you're
thinking about return on the stock versus return
on the index, well, what's really of interest is the excess
return over a risk-free rate. And the efficient
markets models, basically the excess
return of a stock is related to the excess
return of the market as given by a linear
regression model. So we can fit this model. And here's a plot of the excess
returns on a daily basis for GE stock versus the market. So that looks like a
nice sort of point cloud for which a linear
model might fit well. And it does. Well, there are
regression diagnostics, which I'll get to-- well, there
are regression diagnostics which are detailed in the
problem set, where we're looking at how influential are
individual observations, what's their impact on
regression parameters. This display here
basically highlights with a very simple
linear regression model what are the
influential data points. And so I've highlighted
in red those values which are influential. Now, if you look at the
definition of leverage in a linear model,
it's very simple. A simple linear model is
just those observations that are very far from the
mean have large leverage. And so you can confirm
that with your answers to the problem set. This x indicates a
significantly influential point in terms of the
regression parameters given by Cook's distance. And that definition is also
given in the case notes. AUDIENCE: [INAUDIBLE]. PROFESSOR: By computing
the individual leverages with a function
that's given here, and by selecting out those
that exceed a given magnitude. Now, with this very,
very simple model of stocks depending
on one unknown factor, risk factor given the market. In modeling equity
returns, there are many different factors that
can have an impact on returns. So what I've done
in the case study is to look at adding
another factor which is just the return on crude oil. And so-- I need to go down here. So let me highlight
something for you here. With GE stock, what would you
expect the impact of, say, a high return on crude oil to
be on the return of GE stock? Would you expect it to
be positively related or negatively related? OK. Well, GE is a stock that's
just a broad stock invested in many different industries. And it really reflects the
overall market, to some extent. Many years ago,
10, 15 years ago, GE represented maybe 3% of
the GNP of the US market. So it was really highly related
to how well the market does. Now, crude oil is a commodity. And oil is used to drive cars,
to fuel energy production. So if you have an
increase in oil prices, then the cost of essentially
doing business goes up. So it is associated with
an inflation factor. Prices are rising. So if you can see here,
the regression estimate, if we add in a factor of
the return on crude oil, it's negative 0.03. And it has a t value
of minus 3.561. So in fact, the market, in
a sense, over this period, for this analysis, was not
efficient in explaining the return on GE; crude oil
is another independent factor that helps explain returns. So that's useful to know. And if you are clever about
defining and identifying and evaluating
different factors, you can build
factor asset pricing models that are
very, very useful for investing and trading. Now, as a comparison
to this case study, also applied the same
analysis to Exxon Mobil. Now, Exxon Mobil
is an oil company. So let me highlight this here. We basically are
fitting this model. Now let's highlight it. Here, if we consider
this two-factor model, the regression
parameter corresponding to the crude oil factor is
plus 0.13 with a t value of 16. So crude oil definitely
has an impact on the return of Exxon Mobil,
because it goes up and down with oil prices. This case study closes
with a scatter plot of the independent variables
and highlighting where the influential values are. And so just in the same way that
with a simple linear regression it was those that were far
away from the mean of the data were influential, in a
multivariate setting-- here, it's bivariate-- the
influential observations are those that are very
far away from the centroid. And if you look at one of the
problems in the problem set, it actually goes
through and you can see where these
leveraged values are and how it indicates influences
associated with the Mahalanobis distance of cases
from the centroid of the independent variables. So if you're a visual
type mathematician as opposed to an algebraic
type mathematician, I think these
kinds of graphs are very helpful in understanding
what is really going on. And the degree of influence
is associated with the fact that we're basically taking
least squares estimates, so we have the quadratic
form associated with the overall process. There's another
case study that I'll be happy to discuss after
class or during office hours. I don't think we have time
today during the lecture. But it concerns
exchange rate regimes. And the second case study
looks at the Chinese yuan, which was basically pegged
to the dollar for many years. And then I guess through
political influence from other countries,
they started to let the yuan vary
from the dollar, but perhaps pegged
it to some basket of securities-- of currencies. And so how would you determine
what that basket of currencies is? Well, there are
regression methods that have been
developed by economists that help you do that. And that case study goes
through the analysis of that. So check that out to see how
you can get immediate access to currency data and be
fitting these regression models and looking at the
different results and trying to evaluate those. So let's turn now
to the main topic-- let's see here-- which
is time series analysis. Today in the rest
of the lecture, I want to talk about univariate
time series analysis. And so we're thinking of
basically a random variable that is observed over time and
it's a discrete time process. And we'll introduce you
to the Wold representation theorem and definitions
of stationarity and its relationship there. Then, look at the classic
models of autoregressive moving average models. And then extending those
to non-stationarity with integrated autoregressive
moving average models. And then finally, talk about
estimating stationary models and how we test
for stationarity. So let's begin from
basically first principles. We have a stochastic process,
a discrete time stochastic process, X, which consists
of random variables indexed by time. And we're thinking
now discrete time. The stochastic behavior
of this sequence is determined by specifying
the density or probability mass functions for all finite
collections of time indexes. And so if we could specify
all finite.dimensional distributions of
this process, we would specify this
probability model for the stochastic process. Now, this stochastic process
is strictly stationary if the density function for
any collection of times, t_1 through t_m, is equal to
the density function for a tau translation of that. So the density function for any
finite-dimensional distribution is stationary, is constant
under arbitrary translations. So that's a very
strong property. But it's a reasonable
property to ask for if you're doing statistical modeling. And what do you want to do
when you're estimating models? You want to estimate
things that are constant. Constants are nice
things to estimate. And parameters of
models are constant. So we really want the underlying
structure of the distributions to be the same. That was strict
stationarity, which requires knowledge of
the entire distribution of the stochastic process. We're now going to introduce
a weaker definition, which is covariance stationarity. And a covariance
stationary process has a constant mean,
mu; a constant variance, sigma squared; and a
covariance over increments tau, given by a function gamma of
tau, that is also constant. Gamma isn't a constant function,
but basically for all t, covariance of X_t, X_(t+tau)
is this gamma of tau function. And we also can introduce
the autocorrelation function of the stochastic
process, rho of tau. And so the correlation
of two random variables is the covariance of those
random variables divided by the square root of the
product of the variances. And Choongbum I think
introduced that a bit. in one of his lectures,
where we were talking about the correlation function. But essentially, the
correlation function is if you standardize the
data or the random variables to have mean 0-- so
subtract off the means and then divide through by
their standard deviations. So those translated variables
have mean 0 and variance 1. Then the correlation
coefficient is the covariance between those standardized
random variables. So this is going to come up
again and again in time series analysis. Now, the Wold
representation theorem is a very, very powerful theorem
about covariance stationary processes. It basically states that if
we have a zero-mean covariance stationary time
series, then it can be decomposed into two
components with a very nice structure. Basically, X_t can be
decomposed into V_t plus S_t. V_t is going to be a linearly
deterministic process, meaning that past values of
V_t perfectly predict what V_t is going to be. So this could be like a
linear trend or some fixed function of past values. It's basically a
deterministic process. So there's nothing
random in V_t. It's something that's
fixed, without randomness. And S_t is a sum
of coefficients, psi_i times eta_(t-i), where
the eta_t's are linearly unpredictable white noise. So what we have is S_t
is a weighted average of white noise with
coefficients given by the psi_i. And the coefficients psi_i
are such that psi_0 is 1, and the sum of the
squared psi_i's is finite. And the white noise
eta_t-- what's white noise? It has expectation zero. It has variance, given by
sigma squared, that's constant. And it has covariance across
different white noise elements that's 0 for all t and s. So eta_t's are uncorrelated
with themselves, and of course, they
are uncorrelated with the deterministic process. So this is really a very,
very powerful concept. If you are modeling
a process and it has covariance
stationarity, then there exists a representation
like this of the function. So it's a very
compelling structure, which we'll see how it applies
in different circumstances. Now, before getting into the
definition of autoregressive moving average
models, I just want to give you an intuitive
understanding of what's going on with the Wold decomposition. And this, I think,
will help motivate why the Wold
decomposition should exist from a mathematical standpoint. So consider just some
univariate stochastic process, some time series X_t
that we want to model. And we believe that it's
covariance stationary. And so we want to
specify essentially the Wold decomposition of that. Well, what we could
do is initialize a parameter p, the number
of past observations, in the linearly
deterministic term. And then estimate the linear
projection of X_t on the last p lag values. And so what I want to do
is consider estimating that relationship using
a sample of size n with some ending point t_0
less than or equal to T. And so we can consider y
values like a response variable being given by the successive
values of our time series. And so our response variables
y_j can be considered to be x t_0 minus n plus j. And define a y vector and
a Z matrix as follows. So we have values of our
stochastic process in y. And then our Z matrix,
which is essentially a matrix of
independent variables, is just the lagged
values of this process. So let's apply
ordinary least squares to specify the projection. This projection matrix
should be familiar now. And that basically gives
us a prediction of y hat depending on p lags. And we can compute the
projection residual from that fit. Well, we can conduct
time series methods to analyze these residuals,
which we'll be introducing here in a few minutes, to specify
a moving average model. We can then have estimates of
the underlying coefficients psi and estimates of
these residuals eta_t. And then we can evaluate whether
this is a good model or not. What does it mean to be
an appropriate model? Well, the residual should
be orthogonal to longer lags than t minus s, or
longer lags than p. So we basically shouldn't
have any dependence of our residuals on lags
of the stochastic process that weren't included
in the model. Those should be orthogonal. And the eta_t hats should be
consistent with white noise. So those issues
can be evaluated. And if there's
evidence otherwise, then we can change the
specification of the model. We can add additional lags. We can add additional
deterministic variables if we can identify
what those might be. And proceed with this process. But essentially that is
how the Wold decomposition could be implemented. And theoretically, as
our sample gets large, if we're observing this time
series for a long time, then well certainly the
limit of the projections as p, the number of lags
we include, gets large, should be essentially
the projection of our data on its history. And that, in fact, is the
projection corresponding to, defining, the
coefficient's psi_i. And so in the limit, that
projection will converge and it will converge
in the sense that the coefficients of
the projection definition correspond to the psi_i. And now if p goes to
infinity is required, now p means that there's
basically a long term dependence in the process. Basically, it doesn't
stop at a given lag. The dependence
persists over time. Then we may require
that p goes to infinity. Now, what happens when
p goes to infinity? Well, if you let p go
to infinity too quickly, you run out of
degrees of freedom to estimate your models. And so from an
implementation standpoint, you need to let p/n
go to 0 so that you have essentially more
data than parameters that you're estimating. And so that is required. And in time series
modeling, what we look for are models where
finite values of p are required. So we're only estimating a
finite number of parameters. Or if we have a moving
average model which has coefficients that
are infinite in number, perhaps those can be defined by
a small number of parameters. So we'll be looking for
that kind of feature in different models. Let's turn to talking
about the lag operator. The lag operator is
a fundamental tool in time series models. We consider the operator L
that shifts a time series back by one time increment. And applying this
operator recursively, we get, if it's operating
0 times, there's no lag, one time, there's
one lag, two times, two lags-- doing
that iteratively. And in thinking of these,
what we're dealing with is like a transformation on
infinite dimensional space, where it's like
the identity matrix sort of shifted by
one element-- or not the identity, but an element. It's like the identity
matrix shifted by one column or two columns. So anyway, inverses
of these operators are well defined in terms
of what we get from them. So we can represent
the Wold representation in terms of these lag
operators by saying that our stochastic
process X_t is equal to V_t plus this
psi of L function, basically a
functional of the lag operator, which is a potentially
infinite-order polynomial of the lags. So this notation is
something that you need to get very
familiar with if you're going to be comfortable with
the different models that are introduced with
ARMA and ARIMA models. Any questions about that? Now relating to
this-- let me just introduce now, because this
will come up somewhat later. But there's the impulse
response function of the covariance
stationary process. If we have a stochastic process
X_t which is given by this Wold representation, then
you can ask yourself what happens to the innovation
at time t, which is eta_t, how does that affect
the process over time? And so, OK, pretend that you are
chairman of the Federal Reserve Bank. And you're interested in the GNP
or basically economic growth. And you're considering
changing interest rates to help the economy. Well, you'd like to
know what an impact is of your change in
this factor, how that's going to affect the
variable of interest, perhaps GNP. Now, in this case,
we're thinking of just a simple covariance
stationary stochastic process. It's basically a process that
is a random-- a weighted sum, a moving average of
innovations eta_t. But the question is, basically
any covariance stationary process could be
represented in this form. And the impulse
response function relates to what is
the impact of eta_t. What's its impact over time? Basically, it affects
the process at time t. That, because of the
moving average process, it affects it at t plus
1, affects it at t plus 2. And so this impulse
response is basically the derivative of the
value of the process with the j-th previous
innovation is given by psi_j. So the different
innovations have an impact on the current value given by
this impulse response function. So looking backward,
that definition is pretty well defined. But you can also
think about how does an impact of the
innovation affect the process going forward. And the long-run
cumulative response is essentially what is the
impact of that innovation in the process ultimately? And eventually, it's
not going to change the value of the process. But what is the value to
which the process is moving because of that one innovation? And so the long run
cumulative response is given by basically the
sum of these individual ones. And it's given by the
sum of the psi_i's. So that's the polynomial of
psi with lag operator, where we replace the lag operator by 1. We'll see this
again when we talk about vector
autoregressive processes with multivariate time series. Now, the Wold
representation, which is a infinite-order moving
average, possibly infinite order, can have an
autoregressive representation. Suppose that there is
another polynomial psi_i star of the lags, which we're
going to call psi inverse of L, which satisfies the fact if you
multiply that with psi of L, you get the identity lag 0. Then this psi inverse,
if that exists, is basically the
inverse of the psi of L. So if we start with psi of
L, if that's invertible, then there exists
a psi inverse of L, with coefficients psi_i star. And one can basically take
our original expression for the stochastic process,
which is as this moving average of the eta's, and express it
as this essentially moving averages of the X's. And so we've essentially
inverted the process and shown that the
stochastic process can be expressed as an infinite
order autoregressive representation. And so this infinite order
autoregressive representation corresponds to that intuitive
understanding of how the Wold representation exists. And it actually works with the--
the regression coefficients in that projection several
slides back corresponds to this inverse operator. So let's turn to some
specific time series models that are widely used. The class of autoregressive
moving average processes has this mathematical
definition. We define the X_t to be equal
to a linear combination of lags of X, going back p
lags, with coefficients phi_1 through phi_p. And then there are
residuals which are expressed in terms of a
q-th order moving average. So in this framework, the
eta_t's are white noise. And white noise, to reiterate,
has mean 0, constant variance, zero covariance between those. In this representation, I've
simplified things a little bit by subtracting off the
mean from all of the X's. And that just makes the formulas
a little bit more simpler. Now, with lag operators, we
can write this ARMA model as phi of L, p-th order
polynomial of lag L given with coefficients 1,
phi_1 up to phi_p, and theta of L given
by 1, theta_1, theta_2, up to theta_q. This is basically
a representation of the ARMA time series model. Basically, we're
taking a set of lags of the values of the stochastic
process up to order p. And that's equal to a weighted
average of the eta_t's. If we multiply by the inverse
of phi of L, if that exists, then we get this
representation here, which is simply the
Wold decomposition. So the ARMA models basically
have a Wold decomposition if this phi of L is invertible. And we'll explore
these by looking at simpler cases
of the ARMA models by just focusing on
autoregressive models first and then moving
average processes second so that
you'll get a better feel for how these things are
manipulated and interpreted. So let's move on to the p-th
order autoregressive process. So we're going to consider
ARMA models that just have autoregressive terms in them. So we have phi of L X_t
minus mu is equal to eta_t, which is white noise. So a linear combination of
the series is white noise. And X_t follows then a linear
regression model on explanatory variables, which are
lags of the process X. And this could be expressed
as X_t equal to c plus the sum from 1 to p of phi_j X_(t-j),
which is a linear regression model with regression
parameters phi_j. And c, the constant term, is
equal to mu times phi of 1. Now, if you basically take
expectations of the process, you basically have
coefficients of mu coming in from all the terms. And phi of 1 times mu is the
regression coefficient there. So with this
autoregressive model, we now want to go over what are
the stationarity conditions. Certainly, this
autoregressive model is one where, well,
a simple random walk follows an autoregressive
model but is not stationary. We'll highlight that
in a minute as well. But if you think
it, that's true. And so stationarity is something
to be understood and evaluated. This polynomial
function phi, where if we replace the
lag operator L by z, a complex variable, the
equation phi of z equal to 0 is the characteristic
equation associated with this autoregressive model. And it turns out that we'll
be interested in the roots of this characteristic equation. Now, if we consider
writing phi of L as a function of the
roots of the equation, we get this expression
where you'll notice if you multiply
all those terms out, the 1's all multiply out
together, and you get 1. And with the lag operator
L to the p-th power, that would be the product
of 1 over lambda_1 times 1 over lambda_2,
or actually negative 1 over lambda_1 times
negative 1 over lambda_2, and so forth-- negative
1 over lambda_p. Basically, if there are
p roots to this equation, this is how it would
be written out. And the process
X_t is covariance stationary if and
only if all the roots of this characteristic equation
lie outside the unit circle. So what does that mean? That means that the norm
modulus of the complex z is greater than 1. So they're outside
the unit circle where it's less
than or equal to 1. And the roots, if they are
outside the unit circle, then the modulus of the
lambda_j's is greater than 1. And if we then consider
taking a complex number lambda, basically
the root, and have an expression for 1 minus
1 over lambda L inverse, we can get this series
expression for that inverse. And that series will exist and
be bounded if the lambda_i are greater than 1 in magnitude. So we can actually compute
an inverse of phi of L by taking the inverse
of each of the component products in that polynomial. So in introductory
time series courses, they talk about
stationarity and unit roots, but they don't
really get into it, because people don't
know complex math, don't know about roots. So anyway, but this
is just very simply how that framework is applied. So we have a
polynomial equation, the characteristic equation,
whose roots we're looking for. Those roots have to
be outside the unit circle for stationarity
of the process. Well, it's basically
conditions for invertibility of the process, of the
autoregressive process. And that invertibility renders
the process an infinite-order moving average process. So let's go through
these results for the autoregressive
process of order one, where things-- always start
with the simplest cases to understand things. The characteristic equation
for this model is just 1 minus phi z. The root is 1/phi. So lambda is greater than
1-- if the modulus of lambda is greater than 1,
meaning the root is outside the unit circle,
then phi is less than 1. So for covariance stationarity
of this autoregressive process, we need the magnitude of phi
to be less than 1 in magnitude. The expected value of X is mu. The variance of X
is sigma squared X. This has this form, sigma
squared over 1 minus phi. That expression is
basically obtained by looking at the infinite order
moving average representation. But notice that if
phi is positive, then the variance
of X is actually greater than the variance
of the innovations. And if phi is less than 0,
then it's going to be smaller. So the innovation variance
basically is scaled up a bit in the autoregressive process. The covariance matrix is
phi times sigma squared X. You'll be going through
this in the problem set. And the covariance of X is phi
to the j power sigma squared X. And these expressions can
all be easily evaluated by simply writing out the
definition of these covariances in terms of the original
model and looking at what terms are independent,
cancel out, and that proceeds. Let's just go
through these cases. Let's show it all here. So we have if phi
is between 0 and 1, then the process experiences
exponential mean reversion to mu. So an autoregressive
process with phi between 0 on 1 corresponds to a
mean-reverting process. This process is
actually one that has been used theoretically
for interest rate models and a lot of theoretical
work in finance. The Vasicek model is
actually an example of the Ornstein-Uhlenbeck
process, which is basically a
mean-reverting Brownian motion. And any variables
that exhibit or could be thought of as
exhibiting mean reversion, this model can be
applied to those processes, such as interest rate
spreads or real exchange rates, variables where one can
expect that things never get too large or too small. They come back to some mean. Now, the challenge
is, that usually may be true over
short periods of time. But over very long
periods of time, the point to which you're
reverting to changes. So these models tend to
not have broad application over long time ranges. You need to adapt. Anyway, with the AR
process, we can also have negative
values of phi, which results in exponential mean
reversion that's oscillating in time, because the
autoregressive coefficient basically is a negative value. And for phi equal to 1, the Wold
decomposition doesn't exist. And the process is the
simple random walk. So basically, if
phi is equal to 1, that means that basically just
changes in value of the process are independent and identically
distributed white noise. And that's the
random walk process. And that process, as was
covered in earlier lectures, is non-stationary. If phi is greater than 1, then
you have an explosive process, because basically the
values are scaling up every time increment. So those are features
of the AR(1) model. For a general autoregressive
process of order p, there's a method-- well, we
can look at the second order moments of that process, which
have a very nice structure, and then use those to
solve for estimates of the ARMA parameters, or
autoregressive parameters. And those happen to be
specified by what are called the Yule-Walker equations. So the Yule-Walker equations
is a standard topic in time series analysis. What is it? What does it correspond to? Well, we take our original
autoregressive process of order p. And we write out the
formulas for the covariance at lag j between
two observations. So what's the covariance
between X_t and X_(t-j)? And that expression is
given by this equation. And so this equation for gamma
of j is determined simply by evaluating the expectations
where we're taking the expectation of X_t in the
autoregressive process times the fix X_(t-j) minus mu. So just evaluating
those terms, you can validate that
this is the equation. If we look at the equations
corresponding to j equals 1-- so lag 1 up through
lag p-- this is what those equations look like. Basically, the left-hand side
is gamma_1 through gamma_p. The covariance to
lag 1 up to lag p is equal to basically
linear functions given by the phi of
the other covariances. Who can tell me what the
structure is of this matrix? It's not a diagonal matrix? What kind of matrix is this? Math trivia question here. It has a special name. Anyone? It's a Toeplitz matrix. The off diagonals are
all the same value. And in fact, because of the
symmetry of the covariance, basically the gamma of 1 is
equal to gamma of minus 1. Gamma of minus 2 is
equal to gamma plus 2. Because of the
covariant stationarity, it's actually also symmetric. So these equations allow
us to solve for the phis so long as we have estimates
of these covariances. So if we have a
system of estimates, we can plug these in in
an attempt to solve this. If they're consistent
estimates of the covariances, then there will be a solution. And then the 0th
equation, which was not part of the series
of equations-- if you go back and look
at the 0th equation, that allows you to get an estimate
for the sigma squared. So these Yule-Walker
equations are the way in which many ARMA
models are specified in different statistics packages
and in terms of what principles are being applied. Well, if we're using unbiased
estimates of these parameters, then this is applying
what's called the method of moments principle
for statistical estimation. And with complicated models,
where sometimes the likelihood functions are very hard
to specify and compute, and then to do optimization
over those is even harder. It can turn out that
there are relationships between the moments of the
random variables, which are functions of the
unknown parameters. And you can solve for basically
the sample moments equalling the theoretical moments
and you apply the method of moments estimation method. Econometrics is rich with many
applications of that principle. The next section goes through
the moving average model. Let me highlight this. So with an order
q moving average, we basically have a polynomial
in the lag operator L, which is operated
upon the eta_t's. And if you write out
the expectations of X_t, you get mu. The variance of X_t,
which is gamma 0, is sigma squared times 1 plus
the squares of the coefficients in the polynomial. And so this feature,
this property here is due to the fact that we have
uncorrelated innovations in the eta_t's. The eta t's are white noise. So the only thing that comes
through in the square of X_t and the expectation of
that is the squared powers of the etas, which
have coefficients given by the theta_i squared. So these properties are left--
I'll leave you just to verify, very straightforward. But let's now turn to the
final minutes of the lecture today to accommodating
non-stationary behavior in time series. The original approaches
with time series was to focus on
estimation methodologies for covariance
stationary process. So if the series is not
covariance stationary, then we would want to
do some transformation of the data, of the
series, into a stationary so that the resulting
process is stationary. And with the
differencing operators, delta, Box and Jenkins
advocated moving non-stationary trending
behavior, which is exhibited often in
economic time series, by using a first difference,
maybe a second difference, or a k-th order difference. So these operators are
defined in this way. Basically with the
k-th order operator having this
expression here, this is the binomial expansion
of a k-th power, which can be useful. It comes up all the time
in probability theory. And if a process has
a linear time trend, then delta X_t is going to
have no time trend at all, because you're
basically taking out that linear component by
taking successive differences. Sometimes, if you
have a real series and you look at the difference,
it appears non-stationary, you look at first differences,
that can still not appear to be growing
over time, in which case sometimes the second
difference will result in a process with no trend. So these are sort of
convenient tricks, techniques to render
the series stationary. And let's see. There's examples here of
linear trend reversion models which are rendered
covariance stationary under first differencing. In this case, this is an
example where you have a deterministic time trend. But then you have reversion
to the time trend over time. So we basically have
eta_t, the error about the deterministic trend,
is a first order autoregressive process. And the moments here
can be derived this way. Leave that as an exercise. One could also consider
the pure integrated process and talk about
stochastic trends. And basically,
random walk processes are often referred
to in econometrics as stochastic trends. And you may want to try and
remove those from the data, or accommodate them. And so the stochastic
trend process is basically given by the first difference
X_t is just equal to eta_t. And so we have essentially
this random walk from a given starting point. And it's easy to verify it if
you knew the 0th point, then the variance of the t-th time
point would be t sigma squared, because we're summing t
independent innovations. And the covariance between
t and lag t minus j is simply t minus
j sigma squared. And the correlation between
those has this form. What you can see is that this
definitely depends on time. So it's not a
stationary process. So this first differencing
results in stationarity. And the end difference
process has those features. Let's see where we are. Final topic for
today is just how you incorporate non-stationary
process into ARMA processes. Well, if you take
first differences or second differences
and the resulting process is covariance
stationary, then we can just incorporate that
differencing into the model specification itself, and define
ARIMA models, Autoregressive Integrated Moving
Average Processes. And so to specify
these models, we need to determine the order
of the differencing required to move trends,
deterministic or stochastic, and then estimating
the unknown parameters, and then applying model
selection criteria. So let me go very
quickly through this and come back to it the
beginning of next time. But in specifying the
parameters of these models, we can apply maximum
likelihood, again, if we assume normality of
these innovations eta_t. And we can express
the ARMA model in state space
form, which results in a form for the
likelihood function, which we'll see a few lectures ahead. But then we can apply limited
information maximum likelihood, where we just condition on the
first observations of the data and maximize the likelihood. Or not condition on the first
few observations, but also use their information as well,
and look at their density functions, incorporating
those into the likelihood relative to the stationary
distribution for their values. And then the issue
becomes, how do we choose amongst different models? Now, last time we talked about
linear regression models, how you'd specify a
given model, here, we're talking about autoregressive,
moving average, and even integrated
moving average processes and how do we specify
those, well, with the method of maximum likelihood,
there are procedures which-- there are measures of
how effectively a fitted model is, given by an
information criterion that you would want to minimize
for a given fitted model. So we can consider
different sets of models, different numbers of
explanatory variables, different orders of
autoregressive parameters, moving average parameters,
and compute, say, the Akaike information criterion
or the Bayes information criterion or the
Hannan-Quinn criterion as different ways of judging
how good different models are. And let me just finish
today by pointing out that what these
information criteria are is basically a function of the
log likelihood function, which is something we're
trying to maximize with maximum
likelihood estimates. And then adding some penalty
for how many parameters we're estimating. And so what I'd like you to
think about for next time is what kind of a penalty
is appropriate for adding an extra parameter. Like, what evidence is
required to incorporate extra parameters, extra
variables, in the model. Would it be t statistics
that exceeds some threshold or some other criteria. Turns out that these are
all related to those issues. And it's very interesting
how those play out. And I'll say that for those
of you who have actually seen these before, the
Bayes information criterion corresponds to an
assumption that there is some finite number of
variables in the model. And you know what those are. The Hannan-Quinn criterion
says maybe there's an infinite number of
variables in the model, but you want to be
able to identify those. And so anyway, it's a
very challenging problem with model selection. And these criteria can
be used to specify those. So we'll go through
that next time.
