The following content is
provided under a Creative Commons license. Your support will help MIT
OpenCourseWare continue to offer high quality educational
resources for free. To make a donation or view
additional materials from hundreds of MIT courses, visit
MIT OpenCourseWare at ocw.mit.edu PROFESSOR: So we're going to
finish today our discussion of Bayesian Inference, which
we started last time. As you probably saw there's
not a huge lot of concepts that we're introducing at this
point in terms of specific skills of calculating
probabilities. But, rather, it's more of an
interpretation and setting up the framework. So the framework in Bayesian
estimation is that there is some parameter which is not
known, but we have a prior distribution on it. These are beliefs about what
this variable might be, and then we'll obtain some
measurements. And the measurements are
affected by the value of that parameter that we don't know. And this effect, the fact that
X is affected by Theta, is captured by introducing a
conditional probability distribution-- the distribution of X
depends on Theta. It's a conditional probability
distribution. So we have formulas for these
two densities, the prior density and the conditional
density. And given that we have these,
if we multiply them we can also get the joint density
of X and Theta. So we have everything
that's there is to know in this second. And now we observe the random
variable X. Given this random variable what can we
say about Theta? Well, what we can do is we
can always calculate the conditional distribution of
theta given X. And now that we have the specific value of
X we can plot this as a function of Theta. OK. And this is the complete
answer to a Bayesian Inference problem. This posterior distribution
captures everything there is to say about Theta, that's
what we know about Theta. Given the X that we have
observed Theta is still random, it's still unknown. And it might be here, there,
or there with several probabilities. On the other hand, if you want
to report a single value for Theta then you do
some extra work. You continue from here, and you
do some data processing on X. Doing data processing means
that you apply a certain function on the data,
and this function is something that you design. It's the so-called estimator. And once that function is
applied it outputs an estimate of Theta, which we
call Theta hat. So this is sort of the big
picture of what's happening. Now one thing to keep in mind
is that even though I'm writing single letters here, in
general Theta or X could be vector random variables. So think of this-- it could be a collection
Theta1, Theta2, Theta3. And maybe we obtained several
measurements, so this X is really a vector X1,
X2, up to Xn. All right, so now how do we
choose a Theta to report? There are various ways
of doing it. One is to look at the posterior
distribution and report the value of Theta, at
which the density or the PMF is highest. This is called the maximum
a posteriori estimate. So we pick a value of theta for
which the posteriori is maximum, and we report it. An alternative way is to try to
be optimal with respects to a mean squared error. So what is this? If we have a specific estimator,
g, this is the estimate it's going
to produce. This is the true value of
Theta, so this is our estimation error. We look at the square of the
estimation error, and look at the average value. We would like this squared
estimation error to be as small as possible. How can we design our estimator
g to make that error as small as possible? It turns out that the answer is
to produce, as an estimate, the conditional expectation
of Theta given X. So the conditional expectation is the
best estimate that you could produce if your objective is to
keep the mean squared error as small as possible. So this statement here is a
statement of what happens on the average over all Theta's and
all X's that may happen in our experiment. The conditional expectation as
an estimator has an even stronger property. Not only it's optimal on the
average, but it's also optimal given that you have made a
specific observation, no matter what you observe. Let's say you observe the
specific value for the random variable X. After that point if
you're asked to produce a best estimate Theta hat that
minimizes this mean squared error, your best estimate
would be the conditional expectation given the specific
value that you have observed. These two statements say almost
the same thing, but this one is a bit stronger. This one tells you no matter
what specific X happens the conditional expectation
is the best estimate. This one tells you on the
average, over all X's may happen, the conditional expectation is the best estimator. Now this is really a consequence
of this. If the conditional expectation
is best for any specific X, then it's the best one even when
X is left random and you are averaging your error
over all possible X's. OK so now that we know what is
the optimal way of producing an estimate let's do a
simple example to see how things work out. So we have started with an
unknown random variable, Theta, which is uniformly
distributed between 4 and 10. And then we have an observation
model that tells us that given the value of
Theta, X is going to be a random variable that ranges
between Theta - 1, and Theta + 1. So think of X as a noisy
measurement of Theta, plus some noise, which is
between -1, and +1. So really the model that we are
using here is that X is equal to Theta plus U -- where U is uniform
on -1, and +1. one, and plus one. So we have the true value of
Theta, but X could be Theta - 1, or it could be all the
way up to Theta + 1. And the X is uniformly
distributed on that interval. That's the same as saying that
U is uniformly distributed over this interval. So now we have all the
information that we need, we can construct the
joint density. And the joint density is, of
course, the prior density times the conditional density. We go both of these. Both of these are constants, so
the joint density is also going to be a constant. 1/6 times 1/2, this
is one over 12. But it is a constant,
not everywhere. Only on the range of possible
x's and thetas. So theta can take any value
between four and ten, so these are the values of theta. And for any given value of theta
x can take values from theta minus one, up
to theta plus one. So here, if you can imagine, a
line that goes with slope one, and then x can take that value
of theta plus or minus one. So this object here, this is
the set of possible x and theta pairs. So the density is equal to one
over 12 over this set, and it's zero everywhere else. So outside here the density is
zero, the density only applies at that point. All right, so now we're
asked to estimate theta in terms of x. So we want to build an estimator
which is going to be a function from the
x's to the thetas. That's why I chose the axis
this way-- x to be on this axis, theta on that axis-- Because the estimator we're
building is a function of x. Based on the observation that
we obtained, we want to estimate theta. So we know that the optimal
estimator is the conditional expectation, given
the value of x. So what is the conditional
expectation? If you fix a particular value of
x, let's say in this range. So this is our x, then what
do we know about theta? We know that theta lies
in this range. Theta can only be sampled
between those two values. And what kind of distribution
does theta have? What is the conditional
distribution of theta given x? Well, remember how we built
conditional distributions from joint distributions? The conditional distribution is
just a section of the joint distribution applied to
the place where we're conditioning. So the joint is constant. So the conditional is also going
to be a constant density over this interval. So the posterior distribution
of theta is uniform over this interval. So if the posterior of theta is
uniform over that interval, the expected value of theta is
going to be the meet point of that interval. So the estimate which
you report-- if you observe that theta-- is going to be this particular
point here, it's the midpoint. The same argument goes through
even if you obtain an x somewhere here. Given this x, theta
can take a value between these two values. Theta is going to have a uniform
distribution over this interval, and the conditional
expectation of theta given x is going to be the midpoint
of that interval. So now if we plot our estimator
by tracing midpoints in this diagram what you're
going to obtain is a curve that starts like this, then
it changes slope. So that it keeps track of the
midpoint, and then it goes like that again. So this blue curve here is
our g of x, which is the conditional expectation of
theta given that x is equal to little x. So it's a curve, in our example
it consists of three straight segments. But overall it's non-linear. It's not a single line
through this diagram. And that's how things
are in general. g of x, our optimal estimate has
no reason to be a linear function of x. In general it's going to be
some complicated curve. So how good is our estimate? I mean you reported your x, your
estimate of theta based on x, and your boss asks you
what kind of error do you expect to get? Having observed the particular
value of x, what you can report to your boss is what you
think is the mean squared error is going to be. We observe the particular
value of x. So we're conditioning, and we're
living in this universe. Given that we have made this
observation, this is the true value of theta, this is the
estimate that we have produced, this is the expected
squared error, given that we have made the particular
observation. Now in this conditional universe
this is the expected value of theta given x. So this is the expected value of
this random variable inside the conditional universe. So when you take the mean
squared of a random variable minus the expected value, this
is the same thing as the variance of that random
variable. Except that it's the
variance inside the conditional universe. Having observed x, theta is
still a random variable. It's distributed according to
the posterior distribution. Since it's a random variable,
it has a variance. And that variance is our
mean squared error. So this is the variance of the
posterior distribution of Theta given the observation
that we have made. OK, so what is the variance
in our example? If X happens to be here, then
Theta is uniform over this interval, and this interval
has length 2. Theta is uniformly distributed
over an interval of length 2. This is the posterior
distribution of Theta. What is the variance? Then you remember the formula
for the variance of a uniform random variable, it is the
length of the interval squared divided by 12, so this is 1/3. So the variance of Theta -- the mean squared error-- is
going to be 1/3 whenever this kind of picture applies. This picture applies when
X is between 5 and 9. If X is less than 5, then the
picture is a little different, and Theta is going
to be uniform over a smaller interval. And so the variance of
theta is going to be smaller as well. So let's start plotting our
mean squared error. Between 5 and 9 the variance
of Theta -- the posterior variance-- is 1/3. Now when the X falls in here
Theta is uniformly distributed over a smaller interval. The size of this interval
changes linearly over that range. And so when we take the square
size of that interval we get a quadratic function of
how much we have moved from that corner. So at that corner what is
the variance of Theta? Well if I observe an X that's
equal to 3 then I know with certainty that Theta
is equal to 4. Then I'm in very good shape, I
know exactly what Theta is going to be. So the variance, in this
case, is going to be 0. If I observe an X that's a
little larger than Theta is now random, takes values on
a little interval, and the variance of Theta is going to be
proportional to the square of the length of that
little interval. So we get a curve that
starts rising quadratically from here. It goes up forward 1/3. At the other end of the picture
the same is true. If you observe an X which is
11 then Theta can only be equal to 10. And so the error in Theta
is equal to 0, there's 0 error variance. But as we obtain X's that are
slightly less than 11 then the mean squared error again
rises quadratically. So we end up with a
plot like this. What this plot tells us is that
certain measurements are better than others. If you're lucky, and you see X
equal to 3 then you're lucky, because you know Theta
exactly what it is. If you see an X which is equal
to 6 then you're sort of unlikely, because it
doesn't tell you Theta with great precision. Theta could be anywhere
on that interval. And so the variance
of Theta -- even after you have
observed X -- is a certain number,
1/3 in our case. So the moral to keep out
of that story is that the error variance-- or the mean squared error-- depends on what particular
observation you happen to obtain. Some observations may be very
informative, and once you see a specific number than you know
exactly what Theta is. Some observations might
be less informative. You observe your X, but it could
still leave a lot of uncertainty about Theta. So conditional expectations are
really the cornerstone of Bayesian estimation. They're particularly
popular, especially in engineering contexts. There used a lot in signal
processing, communications, control theory, so on. So that makes it worth playing
a little bit with their theoretical properties, and get
some appreciation of a few subtleties involved here. No new math in reality, in what
we're going to do here. But it's going to be a good
opportunity to practice manipulation of conditional
expectations. So let's look at the expected
value of the estimation error that we obtained. So Theta hat is our estimator,
is the conditional expectation. Theta hat minus Theta is what
kind of error do we have? If Theta hat, is bigger than
Theta then we have made the positive error. If not, if it's on the other
side, we have made the negative error. Then it turns out that on the
average the errors cancel each other out, on the average. So let's do this calculation. Let's calculate the expected
value of the error given X. Now by definition the error is
expected value of Theta hat minus Theta given X. We use linearity of expectations
to break it up as expected value of Theta hat
given X minus expected value of Theta given X.
And now what? Our estimate is made on the
basis of the data of the X's. If I tell you X then you
know what Theta hat is. Remember that the conditional
expectation is a random variable which is a function
of the random variable, on which you're conditioning on. If you know X then you know the
conditional expectation given X, you know what Theta
hat is going to be. So Theta hat is a function of
X. If it's a function of X then once I tell you X
you know what Theta hat is going to be. So this conditional expectation
is going to be Theta hat itself. Here this is-- just
by definition-- Theta hat, and so we
get equality to 0. So what we have proved is that
no matter what I have observed, and given that I have
observed something on the average my error is
going to be 0. This is a statement involving
equality of random variables. Remember that conditional
expectations are random variables because they depend
on the thing you're conditioning on. 0 is sort of a trivial
random variable. This tells you that this random
variable is identically equal to the 0 random
variable. More specifically it tells you
that no matter what value for X you observe, the conditional
expectation of the error is going to be 0. And this takes us to this
statement here, which is inequality between numbers. No matter what specific value
for capital X you have observed, your error, on
the average, is going to be equal to 0. So this is a less abstract
version of these statements. This is inequality between
two numbers. It's true for every value of
X, so it's true in terms of these random variables being
equal to that random variable. Because remember according to
our definition this random variable is the random variable
that takes this specific value when capital
X happens to be equal to little x. Now this doesn't mean that your
error is 0, it only means that your error is as likely, in
some sense, to fall on the positive side, as to fall
on the negative side. So sometimes your error will be positive, sometimes negative. And on the average these
things cancel out and give you a 0 --. on the average. So this is a property that's
sometimes giving the name we say that Theta hat
is unbiased. So Theta hat, our estimate, does
not have a tendency to be on the high side. It does not have a tendency
to be on the low side. On the average it's
just right. So let's do a little
more playing here. Let's see how our error is
related to an arbitrary function of the data. Let's do this in a conditional
universe and look at this quantity. In a conditional universe
where X is known then h of X is known. And so you can pull it outside
the expectation. In the conditional universe
where the value of X is given this quantity becomes
just a constant. There's nothing random
about it. So you can pull it out,
the expectation, and write things this way. And we have just calculated
that this quantity is 0. So this number turns out
to be 0 as well. Now having done this,
we can take expectations of both sides. And now let's use the law of
iterated expectations. Expectation of a conditional
expectation gives us the unconditional expectation, and
this is also going to be 0. So here we use the law of
iterated expectations. OK. OK, why are we doing this? We're doing this because I would
like to calculate the covariance between Theta
tilde and Theta hat. Theta hat is, ask the question
-- is there a systematic relation between the error
and the estimate? So to calculate the covariance
we use the property that we can calculate the covariances
by calculating the expected value of the product minus
the product of the expected values. And what do we get? This is 0, because of
what we just proved. And this is 0, because of
what we proved earlier. That the expected value of
the error is equal to 0. So the covariance between the
error and any function of X is equal to 0. Let's use that to the case where
the function of X we're considering is Theta
hat itself. Theta hat is our estimate, it's
a function of X. So this 0 result would still apply,
and we get that this covariance is equal to 0. OK, so that's what we proved. Let's see, what are the morals
to take out of all this? First is you should be very
comfortable with this type of calculation involving
conditional expectations. The main two things that we're
using are that when you condition on a random variable
any function of that random variable becomes a constant,
and can be pulled out the conditional expectation. The other thing that we are
using is the law of iterated expectations, so these are
the skills involved. Now on the substance, why is
this result interesting? This tells us that the error is uncorrelated with the estimate. What's a hypothetical situation
where these would not happen? Whenever Theta hat is positive
my error tends to be negative. Suppose that whenever Theta hat
is big then you say oh my estimate is too big, maybe the
true Theta is on the lower side, so I expect my error
to be negative. That would be a situation that
would violate this condition. This condition tells you that
no matter what Theta hat is, you don't expect your error to
be on the positive side or on the negative side. Your error will still
be 0 on the average. So if you obtain a very high
estimate this is no reason for you to suspect that
the true Theta is lower than your estimate. If you suspected that the true
Theta was lower than your estimate you should have
changed your Theta hat. If you make an estimate and
after obtaining that estimate you say I think my estimate
is too big, and so the error is negative. If you thought that way then
that means that your estimate is not the optimal one, that
your estimate should have been corrected to be smaller. And that would mean that there's
a better estimate than the one you used, but the
estimate that we are using here is the optimal one in terms
of mean squared error, there's no way of
improving it. And this is really captured
in that statement. That is knowing Theta hat
doesn't give you a lot of information about the error, and
gives you, therefore, no reason to adjust your estimate
from what it was. Finally, a consequence
of all this. This is the definition
of the error. Send Theta to this side, send
Theta tilde to that side, you get this relation. The true parameter is composed
of two quantities. The estimate, and the
error that they got with a minus sign. These two quantities are
uncorrelated with each other. Their covariance is 0, and
therefore, the variance of this is the sum of the variances
of these two quantities. So what's an interpretation
of this equality? There is some inherent
randomness in the random variable theta that we're
trying to estimate. Theta hat tries to estimate it,
tries to get close to it. And if Theta hat always stays
close to Theta, since Theta is random Theta hat must also be
quite random, so it has uncertainty in it. And the more uncertain Theta
hat is the more it moves together with Theta. So the more uncertainty
it removes from Theta. And this is the remaining
uncertainty in Theta. The uncertainty that's left
after we've done our estimation. So ideally, to have a small
error we want this quantity to be small. Which is the same as
saying that this quantity should be big. In the ideal case Theta hat
is the same as Theta. That's the best we
could hope for. That corresponds to 0 error,
and all the uncertainly in Theta is absorbed by the
uncertainty in Theta hat. Interestingly, this relation
here is just another variation of the law of total variance
that we have seen at some point in the past. I will skip that derivation, but
it's an interesting fact, and it can give you an
alternative interpretation of the law of total variance. OK, so now let's return
to our example. In our example we obtained the
optimal estimator, and we saw that it was a nonlinear curve,
something like this. I'm exaggerating the corner
of a little bit to show that it's nonlinear. This is the optimal estimator. It's a nonlinear function
of X -- nonlinear generally
means complicated. Sometimes the conditional
expectation is really hard to compute, because whenever you
have to compute expectations you need to do some integrals. And if you have many random
variables involved it might correspond to a
multi-dimensional integration. We don't like this. Can we come up, maybe,
with a simpler way of estimating Theta? Of coming up with a point
estimate which still has some nice properties, it
has some good motivation, but is simpler. What does simpler mean? Perhaps linear. Let's put ourselves in a
straitjacket and restrict ourselves to estimators that's
are of these forms. My estimate is constrained
to be a linear function of the X's. So my estimator is going to be
a curve, a linear curve. It could be this, it could be
that, maybe it would want to be something like this. I want to choose the best
possible linear function. What does that mean? It means that I write my
Theta hat in this form. If I fix a certain a and b I
have fixed the functional form of my estimator, and this
is the corresponding mean squared error. That's the error between the
true parameter and the estimate of that parameter, we
take the square of this. And now the optimal linear
estimator is defined as one for which these mean squared
error is smallest possible over all choices of a and b. So we want to minimize
this expression over all a's and b's. How do we do this
minimization? Well this is a square,
you can expand it. Write down all the terms in the
expansion of the square. So you're going to get
the term expected value of Theta squared. You're going to get
another term-- a squared expected value of X
squared, another term which is b squared, and then you're
going to get to various cross terms. What you have here is really a
quadratic function of a and b. So think of this quantity that
we're minimizing as some function h of a and b, and it
happens to be quadratic. How do we minimize a
quadratic function? We set the derivative of this
function with respect to a and b to 0, and then
do the algebra. After you do the algebra you
find that the best choice for a is this 1, so this is the
coefficient next to X. This is the optimal a. And the optimal b corresponds
of the constant terms. So this term and this times that
together are the optimal choices of b. So the algebra itself is
not very interesting. What is really interesting is
the nature of the result that we get here. If we were to plot the result on
this particular example you would get the curve that's
something like this. It goes through the middle
of this diagram and is a little slanted. In this example, X and Theta
are positively correlated. Bigger values of X generally
correspond to bigger values of Theta. So in this example the
covariance between X and Theta is positive, and so our estimate
can be interpreted in the following way: The expected
value of Theta is the estimate that you would come up
with if you didn't have any information about Theta. If you don't make any
observations this is the best way of estimating Theta. But I have made an observation,
X, and I need to take it into account. I look at this difference, which
is the piece of news contained in X? That's what X should
be on the average. If I observe an X which is
bigger than what I expected it to be, and since X and Theta
are positively correlated, this tells me that Theta should
also be bigger than its average value. Whenever I see an X that's
larger than its average value this gives me an indication
that theta should also probably be larger than
its average value. And so I'm taking that
difference and multiplying it by a positive coefficient. And that's what gives
me a curve here that has a positive slope. So this increment-- the new information contained
in X as compared to the average value we expected
apriori, that increment allows us to make a correction to our
prior estimate of Theta, and the amount of that correction is
guided by the covariance of X with Theta. If the covariance of X with
Theta were 0, that would mean there's no systematic relation
between the two, and in that case obtaining some information
from X doesn't give us a guide as to how to
change the estimates of Theta. If that were 0, we would
just stay with this particular estimate. We're not able to make
a correction. But when there's a non zero
covariance between X and Theta that covariance works as a
guide for us to obtain a better estimate of Theta. How about the resulting
mean squared error? In this context turns out that
there's a very nice formula for the mean squared
error obtained from the best linear estimate. What's the story here? The mean squared error that we
have has something to do with the variance of the original
random variable. The more uncertain our original
random variable is, the more error we're
going to make. On the other hand, when the two
variables are correlated we explored that correlation
to improve our estimate. This row here is the correlation
coefficient between the two random
variables. When this correlation
coefficient is larger this factor here becomes smaller. And our mean squared error
become smaller. So think of the two
extreme cases. One extreme case is when
rho equal to 1 -- so X and Theta are perfectly
correlated. When they're perfectly
correlated once I know X then I also know Theta. And the two random variables
are linearly related. In that case, my estimate is
right on the target, and the mean squared error
is going to be 0. The other extreme case is
if rho is equal to 0. The two random variables
are uncorrelated. In that case the measurement
does not help me estimate Theta, and the uncertainty
that's left-- the mean squared error-- is just the original
variance of Theta. So the uncertainty in Theta
does not get reduced. So moral-- the estimation error is a
reduced version of the original amount of uncertainty
in the random variable Theta, and the larger the correlation
between those two random variables, the better we can
remove uncertainty from the original random variable. I didn't derive this formula,
but it's just a matter of algebraic manipulations. We have a formula for
Theta hat, subtract Theta from that formula. Take square, take expectations,
and do a few lines of algebra that you can
read in the text, and you end up with this really neat
and clean formula. Now I mentioned in the beginning
of the lecture that we can do inference with Theta's
and X's not just being single numbers, but they could
be vector random variables. So for example we might have
multiple data that gives us information about X. There are no vectors here, so
this discussion was for the case where Theta and X were just
scalar, one-dimensional quantities. What do we do if we have
multiple data? Suppose that Theta is still a
scalar, it's one dimensional, but we make several
observations. And on the basis of these
observations we want to estimate Theta. The optimal least mean squares
estimator would be again the conditional expectation of
Theta given X. That's the optimal one. And in this case X is a
vector, so the general estimator we would use
would be this one. But if we want to keep things
simple and we want our estimator to have a simple
functional form we might restrict to estimator that are
linear functions of the data. And then the story is
exactly the same as we discussed before. I constrained myself to
estimating Theta using a linear function of the data,
so my signal processing box just applies a linear
function. And I'm looking for the best
coefficients, the coefficients that are going to result
in the least possible squared error. This is my squared error, this
is (my estimate minus the thing I'm trying to estimate)
squared, and then taking the average. How do we do this? Same story as before. The X's and the Theta's get
averaged out because we have an expectation. Whatever is left is just a
function of the coefficients of the a's and of b's. As before it turns out to
be a quadratic function. Then we set the derivatives of
this function of a's and b's with respect to the
coefficients, we set it to 0. And this gives us a system
of linear equations. It's a system of linear
equations that's satisfied by those coefficients. It's a linear system because
this is a quadratic function of those coefficients. So to get closed-form formulas
in this particular case one would need to introduce vectors,
and matrices, and metrics inverses and so on. The particular formulas are not
so much what interests us here, rather, the interesting
thing is that this is simply done just using straightforward
solvers of linear equations. The only thing you need to do
is to write down the correct coefficients of those non-linear
equations. And the typical coefficient
that you would get would be what? Let say a typical quick
equations would be -- let's take a typical
term of this quadratic one you expanded. You're going to get the terms
such as a1x1 times a2x2. When you take expectations
you're left with a1a2 times expected value of x1x2. So this would involve terms such
as a1 squared expected value of x1 squared. You would get terms such as
a1a2, expected value of x1x2, and a lot of other terms
here should have a too. So you get something that's
quadratic in your coefficients. And the constants that show up
in your system of equations are things that have to do with
the expected values of squares of your random
variables, or products of your random variables. To write down the numerical
values for these the only thing you need to know are the
means and variances of your random variables. If you know the mean and
variance then you know what this thing is. And if you know the covariances
as well then you know what this thing is. So in order to find the optimal
linear estimator in the case of multiple data you do
not need to know the entire probability distribution
of the random variables that are involved. You only need to know your
means and covariances. These are the only quantities
that affect the construction of your optimal estimator. We could see this already
in this formula. The form of my optimal estimator
is completely determined once I know the
means, variance, and covariance of the random
variables in my model. I do not need to know how the
details distribution of the random variables that
are involved here. So as I said in general, you
find the form of the optimal estimator by using a linear
equation solver. There are special examples
in which you can get closed-form solutions. The nicest simplest estimation
problem one can think of is the following-- you have some uncertain
parameter, and you make multiple measurements
of that parameter in the presence of noise. So the Wi's are noises. I corresponds to your
i-th experiment. So this is the most common
situation that you encounter in the lab. If you are dealing with some
process, you're trying to measure something you measure
it over and over. Each time your measurement
has some random error. And then you need to take all
your measurements together and come up with a single
estimate. So the noises are assumed to be
independent of each other, and also to be independent
from the value of the true parameter. Without loss of generality we
can assume that the noises have 0 mean and they have
some variances that we assume to be known. Theta itself has a prior
distribution with a certain mean and the certain variance. So the form of the
optimal linear estimator is really nice. Well maybe you cannot see it
right away because this looks messy, but what is it really? It's a linear combination of
the X's and the prior mean. And it's actually a weighted
average of the X's and the prior mean. Here we collect all of
the coefficients that we have at the top. So the whole thing is basically
a weighted average. 1/(sigma_i-squared) is the
weight that we give to Xi, and in the denominator we have the
sum of all of the weights. So in the end we're dealing
with a weighted average. If mu was equal to 1, and all
the Xi's were equal to 1 then our estimate would also
be equal to 1. Now the form of the weights that
we have is interesting. Any given data point is
weighted inversely proportional to the variance. What does that say? If my i-th data point has a lot
of variance, if Wi is very noisy then Xi is not very
useful, is not very reliable. So I'm giving it
a small weight. Large variance, a lot of error
in my Xi means that I should give it a smaller weight. If two data points have the
same variance, they're of comparable quality,
then I'm going to give them equal weight. The other interesting thing is
that the prior mean is treated the same way as the X's. So it's treated as an additional
observation. So we're taking a weighted
average of the prior mean and of the measurements that
we are making. The formula looks as if the
prior mean was just another data point. So that's the way of thinking
about Bayesian estimation. You have your real data points,
the X's that you observe, you also had some
prior information. This plays a role similar
to a data point. Interesting note that if all
random variables are normal in this model these optimal linear
estimator happens to be also the conditional
expectation. That's the nice thing about
normal random variables that conditional expectations
turn out to be linear. So the optimal estimate and the
optimal linear estimate turn out to be the same. And that gives us another
interpretation of linear estimation. Linear estimation is essentially
the same as pretending that all random
variables are normal. So that's a side point. Now I'd like to close
with a comment. You do your measurements and
you estimate Theta on the basis of X. Suppose that instead
you have a measuring device that's measures X-cubed
instead of measuring X, and you want to estimate Theta. Are you going to get to
different a estimate? Well X and X-cubed contained
the same information. Telling you X is the
same as telling you the value of X-cubed. So the posterior distribution
of Theta given X is the same as the posterior distribution
of Theta given X-cubed. And so the means of these
posterior distributions are going to be the same. So doing transformations through
your data does not matter if you're doing optimal
least squares estimation. On the other hand, if you
restrict yourself to doing linear estimation then using a
linear function of X is not the same as using a linear
function of X-cubed. So this is a linear estimator,
but where the data are the X-cube's, and we have a linear
function of the data. So this means that when you're
using linear estimation you have some choices to make
linear on what? Sometimes you want to plot your
data on a not ordinary scale and try to plot
a line through them. Sometimes you plot your data
on a logarithmic scale, and try to plot a line
through them. Which scale is the
appropriate one? Here it would be
a cubic scale. And you have to think about
your particular model to decide which version would be
a more appropriate one. Finally when we have multiple
data sometimes these multiple data might contain the
same information. So X is one data point,
X-squared is another data point, X-cubed is another
data point. The three of them contain the
same information, but you can try to form a linear
function of them. And then you obtain a linear
estimator that has a more general form as a
function of X. So if you want to estimate your
Theta as a cubic function of X, for example, you can set
up a linear estimation model of this particular form and find
the optimal coefficients, the a's and the b's. All right, so the last slide
just gives you the big picture of what's happening in Bayesian
Inference, it's for you to ponder. Basically we talked about three possible estimation methods. Maximum posteriori, mean squared
error estimation, and linear mean squared error
estimation, or least squares estimation. And there's a number of standard
examples that you will be seeing over and over in
the recitations, tutorial, homework, and so on, perhaps
on exams even. Where we take some nice priors
on some unknown parameter, we take some nice models for the
noise or the observations, and then you need to work out
posterior distributions in the various estimates and
compare them.
