The following content is
provided under a Creative Commons license. Your support will help MIT
OpenCourseWare continue to offer high-quality, educational resources for free. To make a donation or view
additional materials from hundreds of MIT courses, visit
MIT OpenCourseWare at ocw.mit.edu. PROFESSOR: Good morning. So today we're going
to continue the subject from last time. So we're going to talk about
derived distributions a little more, how to derive the
distribution of a function of a random variable. So last time we discussed a
couple of examples in which we had a function of a
single variable. And we found the distribution
of Y, if we're told the distribution of X. So today we're going to do an
example where we deal with the function of two random
variables. And then we're going to consider
the most interesting example of this kind, in which
we have a random variable of the form W, which is
the sum of two independent, random variables. That's a case that shows
up quite often. And so we want to see what
exactly happens in this particular case. Just one comment that
I should make. The material that we're covering
now, chapter four, is sort of conceptually a little
more difficult than one we have been doing before. So I would definitely encourage
you to read the text before you jump and try
to do the problems in your problem sets. OK, so let's start with our
example, in which we're given two random variables. They're jointly continuous. And their distribution
is pretty simple. They're uniform on
the unit square. In particular, each one of the
random variables is uniform on the unit interval. And the two random variables
are independent. What we're going to find is the
distribution of the ratio of the two random variables. How do we go about it? , Well,
the same cookbook procedure that we used last time
for the case of a single random variable. The cookbook procedure that
we used for this case also applies to the case where you
have a function of multiple random variables. So what was the cookbook
procedure? The first step is to find the
cumulative distribution function of the random variable
of interest and then take the derivative in order
to find the density. So let's find the cumulative. So, by definition, the
cumulative is the probability that the random variable is
less than or equal to the argument of the cumulative. So if we write this event in
terms of the random variable of interest, this is the
probability that our random variable is less than
or equal to z. So what is that? OK, so the ratio is going to be
less than or equal to z, if and only if the pair, (x,y),
happens to fall below the line that has a slope z. OK, so we draw a line
that has a slope z. The ratio is less than this
number, if and only if we get the pair of x and y that falls
inside this triangle. So we're talking about
the probability of this particular event. Since this line has a slope of
z, the height at this point is equal to z. And so we can find the
probability of this event. It's just the area
of this triangle. And so the area is 1
times z times 1/2. And we get the answer, z/2. Now, is this answer
always correct? Now, this answer is going to be
correct only if the slope happens to be such that we get
a picture of this kind. So when do we get a picture
of this kind? When the slope is less than 1. If I consider a different slope,
a number, little z -- that happens to be a slope
of that kind -- then the picture changes. And in that case, we
get a picture of this kind, let's say. So this is a line here
of slope z, again. And this is the second case in
which our number, little z, is bigger than 1. So how do we proceed? Once more, the cumulative is the
probability that the ratio is less than or equal
to that number. So it's the probability that
we fall below the red line. So we're talking about the
event, about this event. So to find the probability of
this event, we need to find the area of this red shape. And one way of finding this area
is to consider the whole area and subtract the area
of this triangle. So let's do it this way. It's going to be 1 minus the
area of the triangle. Now, what's the area
of the triangle? It's 1/2 times this side, which
is 1 times this side. How big is that side? Well, if y and the slope is z,
now z is the ratio y over x. So if y over x-- at this point we have
y/x = z and y =1. This means that z is 1/x. So the coordinate of
this point is 1/x. And this means that
we're going to-- 1/z So here we get the
factor of 1/z. And we're basically done. I guess if you want to have a
complete answer, you should also give the formula
for z less than 0. What is the cumulative when
z is less than 0, the probability that you get the
ratio that's negative? Well, since our random variables
are positive, there's no way that you can
get a negative ratio. So the cumulative down
there is equal to 0. So we can plot the cumulative. And we can take its derivative
in order to find the density. So the cumulative that
we got starts at 0, when z's are negative. Then it starts going up
in proportion to z, at the slope of 1/2. So this takes us up to 1. And then it starts increasing
towards 1, according to this function. When you let z go to infinity,
the cumulative is going to go to 1. And it has a shape of, more
or less, this kind. So now to get the density, we
just take the derivative. And the density is, of
course, 0 down here. Up here the derivative
is just 1/2. And beyond that point we need to
take the derivative of this expression. And the derivative is going to
be 1/2 times 1 over z-squared. So it's going to be a
shape of this kind. And we're done. So you see that problems
involving functions of multiple random variables are
no harder than problems that deal with the functional of
a single random variable. The general procedure is,
again, exactly the same. You first find the cumulative,
and then you differentiate. The only extra difficulty will
be that when you calculate the cumulative, you need to find
the probability of an event that involves multiple
random variables. And sometimes this could be
a little harder to do. By the way, since we dealt
with this example, just a couple of questions. What do you think is going to
be the expected value of the random variable Z? Let's see, the expected value
of the random variable Z is going to be the integral
of z times the density. And the density is equal to 1/2
for z going from 0 to 1. And then there's another contribution from 1 to infinity. There the density is
1/(2z-squared). And we get the z, since we're
dealing with expectations, dz. So what is this integral? Well, if you look here, you're
integrating 1/z, all the way to infinity. 1/z has an integral, which
is the logarithm of z. And since the logarithm goes to
infinity, this means that this integral is
also infinite. So the expectation of the random
variable Z is actually infinite in this example. There's nothing wrong
with this. Lots of random variables have
infinite expectations. If the tail of the density falls
kind of slowly, as the argument goes to infinity, then
it may well turn out that you get an infinite integral. So that's just how
things often are. Nothing strange about it. And now, since we are still in
this example, let me ask another question. Would we reason, on the average,
would it be true that the expected value of Z -- remember that Z is the ratio
Y/X -- could it be that the expected value of Z
is this number? Or could it be that it's
equal to this number? Or could it be that it's
none of the above? OK, so how many people think
this is correct? Small number. How many people think
this is correct? Slightly bigger, but still
a small number. And how many people think
this is correct? OK, that's-- this one wins the vote. OK, let's see. This one is not correct, just
because there's no reason it should be correct. So, in general, you cannot
reason on the average. The expected value of a function
is not the same as the same function of the
expected values. This is only true if you're
dealing with linear functions of random variables. So this is not-- this turns out to
not be correct. How about this one? Well, X and Y are independent,
by assumption. So 1/X and Y are also
independent. Why is this? Independence means that one
random variable does not convey any information
about the other. So Y doesn't give you any
information about X. So Y doesn't give you any information
about 1/X. Or to put it differently, if two
random variables are independent, functions of each
one of those random variables are also independent. If X is independent from
Y, then g(X) is independent of h(Y). So this applies to this case. These two random variables
are independent. And since they are independent,
this means that the expected value of their
product is equal to the product of the expected
values. So this relation actually
is true. And therefore, this
is not true. OK. Now, let's move on. We have this general procedure
of finding the derived distribution by going through
the cumulative. Are there some cases where
we can have a shortcut? Turns out that there is a
special case or a special structure in which we can get
directly from densities to densities using directly
just a formula. And in that case, we don't
have to go through the cumulative. And this case is also
interesting, because it gives us some insight about how one
density changes to a different density and what affects the
shape of those densities. So the case where things easy
is when the transformation from one random variable to
the other is a strictly monotonic one. So there's a one-to-one relation
between x's and y's. Here we can reason directly in
terms of densities by thinking in terms of probabilities
of small intervals. So let's look at the small
interval on the x-axis, like this one, when X ranges from-- where capital X ranges
from a small x to a small x plus delta. So this is a small interval
of length delta. Whenever X happens to fall in
this interval, the random variable Y is going
to fall in a corresponding interval up there. So up there we have a
corresponding interval. And these two intervals, the
red and the blue interval-- this is the blue interval. And that's the red interval. These two intervals should have
the same probability. They're exactly the
same event. When X falls here, g(X) happens
to fall in there. So we can sort of say that the
probability of this little interval is the same
as the probability of that little interval. And we know that probabilities
of little intervals have something to do with
densities. So what is the probability
of this little interval? It's the density of the random
variable X, at this point, times the length of
the interval. How about the probability
of that interval? It's going to be the density of
the random variable Y times the length of that
little interval. Now, this interval
has length delta. Does that mean that
this interval also has length delta? Well, not necessarily. The length of this interval has
something to do with the slope of your function g. So slope is dy by dx. Is how much-- the slope tells
you how big is the y interval when you take an interval
x of a certain length. So the slope is what multiplies
the length of this interval to give you the length
of that interval. So the length of this interval
is delta times the slope of your function. So the length of the interval
is delta times the slope of the function, approximately. So the probability of this
interval is going to be the density of Y times the length
of the interval that we are considering. So this gives us a relation
between the density of X, evaluated at this point, to the
density of Y, evaluated at that point. The two densities are
closely related. If these x's are very likely
to occur, then this is big, which means that that density
will also be big. If these x's are very likely to
occur, then those y's are also very likely to occur. But there's also another
factor that comes in. And that's the slope
of the function at this particular point. So we have this relation between
the two densities. Now, in interpreting this
equation, you need to make sure what's the relation between
the two variables. I have both little x's
and little y's. Well, this formula is true for
an (x,y) pair, that they're related according to this
particular function. So if I fix an x and consider
the corresponding y, then the densities at those x's and
corresponding y's will be related by that formula. Now, in the end, you want to
come up with a formula that just gives you the density
of Y as a function of y. And that means that you need to eliminate x from the picture. So let's see how that would
go in an example. So suppose that we're dealing
with the function y equal to x cubed, in which case our
function, g(x), is the function x cubed. And if x cubed is equal to a
little y, If we have a pair of x's and y's that are related
this way, then this means that x is going to be the
cubic root of y. So this is the formula that
takes us back from y's to x's. This is the direct function from
x, how to construct y. This is essentially the inverse
function that tells us, from a given y what is
the corresponding x. Now, if we write this formula,
it tells us that the density at the particular x is going
to be the density at the corresponding y times the slope
of the function at the particular x that we
are considering. The slope of the function
is 3x squared. Now, we want to end up with a
formula for the density of Y. So I'm going to take this
factor, send it to the other side. But since I want it to be a
function of y, I want to eliminate the x's. And I'm going to eliminate
the x's using this correspondence here. So I'm going to get
the density of X evaluated at y to the 1/3. And then this factor in the
denominator, it's 1/(3y to the power 2/3). So we end up finally with the
formula for the density of the random variable Y. And this is the same answer that
you would get if you go through this exercise using the
cumulative distribution function method. You end up getting
the same answer. But here we sort of
get it directly. Just to get a little more
insight as to why the slope comes in-- suppose that we have a function
like this one. So the function is sort of flat,
then moves quickly, and then becomes flat again. What should be -- and suppose that X has some kind
of reasonable density, some kind of flat density. Suppose that X is a pretty
uniform random variable. What's going to happen to
the random variable Y? What kind of distribution
should it have? What are the typical values
of the random variable Y? Either x falls here, and y is
a very small number, or-- let's take that number here
to be -- let's say 2 -- or x falls in this range, and
y takes a value close to 2. And there's a small chance that
x's will be somewhere in the middle, in which case y
takes intermediate values. So what kind of shape do
you expect for the distribution of Y? There's going to be a fair
amount of probability that Y takes values close to 0. There's a small probability
that Y takes intermediate values. That corresponds to the case
where x falls in here. That's not a lot
of probability. So the probability that Y takes
values between 0 and 2, that's kind of small. But then there's a lot of x's
that produces y's that are close to 2. So there's a significant
probability that Y would take values that are close to 2. So you-- the density of Y would have
a shape of this kind. By looking at this picture, you
can tell that it's most likely that either x will fall
here or x will fall there. So the g(x) is most likely
to be close to 0 or to be close to 2. So since y is most likely to be
close to 0 or close to most of the probability
of y is here. And there's a small probability of being in between. Notice that the y's that get a
lot of probability are those y's associated with flats
regions off your g function. When the g function is flat,
that gives you big densities for Y. So the density of Y is inversely
proportional to the slope of the function. And that's what you
get from here. The density of Y is-- send that term to the other
side-- is inversely proportional to the slope of
the function that you're dealing with. OK, so this formula works nicely
for the case where the function is one-to-one. So we can have a unique
association between x's and y's and through an inverse
function, from y's to x's. It works for the monotonically
increasing case. It also works for the
monotonically decreasing case. In the monotonically decreasing
case, the only change that you need to do is to
take the absolute value of the slope, instead of
the slope itself. OK, now, here's another example
or a special case. Let's talk about the most
interesting case that involves a function of two random
variables. And this is the case where we
have two independent, random variables, and we want to
find the distribution of the sum of the two. We're really interested in
the continuous case. But as a warm-up, it's useful
to look at the discrete case first of discrete random
variables. Let's say we want to find the
probability that the sum of X and Y is equal to a
particular number. And to illustrate this,
let's take that number to be equal to 3. What's the probability that
the sum of the two random variables is equal to 3? To find the probability that
the sum is equal to 3, you consider all possible ways that
you can get the sum of 3. And the different ways are the
points in this picture. And they correspond to a line
that goes this way. So the probability that the
sum is equal to a certain number is the probability
that -- is the sum of the
probabilities of all of those points. What is a typical point
in this picture? In a typical point, the
random variable X takes a certain value. And Y takes the value that's
needed so that the sum is equal to W. Any combination of
an x with a w minus x, any such combination gives
you a sum of w. So the probability that the sum
is w is the sum over all possible x's. That's over all these points of
the probability that we get a certain x. Let's say x equals 2 times the
corresponding probability that random variable Y takes
the value 1. And why am I multiplying
probabilities here? That's where we use the
assumption that the two random variables are independent. So the probability that X takes
a certain value and Y takes the complementary value,
that probability is the product of two probabilities
because of independence. And when we write that into our
usual PMF notation, it's a formula of this kind. So this formula is called
the convolution formula. It's an operation that takes
one PMF and another PMF-- p we're given the PMF's
of X and Y -- and produces a new PMF. So think of this formula as
giving you a transformation. You take two PMF's, you do
something with them, and you obtain a new PMF. This procedure, what this
formula does is -- nicely illustrated sort
of by mechanically. So let me show you a picture
here and illustrate how the mechanics go, in general. So you don't have these slides,
but let's just reason through it. So suppose that you are
given the PMF of X, and it has this shape. You're given the PMF of
Y. It has this shape. And somehow we are going
to do this calculation. Now, we need to do this
calculation for every value of W, in order to get the PMF of
W. Let's start by doing the calculation just for one case. Suppose the W is equal to 0, in
which case we need to find the sum of Px(x) and Py(-x). How do you do this calculation
graphically? It involves the PMF of X. But it
involves the PMF of Y, with the argument reversed. So how do we plot this? Well, in order to reverse the
argument, what you need is to take this PMF and flip it. So that's where it's handy
to have a pair of scissors with you. So you cut this down. And so now you take the PMF
of the random variable Y and just flip it. So what you see here is this
function where the argument is being reversed. And then what do we do? We cross-multiply
the two plots. Any entry here gets multiplied
with the corresponding entry there. And we consider all those
products and add them up. In this particular case, the
flipped PMF doesn't have any overlap with the PMF of X. So
we're going to get an answer that's equal to 0. So for w's equal to 0, the Pw is
going to be equal to 0, in this particular plot. Now if we have a different
value of w -- oops. If we have a different value
of the argument w, then we have here the PMF of Y that's
flipped and shifted by an amount of w. So the correct picture of what
you do is to take this and displace it by a certain
amount of w. So here, how much
did I shift it? I shifted it until one
falls just below 4. So I have shifted by a
total amount of 5. So 0 falls under 5, whereas
0 initially was under 0. So I'm shifting it by 5 units. And I'm now going to
cross-multiply and add. Does this give us
the correct-- does it do the correct thing? Yes, because a typical term will
be the probability that this random variable is 3 times
the probability that this random variable is 2. That's a particular way that
you can get a sum of 5. If you see here, the way that
things are aligned, it gives you all the different ways that
you can get the sum of 5. You can get the sum of 5 by
having 1 + 4, or 2 + 3, or 3 + 2, or 4 + 1. You need to add the
probabilities of all those combinations. So you take this times that. That's one product term. Then this times 0,
this times that. And so 1-- you cross-- you find all the products of the
corresponding terms, and you add them together. So it's a kind of handy
mechanical procedure for doing this calculation, especially
when the PMF's are given to you in terms of a picture. So the summary of these
mechanics are just what we did, is that you put the PMF's
on top of each other. You take the PMF of
Y. You flip it. And for any particular w that
you're interested in, you take this flipped PMF and shift
it by an amount of w. Given this particular shift for
a particular value of w, you cross-multiply terms and
then accumulate them or add them together. What would you expect to happen
in the continuous case? Well, the story is familiar. In the continuous case, pretty
much, almost always things work out the same way,
except that we replace PMF's by PDF's. And we replace sums
by integrals. So there shouldn't be any
surprise here that you get a formula of this kind. The density of W can be obtained
from the density of X and the density of Y by
calculating this integral. Essentially, what this integral
does is it fits a particular w of interest. We're interested in the
probability that the random variable, capital W, takes a
value equal to little w or values close to it. So this corresponds to the
event, which is this particular line on the
two-dimensional space. So we need to find
the sort of odd probabilities along that line. But since the setting is
continuous, we will not add probabilities. We're going to integrate. And for any typical point in
this picture, the probability of obtaining an outcome in this
neighborhood is the-- has something to do with the
density of that particular x and the density of the
particular y that would compliment x, in order
to form a sum of w. So this integral that we have
here is really an integral over this particular line. OK, so I'm going to
skip the formal derivation of this result. There's a couple of derivations
in the text. And the one which is outlined
here is yet a third derivation. But the easiest way to make
sense of this formula is to consider what happens in
the discrete case. So for the rest of the lecture
we're going to consider a few extra, more miscellaneous
topics, a few remarks, and a few more definitions. So let's change-- flip a page and consider
the next mini topic. There's not going to be anything
deep here, but just something that's worth
being familiar with. If you have two independent,
normal random variables with certain parameters, the question
is, what does the joined PDF look like? So if they're independent, by
definition the joint PDF is the product of the
individual PDF's. And the PDF's each one
of them involves an exponential of something. The product of two exponentials
is the exponential of the sum. So you just add the exponents. So this is the formula
for the joint PDF. Now, you look at that formula
and you ask, what does it look like? OK, you can understand it, a
function of two variables by thinking about the contours
of this function. Look at the points at
which the function takes a constant value. Where is it? When is it constant? What's the shape of
the set of points where this is a constant? So consider all x's and y's for
which this expression here is a constant, that this
expression here is a constant. What kind of shape is this? This is an ellipse. And it's an ellipse that's
centered at-- it's centered at mu x, mu y. These are the means of the
two random variables. If those sigmas were equal,
that ellipse would be actually a circle. And you would get contours
of this kind. But if, on the other hand, the
sigmas are different, you're going to get an ellipse that
has contours of this kind. So if my contours are
of this kind, that corresponds to what? Sigma x being bigger than
sigma y or vice versa. OK, contours of this kind
basically tell you that X is more likely to be spread out
than Y. So the range of possible x's is bigger. And X out here is as likely
as a Y up there. So big X's have roughly the same
probability as certain smaller y's. So in a picture of this kind,
the variance of X is going to be bigger than the
variance of Y. So depending on how these
variances compare with each other, that's going
to determine the shape of the ellipse. If the variance of Y we're
bigger, then your ellipse would be the other way. It would be elongated in
the other dimension. Just visualize it
a little more. Let me throw at you a
particular picture. This is one-- this is a picture of
one special case. Here, I think, the variances
are equal. That's the kind of shape
that you get. It looks like a two-dimensional
bell. So remember, for a normal random
variables, for a single random variable you get a
PDF that's bell shaped. That's just a bell-shaped
curve. In the two-dimensional case, we
get the joint PDF, which is bell shaped again. And now it looks more like a
real bell, the way it would be laid out in ordinary space. And if you look at the contours
of this function, the places where the function is
equal, the typcial contour would have this shape here. And it would be an ellipse. And in this case, actually, it
will be more like a circle. So these would be the different
contours for different-- so the contours are
places where the joint PDF is a constant. When you change the value of
that constant, you get the different contours. And the PDF is, of course,
centered around the mean of the two random variables. So in this particular case,
since the bell is centered around the (0, 0) vector, this
is a plot of a bivariate normal with 0 means. OK, there's-- bivariate normals are also
interesting when your bell is oriented differently in space. We talked about ellipses that
are this way, ellipses that are this way. You could imagine also bells
that you take them, you squash them somehow, so that they
become narrow in one dimension and then maybe rotate them. So if you had-- we're not going to go into this
subject, but if you had a joint pdf whose contours were
like this, what would that correspond to? Would your x's and y's
be independent? No. This would indicate that there's
a relation between the x's and the y's. That is, when you have bigger
x's, you would expect to also get bigger y's. So it would be a case of
dependent normals. And we're coming back to
this point in a second. Before we get to that point in
a second that has to do with the dependencies between the
random variables, let's just do another digression. If we have our two normals that
are independent, as we discussed here, we can go and
apply the formula, the convolution formula that we
were just discussing. Suppose you want to find the
distribution of the sum of these two independent normals. How do you do this? There is a closed-form formula
for the density of the sum, which is this one. We do have formulas for the
density of X and the density of Y, because both of them are
normal, random variables. So you need to calculate this
particular integral here. It's an integral with
respect to x. And you have to calculate
this integral for any given value of w. So this is an exercise
in integration, which is not very difficult. And it turns out that after you
do everything, you end up with an answer that
has this form. And you look at that,
and you suddenly recognize, oh, this is normal. And conclusion from this
exercise, once it's done, is that the sum of two independent
normal random variables is also normal. Now, the mean of W is, of
course, going to be equal to the sum of the means of X and
Y. In this case, in this formula I took the
means to be 0. So the mean of W is also
going to be 0. In the more general case, the
mean of W is going to be just the sum of the two means. The variance of W is always the
sum of the variances of X and Y, since we have independent
random variables. So there's no surprise here. The main surprise in this
calculation is this fact here, that the sum of independent
normal random variables is normal. I had mentioned this fact
in a previous lecture. Here what we're doing is to
basically outline the argument that justifies this
particular fact. It's an exercise in integration,
where you realize that when you convolve two
normal curves, you also get back a normal one once more. So now, let's return to the
comment I was making here, that if you have a contour plot
that has, let's say, a shape of this kind, this
indicates some kind of dependence between your
two random variables. So instead of a contour plot,
let me throw in here a scattered diagram. What does this scattered
diagram correspond to? Suppose you have a discrete
distribution, and each one of the points in this diagram
has positive probability. When you look at this diagram,
what would you say? I would say that when
y is big then x also tends to be larger. So bigger x's are sort of
associated with bigger y's in some average, statistical
sense. Whereas, if you have a picture
of this kind, it tells you in association that the positive
y's tend to be associated with negative x's most of the time. Negative y's tend to be
associated mostly with positive x's. So here there's a relation
that when one variable is large, the other one is also
expected to be large. Here there's a relation
of the opposite kind. How can we capture
this relation between two random variables? The way we capture it is by
defining this concept called the covariance, that looks at
the relation of was X bigger than usual? That's the question, whether
this is positive. And how does this relate to the
answer-- to the question, was Y bigger than usual? We're asking-- by calculating
this quantity, we're sort of asking the question, is there a
systematic relation between having a big X with
having a big Y? OK , to understand more
precisely what this does, let's suppose that the random
variable has 0 means, So that we get rid of this-- get rid of some clutter. So the covariance is defined
just as this product. What does this do? If positive x's tends to go
together with positive y's, and negative x's tend to go
together with negative y's, this product will always
be positive. And the covariance will
end up being positive. In particular, if you sit down
with a scattered diagram and you do the calculations,
you'll find that the covariance of X and Y in this
diagram would be positive, because here, most of the time,
X times Y is positive. There's going to be a few
negative terms, but there are fewer than the positive ones. So this is a case of a
positive covariance. It indicates a positive relation
between the two random variables. When one is big, the other
also tends to be big. This is the opposite
situation. Here, when one variable-- here, most of the action happens
in this quadrant and that quadrant, which means that
X times Y, most of the time, is negative. You get a few positive
contributions, but there are few. When you add things up, the
negative terms dominate. And in this case we
have covariance of X and Y being negative. So a positive covariance
indicates a sort of systematic relation, that there's a
positive association between the two random variables. When one is large, the other
also tends to be large. Negative covariance is
sort of the opposite. When one tends to be
large, the other variable tends to be small. OK, so what else is there to
say about the covariance? One observation to make
is the following. What's the covariance
of X with X itself? If you plug in X here, you see
that what we have is expected value of X minus expected
of X squared. And that's just the
definition of the variance of a random variable. So that's one fact
to keep in mind. We had a shortcut formula for
calculating variances. There's a similar shortcut
formula for calculating covariances. In particular, we can calculate
covariances in this particular way. That's just the convenient way
of doing it whenever you need to calculate it. And finally, covariances are
very useful when you want to calculate the variance of a
sum of random variables. We know that if two random
variables are independent, the variance of the sum is the
sum of the variances. When the random variables are
dependent, this is no longer true, and we need to supplement
the formula a little bit. And there's a typo on
the slides that you have in your hands. That term of 2 shouldn't
be there. And let's see where that
formula comes from. Let's suppose that our
random variables are independent of -- not independent -- our random variables
have 0 means. And we want to calculate
the variance. So the variance is going
to be expected value of (X1 plus Xn) squared. What you do is you expand
the square. And you get the expected value
of the sum of the Xi squared. And then you get all
the cross terms. OK. And so now, here, let's
assume for simplicity that we have 0 means. The expected value of this is
the sum of the expected values of the X squared terms. And that gives us
the variance. And then we have all the
possible cross terms. And each one of the possible
cross terms is the expected value of Xi times Xj. This is just the covariance. So if you can calculate all
the variances and the covariances, then you're able to
calculate also the variance of a sum of random variables. Now, if two random variables are
independent, then you look at this expression. Because of independence,
expected value of the product is going to be the product
of the expected values. And the expected value
of just this term is always equal to 0. You're expected deviation
from the mean is just 0. So the covariance will
turn out to be 0. So independent random
variables lead to 0 covariances, although the
opposite fact is not necessarily true. So covariances give you some
indication of the relation between two random variables. Something that's not so
convenient conceptually about covariances is that it
has the wrong units. That's the same comment
that we had made regarding variances. And with variances we got out
of that issue by considering the standard deviation, which
has the correct units. So with the same reasoning, we
want to have a concept that captures the relation between
two random variables and, in some sense, that doesn't have
to do with the units that we're dealing. We want to have a dimensionless
quantity. That tells us how strongly two
random variables are related to each other. So instead of considering the
covariance of just X with Y, we take our random variables
and standardize them by dividing them by their
individual standard deviations and take the expectation
of this. So what we end up doing is the
covariance of X and Y, which has units that are the units of
X times the units of Y. But divide with a standard
deviation, so that we get a quantity that doesn't
have units. This quantity, we call it the
correlation coefficient. And it's a very useful quantity,
a very useful measure of the strength
of association between two random variables. It's very informative, because
it falls always between -1 and +1. This is an algebraic exercise
that you're going to see in recitation. And the way that you interpret
it is as follows. If the two random variables
are independent, the covariance is going to be 0. The correlation coefficient
is going to be 0. So 0 correlation coefficient
basically indicates a lack of a systematic relation between
the two random variables. On the other hand, when rho is
large, either close to 1 or close to -1, this is an
indication of a strong association between the
two random variables. And the extreme case is when
rho takes an extreme value. When rho has a magnitude
equal to 1, it's as big as it can be. In that case, the two
random variables are very strongly related. How strongly? Well, if you know one random
variable, if you know the value of y, you can recover the
value of x and conversely. So the case of a complete
correlation is the case where one random variable is a linear
function of the other random variable. In terms of a scatter plot, this
would mean that there's a certain line and that the only
possible (x,y) pairs that can happen would lie on that line. So if all the possible (x,y)
pairs lie on this line, then you have this relation, and the
correlation coefficient is equal to 1. A case where the correlation
coefficient is close to 1 would be a scatter plot like
this, where the x's and y's are quite strongly aligned with
each other, maybe not exactly, but fairly strongly. All right, so you're going to
hear a little more about correlation coefficients
and covariances in recitation tomorrow.
