The following content is
provided under a Creative Commons license. Your support will help MIT
OpenCourseWare continue to offer high quality educational
resources for free. To make a donation or view
additional materials from hundreds of MIT courses, visit
MIT OpenCourseWare at ocw.mit.edu. JOHN TSITSIKLIS: We're going
to start today a new unit. so we will be talking about
limit theorems. So just to introduce the topic,
let's think of the following situation. There's a population
of penguins down at the South Pole. And if you were to pick a
penguin at random and measure their height, the expected value
of their height would be the average of the heights of
the different penguins in the population. So suppose when you
pick one, every penguin is equally likely. Then the expected value is just
the average of all the penguins out there. So your boss asks you to
find out what that the expected value is. One way would be to
go and measure each and every penguin. That might be a little
time consuming. So alternatively, what you can
do is to go and pick penguins at random, pick a few of them,
let's say a number n of them. So you measure the height
of each one. And then you calculate the
average of the heights of those penguins that you
have collected. So this is your estimate
of the expected value. Now, we called this the sample
mean, which is the mean value, but within the sample that
you have collected. This is something that's sort
of feels the same as the expected value, which
is again, the mean. But the expected value's a
different kind of mean. The expected value is the mean
over the entire population, whereas the sample mean is the
average over the smaller sample that you have measured. The expected value
is a number. The sample mean is a
random variable. It's a random variable because
the sample you have collected is random. Now, we think that this is a
reasonable way of estimating the expectation. So in the limit as n goes to
infinity, it's plausible that the sample mean, the estimate
that we are constructing, should somehow get close
to the expected value. What does this mean? What does it mean
to get close? In what sense? And is this statement true? This is the kind of statement
that we deal with when dealing with limit theorems. That's the subject of limit
theorems, when what happens if you're dealing with lots and
lots of random variables, and perhaps take averages
and so on. So why do we bother
about this? Well, if you're in the sampling
business, it would be reassuring to know that this
particular way of estimating the expected value
actually gets you close to the true answer. There's also a higher level
reason, which is a little more abstract and mathematical. So probability problems are easy
to deal with if you're having in your hands one or
two random variables. You can write down their mass
functions, joints density functions, and so on. You can calculate on paper
or on a computer, you can get the answers. Probability problems become
computationally intractable if you're dealing, let's say, with
100 random variables and you're trying to get the exact
answers for anything. So in principle, the same
formulas that we have, they still apply. But they involve summations
over large ranges of combinations of indices. And that makes life extremely
difficult. But when you push the envelope
and you go to a situation where you're dealing with a
very, very large number of variables, then you can
start taking limits. And when you take limits,
wonderful things happen. Many formulas start simplifying,
and you can actually get useful answers by
considering those limits. And that's sort of the big
reason why looking at limit theorems is a useful
thing to do. So what we're going to do today,
first we're going to start with a useful, simple tool
that allows us to relates probabilities with
expected values. The Markov inequality is the
first inequality we're going to write down. And then using that, we're going
to get the Chebyshev's inequality, a related
inequality. Then we need to define what do
we mean by convergence when we talk about random variables. It's a notion that's a
generalization of the notion of the usual convergence
of limits of a sequence of numbers. And once we have our notion of
convergence, we're going to see that, indeed, the sample
mean converges to the true mean, converges to the expected
value of the X's. And this statement is called the
weak law of large numbers. The reason it's called the weak
law is because there's also a strong law, which is
a statement with the same flavor, but with a somewhat
different mathematical content. But it's a little more abstract,
and we will not be getting into this. So the weak law is all that
you're going to get. All right. So now we start our
digression. And our first tool will be the
so-called Markov inequality. So let's take a random variable
that's always non-negative. No matter what, it gets
no negative values. To keep things simple,
let's assume it's a discrete random variable. So the expected value is the sum
over all possible values that a random variable
can take. The values of the random
variables that can take weighted according to their
corresponding probabilities. Now, this is a sum
over all x's. But x takes non-negative
values. And the PMF is also
non-negative. So if I take a sum over fewer
things, I'm going to get a smaller value. So the sum when I add over
everything is less than or equal to the sum that I will get
if I only add those terms that are bigger than
a certain constant. Now, if I'm adding over x's that
are bigger than a, the x that shows up up there
will always be larger than or equal to a. So we get this inequality. And now, a is a constant. I can pull it outside
the summation. And then I'm left with the
probabilities of all the x's that are bigger than a. And that's just the
probability of being bigger than a. OK, so that's the Markov
inequality. Basically tells us that the
expected value is larger than or equal to this number. It relates expected values
to probabilities. It tells us that if the expected
value is small, then the probability that x is big
is also going to be small. So it's translates a statement
about smallness of expected values to a statement about
smallness of probabilities. OK. What we actually need is a
somewhat different version of this same statement. And what we're going to do is to
apply this inequality to a non-negative random variable
of a special type. And you can think of applying
this same calculation to a random variable of this form, (X
minus mu)-squared, where mu is the expected value of X. Now, this is a non-negative
random variable. So, the expected value of this
random variable, which is the variance, by following the same
thinking as we had in that derivation up to there, is
bigger than the probability that this random variable
is bigger than some-- let me use a-squared
instead of an a times the value a-squared. So now of course, this
probability is the same as the probability that the absolute
value of X minus mu is bigger than a times a-squared. And this side is equal to the
variance of X. So this relates the variance of X to the
probability that our random variable is far away
from its mean. If the variance is small, then
it means that the probability of being far away from the
mean is also small. So I derived this by applying
the Markov inequality to this particular non-negative
random variable. Or just to reinforce, perhaps,
the message, and increase your confidence in this inequality,
let's just look at the derivation once more, where I'm
going, here, to start from first principles, but use the
same idea as the one that was used in the proof out here. Ok. So just for variety, now let's
think of X as being a continuous random variable. The derivation is the same
whether it's discrete or continuous. So by definition, the variance
is the integral, is this particular integral. Now, the integral is going to
become smaller if I integrate, instead of integrating over
the full range, I only integrate over x's that are
far away from the mean. So mu is the mean. Think of c as some big number. These are x's that are far
away from the mean to the left, from minus infinity
to mu minus c. And these are the x's that are
far away from the mean on the positive side. So by integrating over
fewer stuff, I'm getting a smaller integral. Now, for any x in this range,
this distance, x minus mu, is at least c. So that squared is at
least c squared. So this term over this
range of integration is at least c squared. So I can take it outside
the integral. And I'm left just with the
integral of the density. Same thing on the other side. And so what factors out is
this term c squared. And inside, we're left with the
probability of being to the left of mu minus c, and then
the probability of being to the right of mu plus c,
which is the same as the probability that the absolute
value of the distance from the mean is larger than
or equal to c. So that's the same inequality
that we proved there, except that here I'm using c. There I used a, but it's
exactly the same one. This inequality was maybe better
to understand if you take that term and send it
to the other side and write it this form. What does it tell us? It tells us that if c is a big
number, it tells us that the probability of being more than
c away from the mean is going to be a small number. When c is big, this is small. Now, this is intuitive. The variance is a measure
of the spread of the distribution, how wide it is. It tells us that if the
variance is small, the distribution is not very wide. And mathematically, this
translates to this statement that when the variance is small,
the probability of being far away is going
to be small. And the further away you're
looking, that is, if c is a bigger number, that probability also becomes small. Maybe an even more intuitive way
to think about the content of this inequality is to,
instead of c, use the number k, where k is positive
and sigma is the standard deviation. So let's just plug k sigma
in the place of c. So this becomes k
sigma squared. These sigma squared's cancel. We're left with 1
over k-square. Now, what is this? This is the event that you are
k standard deviations away from the mean. So for example, this statement
here tells you that if you look at the test scores from a
quiz, what fraction of the class are 3 standard deviations
away from the mean? It's possible, but it's not
going to be a lot of people. It's going to be at most, 1/9
of the class that can be 3 standard deviations or more
away from the mean. So the Chebyshev inequality
is a really useful one. It comes in handy whenever you
want to relate probabilities and expected values. So if you know that your
expected values or, in particular, that your variance
is small, this tells you something about tailed
probabilities. So this is the end of our
first digression. We have this inequality
in our hands. Our second digression is
talk about limits. We want to eventually talk
about limits of random variables, but as a warm up,
we're going to start with limits of sequences. So you're given a sequence
of numbers, a1, a2, a3, and so on. And we want to define the
notion that a sequence converges to a number. You sort of know what this
means, but let's just go through it some more. So here's a. We have our sequence of
values as n increases. What do we mean by the sequence
converging to a is that when you look at those
values, they get closer and closer to a. So this value here is your
typical a sub n. They get closer and closer to
a, and they stay closer. So let's try to make
that more precise. What it means is let's
fix a sense of what it means to be close. Let me look at an interval that
goes from a - epsilon to a + epsilon. Then if my sequence converges
to a, this means that as n increases, eventually the values
of the sequence that I get stay inside this band. Since they converge to a, this
means that eventually they will be smaller than
a + epsilon and bigger than a - epsilon. So convergence means that
given a band of positive length around the number a,
the values of the sequence that you get eventually
get inside and stay inside that band. So that's sort of the picture
definition of what convergence means. So now let's translate this into
a mathematical statement. Given a band of positive length,
no matter how wide that band is or how narrow it
is, so for every epsilon positive, eventually the
sequence gets inside the band. What does eventually mean? There exists a time,
so that after that time something happens. And the something that happens
is that after that time, we are inside that band. So this is a formal mathematical
definition, which actually translates what I was
telling in the wordy way before, and showing in
terms of the picture. Given a certain band, even if
it's narrow, eventually, after a certain time n0, the values
of the sequence are going to stay inside this band. Now, if I were to take epsilon
to be very small, this thing would still be true that
eventually I'm going to get inside of the band, except that
I may have to wait longer for the values to
get inside here. All right, that's what it means
for a deterministic sequence to converge
to something. Now, how about random
variables. What does it mean for a sequence
of random variables to converge to a number? We're just going to twist
a little bit of the word definition. For numbers, we said that
eventually the numbers get inside that band. But if instead of numbers we
have random variables with a certain distribution, so here
instead of a_n we're dealing with a random variable that has
a distribution, let's say, of this kind, what we want is
that this distribution gets inside this band, so it gets
concentrated inside here. What does it means that
the distribution gets inside this band? I mean a random variable
has a distribution. It may have some tails, so
maybe not the entire distribution gets concentrated
inside of the band. But we want that more and more
of this distribution is concentrated in this band. So that -- in a sense that -- the probability of falling
outside the band converges to 0 -- becomes smaller
and smaller. So in words, we're going to say
that the sequence random variables or a sequence of
probability distributions, that would be the same,
converges to a particular number a if the following
is true. If I consider a small band
around a, then the probability that my random variable falls
outside this band, which is the area under this curve,
this probability becomes smaller and smaller as
n goes to infinity. The probability of being
outside this band converges to 0. So that's the intuitive idea. So in the beginning, maybe our
distribution is sitting everywhere. As n increases, the distribution
starts to get concentrating inside the band. When a is even bigger, our
distribution is even more inside that band, so that these
outside probabilities become smaller and smaller. So the corresponding
mathematical statement is the following. I fix a band around
a, a +/- epsilon. Given that band, the probability
of falling outside this band, this probability
converges to 0. Or another way to say it is
that the limit of this probability is equal to 0. If you were to translate this
into a complete mathematical statement, you would have
to write down the following messy thing. For every epsilon positive -- that's this statement -- the limit is 0. What does it mean that the
limit of something is 0? We flip back to the
previous slide. Why? Because a probability
is a number. So here we're talking about
a sequence of numbers convergent to 0. What does it mean for a
sequence of numbers to converge to 0? It means that for any epsilon
prime positive, there exists some n0 such that for every
n bigger than n0 the following is true -- that this probability
is less than or equal to epsilon prime. So the mathematical statement
is a little hard to parse. For every size of that band,
and then you take the definition of what it means for
the limit of a sequence of numbers to converge to 0. But it's a lot easier to
describe this in words and, basically, think in terms
of this picture. That as n increases, the
probability of falling outside those bands just become
smaller and smaller. So the statement is that our
distribution gets concentrated in arbitrarily narrow little
bands around that particular number a. OK. So let's look at an example. Suppose a random variable Yn has
a discrete distribution of this particular type. Does it converge to something? Well, the probability
distribution of this random variable gets concentrated
at 0 -- there's more and more
probability of being at 0. If I fix a band around 0 -- so if I take the band from minus
epsilon to epsilon and look at that band-- the probability of falling
outside this band is 1/n. As n goes to infinity, that
probability goes to 0. So in this case, we do
have convergence. And Yn converges in probability
to the number 0. So this just captures the
facts obvious from this picture, that more and more of
our probability distribution gets concentrated around 0,
as n goes to infinity. Now, an interesting thing to
notice is the following, that even though Yn converges to 0,
if you were to write down the expected value for Yn,
what would it be? It's going to be n times the
probability of this value, which is 1/n. So the expected value
turns out to be 1. And if you were to look at the
expected value of Yn-squared, this would be 0. times this probability, and
then n-squared times this probability, which
is equal to n. And this actually goes
to infinity. So we have this, perhaps,
strange situation where a random variable goes to 0, but
the expected value of this random variable does
not go to 0. And the second moment of that
random variable actually goes to infinity. So this tells us that
convergence in probability tells you something,
but it doesn't tell you the whole story. Convergence to 0 of a random
variable doesn't imply anything about convergence
of expected values or of variances and so on. So the reason is that
convergence in probability tells you that this
tail probability here is very small. But it doesn't tell you how
far does this tail go. As in this example, the tail
probability is small, but that tail acts far away, so it
gives a disproportionate contribution to the expected
value or the expected value squared. OK. So now we've got everything that
we need to go back to the sample mean and study
its properties. So the sad thing is
that we have a sequence of random variables. They're independent. They have the same
distribution. And we assume that they
have a finite mean and a finite variance. We're looking at the
sample mean. Now in principle, you can
calculate the probability distribution of the sample mean,
because we know how to find the distributions
of sums of independent random variables. You use the convolution
formula over and over. But this is pretty
complicated, so let's not look at that. Let's just look at expected
values, variances, and the probabilities that the sample
mean is far away from the true mean. So what is the expected value
of this random variable? The expected value of a sum of
random variables is the sum of the expected values. And then we have this factor
of n in the denominator. Each one of these expected
values is mu, so we get mu. So the sample mean, the average
value of this Mn in expectation is the same as
the true mean inside our population. Now here, this is a fine
conceptual point, there's two kinds of averages involved
when you write down this expression. We understand that
expectations are some kind of average. The sample mean is also an
average over the values that we have observed. But it's two different
kinds of averages. The sample mean is the average
of the heights of the penguins that we collected over
a single expedition. The expected value is to be
thought of as follows, my probabilistic experiment
is one expedition to the South Pole. Expected value here means
thinking on the average over a huge number of expeditions. So my expedition is a random
experiment, I collect random samples, and they record Mn. The average result of an
expedition is what we would get if we were to carry out
a zillion expeditions and average the averages that we
get at each particular expedition. So this Mn is the average during
a single expedition. This expectation is the average
over an imagined infinite sequence
of expeditions. And of course, the other thing
to always keep in mind is that expectations give you numbers,
whereas the sample mean is actually a random variable. All right. So this random variable,
how random is it? How big is its variance? So the variance of a sum of
random variables is the sum of the variances. But since we're dividing by n,
when you calculate variances this brings in a factor
of n-squared. So the variance is sigma-squared
over n. And in particular, the variance
of the sample mean becomes smaller and smaller. It means that when you estimate
that average height of penguins, if you take a
large sample, then your estimate is not going
to be too random. The randomness in your estimates
become small if you have a large sample size. Having a large sample size kind
of removes the randomness from your experiment. Now let's apply the Chebyshev
inequality to say something about tail probabilities
for the sample mean. The probability that you are
more than epsilon away from the true mean is less than or
equal to the variance of this quantity divided by this
number squared. So that's just the translation
of the Chebyshev inequality to the particular context
we've got here. We found the variance. It's sigma-squared over n. So we end up with
this expression. So what does this
expression do? For any given epsilon, if
I fix epsilon, then this probability, which is less
than sigma-squared over n epsilon-squared, converges to
0 as n goes to infinity. And this is just the definition
of convergence in probability. If this happens, that the
probability of being more than epsilon away from the mean, that
probability goes to 0, and this is true no matter how
I choose my epsilon, then by definition we have convergence
in probability. So we have proved that the
sample mean converges in probability to the true mean. And this is what the weak law
of large numbers tells us. So in some vague sense, it
tells us that the sample means, when you take the
average of many, many measurements in your sample,
then the sample mean is a good estimate of the true mean in the
sense that it approaches the true mean as your sample
size increases. It approaches the true mean,
but of course in a very specific sense, in probability,
according to this notion of convergence
that we have used. So since we're talking about
sampling, let's go over an example, which is the typical
situation faced by someone who's constructing a poll. So you're interested in some
property of the population. So what fraction of
the population prefers Coke to Pepsi? So there's a number f, which
is that fraction of the population. And so this is an
exact number. So out of a population of 100
million, 20 million prefer Coke, then f would be 0.2. We want to find out what
that fraction is. We cannot ask everyone. What we're going to do is to
take a random sample of people and ask them for their
preferences. So the ith person either says
yes for Coke or no. And we record that by putting
a 1 each time that we get a yes answer. And then we form the average
of these x's. What is this average? It's the number of 1's that
we got divided by n. So this is a fraction, but
calculated only on the basis of the sample that we have. So you can think of this as
being an estimate, f_hat, based on the sample
that we have. Now, even though we used the
lower case letter here, this f_hat is, of course,
a random variable. f is a number. This is the true fraction in
the overall population. f_hat is the estimate
that we get by using our particular sample. Ok. So your boss told you, I need to
know what f is, but go and do some sampling. What are you going to respond? Unless I ask everyone in the
whole population, there's no way for me to know f exactly. Right? There's no way. OK, so the boss tells you, well
OK, then that'll me f within an accuracy. I want an answer from you,
that's your answer, which is close to the correct answer
within 1 % point. So if the true f is 0.4, your
answer should be somewhere between 0.39 and 0.41. I want a really accurate
answer. What are you going to say? Well, there's no guarantee
that my answer will be within 1 %. Maybe I'm unlucky and I just
happen to sample the wrong set of people and my answer
comes out to be wrong. So I cannot give you a hard
guarantee that this inequality will be satisfied. But perhaps, I can give you a
guarantee that this inequality will be satisfied, this accuracy
requirement will be satisfied, with high
confidence. That is, there's going to be
a smaller probability that things go wrong, that
I'm unlikely and I use a bad sample. But leaving aside that smaller
probability of being unlucky, my answer will be accurate
within the accuracy requirement that you have. So these two numbers are the
usual specs that one has when designing polls. So this number is the accuracy
that we want. It's the desired accuracy. And this number has to do with
the confidence that we want. So 1 minus that number, we could
call it the confidence that we want out
of our sample. So this is really 1
minus confidence. So now your job is to figure out
how large an n, how large a sample should you be using, in
order to satisfy the specs that your boss gave you. All you know at this stage is
the Chebyshev inequality. So you just try to use it. The probability of getting an
answer that's more than 0.01 away from the true answer is, by
Chebyshev's inequality, the variance of this random variable
divided by this number squared. The variance, as we argued
a little earlier, is the variance of the x's
divided by n. So we get this expression. So we would like this
number to be less than or equal to 0.05. OK, here we hit a little
bit off a difficulty. The variance, (sigma_x)-squared,
what is it? (Sigma_x)-squared is, if you
remember the variance of a Bernoulli random variable,
is this quantity. But we don't know it. f is what we're trying to
estimate in the first place. So the variance is not known,
so I cannot plug in a number inside here. What I can do is to be
conservative and use an upper bound of the variance. How large can this number get? Well, you can plot
f times (1-f). It's a parabola. It has a root at 0 and at 1. So the maximum value is going to
be, by symmetry, at 1/2 and when f is 1/2, then this
variance becomes 1/4. So I don't know
(sigma_x)-squared, but I'm going to use the worst case
value for (sigma_x)-squared, which is 4. And this is now an inequality
that I know to be always true. I've got my specs, and my specs
tell me that I want this number to be less than 0.05. And given what I know, the best
thing I can do is to say, OK, I'm going to take
this number and make it less than 0.05. If I choose my n so that this
is less than 0.05, then I'm certain that this probability
is also less than 0.05. What does it take for this
inequality to be true? You can solve for n here, and
you find that to satisfy this inequality, n should be larger
than or equal to 50,000. So you can just let n
be equal to 50,000. So the Chebyshev inequality
tells us that if you take n equal to 50,000, then by the
Chebyshev inequality, we're guaranteed to satisfy the specs
that we were given. Ok. Now, 50,000 is a bit of
a large sample size. Right? If you read anything in the
newspapers where they say so much of the voters think this
and that, this was determined on the basis of a sample of
1,200 likely voters or so. So the numbers that you will
typically see in these news items about polling, they
usually involve sample sizes about the 1,000 or so. You will never see a sample
size of 50,000. That's too much. So where can we cut
some corners? Well, we can cut corners
basically in three places. This requirement is a
little too tight. Newspaper stories will usually
tell you, we have an accuracy of +/- 3 % points, instead
of 1 % point. And because this number comes up
as a square, by making it 3 % points instead of 1, saves
you a factor of 10. Then, the five percent
confidence, I guess that's usually OK. If we use that factor of 10,
then we make our sample that we gain from here, then we get
a sample size of 10,000. And that's, again,
a little too big. So where can we fix things? Well, it turns out that this
inequality that we're using here, Chebyshev's inequality,
is just an inequality. It's not that tight. It's not very accurate. Maybe there's a better way of
calculating or estimating this quantity, which is smaller
than this. And using a more accurate
inequality or a more accurate bound, then we can convince
ourselves that we can settle with a smaller sample size. This more accurate kind of
inequality comes out of a difference limit theorem,
which is the next limit theorem we're going
to consider. We're going to start the
discussion today, but we're going to continue with
it next week. Before I tell you exactly what
that other limit theorem says, let me give you the
big picture of what's involved here. We're dealing with sums of
i.i.d random variables. Each X has a distribution
of its own. So suppose that X has a
distribution which is something like this. This is the density of X. If I
add lots of X's together, what kind of distribution
do I expect? The mean is going to be
n times the mean of an individual X. So if this is mu,
I'm going to get a mean of n times mu. But my variance will
also increase. When I add the random
variables, I'm adding the variances. So since the variance increases,
we're going to get a distribution that's
pretty wide. So this is the density of X1
plus all the way to Xn. So as n increases, my
distribution shifts, because the mean is positive. So I keep adding things. And also, my distribution
becomes wider and wider. The variance increases. Well, we started a different
scaling. We started a scaled version of
this quantity when we looked at the weak law of
large numbers. In the weak law of large
numbers, we take this random variable and divide it by n. And what the weak law tells us
is that we're going to get a distribution that's very highly
concentrated around the true mean, which is mu. So this here would be the
density of X1 plus Xn divided by n. Because I've divided by n, the
mean has become the original mean, which is mu. But the weak law of large
numbers tells us that the distribution of this random
variable is very concentrated around the mean. So we get a distribution
that's very narrow in this kind. In the limit, this distribution
becomes one that's just concentrated
on top of mu. So it's sort of a degenerate
distribution. So these are two extremes, no
scaling for the sum, a scaling where we divide by n. In this extreme, we get the
trivial case of a distribution that flattens out completely. In this scaling, we get a
distribution that gets concentrated around
a single point. Again, we look at some
intermediate scaling that makes things more interesting. Things do become interesting
if we scale by dividing the sum by square root of n instead
of dividing by n. What effect does this have? When we scale by dividing by
square root of n, the variance of Sn over square root of n is
going to be the variance of Sn over sum divided by n. That's how variances behave. The variance of Sn is n
sigma-squared, divide by n, which is sigma squared, which
means that when we scale in this particular way,
as n changes, the variance doesn't change. So the width of our
distribution will be sort of constant. The distribution changes shape,
but it doesn't become narrower as was the case here. It doesn't become wider, kind
of keeps the same width. So perhaps in the limit, this
distribution is going to take an interesting shape. And that's indeed the case. So let's do what
we did before. So we're looking at the sum, and
we want to divide the sum by something that goes like
square root of n. So the variance of Sn
is n sigma squared. The variance of the sigma Sn
is the square root of that. It's this number. So effectively, we're scaling
by order of square root n. Now, I'm doing another
thing here. If my random variable has a
positive mean, then this quantity is going to
have a mean that's positive and growing. It's going to be shifting
to the right. Why is that? Sn has a mean that's
proportional to n. When I divide by square root n,
then it means that the mean scales like square root of n. So my distribution would
still keep shifting after I do this division. I want to keep my distribution
in place, so I subtract out the mean of Sn. So what we're doing here is
a standard technique or transformation where you take
a random variable and you so-called standardize it. I remove the mean of that random
variable and I divide by the standard deviation. This results in a random
variable that has 0 mean and unit variance. What Zn measures is the
following, Zn tells me how many standard deviations am
I away from the mean. Sn minus (n times expected value
of X) tells me how much is Sn away from the
mean value of Sn. And by dividing by the standard
deviation of Sn -- this tells me how many standard
deviations away from the mean am I. So we're going to look at this
random variable, which is just a transformation Zn. It's a linear transformation
of Sn. S And we're going to compare
this random variable to a standard normal random
variable. So a standard normal is the
random variable that you are familiar with, given by the
usual formula, and for which we have tables for it. This Zn has 0 mean and
unit variance. So in that respect, it has the
same statistics as the standard normal. The distribution of Zn
could be anything -- can be pretty messy. But there is this amazing
theorem called the central limit theorem that tells us that
the distribution of Zn approaches the distribution of
the standard normal in the following sense, that
probability is that you can calculate -- of this type -- that you can calculate
for Zn -- is the limit becomes the same as
the probabilities that you would get from the standard
normal tables for Z. It's a statement about
the cumulative distribution functions. This quantity, as a function
of c, is the cumulative distribution function of
the random variable Zn. This is the cumulative
distribution function of the standard normal. The central limit theorem tells
us that the cumulative distribution function of the
sum of a number of random variables, after they're
appropriately standardized, approaches the cumulative
distribution function over the standard normal distribution. In particular, this tells
us that we can calculate probabilities for Zn when n is
large by calculating instead probabilities for Z. And that's
going to be a good approximation. Probabilities for Z are easy to
calculate because they're well tabulated. So we get a very nice shortcut
for calculating probabilities for Zn. Now, it's not Zn that you're
interested in. What you're interested
in is Sn. And Sn -- inverting this relation
here -- Sn is square root n sigma
Zn plus n expected value of X. All right. Now, if you can calculate
probabilities for Zn, even approximately, then you can
certainly calculate probabilities for Sn, because
one is a linear function of the other. And we're going to do a little
bit of that next time. You're going to get, also, some
practice in recitation. At a more vague level, you could
describe the central limit theorem as saying the
following, when n is large, you can pretend that Zn is
a standard normal random variable and do the calculations
as if Zn was standard normal. Now, pretending that Zn is
normal is the same as pretending that Sn is normal,
because Sn is a linear function of Zn. And we know that linear
functions of normal random variables are normal. So the central limit theorem
essentially tells us that we can pretend that Sn is a normal
random variable and do the calculations just as if it
were a normal random variable. Mathematically speaking though,
the central limit theorem does not talk about
the distribution of Sn, because the distribution of Sn
becomes degenerate in the limit, just a very flat
and long thing. So strictly speaking
mathematically, it's a statement about cumulative
distributions of Zn's. Practically, the way you use it
is by just pretending that Sn is normal. Very good. Enjoy the Thanksgiving Holiday.
