The following content is
provided under a Creative Commons license. Your support will help
MIT OpenCourseWare continue to offer high quality
educational resources for free. To make a donation or
view additional materials from hundreds of MIT courses,
visit MIT OpenCourseWare at ocw.mit.edu. PROFESSOR: So today, we're going
to talk about the probability that a random variable
deviates by a certain amount from its expectation. Now, we've seen examples
where a random variable is very unlikely to deviate
much from its expectation. For example, if you
flip 100 mutually independent fair
coins, you're very likely to wind up
with close to 50 heads, very unlikely to
wind up with 25 or fewer heads, for example. We've also seen examples
of distributions where you are very likely to
be far from your expectation, for example, that problem
when we had the communications channel, and we were measuring
the latency of a packet crossing the channel. There, most of the
time, your latency would be 10 milliseconds. But the expected
latency was infinite. So you're very likely to deviate
a lot from your expectation in that case. Last time, we looked
at the variance. And we saw how that gave us
some feel for the likelihood of being far from
the expectation-- high variance meaning you're
more likely to deviate from the expectation. Today, we're going to
develop specific tools for bounding or limiting
the probability you deviate by a specified
amount from the expectation. And the first tool is
known as Markov's theorem. Markov's theorem says that if
the random variable is always non-negative, then it is
unlikely to greatly exceed its expectation. In particular, if R is a
non-negative random variable, then for all x bigger
than 0, the probability that R is at least x is at
most the expected value of R, the mean, divided by x. So in other words, if R is
never negative-- for example, say the expected
value is smaller. Then the probability R is
large will be a small number. Because I'll have a small
number over a big number. So it says that you are
unlikely to greatly exceed the expected value. So let's prove that. Now, from the theorem
of total expectation that you did in
recitation last week, we can compute the
expected value of R by looking at two cases-- the
case when R is at least x, and the case when
R is less than x. That's from the theorem
of total expectation. I look at two cases. R is bigger than x. Take the expected value
there times the probability of this case happening plus
the case when R is less than x. OK, now since R is non-negative,
this is at least 0. R can't ever be negative. So the expectation
can't be negative. A probability can't be negative. So this is at least 0. And this is
trivially at least x. Because I'm taking the
expected value of R in the case when
R is at least x. So R is always at
least x in this case. So its expected
value is at least x. So that means that the
expected value of R is at least x times
the probability R is greater than x, R
is greater or equal to x. And now I can get the theorem
by just dividing by x. I'm less than or equal
to the expected value of R divided by x. So it's a very easy
theorem to prove. But it's going to have
amazing consequences that we're going to build up
through a series of results today. Any questions about Markov's
theorem and the proof? All right, there's a simple
corollary, which is useful. Again, if R is a
non-negative random variable, then for all c bigger
than 0, the probability that R is at least c times its
expected value is at most 1 and c. So the probability you're
twice your expected value is at most 1/2. And the proof is very easy. We just set x to be equal
to c times the expected value of R in the theorem. So I just plug in x is c
times the expected value of R. And I get expected value of R
over c times the expected value of R, which is 1/c. So you just plug in that
value in Markov's theorem, and it comes out. All right, let's
do some examples. Let's let R be the
weight of a random person uniformly selected. And I don't know what the
distribution of weights is in the country. But suppose that the
expected value of R, which is the average
weight, is 100 pounds. So if I average over all people,
their weight is 100 pounds. And suppose I want to
know the probability that the random person
weighs at least 200 pounds. What can I say about
that probability? Do I know it exactly? I don't think so. Because I don't know what the
distribution of weights is. But I can still get an upper
bound on this probability. What bound can I get
on the probability that a random
person has a weight of 200 given the facts here? Yeah. AUDIENCE: [INAUDIBLE] PROFESSOR: Yes, well,
it's 100 over 200, right. It's at most the expected
value, which is 100, over the x, which is 200. And that's equal to 1/2. So the probability that a
random person weighs 200 pounds or more is at most 1/2. Or I could plug it in here. The expected value is 100. 200 is twice that. So c would be 2 here. So the probability of
being twice the expectation is at most 1/2. Now of course,
I'm using the fact that weight is never negative. That's obviously true. But it is implicitly
being used here. So what fraction
of the population now can weigh at
least 200 pounds? Slightly different question. Before I asked you, if
I take a random person, what's the probability they
weigh at least 200 pounds? Now I'm asking, what
fraction of the population can weigh at least 200
pounds if the average is 100? What is it? Yeah? AUDIENCE: At most 1/2. PROFESSOR: At most 1/2. In fact, it's the same answer. And why? Why can't everybody
weigh 200 pounds, so it would be
all the population weighs 200 pounds at least? AUDIENCE: [INAUDIBLE] PROFESSOR: Probability would
be 1, and that can't happen. And in fact,
intuitively, if everybody weighs at least 200
pounds, the average is going to be at
least 200 pounds. And we said the average was 100. And this is illustrating
this interesting thing that probability implies things
about averages and fractions. Because it's really the
same thing in disguise. The connection is, if I've
got a bunch of people, say, in the country, I can
convert a fraction that have some property
into a probability by just selecting
a random person. Yeah. AUDIENCE: [INAUDIBLE] PROFESSOR: No, the
variance could be very big. Because I might have a person
that weighs a million pounds, say. So you have to get into that. But it gets a little
bit more complicated. Yeah. AUDIENCE: [INAUDIBLE] PROFESSOR: No,
there's nothing being assumed about the distribution,
nothing at all, OK? So that's the beauty
of Markov's theorem. Well, I've assumed one thing. I assume that there
is no negative values. That's it. AUDIENCE: [INAUDIBLE] PROFESSOR: That's correct. They can distribute it any
way with positive values. But we have a fact here we've
used, that the average was 100. So that does limit
your distribution. In other words, you couldn't
have a distribution where everybody weighs 200 pounds. Because then the average
would be 200, not 100. But anything else where
they're all positive and they average 100, you know
that at most half can be 200. Because if you
pick a random one, the probability of getting
one that's 200 is at most 1/2, which follows from
Markov's theorem. And that's partly
why it's so powerful. You didn't know anything about
the distribution, really, except its expectation and
that it was non-negative. Any other questions about this? I'll give you some
more examples. All right, here's
another example. Is it possible on the final
exam for everybody in the class to do better than
the mean score? No, of course not. Because if they did, the
mean would be higher. Because the mean is the average. OK, let's do another example. Remember the Chinese
appetizer problem? You're at the restaurant,
big circular table. There's n people at the table. Everybody has one
appetizer in front of them. And then the joker
spins the thing in the middle of the table. So it goes around and around. And it stops in a
random uniform position. And we wanted to know, what's
the expected number of people to get the right appetizer back? What was the answer? Does anybody remember? One. So you expect one person to
get the right appetizer back. Well, say I want to know the
probability that all n people got the right appetizer back. What does Markov tell
you about the probability that all n people get
the right appetizer back? 1/n. The expected value is 1. And now you're asking
the probability that you get R is at least n. So x is n. So it's 1 in n. And what was the probability, or
what is the actual probability? In this case, you
know the distribution, that everybody gets the
right appetizer back, all n. 1 in n. So in the case of the
Chinese appetizer problem, Markov's bound is actually the
right answer, right on target, which gives you an example
where you can't improve it. By itself, if you just
know the expected value, there's no stronger
theorem that way. Because Chinese appetizer is
an example where the bound you get, 1/n, of n people
getting the right appetizer is in fact the true probability. OK, what about the
hat check problem? Remember that? So there's n men put the
hats in the coat closet. They get uniformly
randomly scrambled. So it's a random permutation
applied to the hats. Now each man gets a hat back. What's the expected number of
men to get the right hat back? One, same as the other one. Because you've got n men
each with a 1 in n chance, so it's 1. Markov says the probability that
n men get the right hat back is at most 1 in n, same as before. What's the actual
probability that all n men get the right hat back? AUDIENCE: [INAUDIBLE] PROFESSOR: 1 in n factorial. So in this case, Markov
is way off the mark. It says 1 in n. But in fact the real
bound is much smaller. So Markov is not always tight. It's always an upper bound. But it sometimes is
not the right answer. And to get the right
answer, often you need to know more
about the distribution. OK, what if R can be negative? Is it possible that Markov's
theorem holds there? Because I use the
assumption in the theorem. Can anybody give me an
example where it doesn't work if R can be negative? AUDIENCE: [INAUDIBLE] PROFESSOR: Yeah,
good, so for example, say probability R
equals 1,000 is 1/2, and the probability R
equals minus 1,000 is 1/2. Then the expected
value of R is 0. And say we asked the probability
that R is at least 1,000. Well, that's going to be 1/2. But that does not equal the
expected value of R/1,000, which would be 0. So Markov's theorem really does
need that R to be non-negative. In fact, let's see if we saw
where we used it in the proof. Anybody see where we use
that fact in the proof, that R can't be negative? What is it? AUDIENCE: [INAUDIBLE] PROFESSOR: Well, no,
because x is positive. We said x is positive. So it's not used there. But that's a good
one to look at. Yeah? AUDIENCE: [INAUDIBLE] is
greater than or equal to 0. PROFESSOR: Yeah, if
R can be negative, then this is not necessarily
a positive number. It could be a negative number. And then this
inequality doesn't hold. OK, good. All right, now it
turns out there is a variation of
Markov's theorem you can use when R is negative. Yeah. AUDIENCE: [INAUDIBLE]
but would it be OK just to shift everything up? PROFESSOR: Yeah,
yeah, that's great. If R has a limit on
how negative it can be, then you make an R prime, which
just adds that limit to R, makes it positive
or non-negative. And now use Markov's
theorem there. And that is now an analogous
form of Markov's theorem when R can be negative, but
there's a lower limit to it. And I won't stay to
improve that here. But that's in the
text and something you want to be familiar with. What I do want to do in
class is another case where you can use
Markov's theorem to analyze the probability or
upper bound the probability that R is very small,
less than its expectation. And it's the same idea
as you just suggested. So let's state that. If R is upper bounded, has a
hard limit on the upper bound, by u for some u in
the real numbers, then for all x less than
u, the probability that R is less than or equal
to x is at most u minus the expected value
of R over u minus x. So in this case, we're
getting a probability that R is less than
something instead of R is bigger than something. And we're going to do it using
a simple trick that we'll be sort of using all day, really. The probability that R is
less than x, this event, R is less than x, is the
same as the event u minus R is at least u minus x. So what have I done? I put negative R over here,
subtract x from each side, add u to each side. I've got to put a less
than or equal to here. So R is less than or
equal to x if and only if u minus r is at
least u minus x. It's simple math there. And now I'm going to
apply Markov to this. I'm going to apply Markov
to this random variable. And this will be the
value I would have had for x up in Markov's theorem. Why is it OK to apply
Markov to u minus R? AUDIENCE: You could just
define the new random variable to be u minus R. PROFESSOR: Yeah, so I got
a new random variable. But what do I need to know
about that new random variable to apply Markov? AUDIENCE: u is always
greater than R. PROFESSOR: u is always greater
than R, or at least as big as R. So u minus R is
always non-negative. So I can apply Markov now. And when I apply Markov, I'll
get this is at most-- maybe I'll go over here. The probability that--
ooh, not R here. This is probability. The probability that u minus
R is at least u minus x is at most the expected
value of that random variable over this value. And well now I just use the
linearity of expectation. I've got a scalar here. So this is u minus the expected
value of R over u minus x. So I've used Markov's theorem to
get a different version of it. All right, let's do an example. Say I'm looking at test scores. And I'll let R be the
score of a random student uniformly selected. And say that the
max score is 100. So that's u. All scores are at most 100. And say that I tell
you the class average, or the expected
value of R, is 75. And now I want to know,
what's the probability that a random student
scores 50 or below? Can we figure that out? I don't know anything
about the distribution, just that the max score is 100
and the average score is 75. What's the probability
that a random student scores 50 or less? I want to upper bound that. So we just plug it
into the formula. u is 100. The expected value is 75. u is 100. And x is 50. And that's 25 over 50, is 1/2. So at most half the class
can score 50 or below. And state it as a probability
question or deterministic fact if I know the average is
75 and the max is 100. Of course, another way
of thinking about that is if more than half
the class scored 50 or below, your
average would have had to be lower, even
if everybody else was right at 100. It wouldn't average out to 75. All right, any
questions about that? OK, so sometimes Markov
is dead on right, gives the right answer. For example, half the
class could have scored 50, and half could have gotten
100 to make it be 75. And sometimes it's way off,
like in the hat check problem. Now, if you know more
about the distribution, then you can get better
bounds, especially the cases when you're far off. For example, if you know
the variance in addition to the expectation, or
aside from the expectation, then you can get better
bounds on the probability that the random
variable is large. And in this case, the result is
known as Chebyshev's theorem. I'll do that over here. And it's the analog of Markov's
theorem based on variance. It says, for all
x bigger than 0, and any random variable R--
could even be negative-- the probability that R deviates
from its expected value in either direction
by at least x is at most of the variance
of R divided by x squared. So this is like
Markov's theorem, except that we're now
bounding the deviation in either direction. Instead of expected
value, you have variance. Instead of x, you've got x
squared, but the same idea. In fact, the proof
uses Markov's theorem. Well, the probability
that R deviates from its expected
value by at least x, this is the same event,
or happens if and only if R minus expected value
squared is at least x squared. I'm just going to
square both sides here. OK, I square both sides. And since this is positive
and this is positive, I can square both sides and
maintain the inequality. Now I'm going to
apply Markov's theorem to that random variable. It's a random variable. It's R minus expected
value squared. So it's a random variable. And what's nice about
this random variable that lets me apply Markov's theorem? It's a square. So it's always non-negative. So I can apply Markov's theorem. And my Markov's theorem,
this probability is at most the expected value
of that divided by this. That's what Markov's theorem
says as long as this is always non-negative. All right, what's a
simpler expression for this, the expected value
of the square of the deviation of a random variable? That's the variance. That's the definition
of variance. So that is just the variance
of R over x squared. And we're done. So Chebyshev's theorem is
really just another version of Markov's theorem. But now it's based
on the variance. OK, any questions? OK, so there's a nice corollary
for this, just as with Markov's theorem. It says the probability that the
absolute value, the deviation, is at least c times the
standard deviation of R. So I'm looking at
the probability that R differs from
its expectation by at least some scalar c
times the standard deviation. Well, what's that? Well, that's the variance of R
over the square of this thing-- c squared times the
standard deviation squared. What's the square of
the standard deviation? That's the variance. They cancel, so it's
just 1 over c squared. So the probability of more than
twice the standard deviation off the expectation is
at most 1/4, for example. All right, let's do
some examples of that. Maybe we'll leave
Markov up there. OK, say we're looking at IQs. In this case, we're going to let
R be the IQ of a random person. All right, now we're
going to assume-- and this actually is the case--
that R is always at least 0, despite the fact that
probably most of you have somebody you know who
you think has a negative IQ. They can't be negative. They have to be non-zero. In fact, IQs are adjusted. So the expected IQ is
supposed to be 100, although actually the
averages may be in the 90's. And it's set up so that the
standard deviation of IQ is supposed to be 15. So we're just going to
assume those are facts on IQ. And that's what
it's meant to be. And now we want to know, what's
the probability a random person has an IQ of at least 250? Now Marilyn, from "Ask Marilyn,"
has an IQ pretty close to 250. And she thinks that's
pretty special, pretty rare. So what can we say about that? In particular, say
we used Markov. What could you say about
the probability of having an IQ of at least 250? What does Markov tell us? AUDIENCE: [INAUDIBLE] PROFESSOR: What is it? AUDIENCE: [INAUDIBLE] PROFESSOR: Not quite 1 in 25,
but you're on the right track. It's not quite 2/3. It's the expected
value, which is 100, over the x value, which is 250. So it's 1 in 2.5, or 0.4. So the probability
is at most 0.4, so 40% chance it could
happen, potentially, but no bigger than that. What about Chebyshev? See if you can figure
out what Chebyshev says about the probability of
having an IQ of at least 250. It's a little tricky. You've got to sort of
plug it into the equation there and get it to
fit in the right form. Chebyshev says that-- let's
get in the right form. I've got probability
that R is at least 250. I've got to get it into
that form up there. So that's the probability
that-- well, first R minus 100 is at least 150. So I've got R minus
the expected value. I'm sort of getting it ready
to apply Chebyshev here. And then 150-- how many
standard deviations is 150? 10, all right? So this is the probability that
R minus the expected value of R is at least 10
standard deviations. That's what I'm asking. I'm not quite there. I'm going to use
the corollary there. But I've got to get that
absolute value thing in. But it's upper bounded
by the probability of the absolute value of
R minus expected value bigger than or equal to
10 standard deviations. Because this allows
for two cases. R is 10 standard
deviations high, and R is 10 standard
deviations low or more. So this is upper
bounded by that. And now I can plug in Chebyshev
in the corollary form. And what's the answer
when I do that? 1 in 100-- the
probability of being off by 10 standard deviations
or more is at most 1 in 100, 1 in 10 squared. So it's a lot better bound. It's 1% instead of 40%. So knowing the variance
of the standard deviation gives you a lot more information
and generally gives you much better bounds
on the probability of deviating from the mean. And the reason it
gives you better bounds is because the variance
is squaring deviations. So they count a lot more. All right, now let's look at
this step a little bit more. All right, let's
say here is a line, and here's the
expected value of R. And say here's 10 standard
deviations high here. So this will be more than
10 standard deviations. And this will be 10 standard
deviations on the low side. So here, I'm low. Now, this line here
with the absolute value is figuring out the probability
of being low or high. This is the probability
that the absolute value of R minus its expected
value is at least 10 standard deviations. What we really wanted
to know for bound was just the high side. Now, is it true that then, since
the probability of high or low is 1 in 100, the probability
of being high is at most 1 in 200, half? Is that true? Yeah? AUDIENCE: [INAUDIBLE] PROFESSOR: Yeah, it is
not necessarily true that the high and
the low are equal, and therefore the high
is half the total. It might be true, but
not necessarily true. And that's a mistake
that often gets made where you'll take this
fact as being less than 100 to conclude that's
less than 1 in 200. And that you can't do,
unless the distribution is symmetric around
the expected value. Then you could do it, if
it's a symmetric distribution around the expected value. But usually it's not. Now, there is something
better you can say. So let me tell you what it is. But we won't prove it. I think we might
prove it in the text. I'm not sure. If you just want the high side
or just want the low side, you can do slightly better
than 1 in c squared. That's the following theorem. For any random variable
R, the probability that R is on the high side
by c standard deviations is at most 1 over
c squared plus 1. So it's not 1 over 2c squared. It's 1 over c squared
plus 1, and the same thing for the probability of
being on the low side. Let's see, have I
written this right? Hmm, I want to get this as
less than or equal to negative c times the standard deviation. So here I'm high by c or
more standard deviations. Here I'm low. So R is less than the
expected value by at least c standard deviations. And that is also 1
over c squared plus 1. And it is possible
to find distributions that hit these targets--
not both at the same time, but one or the other,
hit those targets. So that's the best you
can say in general. All right, so using
this bound, what's the probability that
a random person has an IQ of at least 250? It's a little better
than 1 in 100. AUDIENCE: [INAUDIBLE] PROFESSOR: Yeah, 1/101. So in fact, the best we can
say without knowing any more information about IQs is
that it's at most 1/101, slightly better. Now in fact, with IQs, they know
more about the distribution. And the probability
is a lot less. Because you know more
about the distribution than we've assumed here. In fact, I don't think
anybody has an IQ over 250 as far as I know. Any questions about this? OK, all right, say
we give the exam. What fraction of the class
can score more than two standard deviations, get two
standard deviations or more, away from the average,
above or below? Could half the class be
two standard deviations off the mean? No? What's the biggest fraction
that that could happen? What do I do? What fraction of the class
can be two standard deviations or more from the mean? What is it? AUDIENCE: 1/4. PROFESSOR: 1/4, because c is 2. You don't even know
what the mean is. You don't know what the
standard deviation is. You don't need to. I just asked, you're
two standard deviations off or more. At most, 1/4. How many could be two
standard deviations high or better at most? 1/5-- 1 over 4 plus 1, good. OK, this holds
true no matter what the distribution
of test scores is. Yeah? AUDIENCE: [INAUDIBLE] PROFESSOR: Which one? This one? AUDIENCE: Yeah. PROFESSOR: Oh, that's
more complicated. That'll take us several
boards to do, to prove that. And I forget if we put
it in the text or not. It might be in the
text, to prove that. Any other questions? OK so Markov and Chebyshev are
sometimes close, sometimes not. Now, for the rest
of today, we're going to talk about a much
more powerful technique. But it only works
in a special case. Now, the good news
is this special case happens all the
time in practice. And it's the case when you're
analyzing a random variable that itself is
the sum of a bunch of other random variables. And we've seen already
examples like that. And the other random
variables have to be mutually independent. And in this case,
you get a bound that's called a Chernoff bound. And this is the same
Chernoff who figured out how to beat the lottery. And it's interesting. Long after we started
teaching this, originally this stuff was only
taught, for Chernoff bounds, for graduate students. And now we teach it here. Because it's so important. And it really is accessible. It'll be probably the
most complicated proof we've done to establish
a Chernoff bound. But Chernoff himself,
when he discovered this, thought it was no big deal. In fact, he couldn't
figure out why everybody in computer science
was always writing papers with Chernoff bounds in them. And that's because he
didn't put any emphasis on the bounds in his work. But computer scientists
who came later found all sorts of
important applications. And we'll see some
of those today. So let me tell you
what the bound is. And the nice thing
is it really is Markov's theorem
again in disguise, just a little more complicated. Theorem-- it's called
a Chernoff bound. Let T1, T2, up to Tn be
any mutually independent-- that's really important--
random variables such that each of them takes
values only between 0 and 1. And if they don't, just
normalize them so they do. So we're going to take a
bunch of random variables that are mutually independent. And they are all
between 0 and 1. Then we're going to look at the
sum of those random variables, call that T. Then
for any c at least 1, the probability that the sum
random variable is at least c times its expected value. So it's going to be the high
side here-- is at most e to the minus z, and I'll tell
you what that is in a minute, times the expected value of T
where z is c natural log of c plus 1 minus c. And it turns out if c is bigger
than 1, this is positive. So that's a lot,
one of the longest theorems we wrote down here. But what it says is
that probability were high is exponentially small. As the expected value is
big, the chance of being high gets really, really tiny. Now, I'm going to
prove it in a minute. But let's just plug
in some examples to see what's going on here. So for example, suppose the
expected value of T is 100. And suppose c is 2. So we expect to have
100 come out of the sum. The probability we get
at least 200-- well, let's figure out what that is. c being 2 we can evaluate z now. It's 2 natural log
of 2 plus 1 minus 2. And that's close to but a
little larger than 0.38. So we can plug z in,
the exponent up there, and find that the probability
that T is at least twice its expected value,
namely at least 200, is at most e to the
minus 0.38 times 100, which is e to the minus 38,
which is just really small. So that's just way better
than any results you get with Markov or Chebyshev. So if you have a bunch of random
variables between 0 and 1, and they're mutually
independent, you add them up. If you expect 100 as
the answer, the chance of getting 200 or more-- forget
about it, not going to happen. Now, of course Chernoff doesn't
apply to all distributions. It has to be this type. This is a pretty broad class. In fact, it contains the class
of all Bernoulli distributions. So I have binomial
distributions. Because remember a binomial
distribution-- well, remember binomial distributions? That's where T is
the sum of Ti's. In binomial, you
have Tj is 0 or 1. It can't be in between. And with binomial, all Tj's
have the same distribution. With Chernoff, they
can all be different. So Chernoff is much
broader than binomial. The individual guys here can
have different distributions and attain values
anywhere between 0 and 1, as opposed to just
one or the other. Any questions about this theorem
and what it says in the terms there? One nice thing about it is
the number of random variables doesn't even show up
in the answer here. n doesn't even appear. Yeah. AUDIENCE: [INAUDIBLE] PROFESSOR: Does
not apply to what? AUDIENCE: [INAUDIBLE] PROFESSOR: Yeah, when c equals
1, what happens is z is 0. Because I have a log of 1
is 0, and 1 minus 1 is 0. And if z is 0, it
says your probability is upper bounded by 1. Well, not too interesting,
because any probability is upper bounded by 1. So it doesn't give
you any information when c is 0, none at all. But as soon as c starts
being-- sorry, if c is 1. As soon as c starts being bigger
than 1, which is sort of a case you're interested in, you're
bigger than your expectation, then it gives very
powerful results. Yeah. AUDIENCE: [INAUDIBLE] PROFESSOR: Yeah, you can. It's true for n
equals 1 as well. Now, it doesn't give you
a lot of information. Because if c is bigger
than 1 and n was 1, so it's using one variable,
what's the probability that a random variable
exceeds its expectation, c times its expectation? AUDIENCE: [INAUDIBLE] PROFESSOR: Yeah, let's see now. Maybe it does give
you information. Because the random variable
has a distribution on 0, 1. That's right, so it does
give you some information. But I don't think
it gives you a lot. I have to think about that. What happens when
there's just one guy? Because the same thing is true. It's just now for a single
random variable on 0, 1 the chance that your
twice the expected value. I have to think about that. That's a good question. Does it do anything
interesting there? OK, all right, so
let's do an example of how you might apply this. Say that you're playing Pick
4, and 10 million people are playing. And say in this
version of Pick 4, you're picking a four digit
number, four single digits. And you win if you
get an exact match. So the probability of
winning, a person winning, well, they've got to get
all four digits right. That's 1 in 10,000,
10 to the fourth. What's the expected
number of winners? If I got 10 million
people, what's the expected number of winners? What is it? We've got 10 million
over 10,000, right? Because what I'm doing here
is the number of winners, T, is going to be the sum of
10 million indicator variables. And the probability that
any one of these guys wins is 1 in 10,000. So the expected
number of winners is 1 in 10,000 added 10
million times, which is this. Is that OK? Everybody should be
really familiar with how to whip these things out. This for sure will have
probably at least a couple questions where
you're going to need to be able to do that kind
of stuff on the final. All right, say I want to know
the probability of getting at least 2,000 winners,
and I want to upper bound that just with the
information I've given you. Well, any thoughts
about an upper bound? AUDIENCE: [INAUDIBLE] PROFESSOR: What's that? AUDIENCE: [INAUDIBLE] PROFESSOR: Yeah, that's
a good upper bound. What did you have to
assume to get there? e to the minus 380
is a great bound. Because you're going to plug
in expected value is 1,000. And we're asking for more
than twice the expected value. So it's e to the minus
0.38 times 1,000. And that for sure is--
so you computed this. And that equals e
to the minus 380. So that's really small. But what did you have to
assume to apply Chernoff? Mutual independence. Mutual independence of what? AUDIENCE: [INAUDIBLE] PROFESSOR: The
numbers people picked. And we already know, if
people are picking numbers, they don't tend to be
mutually independent. They tend to gang up. But if you had a computer
picking the numbers randomly and mutually independently, then
you would be e to the minus 380 by Chernoff if mutually
independent picks. Everybody see why we did that? Because it's a probability
of twice your expectation. The total number of winners is
the sum of 10 million indicator variables. And indicator
variables are 0 or 1. So they fit that
definition up there. And so we already figured
out z is at least 0.38. And you're multiplying by
the expected value of 1,000. That's e to the minus 380,
so very, very unlikely. What if they weren't
mutually independent? Can you say anything about this,
anything at all better than 1, which we know for
any probability? Yeah? AUDIENCE: It's
possible that everyone chose the same numbers. PROFESSOR: Yes, everyone could
have chosen the same number. But that number only comes
up with a 1 in 10,000 chance. So you can say something. AUDIENCE: You can use Markov. PROFESSOR: Use Markov. What does Markov give you? What does Markov give you? 1/2, yeah. Because you've got
the expected value is 1,000 divided by the
bound threshold, is 2,000, is 1/2 by Markov. And that holds true without
any independence assumption. Now, there is an enormous
difference between 1/2 and e to the minus 380. Independence really
makes a huge difference in the bound you can compute. OK, now there's another way
we could've gone about this. What kind of distribution
does T have in this case? It's binomial. Because it's the sum of
indicator random variables, 0, 1's. Each of these is 0, 1. And they're all the
same distribution. There's a 1 in 10,000 chance of
winning for each one of them. So it's a binomial. So we could have
gone back and used the formulas we had for
the binomial distribution, plugged it all in, and we'd have
gotten something pretty similar here. But Chernoff is so much easier. Remember that pain
we would go through with a binomial distribution,
the approximation, Stirling's formula, [INAUDIBLE] whatever,
the factorials and stuff? And that's a nightmare. This was easy. e to the minus 380 was
very easy to compute. And really at that
point it doesn't matter if it's minus 381
or minus 382 or whatever. Because it's really small. So often, even when you have
a binomial distribution, well, Chernoff will apply. And that's a great way to go. Because it gives you
good bounds generally. All right, let's figure
out the probability of at least 1,100
winners instead of 1,000. So let's look at the probability
of at least 100 extra winners over what we expect
out of 10 million. We've got 10 million people. You expect 1,000. We're going to analyze
the probability of 1,100. What's c in this case? We're going to use Chernoff. 1.1. So this is 1.1 times 1,000. And that means that z is
1.1 times the natural log of 1.1 plus 1 minus 1.1. And that is close to
but at least 0.0048. So this probability is
at most, by Chernoff, e to the minus 0.0048 times
the expected number of winners is 1,000. So that is e to the minus
4.8, which is less than 1%, 1 in 100. So that's pretty powerful. It says, you've got 10
million people who could win. The chance of even having 100
more than the 1,000 you expect is 1% chance at most--
very, very powerful. It says you really
expect to get really close to the mean
in this situation. OK, a lot better--
Markov here gives you, what, 1,000 over 1,100. It says your probability
could be 90% or something-- not very useful. Chebyshev won't give
you much here either. So if you're in a situation
to apply Chernoff, always go there. It gives you the best bounds. Any questions? This of course is why computer
scientists use it all the time. OK, actually, before
I do more examples, let me prove the theorem
in a special case to give you a feel
for what's involved. The full proof is in the text. I'm going to prove it
in the special case where the Tj are 0, 1. So they're indicator
random variables. But they don't have to
have the same distribution. So it's still more
general than you get with a binomial distribution. All right, so we're going
to do a proof of Chernoff for the special case where
the Tj are either 0 or 1. So they're indicator variables. OK, so the first step is going
to seem pretty mysterious. But we've been doing
something like it all day. I'm trying to compute the
probability T is bigger than c times its expectation. Well, what I'm going to do is
exponentiate both of these guys and compute the
probability that c to the T is at least c to the c times
the expected value of T. Now, this is not the first
thing you'd expect to do, probably, if you were
trying to prove this. So it's one of those
divine insights that you'd make this step. And then I'm going to
apply Markov, like we've been doing all day, to this. Now, since T is positive and c
is positive, these are equal. And this is never non-negative. So now by Markov, this
is simply upper bounded by the expected value of that,
expected value of c to the T, divided by this. And that's by Markov. So everything we've done today
is really Markov in disguise. Any questions so far? You start looking at
this, you go, oh my god, I got the random variable
and the exponent. This is looking
like a nightmare. What is the expected
value of c to the T, and this kind of stuff? But we're going to
hack through it. Because it gives you just
an amazingly powerful result when you're done. All right, so we've got to
evaluate the expected value of c to the T. And we're going
to use the fact that T is the sum of the Tj's. And that means that c to the
T equals c to the T1 times c to the T2 times c to the Tn. The weird thing about this
proof is that every step sort of makes it more
complicated looking until we get to the end. So it's one of those that's hard
to figure out the first time. All right, that means the
expected value of c to the T is the expected value of
the product of these things. Now I'm going to use the
product rule for expectation. Now, why can I use
the product rule? What am I assuming to
be able to do that? That they are
mutually independent, that the c to the
Tj's are mutually independent of each other. And that follows, because the
Tj's are mutually independent. So if a bunch of random variable
are mutually independent, then their exponentiations
are mutually independent. So this is by product
rule for expectation and mutual independence. OK, so now we've got to
evaluate the expected value of c to the Tj. And this is where
we're going to make it simpler by assuming that Tj
is just a 0, 1 random variable. So the simplification
comes in here. So the expected value of
Tj-- well, there's two cases. Tj is 1, or it's 0. Because we made
this simplification. If it's 1, I get
c to the 1-- ooh, expected value of c to the Tj. Let's get that right. It could be 1, in which case
I get a contribution of c to the 1 times the probability
Tj equals 1 plus the case at 0. So I get c to the 0 times
the probability Tj is 0. Well, c to the 1 is just c. c to the 0 is 1. And I'm going to
rewrite Tj being 0 as 1 minus the
probability Tj is 1. All right, this equals that. And of course the 1 cancels. Now I'm going to
collect terms here to get 1 plus c minus 1 times
the probability Tj equals 1. OK, then I'm going to
do one more step here. This is 1 plus c minus 1 times
the expected value of Tj. Because if I have an
indicator random variable, the expected value is the same
as the probability that it's 1. Because in the
other case it's 0. And now I'm going to use
the trick from last time. Remember 1 plus x is always at
most e to the x from last time? None of these steps is
obvious why we're doing them. But we're going
to do them anyway. So this is at most e to this,
c minus 1 expected value of Tj. Because 1 plus anything is at
most the exponential of that. And I'm doing this step because
I got a product of these guys. And I want to put
them in the exponent so I can then sum
them so it gets easy. OK, now we just plug
this back in here. So that means that the
expected value of c to the T is at most a product
of expected value of e to the cTj is this-- e to the
c minus 1 expected value of Tj. And now I can convert this
to a sum in the exponent. And this is j equals 1 to n. And what do I do
to simplify that? Linearity of expectation. c minus 1 times the sum j equals
1 to n expected value of Tj. Ooh, let's see, did I? Actually, I used
linearity coming out. I already used linearity. I screwed up here. So here I used the linearity
when I took the sum up here inside the expectation. I've already used linearity. What is the sum of the Tj's? T-- yeah, that's what
I needed to do here. OK, we're now almost done. We've got now an upper bound
on the expected value of c to the T. And it is this. And we just plug
that in back up here. So now this is at most e to the
c minus 1 expected value of T over c to the c times
the expected value of t. And now I just do manipulation. c to something is the
same as e to the log of c times that something. So this is e to the minus c ln
c expected value of T plus that. And then I'm
running out of room. That equals-- I can just pull
out the expected values of T. I get e to the minus c log of
c plus c minus 1 expected value of T. And that's e to the
minus z expected value of T. All right, so that's
a marathon proof. It's the worst proof I think. Well, maybe minimum
spanning tree was worse. But this is one of the worst
proofs we've seen this year. But I wanted to show it to you. Because it's one of the
most important results that we cover, certainly
in probability, that can be very
useful in practice. And it gives you some
feel for, hey, this wasn't so obvious to
do it the first time, and also some of the
techniques that are used, which is really
Markov's theorem. Any questions? Yeah. AUDIENCE: Over there, you
define z as 1 minus c. PROFESSOR: Did I do it wrong? AUDIENCE: c natural
log of c, 1 minus c. Maybe it's plus c? PROFESSOR: Oh, I've
got to change the sign. Because I pulled a
negative out in front. So it's got to be
negative c minus 1, which means negative c plus 1. Yeah, good. Yeah, this was OK. I just made the
mistake going to there. Any other questions? OK, so the common theme here in
using Markov to get Chebyshev, to get Chernoff, to get
the Markov extensions, is always the same. And let me show you
what that theme is. Because you can use it to
get even other results. When we're trying to figure
out the probability that T is at least c times its
expected value, or actually even in general,
even more generally than that, the probability
that A is bigger than B, even more generally,
well, that's the same as the probability that
f of A is bigger than f of B as long as you
don't change signs. And then by Markov, this is at
most the expected value of that as long as it's
non-negative over that. In Chebyshev, what
function f did we use for Chebyshev in
deriving Chebyshev's theorem? What was f doing in Chebyshev? Actually I probably
just erased it. What operation were we
doing with Chebyshev? AUDIENCE: Variance. PROFESSOR: Variance. And that meant we
were squaring it. So the technique used
to prove Chebyshev was f was the square function. For Chernoff, f is the
exponentiation function, which turns out to be--
in fact, when we did it for Chernoff, that's the
optimal choice of functions to get good bounds. All right, any
questions on that? All right, let's do one more
example here with numbers. And this is a load
balancing application for example you might
have with web servers. Say you've got to build
a load balancing device, and it's got to
balance N jobs, B1, B2, to BN, across a set of M
servers, S1, S2, to SN. And say you're doing this
for a decent sized website. So maybe N is 100,000. You get 100,000
requests a minute. And say you've got 10 servers
to handle those requests. And say the requests are--
the time for the j-th request is, say, Bj takes the j-th job. The j-th request takes Lj time. And the time is the
same on any server. The servers are all equivalent. And let's assume it's normalized
so that Lj is between 0 and 1. Maybe the worst job takes
a second to do, let's say. And say that if you sum up
the length of all the jobs, you get L. Total workload
is the sum of all of them. j equals 1 to N. And we're going to assume that
the average job length is 1/4 second. So we're going to assume
that the total amount of work is 25,000 seconds, say. So the average job
length is 1/4 second. And the job is to assign these
tasks to the 10 servers so that hopefully every server
is doing L/M work, which would be 25,000/10, or
2,500 milliseconds of work, something like that. I don't know. Because when you're
doing load balancing, you want to take your load and
spread it evenly and equally among all the servers. Any questions about the problem? You've got a bunch of
jobs, a bunch of servers. You want to assign the
jobs to the servers to balance the load. Well, what is the
simplest algorithm you could think of to do this? AUDIENCE: [INAUDIBLE] PROFESSOR: That's a good
algorithm to do this. In practice, the
first thing people do is, well, take the first N/M
jobs, put them on server one, the next N/M on server two. Or they'll use something called
round robin-- first job goes here, second here, third here,
10th here, back and start over. And they hope that it
will balance the load. But it might well not. Because maybe every
10th job is a big one. So what's much better
to do in practice is to assign them randomly. So a job comes in. You don't even pay
attention to how hard it is, how much time
you think it'll take. You might not even know before
you start the job how long it's going to take to complete. Give it to a random server. Don't even look at how
much work that server has. Just give it to a random one. And it turns out this
does very, very well. Without knowing anything,
just that simple approach does great in practice. And today, state of the
art load balancers do this. We've been doing randomized
kinds of thing like this at Akamai now for a decade. And it's just stunning
how well it works. And so let's see why that is. Of course we're going to use
the Chernoff bound to do it. So let's let Rij be the load
on server Si from job Bj. Now, if Bj is not assigned
to Si, it's zero load. Because it's not even
doing the work there. So we know that Rij
equals the load of Bj if it's assigned to Si. And that happens
with probability 1/M. The job picks one of
the M servers at random. And otherwise, the load is 0. Because it's not
assigned to that server. And that is probability
1 minus 1/M. Now let's look at
how much load gets assigned by this random
algorithm to server i. So we'll let Ri be the sum
of all the load assigned to server i. So we've got this indicator
where the random variables are not 0, 1. They're 0 and whatever
this load happens to be for the j-th job, at most 1. And we sum up the value
for the contribution to Si over all the jobs. So now we compute
the expected value of Ri, the expected
load on the i-th server. So the expected load
on the i-th server is-- well, we use
linearity of expectation. And the expected value
of Rij-- well, 0 or Lj. It's Lj with
probability 1/M. This is just now the sum of Lj over
M. And the sum of Lj is just L. So the expected load
of the i-th server is the total load
divided by the number of servers, which is perfect. It's optimal-- can't
do better than that. It makes sense. If you assign all
the jobs randomly, every server is expecting to
get 1/M of the total load. Now we want to know
the probability it deviates from that,
that you have too much load on the i-th server. All right, so the probability
that the i-th server has c times the optimal load
is at most, by Chernoff, if the jobs are independent,
minus zL over M, minus z times the
expected load where z is c ln c plus 1 minus c. This is Chernoff
now, just straight from the formula of Chernoff,
as long as these loads are mutually independent. All right, so we know that when
c gets to be-- I don't know, you pick 10% above
optimal, c equals 1.1, well, we know that this is
going to be a very small number. L/M is 2,500. And z, in this case,
we found was 0.0048. So we get e to the minus
0.0048 times 2,500. And that is really tiny. That's less than 1 in 160,000. So Chernoff tells
us the probability that any server, a
particular server, gets 10% load more than
you expect is minuscule. Now, we're not quite done. That tells us the probability
the first server gets 10% too much load or the problem
the second server got 10% too much load, and so forth. But what we really care
about is the worst server. If all of them are
good except for one, you're still in trouble. Because the one ruined your day. Because it didn't
get the work done. So what do you do to
bound the probability that any of the servers got
too much load, any of the 10? So what I really want to
know is the probability that the worst server
of M takes more than cL over M. Well, that's the
probability that the first one has more than cL over M union
the second one has more than cL over M union the M-th one. What do I do to get that
probability, the probability of a union of events,
upper bounded? AUDIENCE: [INAUDIBLE] PROFESSOR: Upper bounded by
the sum of the individual guys. It's the sum i equals 1 to M
probability Ri greater than or equal to cL over M.
And so that, each of these is at most 1 in 160,000. This is at most M/160,000. And that is equal
to 1 in 16,000. All right, so now
we have the answer. The chance that any server
got 10% load or more is 1 in 16,000 at most, which
is why randomized load balancing is used a lot in practice. Now tomorrow, you're going to
do a real world example where people use this
kind of analysis, and it led to utter disaster. And the reason was that the
components they were looking at were not independent. And the example has to do with
the subprime mortgage disaster. And I don't have time
today to go through it all. But it's in the text, and
you'll see it tomorrow. But basically what
happened is that they took a whole bunch of
loans, subprime loans, put them into these
things called bonds, and then did an analysis
about how many failures they'd expect to have. And they assumed the loans
were all mutually independent. And they applied
their Chernoff bounds. And that concluded
that the chances of being off from the
expectation were nil, like e to the minus 380. In reality, the loans
were highly dependent. When one failed, a
lot tended to fail. And that led to disaster. And you'll go through
some of the math on that tomorrow in recitation.
