The following content is
provided under a Creative Commons license. Your support will help MIT
OpenCourseWare continue to offer high quality educational
resources for free. To make a donation or view
additional materials from hundreds of MIT courses, visit
MIT OpenCourseWare at ocw.mit.edu JOHN TSISIKLIS: So here's
the agenda for today. We're going to do a
very quick review. And then we're going
to introduce some very important concepts. The idea is that all
information is-- Information is always partial. And the question is what do we
do to probabilities if we have some partial information about
the random experiments. We're going to introduce the
important concept of conditional probability. And then we will see three
very useful ways in which it is used. And these ways basically
correspond to divide and conquer methods for breaking
up problems into simpler pieces. And also one more fundamental
tool which allows us to use conditional probabilities to do
inference, that is, if we get a little bit of information
about some phenomenon, what can we
infer about the things that we have not seen? So our quick review. In setting up a model of a
random experiment, the first thing to do is to come up with
a list of all the possible outcomes of the experiment. So that list is what we
call the sample space. It's a set. And the elements of the
sample space are all the possible outcomes. Those possible outcomes must be distinguishable from each other. They're mutually exclusive. Either one happens or the other
happens, but not both. And they are collectively
exhaustive, that is no matter what the outcome of the
experiment is going to be an element of the sample space. And then we discussed last
time that there's also an element of art in how to choose
your sample space, depending on how much detail
you want to capture. This is usually the easy part. Then the more interesting part
is to assign probabilities to our model, that is to make some
statements about what we believe to be likely and what
we believe to be unlikely. The way we do that is by
assigning probabilities to subsets of the sample space. So as we have our sample space
here, we may have a subset A. And we assign a number to that
subset P(A), which is the probability that this
event happens. Or this is the probability that
when we do the experiment and we get an outcome it's the
probability that the outcome happens to fall inside
that event. We have certain rules that
probabilities should satisfy. They're non-negative. The probability of the overall
sample space is equal to one, which expresses the fact that
we're are certain, no matter what, the outcome is going
to be an element of the sample space. Well, if we set the top right
so that it exhausts all possibilities, this should
be the case. And then there's another
interesting property of probabilities that says that,
if we have two events or two subsets that are disjoint, and
we're interested in the probability, that one or the
other happens, that is the outcome belongs to A or belongs
to B. For disjoint events the total probability of
these two, taken together, is just the sum of their
individual probabilities. So probabilities behave
like masses. The mass of the object
consisting of A and B is the sum of the masses of
these two objects. Or you can think of
probabilities as areas. They have, again, the
same property. The area of A together with B is
the area of A plus the area B. But as we discussed at the end
of last lecture, it's useful to have in our hands a more
general version of this additivity property, which says
the following, if we take a sequence of sets-- A1, A2, A3, A4, and so on. And we put all of those
sets together. It's an infinite sequence. And we ask for the probability
that the outcome falls somewhere in this infinite
union, that is we are asking for the probability that the
outcome belongs to one of these sets, and assuming that
the sets are disjoint, we can again find the probability for
the overall set by adding up the probabilities of the
individual sets. So this is a nice and
simple property. But it's a little more subtle
than you might think. And let's see what's going
on by considering the following example. We had an example last time
where we take our sample space to be the unit square. And we said let's consider a
probability law that says that the probability of a subset is
just the area of that subset. So let's consider this
probability law. OK. Now the unit square is
the set --let me just draw it this way-- the unit square is the union of
one element set consisting all of the points. So the unit square is made up
by the union of the various points inside the square. So union over all x's and y's. OK? So the square is made
up out of all the points that this contains. And now let's do
a calculation. One is the probability of our
overall sample space, which is the unit square. Now the unit square is the union
of these things, which, according to our additivity
axiom, is the sum of the probabilities of all of these
one element sets. Now what is the probability
of a one element set? What is the probability of
this one element set? What's the probability that our
outcome is exactly that particular point? Well, it's the area of that
set, which is zero. So it's just the sum of zeros. And by any reasonable
definition the sum of zeros is zero. So we just proved that
one is equal to zero. OK. Either probability theory is
dead or there is some mistake in the derivation that I did. OK, the mistake is quite
subtle and it comes at this step. We're sort of applied the
additivity axiom by saying that the unit square is the
union of all those sets. Can we really apply our
additivity axiom. Here's the catch. The additivity axiom applies
to the case where we have a sequence of disjoint events
and we take their union. Is this a sequence of sets? Can you make up the whole unit
square by taking a sequence of elements inside it and cover
the whole unit square? Well if you try, if you start
looking at the sequence of one element points, that sequence
will never be able to exhaust the whole unit square. So there's a deeper reason
behind that. And the reason is that infinite
sets are not all of the same size. The integers are an
infinite set. And you can arrange the integers
in a sequence. But the continuous set
like the units square is a bigger set. It's so-called uncountable. It has more elements than
any sequence could have. So this union here is not of
this kind, where we would have a sequence of events. It's a different
kind of union. It's a Union that involves a
union of many, many more sets. So the countable additivity
axiom does not apply in this case. Because, we're not dealing
with a sequence of sets. And so this is the
incorrect step. So at some level you might think
that this is puzzling and awfully confusing. On the other hand, if you think
about areas of the way you're used to them from
calculus, there's nothing mysterious about it. Every point on the unit
square has zero area. When you put all the points
together, they make up something that has
finite area. So there shouldn't be any
mystery behind it. Now, one interesting thing that
this discussion tells us, especially the fact that the
single elements set has zero area, is the following-- Individual points have
zero probability. After you do the experiment and
you observe the outcome, it's going to be an
individual point. So what happened in that
experiment is something that initially you thought had zero
probability of occurring. So if you happen to get some
particular numbers and you say, "Well, in the beginning,
what did I think about those specific numbers? I thought they had
zero probability. But yet those particular
numbers did occur." So one moral from this is that
zero probability does not mean impossible. It just means extremely,
extremely unlikely by itself. So zero probability
things do happen. In such continuous models,
actually zero probability outcomes are everything
that happens. And the bumper sticker version
of this is to always expect the unexpected. Yes? AUDIENCE: [INAUDIBLE]. JOHN TSISIKLIS: Well,
probability is supposed to be a real number. So it's either zero or it's
a positive number. So you can think of the
probability of things just close to that point and those
probabilities are tiny and close to zero. So that's how we're going to
interpret probabilities in continuous models. But this is two chapters
ahead. Yeah? AUDIENCE: How do we interpret
probability of zero? If we can use models that
way, then how about probability of one? That it it's extremely
likely but not necessarily for certain? JOHN TSISIKLIS: That's
also the case. For example, if you ask in this
continuous model, if you ask me for the probability that
x, y, is different than the zero, zero this is
the whole square, except for one point. So the area of this is
going to be one. But this event is not entirely
certain because the zero, zero outcome is also possible. So again, probability of one
means essential certainty. But it still allows the
possibility that the outcome might be outside that set. So these are some of the weird
things that are happening when you have continuous models. And that's why we start to
this class with discrete models, on which would
be spending the next couple of weeks. OK. So now once we have set up our
probability model and we have a legitimate probability law
that has these properties, then the rest is
usually simple. Somebody asks you a question of
calculating the probability of some event. While you were told something
about the probability law, such as for example the
probabilities are equal to areas, and then you just
need to calculate. In these type of examples
somebody would give you a set and you would have
to calculate the area of that set. So the rest is just calculation
and simple. Alright, so now it's time
to start with our main business for today. And the starting point
is the following-- You know something
about the world. And based on what you know when
you set up a probability model and you write down
probabilities for the different outcomes. Then something happens, and
somebody tells you a little more about the world, gives
you some new information. This new information, in
general, should change your beliefs about what happened
or what may happen. So whenever we're given new
information, some partial information about the outcome
of the experiment, we should revise our beliefs. And conditional probabilities
are just the probabilities that apply after the revision
of our beliefs, when we're given some information. So lets make this into
a numerical example. So inside the sample space, this
part of the sample space, let's say has probability 3/6,
this part has 2/6, and that part has 1/6. I guess that means that out here
we have zero probability. So these were our initial
beliefs about the outcome of the experiment. Suppose now that someone
comes and tells you that event B occurred. So they don't tell you the
full outcome with the experiment. But they just tell you that the
outcome is known to lie inside this set B. Well then, you should certainly
change your beliefs in some way. And your new beliefs about what
is likely to occur and what is not is going to be
denoted by this notation. This is the conditional
probability that the event A is going to occur, the
probability that the outcome is going to fall inside the set
A given that we are told and we're sure that the event
lies inside the event B Now once you're told that the
outcome lies inside the event B, then our old sample space
in some ways is irrelevant. We have then you sample space,
which is just the set B. We are certain that the outcome
is going to be inside B. For example, what is this
conditional probability? It should be one. Given that I told you that B
occurred, you're certain that B occurred, so this has
unit probability. So here we see an instance of
revision of our beliefs. Initially, event B had the
probability of (2+1)/6 -- that's 1/2. Initially, we thought B
had probability 1/2. Once we're told that B occurred,
the new probability of B is equal to one. OK. How do we revise the probability
that A occurs? So we are going to have the
outcome of the experiment. We know that it's inside B. So
we will either get something here, and A does not occur. Or something inside here,
and A does occur. What's the likelihood that,
given that we're inside B, the outcome is inside here? Here's how we're going
to think about. This part of this set B, in
which A also occurs, in our initial model was twice as
likely as that part of B. So outcomes inside here
collectively were twice as likely as outcomes out there. So we're going to keep the same
proportions and say, that given that we are inside the set
B, we still want outcomes inside here to be twice as
likely outcomes there. So the proportion of the
probabilities should be two versus one. And these probabilities should
add up to one because together they make the conditional
probability of B. So the conditional probabilities should
be 2/3 probability of being here and 1/3 probability
of being there. That's how we revise
our probabilities. That's a reasonable, intuitively
reasonable, way of doing this revision. Let's translate what we
did into a definition. The definition says the
following, that the conditional probability of A
given that B occurred is calculated as follows. We look at the total probability
of B. And out of that probability that was inside
here, what fraction of that probability is assigned to
points for which the event A also occurs? Does it give us the same numbers
as we got with this heuristic argument? Well in this example,
probability of A intersection B is 2/6, divided by total
probability of B, which is 3/6, and so it's 2/3, which
agrees with this answer that's we got before. So the former indeed matches
what we were trying to do. One little technical detail. If the event B has zero
probability, and then here we have a ratio that doesn't
make sense. So in this case, we say that
conditional probabilities are not defined. Now you can take this definition
and unravel it and write it in this form. The probability of A
intersection B is the probability of B times the
conditional probability. So this is just consequence of
the definition but it has a nice interpretation. Think of probabilities
as frequencies. If I do the experiment over and
over, what fraction of the time is it going to be the case
that both A and B occur? Well, there's going to be a
certain fraction of the time at which B occurs. And out of those times when B
occurs, there's going to be a further fraction of
the experiments in which A also occurs. So interpret the conditional
probability as follows. You only look at those
experiments at which B happens to occur. And look at what fraction of
those experiments where B already occurred, event
A also occurs. And there's a symmetrical
version of this equality. There's symmetry between the
events B and A. So you also have this relation that
goes the other way. OK, so what do we use these
conditional probabilities for? First, one comment. Conditional probabilities
are just like ordinary probabilities. They're the new probabilities
that apply in a new universe where event B is known
to have occurred. So we had an original
probability model. We are told that B occurs. We revise our model. Our new model should still be
legitimate probability model. So it should satisfy all sorts
of properties that ordinary probabilities do satisfy. So for example, if A and B are
disjoint events, then we know that the probability of A
union B is equal to the probability of A plus
probability of B. And now if I tell you that a certain event C
occurred, we're placed in a new universe where
event C occurred. We have new probabilities
for that universe. These are the conditional
probabilities. And conditional probabilities
also satisfy this kind of property. So this is just our usual
additivity axiom but the applied in a new model, in which
we were told that event C occurred. So conditional probabilities
do not taste or smell any different than ordinary
probabilities do. Conditional probabilities, given
a specific event B, just form a probability law
on our sample space. It's a different probability
law but it's still a probability law that has all
of the desired properties. OK, so where do conditional
probabilities come up? They do come up in quizzes
and they do come up in silly problems. So let's start with this. We have this example
from last time. Two rolls of a die, all possible
pairs of roles are equally likely, so every element
in this square has probability of 1/16. So all elements are
equally likely. That's our original model. Then somebody comes and tells us
that the minimum of the two rolls is equal to zero. What's that event? The minimum equal to zero can
happen in many ways, if we get two zeros or if we
get a zero and-- sorry, if we get two
two's, or get a two and something larger. And so the is our new event B.
The red event is the event B. And now we want to calculate
probabilities inside this new universe. For example, you may be
interested in the question, questions about the maximum
of the two rolls. In the new universe, what's
the probability that the maximum is equal to one? The maximum being equal to
one is this black event. And given that we're told that
B occurred, this black events cannot happen. So this probability
is equal to zero. How about the maximum
being equal to two, given that event B? OK, we can use the
definition here. It's going to be the probability
that the maximum is equal to two and B occurs
divided by the probability of B. The probability that the
maximum is equal to two. OK, what's the event that the
maximum is equal to two? Let's draw it. This is going to be
the blue event. The maximum is equal to
two if we get any of those blue points. So the intersection of the two
events is the intersection of the red event and
the blue event. There's only one point in
their intersection. So the probability of
that intersection happening is 1/16. That's the numerator. How about the denominator? The event B consists of five
elements, each one of which had probability of 1/16. So that's 5/16. And so the answer is 1/5. Could we have gotten this
answer in a faster way? Yes. Here's how it goes. We're trying to find the
conditional probability that we get this point, given
that B occurred. B consist of five elements. All of those five elements were
equally likely when we started, so they remain equally
likely afterwards. Because when we define
conditional probabilities, we keep the same proportions
inside the set. So the five red elements
were equally likely. They remain equally likely
in the conditional world. So conditional event B having
happened, each one of these five elements has the
same probability. So the probability that we
actually get this point is going to be 1/5. And so that's the shortcut. More generally, whenever you
have a uniform distribution on your initial sample space,
when you condition on an event, your new distribution is
still going to be uniform, but on the smaller events
of that we considered. So we started with a uniform
distribution on the big square and we ended up with a
uniform distribution just on the red point. Now besides silly problems,
however, conditional probabilities show up in real
and interesting situations. And this example is going
to give you some idea of how that happens. OK. Actually, in this example,
instead of starting with a probability model in terms of
regular probabilities, I'm actually going to define the
model in terms of conditional probabilities. And we'll see how
this is done. So here's the story. There may be an airplane flying
up in the sky, in a particular sector of the sky
that you're watching. Sometimes there is one sometimes
there isn't. And from experience you know
that when you look up, there's five percent probability that
the plane is flying above there and 95% probability that
there's no plane up there. So event A is the event that the
plane is flying out there. Now you bought this wonderful
radar that's looks up. And you're told in the
manufacturer's specs that, if there is a plane out there,
your radar is going to register something, a
blip on the screen with probability 99%. And it will not register
anything with probability one percent. So this particular part of the
picture is a self-contained probability model of what your
radar does in a world where a plane is out there. So I'm telling you that the
plane is out there. So we're now dealing with
conditional probabilities because I gave you some
particular information. Given this information that the
plane is out there, that's how your radar is going to
behave with probability 99% is going to detect it, with
probability one percent is going to miss it. So this piece of the picture
is a self-contained probability model. The probabilities
add up to one. But it's a piece of
a larger model. Similarly, there's the
other possibility. Maybe a plane is not up there
and the manufacturer specs tell you something about
false alarms. A false alarm is the situation
where the plane is not there, but for some reason your radar
picked up some noise or whatever and shows a
blip on the screen. And suppose that this happens
with probability ten percent. Whereas with probability
90% your radar gives the correct answer. So this is sort of a model of
what's going to happen with respect to both the plane --
we're given probabilities about this -- and we're given
probabilities about how the radar behaves. So here I have indirectly
specified the probability law in our model by starting with
conditional probabilities as opposed to starting with
ordinary probabilities. Can we derive ordinary
probabilities starting from the conditional number ones? Yeah, we certainly can. Let's look at this event, A
intersection B, which is the event up here, that there
is a plane and our radar picks it up. How can we calculate
this probability? Well we use the definition of
conditional probabilities and this is the probability of
A times the conditional probability of B given A.
So it's 0.05 times 0.99. And the answer, in
case you care-- It's 0.0495. OK. So we can calculate the
probabilities of final outcomes, which are the leaves
of the tree, by using the probabilities that
we have along the branches of the tree. So essentially, what we ended
up doing was to multiply the probability of this
branch times the probability of that branch. Now, how about the answer
to this question. What is the probability
that our radar is going to register something? OK, this is an event that can
happen in multiple ways. It's the event that consists
of this outcome. There is a plane and the radar
registers something together with this outcome, there is no
plane but the radar still registers something. So to find the probability of
this event, we need the individual probabilities
of the two outcomes. For the first outcome, we
already calculated it. For the second outcome, the
probability that this happens is going to be this probability
95% times 0.10, which is the conditional
probability for taking this branch, given that there
was no plane out there. So we just add the numbers. 0.05 times 0.99 plus 0.95
times 0.1 and the final answer is 0.1445. OK. And now here's the interesting
question. Given that your radar recorded
something, how likely is it that there is an airplane
up there? Your radar registering
something -- that can be caused
by two things. Either there's a plane there,
and your radar did its job. Or there was nothing, but your
radar fired a false alarm. What's the probability that this
is the case as opposed to that being the case? OK. The intuitive shortcut would
be that it should be the probability-- you look at their relative odds
of these two elements and you use them to find out how
much more likely it is to be there as opposed
to being there. But instead of doing this,
let's just write down the definition and just use it. It's the probability of A and
B happening, divided by the probability of B. This is just
our definition of conditional probabilities. Now we have already found
the numerator. We have already calculated
the denominator. So we take the ratio of these
two numbers and we find the final answer -- which is 0.34. OK. There's this slightly
curious thing that's happened in this example. Doesn't this number feel
a little too low? My radar -- So this is a conditional
probability, given that my radar said there is something
out there, that there is indeed something there. So it's sort of the probability
that our radar gave the correct answer. Now, the specs of our radar
we're pretty good. In this situation, it gives
you the correct answer 99% of the time. In this situation, it gives
you the correct answer 90% of the time. So you would think
that your radar there is really reliable. But yet here the radar recorded
something, but the chance that the answer that
you get out of this is the right one, given that it
recorded something, the chance that there is an airplane
out there is only 30%. So you cannot really rely on
the measurements from your radar, even though the specs of
the radar were really good. What's the reason for this? Well, the reason is that false
alarms are pretty common. Most of the time there's
nothing. And there's a ten percent
probability of false alarms. So there's roughly a ten percent
probability that in any given experiment, you
have a false alarm. And there is about the five
percent probability that something out there and
your radar gets it. So when your radar records
something, it's actually more likely to be a false
alarm rather than being an actual airplane. This has probability ten
percent roughly. This has probability roughly
five percent So conditional probabilities
are sometimes counter-intuitive in terms of
the answers that they get. And you can make similar
stories about doctors interpreting the results
of tests. So you tested positive for
a certain disease. Does it mean that you have
the disease necessarily? Well if that disease has been
eradicated from the face of the earth, testing positive
doesn't mean that you have the disease, even if the test
was designed to be a pretty good one. So unfortunately, doctors do get
it wrong also sometimes. And the reasoning that
comes in such situations is pretty subtle. Now for the rest of the lecture,
what we're going to do is to take this example where
we did three things and abstract them. These three trivial calculations
that's we just did are three very important,
very basic tools that you use to solve more general
probability problems. So what's the first one? We find the probability of a
composite event, two things happening, by multiplying
probabilities and conditional probabilities. More general version of this,
look at any situation, maybe involving lots and
lots of events. So here's a story that event A
may happen or may not happen. Given that A occurred, it's
possible that B happens or that B does not happen. Given that B also happens, it's
possible that the event C also happens or that event
C does not happen. And somebody specifies for you
a model by giving you all these conditional probabilities
along the way. Notice what we move along
the branches as the tree progresses. Any point in the tree
corresponds to certain events having happened. And then, given that this
has happened, we specify conditional probabilities. Given that this has happened,
how likely is it for that C also occurs? Given a model of this kind, how
do we find the probability or for this event? The answer is extremely
simple. All that you do is move along
with the tree and multiply conditional probabilities
along the way. So in terms of frequencies, how
often do all three things happen, A, B, and C? You first see how often
does A occur. Out of the times that
A occurs, how often does B occur? And out of the times where both
A and B have occurred, how often does C occur? And you can just multiply those
three frequencies with each other. What is the formal
proof of this? Well, the only thing we have in
our hands is the definition of conditional probabilities. So let's just use this. And-- OK. Now, the definition of
conditional probabilities tells us that the probability
of two things is the probability of one of them
times a conditional probability. Unfortunately, here we have the probability of three things. What can I do? I can put a parenthesis in here
and think of this as the probability of this and that
and apply our definition of conditional probabilities
here. The probability of two things
happening is the probability that the first happens times
the conditional probability that the second happens, given
A and B, given that the first one happened. So this is just the definition
of the conditional probability of an event, given
another event. That other event is a
composite one, but that's not an issue. It's just an event. And then we use the definition
of conditional probabilities once more to break this apart
and make it P(A), P(B given A) and then finally,
the last term. OK. So this proves the formula
that I have up there on the slides. And if you wish to calculate
any other probability in this diagram. For example, if you want to
calculate this probability, you would still multiply the
conditional probabilities along the different branches
of the tree. In particular, here in this
branch, you would have the conditional probability of
C complement, given A intersection B complement,
and so on. So you write down probabilities
along all those tree branches and just multiply
them as you go. So this was the first skill
that we are covering. What was the second one? What we did was to calculate
the total probability of a certain event B that
consisted of-- was made up from different
possibilities, which corresponded to different
scenarios. So we wanted to calculate the
probability of this event B that consisted of those
two elements. Let's generalize. So we have our big model. And this sample space
is partitioned in a number of sets. In our radar example, we had
a partition in two sets. Either a plane is there, or
a plane is not there. Since we're trying to
generalize, now I'm going to give you a picture for the case
of three possibilities or three possible scenarios. So whatever happens in the
world, there are three possible scenarios,
A1, A2, A3. So think of these as there's
nothing in the air, there's an airplane in the air, or there's
a flock of geese flying in the air. So there's three possible
scenarios. And then there's a certain event
B of interest, such as a radar records something or
doesn't record something. We specify this model by giving probabilities for the Ai's-- That's the probability of
the different scenarios. And somebody also gives us the
probabilities that this event B is going to occur, given
that the Ai-th scenario has occurred. Think of the Ai's
as scenarios. And we want to calculate the
overall probability of the event B. What's happening
in this example? Perhaps, instead of this
picture, it's easier to visualize if I go back to the
picture I was using before. We have three possible
scenarios, A1, A2, A3. And under each scenario, B may
happen or B may not happen. And so on. So here we have A2 intersection
B. And here we have A3 intersection B. In the
previous slide, we found how to calculate the probability
of any event of this kind, which is done by multiplying
probabilities here and conditional probabilities
there. Now we are asked to calculate
the total probability of the event B. The event B can happen
in three possible ways. It can happen here. It can happen there. And it can happen here. So this is our event B. It
consists of three elements. To calculate the total
probability of our event B, all we need to do is to add
these three probabilities. So B is an event that consists
of these three elements. There are three ways
that B can happen. Either B happens together with
A1, or B happens together with A2, or B happens together
with A3. So we need to add the
probabilities of these three contingencies. For each one of those
contingencies, we can calculate its probability by
using the multiplication rule. So the probability of A1 and
B happening is this-- It's the probability of A1
and then B happening given that A1 happens. The probability of this
contingency is found by taking the probability that A2 happens
times the conditional probability of A2, given
that B happened. And similarly for
the third one. So this is the general rule
that we have here. The rule is written for the
case of three scenarios. But obviously, it has a
generalization for the case of four or five or more
scenarios. It gives you a way of breaking
up the calculation of an event that can happen in multiple ways
by considering individual probabilities for the different
ways that the event can happen. OK. So-- Yes? AUDIENCE: Does this
have to change for infinite sample space? JOHN TSISIKLIS: No. This is true whether
your sample space is infinite or finite. What I'm using in this argument
that we have a partition into just three
scenarios, three events. So it's a partition to a finite
number of events. It's also true if it's a
partition into an infinite sequence of events. But that's, I think, one of the
theoretical problems at the end of the chapter. You probably may not
need it for now. OK, going back to
the story here. There are three possible
scenarios about what could happen in the world that
are captured here. Event, under each scenario,
event B may or may not happen. And so these probabilities tell
us the likelihoods of the different scenarios. These conditional probabilities
tell us how likely is it for B to happen
under one scenario, or the other scenario, or the
other scenario. The overall probability of
B is found by taking some combination of the probabilities
of B in the different possible
worlds, in the different possible scenarios. Under some scenario, B
may be very likely. Under another scenario, it
may be very unlikely. We take all of these into
account and weigh them according to the likelihood
of the scenarios. Now notice that since A1, A2,
and three form a partition, these three probabilities
have what property? Add to what? They add to one. So it's the probability of this
branch, plus this branch, plus this branch. So what we have here is a
weighted average of the probabilities of the B's into
the different worlds, or in the different scenarios. Special case. Suppose the three scenarios
are equally likely. So P of A1 equals 1/3, equals
to P of A2, P of A3. what are we saying here? In that case of equally likely
scenarios, the probability of B is the average of the
probabilities of B in the three different words, or in the
three different scenarios. OK. So to finally, the last step. If we go back again two slides,
the last thing that we did was to calculate a
conditional probability of this kind, probability of
A given B, which is a probability associated
essentially with an inference problem. Given that our radar recorded
something, how likely is it that the plane was up there? So we're trying to infer whether
a plane was up there or not, based on the information
that we've got. So let's generalize once more. And we're just going to rewrite
what we did in that example, but in terms of general
symbols instead of the specific numbers. So once more, the model that we
have involves probabilities of the different scenarios. These we call them prior
probabilities. They're are our initial beliefs
about how likely each scenario is to occur. We also have a model of our
measuring device that tells us under that scenario how likely
is it that our radar will register something or not. So we're given again these
conditional probabilities. We're given the conditional probabilities for these branches. Then we are told that
event B occurred. And on the basis of this new
information, we want to form some new beliefs about the
relative likelihood of the different scenarios. Going back again to our radar
example, an airplane was present with probability 5%. Given that the radar recorded
something, we're going to change our beliefs. Now, a plane is present
with probability 34%. The radar, since we saw
something, we are going to revise our beliefs as to whether
the plane is out there or is not there. And so what we need to do is to
calculate the conditional probabilities of the different
scenarios, given the information that we got. So initially, we have these
probabilities for the different scenarios. Once we get the information,
we update them and we calculate our revised
probabilities or conditional probabilities given the
observation that we made. OK. So what do we do? We just use the definition
of conditional probabilities twice. By definition the conditional
probability is the probability of two things happening divided
by the probability of the conditioning event. Now, I'm using the definition
of conditional probabilities once more, or rather I use
the multiplication rule. The probability of two things
happening is the probability of the first and the second. So these are things that
are given to us. They're the probabilities of
the different scenarios. And it's the model of our
measuring device, which we assume to be available. And how about the denominator? This is total probability of the
event B. But we just found that's it's easy to calculate
using the formula in the previous slide. To find the overall probability
of event B occurring, we look at the
probabilities of B occurring under the different scenario
and weigh them according to the probabilities of
all the scenarios. So in the end, we have a formula
for the conditional probability, A's given B,
based on the data of the problem, which were
probabilities of the different scenarios and conditional
probabilities of B, given the A's. So what this calculation does
is, basically, it reverses the order of conditioning. We are given conditional
probabilities of these kind, where it's B given A and we
produce new conditional probabilities, where things
go the other way. So schematically, what's
happening here is that we have model of cause and
effect and-- So a scenario occurs and that
may cause B to happen or may not cause it to happen. So this is a cause/effect
model. And it's modeled using
probabilities, such as probability of B given Ai. And what we want to do is
inference where we are told that B occurs, and we want
to infer whether Ai also occurred or not. And the appropriate
probabilities for that are the conditional probabilities
that A occurred, given that B occurred. So we're starting with a causal
model of our situation. It models from a given cause how
likely is a certain effect to be observed. And then we do inference, which
answers the question, given that the effect was
observed, how likely is it that the world was in this
particular situation or state or scenario. So the name of the Bayes rule
comes from Thomas Bayes, a British theologian back
in the 1700s. It actually-- This calculation addresses
a basic problem, a basic philosophical problem, how one
can learn from experience or from experimental data and
some systematic way. So the British at that time
were preoccupied with this type of question. Is there a basic theory that
about how we can incorporate new knowledge to previous
knowledge. And this calculation made an
argument that, yes, it is possible to do that in
a systematic way. So the philosophical
underpinnings of this have a very long history and a lot
of discussion around them. But for our purposes, it's just
an extremely useful tool. And it's the foundation of
almost everything that gets done when you try to do
inference based on partial observations. Very well. Till next time.
