The following content is
provided under a Creative Commons license. Your support will help
MIT OpenCourseWare continue to offer high quality
educational resources for free. To make a donation or to
view additional materials from hundreds of
MIT courses, visit MITOpenCourseWare@OCW.MIT.edu PHILIPPE RIGOLLET: So today
WE'LL actually just do a brief chapter on Bayesian statistics. And there's entire courses
on Bayesian statistics, there's entire books
on Bayesian statistics, there's entire careers
in Bayesian statistics. So admittedly, I'm
not going to be able to do it
justice and tell you all the interesting
things that are happening in Bayesian statistics. But I think it's important
as a statistician to know what it
is, how it works, because it's actually
a weapon of choice for many practitioners. And because it allows them to
incorporate their knowledge about a problem in a
fairly systematic manner. So if you look at like, say the
Bayesian statistics literature, it's huge. And so here I give
you sort of a range of what you can expect to
see in Bayesian statistics from your second edition of
a traditional book, something that involves computation,
some things that involve risk thinking. And there's a lot of
Bayesian thinking. There's a lot of
things that you know talking about sort of like
philosophy of thinking Bayesian. This book, for example,
seems to be one of them. This book is
definitely one of them. This one represents sort of
a wide, a broad literature on Bayesian statistics, for
applications for example, in social sciences. But even in large
scale machine learning, there's a lot of Bayesian
statistics happening, particular using something
called Bayesian parametrics, or hierarchical
Bayesian modeling. So we do have some experts
at MIT in the c-cell. Tamara Broderick for
example, is a person who does quite a bit
of interesting work on Bayesian parametrics. And if that's something you
want to know more about, I urge you to go
and talk to her. So before we go into
more advanced things, we need to start with what
is the Bayesian approach. What do Bayesians
do, and how is it different from what
we've been doing so far? So to understand the
difference between Bayesians and what we've been
doing so far is, we need to first put a name on
what we've been doing so far. It's called
frequentist statistics. Which usually Bayesian versus
frequentist statistics, by versus I don't mean
that there is naturally in opposition to them. Actually, often you will
see the same method that comes out of both approaches. So let's see how
we did it, right. The first thing, we had data. We observed some data. And we assumed that this
data was generated randomly. The reason we did
that is because this would allow us to leverage
tools from probability. So let's say by nature,
measurements, you do a survey, you get some data. Then we made some assumptions
on the data generating process. For example, we
assumed they were iid. That was one of the
recurring things. Sometimes we assume
it was Gaussian. If you wanted to
use say, T-test. Maybe we did some
nonparametric statistics. We assume it was a
smooth function or maybe linear regression function. So those are our modeling. And this was basically
a way to say, well, we're not going to allow for
any distributions for the data that we have. But maybe a small
set of distributions that indexed by some small
parameters, for example. Or at least remove some
of the possibilities. Otherwise, there's
nothing we can learn. And so for example,
this was associated to some parameter of
interest, say data or beta in the regression model. Then we had this unknown
problem and this unknown thing, a known parameter. And we wanted to find it. We wanted to either
estimate it or test it, or maybe find a confidence
interval for the subject. So, so far I should not have
said anything that's new. But this last
sentence is actually what's going to be different
from the Bayesian part. And particular, this
unknown but fixed things is what's going to be changing. In the Bayesian
approach, we still assume that we observe
some random data. But the generating process
is slightly different. It's sort of a
two later process. And there's one
process that generates the parameter and
then one process that, given this parameter
generates the data. So what the first layer
does, nobody really believes that there's
some random process that's happening, about
generating what is going to be the true expected
number of people who turn their head to
the right when they kiss. But this is actually going to
be something that brings us some easiness for
us to incorporate what we call prior belief. We'll see an
example in a second. But often, you actually
have prior belief of what this
parameter should be. When we, say least
squares, we looked over all of the vectors
in all of R to the p, including the ones that
have coefficients equal to 50 million. Those are things that we
might be able to rule out. We might be able to rule out
that on a much smaller scale. For example, well
I'm not an expert on turning your head to
the right or to the left. But maybe you can
rule out the fact that almost everybody
is turning their head in the same direction, or almost
everybody is turning their head to another direction. So we have this prior belief. And this belief is going
to play say, hopefully less and less important role as
we collect more and more data. But if we have a
smaller amount of data, we might want to be able
to use this information, rather than just
shooting in the dark. And so the idea is to
have this prior belief. And then, we want to
update this prior belief into what's called the
posterior belief after we've seen some data. Maybe I believe that
there's something that should be in some range. But maybe after I see data, it's
comforting me in my beliefs. So I'm actually having
maybe a belief that's more. So belief encompasses
basically what you think and how strongly
you think about it. That's what I call belief. So for example, if I have a
belief about some parameter theta, maybe my
belief is telling me where theta should
be and how strongly I believe in it, in the sense
that I have a very narrow region where theta could be. The posterior beliefs, as
well, you see some data. And maybe you're more confident
or less confident about what you've seen. Maybe you've shifted
your belief a little bit. And so that's what we're
going to try to see, and how to do this in
a principal manner. To understand this
better, there's nothing better than an example. So let's talk about another
stupid statistical question. Which is, let's try
to understand p. Of course, I'm not going to
talk about politics from now on. So let's talk about p,
the proportion of women in the population. And so what I could do is
to collect some data, X1, Xn and assume that
they're Bernoulli with some parameter, p unknown. So p is in 0, 1. OK, let's assume that
those guys are iid. So this is just an indicator
for each of my collected data, whether the person I randomly
sample is a woman, I get a one. If it's a man, I get a zero. Now the question is, I
sample these people randomly. I do you know their gender. And the frequentist
approach was just saying, OK, let's just estimate
p hat being Xn bar. And then we could do some tests. So here, there's a test. I want to test maybe if
p is equal to 0.5 or not. That sounds like a pretty
reasonable thing to test. But we want to also
maybe estimate p. But here, this is a case where
we definitely prior belief of what p should be. We are pretty confident that
p is not going to be 0.7. We actually believe
that we should be extremely close to one
half, but maybe not exactly. Maybe this population is not
the population in the world. But maybe this is the
population of, say some college and we want to understand if
this college has half women or not. Maybe we know it's going
to be close to one half, but maybe we're not quite sure. We're going to want to
integrate that knowledge. So I could integrate it in
a blunt manner by saying, discard the data and say
that p is equal to one half. But maybe that's just
a little too much. So how do I do this trade
off between adding the data and combining it with
this prior knowledge? In many instances, essentially
what's going to happen is this one half is going to
act like one new observation. So if you have
five observations, this is just the
sixth observation, which will play a role. If you have a
million observations, you're going to have
a million and one. It's not going to play
so much of a role. That's basically how it goes. But, definitely not
always because we'll see that if I take my prior to
be a point minus one half here, it's basically as if I
was discarding my data. So essentially, there's
also your ability to encompass how strongly
you believe in this prior. And if you believe
infinitely more in the prior than you believe in
the data you collected, then it's not going to act
like one more observation. The Bayesian approach
is a tool to one, include mathematically
our prior. And our prior belief into
statistical procedures. Maybe I have this
prior knowledge. But if I'm a medical
doctor, it's not clear to me how I'm going to turn this into
some principal way of building estimators. And the second
goal is going to be to update this prior belief
into a posterior belief by using the data. How do I do this? And at some point,
I sort of suggested that there's two layers. One is where you draw
the parameter at random. And two, once you
have the parameter, conditionless parameter,
you draw your data. Nobody believed this actually is
happening, that nature is just rolling dice for us and
choosing parameters at random. But what's happening
is that, this idea that the parameter comes
from some random distribution actually captures, very
well, this idea that how you would encompass your prior. How would you say, my
belief is as follows? Well here's an example about p. I'm 90% sure that p is
between 0.4 and 0.6. And I'm 95% sure that p
is between 0.3 and 0.8. So essentially, I have
this possible value of p. And what I know is that, there's
90% here between 0.4 and 0.6. And then I have 0.3 and 0.8. And I know that I'm 95%
sure that I'm in here. If you remember, this sort of
looks like the kind of pictures that I made when I had
some Gaussian, for example. And I said, oh here we have
90% of the observations. And here, we have 95%
of the observations. So in a way, if I
were able to tell you all those ranges for
all possible values, then I would essentially
describe a probability distribution for p. And what I'm saying
is that, p is going to have this kind of shape. So of course, if I tell you
only two twice this information that there's 90% I'm here,
and I'm between here and here. And 95%, I'm between here
and here, then there's many ways I can
accomplish that, right. I could have something that
looks like this, maybe. It could be like this. There's many ways
I can have this. Some of them are
definitely going to be mathematically more
convenient than others. And hopefully, we're
going to have things that I can
parameterize very well. Because if I tell
you this is this guy, then there's basically one,
two three, four, five, six, seven parameters. So I probably don't
want something that has seven parameters. But maybe I can say, oh,
it's a Gaussian and I all I have to do is to tell
you where it's centered and what the standard
deviation is. So the idea of using
this two layer thing, where we think of
the parameter p as being drawn from
some distribution, is really just a way for us
to capture this information. Our prior belief
being, well there's this percentage of
chances that it's there. But the percentage of
this chance, I'm not I'm deliberately not using
probability here. So it's really a way
to get close to this. That's why I say, the true
parameter is not random. But the Bayesian approach
does as if it was random. And then, just spits
out a procedure out of this thought process,
this thought experiment. So when you practice
Bayesian statistics a lot, you start getting automatisms. You start getting some things
that you do without really thinking about
it. just like when you you're a statistician,
the first thing you do is, can I think of this data as
being Gaussian for example? When you're Bayesian
you're thinking about, OK I have a set of parameters. So here, I can
describe my parameter as being theta in
general, in some big space parameter of theta. But what spaces
did we encounter? Well, we encountered
the real line. We encountered the interval
0, 1 for Bernoulli's And we encountered some of
the positive real line for exponential
distributions, etc. And so what I'm
going to need to do, if I want to put some
prior on those spaces, I'm going to have to
have a usual set of tools for this guy, usual set
of tools for this guy, usual sort of
tools for this guy. And by usual set
of tools, I mean I'm going to have to have a
family of distributions that's supported on this. So in particular,
this is the speed in which my parameter
that I usually denote by p for Bernoulli lives. And so what I need is to find a
distribution on the interval 0, 1 just like this guy. The problem with the
Gaussian is that it's not on the interval 0, 1. It's going to spill
out in the end. And it's not going to be
something that works for me. And so the question is, I need
to think about distributions that are probably continuous. Why would I restrict myself
to discrete distributions that are actually convenient and for
Bernoulli, one that's actually basically the main tool
that everybody is using is the so-called
beta distribution. So the beta distribution
has two parameters. So x follows a beta
with parameters a and b if it has
a density, f of x is equal to x to the a minus 1. 1 minus x to the b minus 1,
if x is in the interval 0, 1 and 0 for all other x's. OK? Why is that a good thing? Well, it's a density that's
on the interval 0, 1 for sure. But now I have these two
parameters and a set of shapes that I can get by tweaking those
two parameters is incredible. It's going to be a
unimodal distribution. It's still fairly nice. It's not going to be something
that goes like this and this. Because if you think
about this, what would it mean if your prior
distribution of the interval 0, 1 had this shape? It would mean that, maybe
you think that p is here or maybe you think
that p is here, or maybe you think
that p is here. Which essentially
means that you think that p can come from
three different phenomena. And there's other models
that are called mixers for that, that directly
account for the fact that maybe there are several
phenomena that are aggregated in your data set. But if you think that your
data set is sort of pure, and that everything comes
from the same phenomenon, you want something
that looks like this, or maybe looks like this, or
maybe is sort of symmetric. You want to get all this stuff. Maybe you want something
that says, well if I'm talking about p being the
probability of the proportion of women in the whole world, you
want something that's probably really spiked around one half. Almost the point
math, because you know let's agree that 0.5
is the actual number. So you want something that
says, OK maybe I'm wrong. But I'm sure I'm not going
to be really that way off. So you want something
that's really pointy. But if it's something
you've never checked, and again I can not make
references at this point, but something where you might
have some uncertainty that should be around one half. Maybe you want something
that a little more allows you to say, well, I think
there's more around one half. But there's still some
fluctuations that are possible. And in particular
here, I talk about p, where the two parameters a
and b are actually the same. I call them a. One is called scale. The other one is called shape. Oh sorry, this is not a density. So it actually has
to be normalized. When you integrate
this guy, it's going to be some function
that depends on a and b, actually depends
on this function through the beta function. Which is this combination
of gamma function, so that's why it's
called beta distribution. That's the definition of
the beta function when you integrate this thing anyway. You just have to normalize it. That's just a number that
depends on the a and b. So here, if you
take a equal to b, you have something
that essentially is symmetric around one half. Because what does it look like? Well, so my density f of
x, is going to be what? It's going to be my constant
times x, times one minus x to a minus one. And this function, x times
1 minus x looks like this. We've drawn it before. That was something
that showed up as being the variance
of my Bernoulli. So we know it's something that
takes its maximum at one half. And now I'm just taking
a power of this guy. So I'm really just
distorting this thing into some fairly
symmetric manner. This distribution that
we actually take for p. I assume that p, the
parameter, notice that this is kind of weird. First of all, this is
probably the first time in this entire
course that something has a distribution when it's
actually a lower case letter. That's something you
have to deal with, because we've been using lower
case letters for parameters. And now we want them
to have a distribution. So that's what's
going to happen. This is called the
prior distribution. So really, I should write
something like f of p is equal to a constant times
p, 1 minus p, to the n minus 1. Well no, actually I should not
because then it's confusing. One thing in terms
of notation that I'm going to write, when
I have a constant here and I don't want to
make it explicit. And we'll see in a second why I
don't need to make it explicit. I'm going to write
this as f of x is proportional to x 1
minus x to the n minus 1. That's just to say, equal to
some constant that does not depend on x times this thing. So if we continue
with our experiment where I'm drawing
this data, X1 to Xn, which is Bernoulli p, if
p has some distribution it's not clear what it
means to have a Bernoulli with some random parameter. So what I'm going to do is, then
I'm going to first draw my p. Let's say I get a number, 0.52. And then, I'm going to draw
my data conditionally on p. So here comes the first and
last flowchart of this class. So nature first draws p. p follows some data on a, a. Then I condition on p. And then I draw X1, Xn
that are iid, Bernoulli p. Everybody understand the
process of generating this data? So you first draw a
parameter, and then you just flip those independent biased
coins with this particular p. There's this layered thing. Now conditionally p, right so
here I have this prior about p which was the thing. So this is just the
thought process again, it's not anything that
actually happens in practice. This is my way of thinking about
how the data was generated. And from this, I'm going to try
to come up with some procedure. Just like, if your estimator
is the average of the data, you don't have to
understand probability to say that my estimator
is the average of the data. Anyone outside this
room understands that the average
is a good estimator for some average behavior. And they don't need
to think of the data as being a random
variable, et cetera. So same thing, basically. In this case, you can see that
the posterior distribution is still a beta. What it means is that,
I had this thing. Then, I observed my data. And then, I continue
and here I'm going to update my prior
into some posterior distribution, pi. And here, this guy is
actually also a beta. My posterior
distribution, p, is also a beta distribution
with the parameters that are on this slide. And I'll have the space
to reproduce them. So I start the beginning
of this flowchart as having p, which is a prior. I'm going to get
some observations and then, I'm going to
update what my posterior is. This posterior is
basically something that's, in business
statistics was beautiful is as soon as
you have this distribution, it's essentially capturing all
the information about the data that you want for p. And it's not just the point. It's not just an average. It's actually an
entire distribution for the possible
values of theta. And it's not the same
thing as saying, well if theta hat is equal to Xn
bar, in the Gaussian case I know that this is some mean, mu. And then maybe it has
varying sigma squared over n. That's not what I mean by, this
is my posterior distribution. This is not what I mean. This is going to come from
this guy, the Gaussian thing and the central limit theorem. But what I mean is this guy. And this came exclusively
from the prior distribution. If I had another prior,
I would not necessarily have a beta distribution
on the output. So when I have the same
family of distributions at the beginning and at
the end of this flowchart, I say that beta is
a conjugate prior. Meaning I put in beta as a prior
and I get beta as [INAUDIBLE] And that's why betas
are so popular. Conjugate priors
are really nice, because you know that whatever
you put in, what you're going to get in the end is a beta. So all you have to think
about is the parameters. You don't have to check
again what the posterior is going to look like, what the
PDF of this guy is going to be. You don't have to
think about it. You just have to check
what the parameters are. And there's families
of conjugate priors. Gaussian gives
Gaussian, for example. There's a bunch of them. And this is what drives people
into using specific priors as opposed to others. It has nice
mathematical properties. Nobody believes that p is really
distributed according to beta. But it's flexible enough
and super convenient mathematically. Now let's see for one
second, before we actually go any further. I didn't mention A and
B are both in here, A and B are both
positive numbers. They can be anything positive. So here what I did
is that, I updated A into a plus the sum
of my data, and b into b plus n minus
the sum of my data. So that's essentially, a becomes
a plus the number of ones. Well, that's only
when I have a and a. So the first parameters become
itself plus the number of ones. And the second
one becomes itself plus the number of zeros. And so just as a sanity
check, what does this mean? If a it goes to zero, what
is the beta when a goes to 0? We can actually
read this from here. Actually, let's take a goes to-- no. Sorry, let's just do this. I'll do it when we talk
about non-informative prior, because it's a little too messy. How do we do this? How did I get this posterior
distribution, given the prior? How do I update This well this
is called Bayesian statistics. And you've heard this
word, Bayes before. And the way you've heard
it is in the Bayes formula. What was the Bayes formula? The Bayes formula
was telling you that the probability of A, given
B was equal to something that depended on the probability of
B, given A. That's what it was. You can actually either
remember the formula or you can remember
the definition. And this is what p of A
and B divided by p of B. So this is p of B, given A
times p of A divided by p of B. That's what Bayes
formula is telling you. Agree? So now what I want is to have
something that's telling me how this is going to work. What is going to play the
role of those events, A and B? Well one is going
to be, this is going to be the distribution
of my parameter of theta, given that I see the data. And this is going
to tell me, what is the distribution of the
data, given that I know what my parameter if theta is. But that part, if
this is theta and this is the parameter of
theta, this is what we've been doing all along. The distribution of the data,
given the parameter here was n iid Bernoulli p. I knew exactly what their joint
probability mass function is. Then, that was what? So we said that this
is going to be my data and this is going
to be my parameter. So that means that, this is
the probability of my data, given the parameter. This is the probability
of the parameter. What is this? What did we call this? This is the prior. It's just the distribution
of my parameter. Now what is this? Well, this is just
the distribution of the data, itself. This is essentially the
distribution of this, if this was indeed
not conditioned on p. So if I don't condition
on p, this data is going to be a bunch of iid,
Bernoulli with some parameter. But the perimeter
is random, right. So for different realization
of this data set, I'm going to get different
parameters for the Bernoulli. And so that leads to
some sort of convolution. It's not really a
convolution in this case, but it's like some sort of
composition of distributions. I have the randomness that
comes from here and then, the randomness that comes
from realizing the Bernoulli. That's just the
marginal distribution. It actually might be painful to
understand what this is, right. In a way, it's sort of a
mixture and it's not super nice. But we'll see that this
actually won't matter for us. This is going to be some number. It's going to be there. But it will matter
for us, what it is. Because it actually does
not depend on the parameter. And that's all
that matters to us. Let's put some names
on those things. This was very informal. So let's put some actual
names on what we call prior. So what is the formal
definition of a prior, what is the formal
definition of a posterior, and what are the
rules to update it? So I'm going to have my data,
which is going to be X1, Xn. Let's say they are iid, but
they don't actually have to. And so I'm going to
have given, theta. And when I say
given, it's either given like I did in the
first part of this course in all previous chapters,
or conditionally on. If you're thinking like a
Bayesian, what I really mean is conditionally on
this random parameter. It's as if it was
a fixed number. They're going to
have a distribution, X1, Xn is going to
have some distribution. Let's assume for now
it's a PDF, pn of X1, Xn. I'm going to write
theta like this. So for example, what is this? Let's say this is a PDF. It could be a PMF. Everything I say, I'm going to
think of them as being PDF's. I'm going to combine
PDF's with PDF's, but I could combine PDF it PMF, PMF
with PDF's or PMF with PMF. So everywhere you see
a D could be an M. Now I have those things. So what does it mean? So here is an example. X1, Xn or iid, and theta 1. Now I know exactly what the
joint PDF of this thing is. It means that pn of X1, Xn
given theta is equal to what? Well it's 1 over
2pi to the power n e, to the minus sum
from i equal 1 to n of xi minus theta
squared divided by 2. So that's just the joint
distribution of n iid and theta 1, random variables. That's my pn given theta. Now this is what we denoted
by f sub theta before. We had the subscript before, but
now we just put a bar in theta because we want to remember
that this is actually conditioned on theta. But this is just notation. You should just think of this
as being, just the usual thing that you get from some
statistical model. Now, that's going to be pn. Theta has prior
distribution, pi. For example, so think of it
as either PDF or PMF again. For example, pi
of theta was what? Well it was some constant
times theta to the a minus 1, 1 minus theta to a minus 1. So it has some
prior distribution, and that's another PMF. So now I'm given the
distribution of my, x is given theta and given
the distribution of my theta. I'm given this guy. That's this guy. I'm given that guy,
which is my pi. So that's my pn of
X1, Xn given theta. That's my pi of theta. Well, this is just
the integral of pn of X1, Xn times pi
of theta, d theta, over all possible sets of theta. That's just when I
integrate out my theta, or I compute the
marginal distribution, I did this by integrating. That's just basic probability,
conditional probabilities. Then if I had the
PMF, I would just sum over the values of thetas. Now what I want is to
find what's called, so that's the
prior distribution, and I want to find the
posterior distribution. It's pi of theta, given X1, Xn. If I use Bayes' rule
I know that this is pn of X1, Xn, given
theta times pi of theta. And then it's divided
by the distribution of those guys, which I will
write as integral over theta of pn, X1, Xn, given theta
times pi of theta, d theta. Everybody's with me, still? If you're not
comfortable with this, it means that you probably need
to go read your couple of pages on conditional densities
and conditional PMF's from your probably class. There's really not much there. It's just a matter of being able
to define those quantities, f density of x, given y. This is just what's called
a conditional density. You need to understand
what this object is and how it relates to the
joint distribution of x and y, or maybe the distribution of
x or the distribution of y. But it's the same rules. One way to actually
remember this is, this is exactly
the same rules as this. When you see a bar, it's the
same thing as the probability of this and this guy. So for densities,
it's just a comma divided by the second the
probably the second guy. That's it. So if you remember this, you can
just do some pattern matching and see what I just wrote here. Now, I can compute every
single one of these guys. This something I get
from my modeling. So I did not write this. It's not written in the slides. But I give a name to this guy
that was my prior distribution. And that was my
posterior distribution. In chapter three, maybe
what did we call this guy? The one that does not have a
name and that's in the box. What did we call it? AUDIENCE: [INAUDIBLE] PHILLIPE RIGOLLET: It is the
joint distribution of the Xi's. And we gave it a name. AUDIENCE: [INAUDIBLE] PHILLIPE RIGOLLET: It's
the likelihood, right? This is exactly the likelihood. This was the
likelihood of theta. And this is something that's
very important to remember, and that really reminds you
that these things are really not that different. Maximum likelihood estimation
and Bayesian estimation, because your posterior is really
just your likelihood times something that's just putting
some weights on the thetas, depending on where you
think theta should be. If I had, say a maximum
likelihood estimate, and my likelihood and
theta looked like this, but my prior and theta
looked like this. I said, oh I really want
thetas that are like this. So what's going to
happen is that, I'm going to turn this into some
posterior that looks like this. So I'm just really
waiting, this posterior, this is a constant that does
not depend on theta right? Agreed? I integrated over
theta, so theta is gone. So forget about this guy. I have basically, that the
posterior distribution up to scaling, because it has to
be a probability density and not just anything any
function that's positive, is the product of this guy. It's a weighted version
of my likelihood. That's all it is. I'm just weighing
the likelihood, using my prior belief on theta. And so given this guy
a natural estimator, if you follow the maximum
likelihood principle, would be the maximum
of this posterior. Agreed? That would basically be doing
exactly what maximum likelihood estimation is telling you. So it turns out that you can. It's called Maximum
A Posteriori, and I won't talk much
about this, or MAP. That's Maximum a Posteriori. So it's just the
theta hat is the arc max of pi theta, given X1, Xn. And it sounds like it's OK. I'll give you a
density and you say, OK I have a density for all
values of my parameters. You're asking me to
summarize it into one number. I'm just going to take the most
likely number of those guys. But you could summarize
it, otherwise. You could take the average. You could take the median. You could take a
bunch of numbers. And the beauty of
Bayesian statistics is that, you don't have to
take any number in particular. You have an entire
posterior distribution. This is not only telling
you where theta is, but it's actually telling
you the difference if you actually
give as something that gives you the posterior. Now, let's say the theta
is p between 0 and 1. If my posterior distribution
looks like this, or my posterior distribution
looks like this, then those two guys
have one, the same mode. This is the same value. And their symmetric, so they'll
also have the same mean. So these two posterior
distributions give me the same
summary into one number. However clearly, one
is much more confident than the other one. So I might as well just
spit it out as a solution. You can do even better. People actually do things,
such as drawing a random number from this distribution. Say, this is my number. That's kind of
dangerous, but you can imagine you could do this. This is what works. That's what we went through. So here, as you notice I don't
care so much about this part here. Because it does not
depend on theta. So I know that given the
product of those two things, this thing is only the
constant that I need to divide so that when I integrate
this thing over theta, it integrates to one. Because this has to be a
probability density on theta. I can write this and just
forget about that part. And that's what's written
on the top of this slide. This notation, this sort of
weird alpha, or I don't know. Infinity sign
propped to the right. Whatever you want
to call this thing is actually just really
emphasizing the fact that I don't care. I write it because I can,
but you know what it is. In some instances, you have
to compute the integral. In some instances, you don't
have to compute the integral. And a lot of
Bayesian computation is about saying,
OK it's actually really hard to
compute this integral, so I'd rather not doing it. So let me try to find some
methods that will allow me to sample from the
posterior distribution, without having to compute this. And that's what's called
Monte-Carlo Markov chains, or MCMC, and that's
exactly what they're doing. They're just using
only ratios of things, like that for different thetas. And which means that
if you take ratios, the normalizing constant
is gone and you don't need to find this integral. So we won't go into
those details at all. That would be the purpose
of an entire course on Bayesian inference. Actually, even
Bayesian computations would be an entire
course on its own. And there's some very
interesting things that are going on there,
the interface of stats and computation. So let's go back to our example
and see if we can actually compute any of those things. Because it's very nice to give
you some data, some formulas. Let's see if we
can actually do it. In particular, can I
actually recover this claim that the posterior associated
to a beta prior with a Bernoulli likelihood is actually
giving me a beta again? What was my prior? So p was following
a beta AA, which means that p, the density. That was pi of theta. Well I'm going to
write this as pi of p-- was proportional to p to the
A minus 1 times 1 minus p to the A minus 1. So that's the first ingredient
I need to complete my posterior. I really need only two, if I
wanted to bound up to constant. The second one was p hat. We've computed that many times. And we had even a nice
compact way of writing it, which was that pn of X1,
Xn, given the parameter p. So the joint density of my data,
given p, that's my likelihood. The likelihood of p was what? Well it was p to
the sum of Xi's. 1 minus p to the n
minus some of the Xi's. Anybody wants me
to parse this more? Or do you remember seeing
that from maximum likelihood estimation? Yeah? AUDIENCE: [INAUDIBLE] PHILLIPE RIGOLLET: That's
what conditioning does. AUDIENCE: [INAUDIBLE]
previous slide. [INAUDIBLE] bottom
there, it says D pi of t. Shouldn't it be dt pi of t? PHILLIPE RIGOLLET:
So D pi of T is a measure theoretic notation,
which I used without thinking. And I should not because
I can see it upsets you. D pi of T is just a
natural way to say that I integrate
against whatever I'm given for the prior of theta. In particular, if theta is just
the mix of a PDF and a point mass, maybe I say
that my p takes value 0.5 with probability 0.5. And then is uniform on the
interval with probability 0.5. For this, I neither
have a PDF nor a PMF. But I can still talk about
integrating with respect to this, right? It's going to look like, if
I take a function f of T, D pi of T is going to be
one half of f of one half. That's the point mass
with probability one half, at one half. Plus one half of the integral
between 0 and 1, of f of TDT. This is just the notation, which
is actually funnily enough, interchangeable with pi of DT. But if you have a
density, it's really just the density pi of TDT. If pi is really a
density, but that's when it's when pi is and
measure and not a density. Everybody else,
forget about this. This is not something
you should really worry about at this point. This is more graduate
level probability classes. But yeah, it's called
measure theory. And that's when you think
of pi as being a measure in an abstract fashion. You don't have to worry
whether it's a density or not, or whether
it has a density. So everybody is OK with this? Now I need to
compute my posterior. And as I said, my
posterior is really just the product of
the likelihood weighted by the prior. Hopefully, at this stage
of your application, you can multiply two functions. So what's happening is,
if I multiply this guy with this guy, p gets
this guy to the power this guy plus this guy. And then 1 minus p gets the
power n minus some of Xi's. So this is always
from I equal 1 to n. And then plus A minus 1 as well. This is up to constant, because
I still need to solve this. And I could try to do it. But I really don't
have to, because I know that if my density
has this form, then it's a beta distribution. And then I can just
go on Wikipedia and see what should be
the normalization factor. But I know it's going to
be a beta distribution. It's actually the
beta with parameter. So this is really my beta
with parameter, sum of Xi, i equal 1 to n plus A minus 1. And then the second
parameter is n minus sum of the Xi's plus A minus 1. I just wrote what was here. What happened to my one? Oh no, sorry. Beta has the power minus 1. So that's the
parameter of the beta. And this is the
parameter of the beta. Beta is over there, right? So I just replace
A by what I see. A is just becoming
this guy plus this guy and this guy plus this guy. Everybody is comfortable
with this computation? We just agreed that beta priors
for Bernoulli observations are certainly convenient. Because they are just
conjugate, and we know that's what is going
to come out in the end. That's going to
be a beta as well. I just claim it was convenient. It was certainly convenient
to compute this, right? There was certainly
some compatibility when I had to multiply this
function by that function. And you can imagine that things
could go much more wrong, than just having p to some power
and p to some power, 1 minus p to some power, when it might
just be some other power. Things were nice. Now this is nice, but I can also
question the following things. Why beta, for one? The beta tells me something. That's convenient, but
then how do I pick A? I know that A should definitely
capture the fact that where I want to have my p
most likely located. But it also actually
also captures the variance of my beta. And so choosing
different As is going to have different functions. If I have A and B, If I started
with the beta with parameter. If I started with a B here, I
would just pick up the B here. Agreed? And that would just
be a symmetric. But they're going to
capture mean and variance of this thing. And so how do I pick those guys? If I'm a doctor and
you're asking me, what do you think the
chances of this drug working in this kind of patients is? And I have to spit out the
parameters of a beta for you, it might be a bit of a
complicated thing to do. So how do you do this,
especially for problems? So by now, people
have actually mastered the art of coming up with how
to formulate those numbers. But in new problems that
come up, how do you do this? What happens if you want
to use Bayesian methods, but you actually do not
know what you expect to see? To be fair, before we started
this class, I hope all of you had no idea whether people tend
to bend their head to the right or to the left before kissing. Because if you did, well
you have too much time on your hands and I should
double your homework. So in this case,
maybe you still want to use the Bayesian machinery. Maybe you just want
to do something nice. It's nice right, I mean
it worked out pretty well. What if you want to do? Well you actually want
to use some priors that carry no information, that
basically do not prefer any theta to another theta. Now, you could read
this slide or you could look at this formula. We just said that this
pi here was just here to weigh some thetas more
than others, depending on their prior belief. If our prior belief
does not want to put any preference towards
some thetas than to others, what do I do? AUDIENCE: [INAUDIBLE] PHILLIPE RIGOLLET:
Yeah, I remove it. And the way to remove
something we multiply by, is just replace it by one. That's really what we're doing. If this was a constant
not depending on theta, then that would mean that
we're not preferring any theta. And we're looking
at the likelihood. But not as a function that
we're trying to maximize, but it is a function that
we normalize in such a way that it's actually
a distribution. So if I have pi,
which is not here, this is really just taking
the like likelihood, which is a positive function. It may not integrate
to 1, so I normalize it so that it integrates to 1. And then I just say, well this
is my posterior distribution. Now I could just
maximize this thing and spit out my maximum
likelihood estimator. But I can also
integrate and find what the expectation
of this guy is. I can find what the
median of this guy is. I can sample data from this guy. I can build, understand what
the variance of this guy is. Which is something we did
not do when we just did maximum likelihood estimation
because given a function, all we cared about was the
arc max of this function. These priors are
called uninformative. This is just replacing this
number by one or by a constant. Because it still
has to be a density. If I have a bounded
set, I'm just looking for the
uniform distribution on this bounded set, the
one that puts constant one over the size of this thing. But if I have an
invalid set, what is the density that
takes a constant value on the entire real
line, for example? What is this density? AUDIENCE: [INAUDIBLE] PHILLIPE RIGOLLET:
Doesn't exist, right? It just doesn't exist. The way you can think
of it is a Gaussian with the variance going
to infinity, maybe, or something like this. But you can think
of it in many ways. You can think of the limit of
the uniform between minus T and T, with T going to infinity. But this thing is actually zero. There's nothing there. You can actually
still talk about this. You could always talk
about this thing, where you think of this guy
as being a constant, remove this thing from this
equation, and just say, well my posterior is
just the likelihood divided by the integral of
the likelihood over theta. And if theta is the entire
real line, so be it. As long as this
integral converges, you can still talk
about this stuff. This is what's called
an improper prior. An improper prior is just a
non-negative function defined in theta, but it does not have
to integrate neither to one, nor to anything. If I integrate the
function equal to 1 on the entire real
line, what do I get? Infinity. It's not a proper prior, and
it's called and improper prior. And those improper
priors are usually what you see when you start
to want non-informative priors on infinite sets of datas. That's just the nature of it. You should think of them as
being the uniform distribution of some infinite set, if
that thing were to exist. Let's see some examples
about non-informative priors. If I'm in the interval 0,
1 this is a finite set. So I can talk about
the uniform prior on the interval 0, 1 for a
parameter, p of a Bernoulli. If I want to talk
about this, then it means that my prior is p follows
some uniform on the interval 0, 1. So that means that f of
x is 1 if x is in 0, 1. Otherwise, there is actually
not even a normalization. This thing integrates to 1. And so now if I look
at my likelihood, it's still the same thing. So my posterior
becomes theta X1, Xn. That's my posterior. I don't write the
likelihood again, because we still have it-- well we don't have
it here anymore. The likelihood is given here. Copy, paste over there. The posterior is just
this thing times 1. So you will see it in a second. So it's p to the power sum
of the Xi's, one minus p to the power, n minus
sum of the Xi's. And then it's multiplied by
1, and then divided by this integral between 0 and
1 of p, sum of the Xi's. 1 minus p, n minus
sum of the Xi's. Dp, which does not depend on p. And I really don't care
what the thing actually is. That's posterior of p. And now I can see,
well what is this? It's actually just the
beta with parameters. This guy plus 1. And this guy plus 1. I didn't tell you what the
expectation of a beta was. We don't know what the
expectation of a beta is, agreed? If I wanted to find say, the
expectation of this thing that would be some good
estimator, we know that the maximum
of this guy-- what is the maximum of this thing? Well, it's just this thing,
it's the average of the Xi's. That's just the maximum
likelihood estimator for Bernoulli. We know it's the average. Do you think if I take the
expectation of this thing, I'm going to get the average? So actually, I'm not
going to get the average. I'm going to get this guy plus
this guy, divided by n plus 1. Let's look at what
this thing is doing. It's looking at the number
of ones and it's adding one. And this guy is looking
at the number of zeros and it's adding one. Why is it adding this one? What's going on here? This is going to matter
mostly when the number of ones is actually zero, or the
number of zeros is zero. Because what it does is just
pushes the zero from non-zero. And why is that something that
this Bayesian method actually does for you automatically? It's because when we
put this non-informative prior on p, which was
uniform on the interval 0, 1. In particular, we know
that the probability that p is equal to 0 is zero. And the probability p
is equal to 1 is zero. And so the problem
is that if I did not add this 1 with some
positive probability, I wouldn't be allowed to spit
out something that actually had p hat, which was equal to 0. If by chance, let's say
I have n is equal to 3, and I get only 0, 0, 0, that
could happen with probability. 1 over pq, one over 1 minus pq. That's not something
that I want. And I'm using my priors. My prior is not informative,
but somehow it captures the fact that I don't want to
believe p is going to be either equal to 0 or 1. So that's sort of
taken care of here. So let's move away a little
bit from the Bernoulli example, shall we? I think we've seen enough of it. And so let's talk about
the Gaussian model. Let's say I want to
do Gaussian inference. I want to do inference
in a Gaussian model, using Bayesian methods. What I want is that Xi,
X1, Xn, or say 0, 1 iid. Sorry, theta 1, iid
conditionally on theta. That means that pn of
X1, Xn, given theta is equal to exactly
what I wrote before. So 1 square root to pi, to the
n exponential minus one half sum of Xi minus theta squared. So that's just the
joint distribution of my Gaussian with mean data. And the another
question is, what is the posterior distribution? Well here I said, let's use
the uninformative prior, which is an improper prior. It puts weight on everyone. That's the so-called uniform
on the entire real line. So that's certainly
not a density. But it can still just use this. So all I need to do
is get this divided by normalizing this thing. But if you look at
this, essentially I want to understand. So this is proportional
to the exponential minus one half
sum from I equal 1 to n of Xi minus theta squared. And now I want to see
this thing as a density, not on the Xi's but on theta. What I want is a
density on theta. So it looks like I have
chances of getting something that looks like a Gaussian. To have a Gaussian, I would
need to see minus one half. And then I would need to
see theta minus something here, not just the sum of
something minus thetas. So I need to work
a little bit more, to expand the square here. So this thing here
is going to be equal to exponential minus
one half sum from I equal 1 to n of Xi squared minus 2Xi
theta plus theta squared. Now what I'm going to do
is, everything remember is up to this little sign. So every time I see a term
that does not depend on theta, I can just push it in there
and just make it disappear. Agreed? This term here, exponential
minus one half sum of Xi squared, does it
depend on theta? No. So I'm just pushing it here. This guy, yes. And the other one, yes. So this is proportional to
exponential sum of the Xi. And then I'm going to pull out
my theta, the minus one half canceled with the minus 2. And then I have minus
one half sum from I equal 1 to n of theta squared. Agreed? So now what this
thing looks like, this looks very much like some
theta minus something squared. This thing here is really
just n over 2 times theta. Sorry, times theta squared. So now what I need to do is to
write this of the form, theta minus something. Let's call it mu, squared,
divided by 2 sigma squared. I want to turn this into
that, maybe up to terms that do not depend on theta. That's what I'm
going to try to do. So that's called
completing the squaring. That's some exercises you do. You've done it probably,
already in the homework. And that's something
you do a lot when you do Bayesian
statistics, in particular. So let's do this. What is it going to
be the leading term? Theta squared is going to
be multiplied by this thing. So I'm going to pull
out my n over 2. And then I'm going to write
this as minus theta over 2. And then I'm going to write
theta minus something squared. And this something is going
to be one half of what I see in the cross-product. I need to actually
pull this thing out. So let me write it
like that first. So that's theta squared. And then I'm going to write it
as minus 2 times 1 over n sum from I equal 1 to n
of Xi's times theta. That's exactly just a rewriting
of what we had before. And that should look
much more familiar. A squared minus 2 blap A,
and then I missed something. So this thing, I'm going
to be able to rewrite as theta minus Xn bar squared. But then I need to remove
the square of Xn bar. Because it's not here. So I just complete the square. And then I actually really don't
care with this thing actually was, because it's going to go
again in the little Alpha's sign over there. So this thing
eventually is going to be proportional
to exponential of minus n over 2 times theta
of minus Xn bar squared. And so we know that if
this is a density that's proportional to this guy, it has
to be some n with mean, Xn bar. And variance, this is supposed
to be 1 over sigma squared. This guy over here, this n. So that's really just 1 over n. So the posterior
distribution is a Gaussian centered at the average
of my observations. And with variance, 1 over n. Everybody's with me? Why I'm saying this, this was
the output of some computation. But it sort of
makes sense, right? It's really telling me that
the more observations I have, the more concentrated
this posterior is. Concentrated around what? Well around this Xn bar. That looks like something
we've sort of seen before. But it does not have the
same meaning, somehow. This is really just the
posterior distribution. It's sort of a sanity check,
that I have this 1 over n when I have Xn bar. But it's not the
same thing as saying that the variance of Xn bar was
1 over n, like we had before. As an exercise,
I would recommend if you don't get it,
just try pi of theta to be equal to some n mu 1. Here, the prior that we used
was completely non-informative. What happens if I take my prior
to be some Gaussian, which is centered at mu and
it has the same variance as the other guys? So what's going to
happen here is that we're going to put a weight. And everything
that's away from mu is going to actually
get less weight. I want to know how I'm
going to be updating this prior into a posterior. Everybody sees what
I'm saying here? So that means that pi of theta
has the density proportional to exponential minus one
half theta minus mu squared. So I need to multiply
my posterior with this, and then see. It's actually going
to be a Gaussian. This is also a conjugate prior. It's going to spit
out another Gaussian. You're going to have to complete
a square again, and just check what it's actually giving you. And so spoiler alert,
it's going to look like you get an extra
observation, which is actually equal to mu. It's going to be the average
of n plus 1 observations. The first n1's being X1 to Xn. And then, the last one being mu. And it sort of makes sense. That's actually a
fairly simple exercise. Rather than going
into more computation, this is something
you can definitely do when you're in the
comfort of your room. I want to talk about
other types of priors. The first thing I said is,
there's this beta prior that I just pulled out of my hat
and that was just convenient. Then there was this
non-informative prior. It was convenient. It was non-informative, so
if you don't know anything else maybe that's
what you want to do. The question is, are there
any other priors that are sort of principled
and generic, in the sense that the uninformative
prior was generic, right? It was equal to 1, that's
as generic as it gets. So is there anything
that's generic as well? Well, there's this priors that
are called Jeffrey's priors. And Jeffrey's prior, which is
proportional to square root of the determinant of the
Fisher information of theta. This is actually a
weird thing to do. It says, look at your model. Your model is going to
have a Fisher information. Let's say it exists. Because we know it
does not always exist. For example, in the
multinomial model, we didn't have a
Fisher information. The determinant of
a matrix is somehow measuring the size of a matrix. If you don't trust
me, just think about the matrix being
of size one by one, then the determinant is just
the number that you have there. And so this is really something
that looks like the Fisher information. It's proportional to the
amount of information that you have at
a certain point. And so what my prior
is saying well, I want to put more weights
on those thetas that are going to just extract more
information from the data. You can actually
compute those things. In the first example,
Jeffrey's prior is something that
looks like this. In one dimension,
Fisher information is essentially one
the word variance. That's just 1 over the
square root of the variance, because I have the square root. And when I have the Jeffrey's
prior, when I have the Gaussian case, this is the
identity matrix that I would have in
the Gaussian case. The determinant of
the identities is 1. So square root of 1 is 1, and
so I would basically get 1. And that gives me my improper
prior, my uninformative prior that I had. So the uninformative
prior 1 is fine. Clearly, all the thetas
carry the same information in the Gaussian model. Whether I translate
it here or here, it's pretty clear none
of them is actually better than the other. But clearly for
the Bernoulli case, the p's that are closer
to the boundary carry more information. I sort of like those
guys, because they just carry more information. So what I do is, I
take this function. So p1 minus p. Remember, it's something
that looks like this. On the interval 0, 1. This guy, 1 over square
root of p1 minus p is something that
looks like this. Agreed What it's doing is
sort of wants to push towards the piece that actually
carry more information. Whether you want to
bias your data that way or not, is something
you need to think about. When you put a prior on your
data, on your parameter, you're sort of biasing
towards this idea your data. That's maybe not
such a good idea, when you have some p that's
actually close to one half, for example. You're actually
saying, no I don't want to see a p that's
close to one half. Just make a decision,
one way or another. But just make a decision. So it's forcing you to do that. Jeffrey's prior, I'm
running out of time so I don't want to go
into too much detail. We'll probably stop
here, actually. So Jeffrey's priors have
this very nice property. It's that they actually do not
care about the parameterization of your space. If you actually have
p and you suddenly decide that p is not the
right parameter for Bernoulli, but it's p squared. You could decide to
parameterize this by p squared. Maybe your doctor is
actually much more able to formulate some prior
assumption on p squared, rather than p. You never know. And so what happens is
that Jeffrey's priors are an invariant in this. And the reason is because
the information carried by p is the same as the information
carried by p squared, somehow. They're essentially
the same thing. You need to have one to one map. Where you basically for
each parameter, before you have another parameter. Let's call Eta the
new parameters. The PDF of the new prior
indexed by Eta this time is actually also
Jeffrey's prior. But this time, the
new Fisher information is not the Fisher information
with respect to theta. But it's this Fisher
information associated to this statistical
model indexed by Eta. So essentially, when you
change the parameterization of your model, you still
get Jeffrey's prior for the new parameterization. Which is, in a way,
a desirable property. Jeffrey's prior is just
an uninformative priors, or priors you want
to use when you want a systematic way without
really thinking about what to pick for your mile. I'll finish this next time. And we'll talk about
Bayesian confidence regions. We'll talk about
Bayesian estimation. Once I have a posterior,
what do I get? And basically, the
only message is going to be that you
might want to integrate against the posterior. Find the posterior, the
expectation of your posterior distribution. That's a good point
estimator for theta. We'll just do a
couple of computation.
