The following content is
provided under a Creative Commons license.
Your support will help MIT OpenCourseWare continue to offer
high quality educational resources for free.
To make a donation or to view additional materials from
hundreds of MIT courses, visit MIT OpenCourseWare at
ocw.mit.edu. Today we are going to see how
to use what we saw last time about partial derivatives to
handle minimization or maximization problems involving
functions of several variables. Remember last time we said that
when we have a function, say, of two variables, x and y,
then we have actually two different derivatives,
partial f, partial x, also called f sub x,
the derivative with respect to x keeping y constant.
And we have partial f, partial y, also called f sub y,
where we vary y and we keep x as a constant.
And now, one thing I didn't have time to tell you about but
hopefully you thought about in recitation yesterday,
is the approximation formula that tells you what happens if
you vary both x and y. f sub x tells us what happens
if we change x a little bit, by some small amount delta x.
f sub y tells us how f changes, if you change y by a small
amount delta y. If we do both at the same time
then the two effects will add up with each other,
because you can imagine that first you will change x and then
you will change y. Or the other way around.
It doesn't really matter. If we change x by a certain
amount delta x, and if we change y by the
amount delta y, and let's say that we have z=
f(x, y) then that changes by an amount which is approximately f
sub x times delta x plus f sub y times delta y.
And that is one of the most important formulas about partial
derivatives. The intuition for this,
again, is just the two effects of if I change x by a small
amount and then I change y. Well, first changing x will
modify f, how much does it modify f?
The answer is the rate change is f sub x.
And if I change y then the rate of change of f when I change y
is f sub y. So all together I get this
change as a value of f. And, of course,
that is only an approximation formula.
Actually, there would be higher order terms involving second and
third derivatives and so on. One way to justify this --
Sorry. I was distracted by the
microphone. OK.
How do we justify this formula? Well, one way to think about it
is in terms of tangent plane approximation.
Let's think about the tangent plane with regard to a function
f. We have some pictures to show
you. It will be easier if I show you
pictures. Remember, partial f,
partial x was obtained by looking at the situation where y
is held constant. That means I am slicing the
graph of f by a plane that is parallel to the x,
z plane. And when I change x,
z changes, and the slope of that is going to be the
derivative with respect to x. Now, if I do the same in the
other direction then I will have similarly the slope in a slice
now parallel to the y, z plane that will be partial f,
partial y. In fact, in each case,
I have a line. And that line is tangent to the
surface. Now, if I have two lines
tangent to the surface, well, then together they
determine for me the tangent plane to the surface.
Let's try to see how that works. We know that f sub x and f sub
y are the slopes of two tangent lines to this plane,
two tangent lines to the graph. And let's write down the
equations of these lines. I am not going to write
parametric equations. I am going to write them in
terms of x, y, z coordinates.
Let's say that partial f of a partial x at the given point is
equal to a. That means that we have a line
given by the following conditions.
I am going to keep y constant equal to y0.
And I am going to change x. And, as I change x,
z will change at the rate that is equal to a.
That would be z = 0 a(x - x0). That is how you would describe
a line that, I guess, the one that is plotted in
green here, been dissected with the slice parallel to the x,
z plane. I hold y constant equal to y0.
And z is a function of x that varies with a rate of a.
And now if I look similarly at the other slice,
let's say that the partial with respect to y is equal to b,
then I get another line which is obtained by the fact that z
now will depend on y. And the rate of change with
respect to y will be b. While x is held constant equal
to x0. These two lines are both going
to be in the tangent plane to the surface. They are both tangent to the
graph of f and together they determine the plane. And that plane is just given by
the formula z = z0 a( x - x0) b ( y - y0).
If you look at what happens -- This is the equation of a plane.
z equals constant times x plus constant times y plus constant.
And if you look at what happens if I hold y constant and vary x,
I will get the first line. If I hold x constant and vary
y, I get the second line. Another way to do it,
of course, would provide actually
parametric equations of these lines,
get vectors along them and then take the cross-product to get
the normal vector to the plane. And then get this equation for
the plane using the normal vector.
That also works and it gives you the same formula.
If you are curious of the exercise, do it again using
parametrics and using cross-product to get the plane
equation. That is how we get the tangent
plane. And now what this approximation
formula here says is that, in fact, the graph of a
function is close to the tangent plane.
If we were moving on the tangent plane,
this would be an actual equality.
Delta z would be a linear function of delta x and delta y.
And the graph of a function is near the tangent plane,
but is not quite the same, so it is only an approximation
for small delta x and small delta y.
The approximation formula says the graph of f is close to its
tangent plane. And we can use that formula
over here now to estimate how the value of f changes if I
change x and y at the same time. Questions about that?
Now that we have caught up with what we were supposed to see on
Tuesday, I can tell you now about max and min problems. That is going to be an
application of partial derivatives to look at
optimization problems. Maybe ten years from now,
when you have a real job, your job might be to actually
minimize the cost of something or maximize the profit of
something or whatever. But typically the function that
you will have to strive to minimize or maximize will depend
on several variables. If you have a function of one
variable, you know that to find its minimum or its maximum you
look at the derivative and set that equal to zero.
And you try to then look at what happens to the function.
Here it is going to be kind of similar, except,
of course, we have several derivatives.
For today we will think about a function of two variables,
but it works exactly the same if you have three variables,
ten variables, a million variables.
The first observation is that if we have a local minimum or a
local maximum then both partial derivatives,
so partial f partial x and partial f partial y,
are both zero at the same time. Why is that?
Well, let's say that f of x is zero.
That means when I vary x to first order the function doesn't
change. Maybe that is because it is
going through... If I look only at the slice
parallel to the x-axis then maybe I am going through the
minimum. But if partial f,
partial y is not 0 then actually, by changing y,
I could still make a value larger or smaller.
That wouldn't be an actual maximum or minimum.
It would only be a maximum or minimum if I stay in the slice.
But if I allow myself to change y that doesn't work.
I need actually to know that if I change y the value will not
change either to first order. That is why you also need
partial f, partial y to be zero. Now, let's say that they are
both zero. Well, why is that enough?
It is essentially enough because of this formula telling
me that if both of these guys are zero then to first order the
function doesn't change. Then, of course,
there will be maybe quadratic terms that will actually turn
that, you know, this won't really say that your
function is actually constant. It will just tell you that
maybe it will actually be quadratic or higher order in
delta x and delta y. That is what you expect to have
at a maximum or a minimum. The condition is the same thing
as saying that the tangent plane to the graph is actually going
to be horizontal. And that is what you want to
have. Say you have a minimum,
well, the tangent plane at this point, at the bottom of the
graph is going to be horizontal. And you can see that on this
equation of a tangent plane, when both these coefficients
are 0 that is when the equation becomes z equals constant:
the horizontal plane. Does that make sense?
We will have a name for this kind of point because,
actually, what we will see very soon is
that these conditions are necessary but are not
sufficient. There are actually other kinds
of points where the partial derivatives are zero.
Let's give a name to this. We say the definition is (x0,
y0) is a critical point of f -- -- if the partial derivative,
with respect to x, and partial derivative with
respect to y are both zero. Generally, you would want all
the partial derivatives, no matter how many variables
you have, to be zero at the same time.
Let's see an example. Let's say I give you the
function f(x;y)= x^2 - 2xy 3y^2 2x - 2y.
And let's try to figure out whether we can minimize or
maximize this. What we would start doing
immediately is taking the partial derivatives.
What is f sub x? It starts with 2x - 2y 0 2.
Remember that y is a constant so this differentiates to zero.
Now, if we do f sub y, that is going to be 0-2x 6y-2.
And what we want to do is set these things to zero.
And we want to solve these two equations at the same time.
An important thing to remember, and maybe I should have told
you a couple of weeks ago already,
if you have two equations to solve, well,
it is very good to try to simplify them by adding them
together or whatever, but you must keep two equations.
If you have two equations, you shouldn't end up with just
one equation out of nowhere. For example here,
we can certainly simplify things by summing them together.
If we add them together, well, the x's cancel and the
constants cancel. In fact, we are just left with
4y for zero. That is pretty good.
That tells us y should be zero. But then we should,
of course, go back to these and see what else we know.
Well, now it tells us, if you put y = 0 it tells you
2x 2 = 0. That tells you x = - 1.
We have one critical point that is (x, y) = (- 1;
0). Any questions so far?
No. Well, you should have a
question. The question should be how do
we know if it is a maximum or a minimum?
Yeah. If we had a function of one
variable, we would decide things based on the second derivative.
And, in fact, we will see tomorrow how to do
things based on the second derivative.
But that is kind of tricky because there are a lot of
second derivatives. I mean we already have two
first derivatives. You can imagine that if you
keep taking partials you may end up with more and more,
so we will have to figure out carefully what the condition
should be. We will do that tomorrow.
For now, let's just try to look a bit at how do we understand
these things by hand? In fact, let me point out to
you immediately that there is more than maxima and minima.
Remember, we saw the example of x^2 y^2.
That has a critical point. That critical point is
obviously a minimum. And, of course,
it could be a local minimum because it could be that if you
have a more complicated function there is indeed a minimum here,
but then elsewhere the function drops to a lower value.
We call that just a local minimum to say that it is a
minimum if you stick two values that are close enough to that
point. Of course, you also have local
maximum, which I didn't plot, but it is easy to plot.
That is a local maximum. But there is a third example of
critical point, and that is a saddle point.
The saddle point, it is a new phenomena that you
don't really see in single variable calculus.
It is a critical point that is neither a minimum nor a maximum
because, depending on which direction you look in,
it's either one or the other. See the point in the middle,
at the origin, is a saddle point.
If you look at the tangent plane to this graph,
you will see that it is actually horizontal at the
origin. You have this mountain pass
where the ground is horizontal. But, depending on which
direction you go, you go up or down.
So, we say that a point is a saddle point if it is neither a
minimum or a maximum. Possibilities could be a local
min, a local max or a saddle. Tomorrow we will see how to
decide which one it is, in general, using second
derivatives. For this time,
let's just try to do it by hand.
I just want to observe, in fact, I can try to,
you know, these examples that I have
here, they are x^2 y^2, y^2 - x^2,
they are sums or differences of squares.
And, if we know that we can put things as sum of squares for
example, we will be done. Let's try to express this maybe
as the square of something. The main problem is this 2xy.
Observe we know something that starts with x^2 - 2xy but is
actually a square of something else.
It would be x^2 - 2xy y^2, not plus 3y2.
Let's try that. So, we are going to complete
the square. I am going to say it is x minus
y squared, so it gives me the first two terms and also the y2.
Well, I still need to add two more y^2, and I also need to
add, of course, the 2x and - 2y.
It is still not simple enough for my taste.
I can actually do better. These guys look like a sum of
squares, but here I have this extra stuff, 2x - 2y.
Well, that is 2 (x - y). It looks like maybe we can
modify this and make this into another square.
So, in fact, I can simplify this further to
(x - y 1)^2. That would be (x - y)^2 2( x -
y), and then there is a plus one.
Well, we don't have a plus one so let's remove it by
subtracting one. And I still have my 2y^2.
Do you see why this is the same function?
Yeah. Again, if I expand x minus y
plus one squared, I get (x - y)^2 2 (x - y) 1.
But I will have minus one that will cancel out and then I have
a plus 2y^2. Now, what I know is a sum of
two squared minus one. And this critical point,
(x,y) = (-1;0), that is actually when this is
zero and that is zero, so that is the smallest value.
This is always greater or equal to zero, the same with that one,
so that is always at least minus one.
And minus one happens to be the value at the critical point.
So, it is a minimum. Now, of course here I was very
lucky. I mean, generally,
I couldn't expect things to simplify that much.
In fact, I cheated. I started from that,
I expanded, and then that is how I got my example.
The general method will be a bit different,
but you will see it will actually also involve completing
squares. Just there is more to it than
what we have seen. We will come back to this
tomorrow. Sorry?
How do I know that this equals -- How do I know that the whole
function is greater or equal to negative one?
Well, I wrote f of x, y as something squared plus
2y^2 - 1. This squared is always a
positive number and not a negative.
It is a square. The square of something is
always non-negative. Similarly, y^2 is also always
non-negative. So if you add something that is
at least zero plus something that is at least zero and you
subtract one, you get always at least minus
one. And, in fact,
the only way you can get minus one is if both of these guys are
zero at the same time. That is how I get my minimum.
More about this tomorrow. In fact,
what I would like to tell you about now instead is a nice
application of min, max problems that maybe you
don't think of as a min, max problem that you will see.
I mean you will think of it that way because probably your
calculator can do it for you or, if not, your computer can do it
for you. But it is actually something
where the theory is based on minimization in two variables.
Very often in experimental sciences you have to do
something called least-squares intercalation.
And what is that about? Well, it is the idea that maybe
you do some experiments and you record some data.
You have some data x and some data y.
And, I don't know, maybe, for example,
x is -- Maybe your measuring frogs and you're trying to
measure how bit the frog leg is compared to the eyes of the
frog, or you're trying to measure
something. And if you are doing chemistry
then it could be how much you put of some reactant and how
much of the output product that you wanted to synthesize
generated. All sorts of things.
Make up your own example. You measure basically,
for various values of x, what the value of y ends up
being. And then you like to claim
these points are kind of aligned.
And, of course, to a mathematician they are not
aligned. But, to an experimental
scientist, that is evidence that there is a relation between the
two. And so you want to claim -- And
in your paper you will actually draw a nice little line like
that. The functions depend linearly
on each of them. The question is how do we come
up with that nice line that passes smack in the middle of
the points? The question is,
given experimental data xi, yi -- Maybe I should actually
be more precise. You are given some experimental
data. You have data points x1,
y1, x2, y2 and so on, xn, yn,
the question would be find the "best fit"
line of a form y equals ax b that somehow approximates very
well this data. You can also use that right
away to predict various things. For example,
if you look at your new homework,
actually the first problem asks you to predict how many iPods
will be on this planet in ten years looking at past sales and
how they behave. One thing, right away,
before you lose all the money that you don't have yet,
you cannot use that to predict the stock market.
So, don't try to use that to make money.
It doesn't work. One tricky thing here that I
want to draw your attention to is what are the unknowns here?
The natural answer would be to say that the unknowns are x and
y. That is not actually the case.
We are not going to solve for some x and y.
I mean we have some values given to us.
And, when we are looking for that line, we don't really care
about the perfect value of x. What we care about is actually
these coefficients a and b that will tell us what the relation
is between x and y. In fact, we are trying to solve
for a and b that will give us the nicest possible line for
these points. The unknowns,
in our equations, will have to be a and b,
not x and y. The question really is find the
"best" a and b.
And, of course, we have to decide what we mean
by best. Best will mean that we minimize
some function of a and b that measures the total errors that
we are making when we are choosing this line compared to
the experimental data. Maybe, roughly speaking,
it should measure how far these points are from the line.
But now there are various ways to do it.
And a lot of them are valid they give you different answers.
You have to decide what it is that you prefer.
For example, you could measure the distance
to the line by projecting perpendicularly.
Or you could measure instead, for a given value of x,
the difference between the experimental value of y and the
predicted one. And that is often more relevant
because these guys actually may be expressed in different units.
They are not the same type of quantity.
You cannot actually combine them arbitrarily.
Anyway, the convention is usually we measure distance in
this way. Next, you could try to minimize
the largest distance. Say we look at who has the
largest error and we make that the smallest possible.
The drawback of doing that is experimentally very often you
have one data point that is not good because maybe you fell
asleep in front of the experiment.
And so you didn't measure the right thing.
You tend to want to not give too much importance to some data
point that is far away from the others.
Maybe instead you want to measure the average distance or
maybe you want to actually give more weight to things that are
further away. And then you don't want to do
the distance with a square of the distance.
There are various possible answers, but one of them gives
us actually a particularly nice formula for a and b.
And so that is why it is the universally used one.
Here it says list squares. That's because we will measure,
actually, the sum of the squares of the errors.
And why do we do that? Well, part of it is because it
looks good. When you see this plot in
scientific papers they really look like the line is indeed the
ideal line. And the second reason is
because actually the minimization problem that we
will get is particularly simple, well-posed and easy to solve.
So we will have a nice formula for the best a and the best b.
If you have a method that is simple and gives you a good
answer then that is probably good.
We have to define best. Here it is in the sense of
minimizing the total square error.
Or maybe I should say total square deviation instead.
What do I mean by this? The deviation for each data
point is the difference between what you have measured and what
you are predicting by your model.
That is the difference between y1 and axi plus b.
Now, what we will do is try to minimize the function capital D,
which is just the sum for all the data points of the square of
a deviation. Let me go over this again.
This is a function of a and b. Of course there are a lot of
letters in here, but xi and yi in real life
there will be numbers given to you.
There will be numbers that you have measured.
You have measured all of this data.
They are just going to be numbers.
You put them in there and you get a function of a and b.
Any questions? How do we minimize this
function of a and b? Well, let's use your knowledge.
Let's actually look for a critical point.
We want to solve for partial d over partial a= 0,
partial d over partial b = 0. That is how we look for
critical points. Let's take the derivative of
this with respect to a. Well, the derivative of a sum
is sum of the derivatives. And now we have to take the
derivative of this quantity squared.
Remember, we take the derivative of the square.
We take twice this quantity times the derivative of what we
are squaring. We will get 2(yi - axi) b times
the derivative of this with respect to a.
What is the derivative of this with respect to a?
Negative xi, exactly. And so we will want this to be
0. And partial d over partial b,
we do the same thing, but different shading with
respect to b instead of with respect to a.
Again, the sum of squares twice yi minus axi equals b times the
derivative of this with respect to b is, I think,
negative one. Those are the equations we have
to solve. Well, let's reorganize this a
little bit. The first equation.
See, there are a's and there are b's in these equations.
I am going to just look at the coefficients of a and b.
If you have good eyes, you can see probably that these
are actually linear equations in a and b.
There is a lot of clutter with all these x's and y's all over
the place. Let's actually try to expand
things and make that more apparent.
The first thing I will do is actually get rid of these
factors of two. They are just not very
important. I can simplify things.
Next, I am going to look at the coefficient of a.
I will get basically a times xi squared.
Let me just do it and should be clear.
I claim when we simplify this we get xi squared times a plus
xi times b minus xiyi. And we set this equal to zero.
Do you agree that this is what we get when we expand that
product? Yeah. Kind of?
OK. Let's do the other one. We just multiply by minus one,
so we take the opposite of that which would be axi plus b.
I will write that as xia plus b minus yi.
Sorry. I forgot the n here. And let me just reorganize that
by actually putting all the a's together.
That means I will have sum of all the xi2 times a plus sum of
xib minus sum of xiyi equal to zero. If I rewrite this,
it becomes sum of xi2 times a plus sum of the xi's time b,
and let me move the other guys to the other side,
equals sum of xiyi. And that one becomes sum of xi
times a. Plus how many b's do I get on
this one? I get one for each data point.
When I sum them together, I will get n.
Very good. N times b equals sum of yi.
Now, this quantities look scary, but they are actually
just numbers. For example,
this one, you look at all your data points.
For each of them you take the value of x and you just sum all
these numbers together. What you get,
actually, is a linear system in a and b, a two by two linear
system. And so now we can solve this
for a and b. In practice,
of course, first you plug in the numbers for xi and yi and
then you solve the system that you get.
And we know how to solve two by two linear systems,
I hope. That's how we find the best fit
line. Now, why is that going to be
the best one instead of the worst one?
We just solved for a critical point.
That could actually be a maximum of this error function
D. We will have the answer to that
next time, but trust me. If you really want to go over
the second derivative test that we will see tomorrow and apply
it in this case, it is quite hard to check,
but you can see it is actually a minimum.
I will just say -- -- we can show that it is a minimum.
Now, the event with the linear case is the one that we are the
most familiar with. Least-squares interpolation
actually works in much more general settings.
Because instead of fitting for the best line,
if you think it has a different kind of relation then maybe you
can fit in using a different kind of formula.
Let me actually illustrate that with an example.
I don't know if you are familiar with Moore's law.
It is something that is supposed to tell you how quickly
basically computer chips become smarter faster and faster all
the time. It's a law that says things
about the number of transistors that you can fit onto a computer
chip. Here I have some data about --
Here is data about the number of transistors on a standard PC
processor as a function of time. And if you try to do a
best-line fit, well, it doesn't seem to follow
a linear trend. On the other hand,
if you plug the diagram in the log scale,
the log of a number of transitions as a function of
time, then you get a much better line.
And so, in fact, that means that you had an
exponential relation between the number of transistors and time.
And so, actually that's what Moore's law says.
It says that the number of transistors in the chip doubles
every 18 months or every two years.
They keep changing the statement.
How do we find the best exponential fit?
Well, an exponential fit would be something of a form y equals
a constant times exponential of a times x.
That is what we want to look at. Well, we could try to minimize
a square error like we did before.
That doesn't work well at all. The equations that you get are
very complicated. You cannot solve them.
But remember what I showed you on this log plot.
If you plot the log of y as a function of x then suddenly it
becomes a linear relation. Observe, this is the same as ln
of y equals ln of c plus ax. And that is the linear best fit.
What you do is you just look for the best straight line fit
for the log of y. That is something we already
know. But you can also do,
for example, let's say that we have
something more complicated. Let's say that we have actually
a quadratic law. For example,
y is of the form ax^2 bx c. And, of course,
you are trying to find somehow the best.
That would mean here fitting the best parabola for your data
points. Well, to do that,
you would need to find a, b and c.
And now you will have actually a function of a,
b and c, which would be the sum of the old data points of the
square deviation. And, if you try to solve for
critical points, now you will have three
equations involving a, b and c,
in fact, you will find a three by three linear system.
And it works the same way. Just you have a little bit more
data. Basically, you see that this
best fit problems are an example of a minimization problem that
maybe you didn't expect to see minimization problems come in.
But that is really the way to handle these questions.
Tomorrow we will go back to the question of how do we decide
whether it is a minimum or a maximum.
And we will continue exploring in terms of several variables.
