CHRISTINE BREINER: Welcome
back to recitation. In this video, what
I'd like you to do is use least squares to
fit a line to the following data, which includes three
points: the point (0, 1), the point (2, 1),
and the point (3, 4). And I've drawn a rough
picture where these points are on a graph, and I'll be
talking a little bit about that after you try this problem. So why don't you try to
solve the problem based on the technique you
learned in class, and then bring
the video back up. I'll show you again what
you're actually doing, and then I will also solve
the problem and you can see, you can compare
your answer to mine. OK, welcome back. So the first thing I want to do
when we're talking about least squares to fit a line to these
data, what I'd like to do is I'm going to
draw a line on here, and we're going to talk about
what the least squares actually is trying to do. And so I'm going to draw this
line, say, something like this. That's my first
guess on what might be the actual least squares
line for these data. And if you recall, what
you're actually trying to do is you're trying to
minimize a certain quantity, and the quantity you're
trying to minimize is the difference between
the actual value you get and the expected value you
get, the square of that. So this is one thing
we're going to square. So even pictorially,
we could say we have something that
contributes some area, OK, and then this is
another thing that's the difference between
the expected value in y and the actual value you get. So let me try and
make that squared, because we square that value. And then this last
one is pretty small. The difference between
the expected value and the actual value is
very small over in this case from the picture I drew. And so what you're
trying to do is you're trying to
find the line that minimizes the area of
the sum of these three, in this case, three squares. So if I, you know, if I tip
the line down further here, but I kept it fixed there,
the area of this square would get bigger, because this
distance would get bigger. But, of course, the area of
this square would get smaller. And so you're
trying to figure out a way to optimize, in
this case, minimize, the area of these--
sorry, the sum of the areas of these
squares by figuring out where to put the line. Now, I mentioned if I could
fix this here and move it up and down, then that
would change the slope but fix the intercept. But I could also move the
intercept up and down, and I could slide
the line up and down, and that would fix the slope
but vary the intercept, or I could do both. And ultimately, that's
what you're doing. You're trying to find the best
y-intercept and the best slope at the same time, and that's
why you learned this process in multivariable
calculus, because you're trying to optimize
something that has two quantities that you can change. You can change slope and you
can change the y-intercept, so that's why you
learned it now instead of in single-variable calculus. Now, if we come over
here, I wrote down ahead of time what the equations
are you get when you optimize, when you're trying to optimize
the least squares line. So if you write down,
you know, the function as a function of
a and b, and you take the derivative
with respect to a and you set it equal to zero,
you get the first equation. When you take the
derivative with respect to b and you set
it equal to zero, you get the second equation. And so I'm just going
to fill in the details and then tell you
what the solution is, and then I'm going to mention a
little more sophisticated thing we can do, or I guess
maybe the next level of least squares we can do,
that's not a linear problem. So I'll do that last. So what I want to do is I put
all the values I need over here on the right. So all the values we
need are on the right. We're going to need the sum
of all the x_i's and that's 5. The sum of all the y_i's is 6. The sum of the x_i
squareds is 13, and the sum of the
x_i*y_i product is 14. Now, where are
these coming from? Remember, what is
x_i and what is y_i? If we come back over here,
this is (x_1, y_1), (x_2, y_2), (x_3, y_3), OK? So that's sort of how we get--
if we switch the whole pairs around, it doesn't matter. Obviously, if we switched
x- and y-values together, it does matter, and that you
see actually from the equation does matter. Right here you have
a product of them. So let me write this out. And again, we're solving for
a and b, so in this case, a is the slope and b is
the intercept, right? So let me write everything in. So we actually get that we want
to solve the following system: 13a plus 5b is equal to 14. Make sure I put in
everything correctly. And then the second one is
5a plus-- n in this case is 3, because there were three
points-- plus 3b is equal to 6. OK, this is a
system of equations. It has two equations
and two unknowns, and if you solve this-- I
should look at my notes, because I didn't
memorize it and I'm not going to do all the algebra. If you go to solve this,
what you actually get is that a is equal to 6/7
and b is equal to 4/7. So you get that the line--
let me write it here. You get a is equal to 6/7
and b is equal to 4/7, OK? And so what that tells you is
if you come back over here, the line we actually want-- I'll
write the solution right here-- the line we actually want is the
line y equals 6/7 x plus 4/7. That is our least squares
line that is our solution. OK? So that is actually-- solving
for a line to fit these data, you get the following solution. But I want to point out,
and what we're going to do, after this video there will be
a challenge problem, to have you actually finish this. I want to point out
that you can actually also fit a higher-degree
polynomial to these data. For instance, you
could try and use the technique of least
squares to fit a parabola to these data. And let me point out
what the function would look like in this case. So this is for the
challenge problem. For the challenge
problem, it now will be a function
of three variables, so it will look
something like this. We'll say a comma b comma
c, and the point that we'll be wanting to minimize,
or the function we'll be wanting to minimize,
we're summing over i, again from 1 to 3-- I didn't
write that down above, but it was fairly
obvious that that was what we were summing
over-- the following thing. We have y_i minus
this big quantity: a x_i squared plus b*x_i plus c
and we square the whole thing. And so what this
is actually giving you is, just as in
the picture before, this is giving
you the difference between the expected value
and the actual value. So this is your actual y-value,
and based on the parabola you're looking for,
what's in parentheses is your expected y-value, right? So this would actually
be my parabola. When I evaluate it at
x_i, I get a number. I get a value, and that value
will be on the parabola. This will be the
y-value associated to the x-value from the data. So this is the actual value. This is the expected value. We take the difference
and square it. And because we want
to minimize how we're going to
solve this problem and this is what
you'll have to do is, just like with
the line situation, you're going to take the
derivative of this function with respect to each of these
three variables separately. So you'll take f sub a, and then
you'll set that equal to zero. And then you'll take f sub b,
you'll set that equal to zero. And then you'll take
f sub c, and you'll set that equal to zero. Again, we set them
all equal to zero, because this is really
an optimization problem. We want to minimize something. We want to minimize
this quantity. So you take each of
those three derivatives, partial derivatives,
set them equal to zero, and you have a system of three
equations with three variables. And you know how to solve such
a system, by using matrices, for instance. That would be one way
to solve such a system. And so then you can
actually come up with a formula for the sort
of parabola of best fit, you could think of it as. And so I'm going to stop there,
because I think from here, you guys will do it. But I just want
to point out this is where you're
going to be starting this next sort of
challenge problem to find the quadratic of
best fit for these three data points. So that's where I'll end it.
