Least squares using matrices | Lecture 26 | Matrix Algebra for Engineers

in this video in the next video I want to apply our knowledge of matrices and vector spaces to a well known problem in statistics that's called the least squares problem the idea is that you have some data of some y versus x and when you plot the data it looks like it's almost aligned but the Y data has some noise in it so you can't draw an exact line through these points but you can draw an approximate line so my my daughter who's 16 years old actually had exactly this problem this week and this is what she did so she had the data from her chemistry experiment and she just took her software and just drew a line right that's what she did and said that looks like the best line so I told her well you drew that line maybe someone else would draw a different line you have to do this and some more systematic fashion so that it's reproducible and that's what we're going to do here usually in statistics what you do then is you find the distance between each of these points and the line or a line that you may draw and then you square the distances and sum them up and then you try to find the line that minimizes the sum of the squares of those distances that's why it's called the least squares problem so that's what we're going to do here but we're going to do it in a different way we're going to do it from the perspective of matrices so let me you formulate the problem so we have data so the data I'll write as values of x1 y1 x2 y2 and then we have n data points so the data point will be xn YN so those are our data and then in the standard sort of least squares problem we're going to assume that the x's are exact so the x coordinate the x coordinates of this data unknown that's considered the independent variable and then the y's are noisy data okay those are the dependent variables okay so how do we formulate this least squares problem in a make matrix sense even though we know the line cannot go through the data points right a line can only go through two points if you have many points in a data set there's no way you can have the line go through all the points even so we can write down the equations so if lines went through the points we can write down the equations what we knew need to specify is the form of the line so we're going to let our line be y equals beta naught plus beta 1 times X right beta naught is the y-intercept beta 1 is the slope of the line so if this is our assumption for this Purple Line then we can write down the equations the equations would be the first data point would be y1 equals beta naught plus beta 1 X 1 right so if this line y equals beta naught plus beta 1 X went through this first point it would satisfy this equation the second point would be y2 equals beta naught plus beta 1 X 2 and then so on and the last point would be Y N equals beta and plus beta 1 X and this is a matrix equation so let me write the equation this is the the left-hand side here is a column vector so this is y 1 y 2 through y n right and right-hand side then is a multiplication of two matrices the unknown here is beta naught and beta 1 okay so we have some matrix here times our unknown vector the unknown vector is beta naught and beta 1 and then this matrix is 1 times beta naught plus x1 times beta 1 equals y 1 so 1 times beta naught plus X 1 times beta 1 equals y 1 so that's the first equation the second equation is 1 times beta naught times X 2 times beta 1 equals y 2 and so on so then we have 1 down to X and ok so this is our matrix equation of this system of equations this is what we usually call ax equals B right so let me just our old notation would be this is the equation ax equals B ok so we can generalize this so what we want to solve then we want to solve an equation ax equals B right this is the equation where the problem here is that there are many more equations than there are unknowns there's only two unknowns but there are n equations ok so this is considered over the tournament so this is over our determined okay so how do we do that well we use what we've learned from vector spaces so the product a times a column vector X is in the vector space of the the column space of a right so ax gives you a vector that's in the column space of a if ax equals B is over determined that means B is not in the column space of a there's no solution of this equation for the unknown X the unknown X is our beta naught beta one so B is not in the column space of a what define the best solution of this equation what we can do is we can project B onto the column space of a so instead of we cannot solve this one right cannot solve right there's no solution so instead we solve the equation ax equals B projected onto the column space okay and when we do that we will find that the solution for X which is a beta naught beta 1 is indeed the least squares solution the solution that minimizes the square of the distance between the data points and the line okay we'll solve that in the next video let me quickly recap here what we're doing so we have some noisy data we want to fit a line to it okay the X's are assumed to be exact values the Y's have noise in them so we fit a line we call the line y equals B not plus b1x x is the x1 x2 value why is the y1 y2 values this is a system of n equations if we say that the line actually goes through this points we get n equations but only two unknowns the only unknowns are beta naught and beta 1 so this is the system of n equations and two unknowns it's over determined because there are too many equations only two unknowns so you cannot solve an over determined system when we pull back our ideas of vector spaces we know that B then is not in the column space of a that's why there is no solution so in order to find the solution we need to project B onto the column space of a that is what we do in this equation it will turn out this is the equation we solve and when we solve this equation we will have will have solved the least squares problem okay I'm Jeff Chazz Knopf thanks for watching and let's solve this problem in the next video you

Channel: Jeffrey Chasnov

Views: 11,968

Rating: 4.9626169 out of 5

Keywords: linear algebra, matrix algebra, matrices, least squares, linear regression

Id: RlQBEhLhM8Y

Length: 10min 15sec (615 seconds)

Published: Mon Jul 09 2018