Scribd
Upload
61 views

Math416 LeastSquares

The document discusses using the least squares method to find the best fit line through given data points and the best approximation of a function by polynomials of a certain degree in an inner product space. It provides examples of using the method to find the line of best fit through three points and the best degree one polynomial approximation of the function t^2.

Uploaded by

Priya Sharma
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
61 views

Math416 LeastSquares

The document discusses using the least squares method to find the best fit line through given data points and the best approximation of a function by polynomials of a certain degree in an inner product space. It provides examples of using the method to find the line of best fit through three points and the best degree one polynomial approximation of the function t^2.

Uploaded by

Priya Sharma
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 3

Math 416 - Abstract Linear Algebra

Fall 2011, section E1


Least squares solution
1. Curve tting
The least squares solution can be used to t certain functions through data points.
Example: Find the best t line through the points (1, 0), (2, 1), (3, 1).
Solution: We are looking for a line with equation y = a + bx that would ideally go through
all the data points, i.e. satisfy all the equations
_

_
a + b(1) = 0
a + b(2) = 1
a + b(3) = 1.
In matrix form, we want the unknown coecients
_
a
b
_
to satisfy the system
_
_
1 1
1 2
1 3
_
_
_
a
b
_
=
_
_
0
1
1
_
_
but the system has no solution. Instead, we nd the least squares t, i.e. minimize the sum of
the squares of the errors
3

i=1
|(a + bx
i
) y
i
|
2
which is precisely nding the least squares solution of the system above. Writing the system as
Ac = y, the normal equation is
A
T
Ac = A
T
y
and we compute
A
T
A =
_
1 1 1
1 2 3
_
_
_
1 1
1 2
1 3
_
_
=
_
3 6
6 14
_
A
T
y =
_
1 1 1
1 2 3
_
_
_
0
1
1
_
_
=
_
2
5
_
.
The normal equation has the unique solution
c =
_
3 6
6 14
_
1
_
2
5
_
=
1
6
_
14 6
6 3
_ _
2
5
_
=
1
6
_
2
3
_
=
_

1
3
1
2
_
so that the best t line through the data points is y =
1
3
+
1
2
x.
1
Remark: If we hate the formula for the inverse of a 2 2 matrix, or if we need to solve a
bigger system, we can always use Gauss-Jordan:
_
3 6 2
6 14 5
_

_
3 6 2
0 2 1
_

_
3 0 1
0 2 1
_

_
1 0
1
3
0 1
1
2
_
.
The unique solution to the system is indeed
_

1
3
1
2
_
.
2. Arbitrary inner product spaces
Just like Gram-Schmidt, the least squares method works in any inner product space V , not just
R
n
(or C
n
). Assume that the subspace E V onto which we are projecting is nite-dimensional.
Example: Consider the real inner product space C[0, 1] := {f : [0, 1] R | f is continuous }
with its usual inner product
(f, g) =
_
1
0
f(t)g(t) dt.
Find the best approximation of the function t
2
by a polynomial of degree at most one.
Solution using least squares: We are looking for a polynomial of degree at most one a+bt
that would ideally satisfy
a + bt = t
2
which is clearly impossible, i.e. t
2
/ Span{1, t}. The best approximation is the vector in
Span{1, t} minimizing the error vector
a + bt t
2
which is achieved exactly when the error vector is orthogonal to Span{1, t}. This imposes two
conditions:
_
(a + bt t
2
, 1) = 0
(a + bt t
2
, t) = 0
which we can rewrite as
_
a(1, 1) + b(t, 1) = (t
2
, 1)
a(1, t) + b(t, t) = (t
2
, t)
or in matrix form:
_
(1, 1) (t, 1)
(t, 1) (t, t)
_ _
a
b
_
=
_
(t
2
, 1)
(t
2
, t)
_
.
(This is the normal equation. The coecient matrix here plays the role of A
T
A in the previous
example, i.e. the square matrix of all possible inner products between vectors in the basis of E,
in this case {1, t}. Likewise, the right-hand side plays the role of A
T
y in the previous example,
i.e. the list of all possible inner products between the basis vectors {1, t} of E and the vector
t
2
not in E which we want to project down to E.)
Computing the inner products involved, the system can be written as
_
1
1
2
1
3
1
2
1
3
1
4
_
2
which we now solve:
_
1
1
2
1
3
1
2
1
3
1
4
_

_
1
1
2
1
3
0
1
12
1
12
_

_
1
1
2
1
3
0 1 1
_

_
1 0
1
6
0 1 1
_
.
The least squares solution is
_
a
b
_
=
_

1
6
1
_
so that the best approximation of t
2
by a polynomial
of degree at most one is
1
6
+ t.
Remark: If we hate Gauss-Jordan, we can always use the formula for the inverse of a 2 2
matrix, so that the unique solution to the system is
_
1
1
2
1
2
1
3
_
1
_
1
3
1
4
_
=
1
1/12
_
1
3

1
2

1
2
1
_ _
1
3
1
4
_
=
_
1
3

1
2

1
2
1
_ _
4
3
_
=
1
6
_
2 3
3 6
_ _
4
3
_
=
1
6
_
1
6
_
=
_

1
6
1
_
.
http://www.youtube.com/watch?v=lBdASZNPIv8
Solution using Gram-Schmidt: In a previous exercise, we obtained the orthonormal basis
{u
1
= 1, u
2
=

3(2t 1)} of Span{1, t}. Using this, we compute the projection


Proj
{1,t}
(t
2
) = Proj
{u
1
,u
2
}
(t
2
)
= (t
2
, u
1
)u
1
+ (t
2
, u
2
)u
2
= (t
2
, 1) 1 + (t
2
,

3(2t 1))

3(2t 1)
=
1
3
+ 3
_
2(t
2
, t) (t
2
, 1)
_
(2t 1)
=
1
3
+ 3
_
2
4

1
3
_
(2t 1)
=
1
3
+
1
2
(2t 1)
=
1
6
+ t.
3

You might also like