Chapter 5
Least Squares
The term
least squares
describes a frequently used approach to solving overdeter
mined or inexactly speciﬁed systems of equations in an approximate sense. Instead
of solving the equations exactly, we seek only to minimize the sum of the squares
of the residuals.
The least squares criterion has important statistical interpretations. If ap
propriate probabilistic assumptions about underlying error distributions are made,
least squares produces what is known as the
maximumlikelihood
estimate of the pa
rameters. Even if the probabilistic assumptions are not satisﬁed, years of experience
have shown that least squares produces useful results.
The computational techniques for linear least squares problems make use of
orthogonal matrix factorizations.
5.1
Models and Curve Fitting
A very common source of least squares problems is curve ﬁtting. Let
t
be the
independent variable and let
y
(
t
) denote an unknown function of
t
that we want
to approximate. Assume there are
m
observations
, i.e., values of
y
measured at
speciﬁed values of
t
:
y
i
=
y
(
t
i
)
, i
= 1
,...,m.
The idea is to model
y
(
t
) by a linear combination of
n
basis functions
:
y
(
t
)
≈
β
1
φ
1
(
t
) +
···
+
β
n
φ
n
(
t
)
.
The
design matrix
X
is a rectangular matrix of order
m
by
n
with elements
x
i,j
=
φ
j
(
t
i
)
.
The design matrix usually has more rows than columns. In matrixvector notation,
the model is
y
≈
Xβ.
February 15, 2008
1
This preview has intentionally blurred sections. Sign up to view the full version.
View Full Document2
Chapter 5. Least Squares
The symbol
≈
stands for “is approximately equal to.” We are more precise about
this in the next section, but our emphasis is on
least squares
approximation.
The basis functions
φ
j
(
t
) can be nonlinear functions of
t
, but the unknown
parameters,
β
j
, appear in the model linearly. The system of linear equations
Xβ
≈
y
is
overdetermined
if there are more equations than unknowns. The
Matlab
back
slash operator computes a least squares solution to such a system.
beta = X\y
The basis functions might also involve some nonlinear parameters,
α
1
,...,α
p
.
The problem is
separable
if it involves both linear and nonlinear parameters:
y
(
t
)
≈
β
1
φ
1
(
t,α
) +
···
+
β
n
φ
n
(
t,α
)
.
The elements of the design matrix depend upon both
t
and
α
:
x
i,j
=
φ
j
(
t
i
,α
)
.
Separable problems can be solved by combining backslash with the
Matlab
func
tion
fminsearch
or one of the nonlinear minimizers available in the Optimization
Toolbox. The new Curve Fitting Toolbox provides a graphical interface for solving
nonlinear ﬁtting problems.
Some common models include the following:
•
Straight line: If the model is also linear in
t
, it is a straight line:
y
(
t
)
≈
β
1
t
+
β
2
.
This is the end of the preview.
Sign up
to
access the rest of the document.
 Spring '08
 BURKHAUSER
 Linear Algebra, matlab, Least Squares, Linear least squares

Click to edit the document details