5.3 LEAST SQUARES PROBLEMS
KIAM HEONG KWA
pp. 223224 In practice, the coefficient matrix
A
and the righthand side
b
of an
m
×
n
system
A
x
=
b
are obtained emprically. In consequence, the
system
A
x
=
b
is usually inconsistent. In this case, we look for a
vector
b
x
∈
R
n
for which
A
b
x
is closest to
b
in the following sense.
(Clearly, if
x
is a solution of
A
x
=
b
, then it satisfies the desired
property.) Define the
residual
r
:
R
n
→
R
m
of the system
A
x
=
b
by setting
r
(
x
) =
b

A
x
and consider the norm of the residual
k
r
k
:
R
n
→
R
given by
k
r
(
x
)
k
=
k
b

A
x
k
for each
x
∈
R
n
, where
k · k
is the Euclidean norm. Then for a
vector
b
x
∈
R
n
,
A
b
x
is closest to
b
if and only if
k
r
(
b
x
)
k
is a (global)
minimum, i.e.,
k
r
(
b
x
)
k ≤ k
r
(
x
)
k
for all
x
∈
R
n
. Such a vector
b
x
that minimizes
k
r
k
is called a
least
squares solution
of the system
A
x
=
b
. Note that minimizing
k
r
k
is equivalent to minimizing
k
r
k
2
.
Let
b
x
be a least squares solution of
A
x
=
b
. Then
p
=
A
b
x
∈
R
(
A
)
, the column space of
A
. By the definition of a least squares
solution,
p
is an element of
R
(
A
)
that is closest to
b
. Since
R
(
A
)
⊆
R
m
is a subspace, such a closest vector
p
always exists and is unique.
Moreover, it is characterized by the inclusion
b

p
∈
R
(
A
)
⊥
=
N
(
A
T
)
1
. In other words,
b
x
is
a
least
squares
solution
of
A
x
=
b
if
and
only
if
r
(
b
x
) =
b

A
b
x
∈
R
(
A
)
⊥
=
N
(
A
T
)
.
This is a consequence of the following general statement.
Let
S
⊆
R
m
be a subspace and let
b
∈
R
m
but otherwise arbi
trary.
There
is
a
unique
element
p
∈
S
,
called
the
projection
of
b
Therem 5.3.1
Date
: July 15, 2011.
1
The fundamental subspaces theorem from the last section says that
N
(
A
) =
R
(
A
T
)
⊥
and
N
(
A
T
) =
R
(
A
)
⊥
for any matrix
A
.
1
This preview has intentionally blurred sections. Sign up to view the full version.
View Full Document
2
KIAM HEONG KWA
onto
S
,
such
that
k
b

p
k
<
k
b

y
k
for
any
y
∈
S
,
y
6
=
p
. In addition, such an element
p
of
S
is char
acterized by the condition
b

p
∈
S
⊥
.
Proof.
To begin with, recall that
R
m
=
S
⊕
S
⊥
, so that
b
has the
unique representation
b
=
p
+
q
,
where
p
∈
S
and
q
∈
S
⊥
. Hence if
y
∈
S
, so that
p

y
∈
S
, we
have
k
b

y
k
2
=
k
(
b

p
) + (
p

y
)
k
2
=
k
b

p
k
2
+
k
p

y
k
2
because
b

p
=
q
⊥
p

y
. Thus we see that
k
b

y
k ≥ k
b

p
k
and the equality holds if and only if
k
p

y
k
= 0
or
p
=
y
. This
shows the existence and the uniqueness of such an element
p
of
S
in the statement of the theorem.
In fact, the above calculation also shows that if
p
∈
S
is such that
b

p
∈
S
⊥
, then
k
b

y
k ≥ k
b

p
k
for any
y
∈
S
. Conversely,
let
z
∈
S
be such that
b

z
6∈
S
⊥
. Then, in particular,
z
6
=
p
and,
as in the above calculation, we have
k
b

z
k
2
=
k
(
b

p
) + (
p

z
)
k
2
=
k
b

p
k
2
+
k
p

z
k
2
,
so that
k
b

z
k
>
k
b

p
k
.
This shows that the element
p
of
S
such that
k
b

p
k
is minimized
is characterized by the inclusion
b

p
∈
S
⊥
.
pp. 224225 Let
b
x
be a least squares solution of the system
A
x
=
b
. Recall that
b
x
is characterized by the inclusion
r
(
b
x
) =
b

A
b
x
∈
R
(
A
)
⊥
=
N
(
A
)
T
. That is,
A
T
(
b

A
b
x
) =
Ar
(
b
x
) =
0
or equivalently
A
T
A
b
x
=
A
T
b
.
This is the end of the preview.
Sign up
to
access the rest of the document.
 Summer '08
 KIM
 Linear Algebra, Algebra, Least Squares, ax, squares solution

Click to edit the document details