We want to find Δ
A
,
Δ
y
,
x
such that
(
y
+ Δ
y
) = (
A
+ Δ
A
)
x
,
for Δ
y
,
Δ
A
of minimal size. Rewrite this as
(
A
+ Δ
A
)
x

(
y
+ Δ
y
) =
0
⇒
A
+ Δ
A y
+ Δ
y
x

1
=
0
⇒
(
C
+
Δ
)
x

1
=
0
where
C
=
A y
,
Δ
=
Δ
A
Δ
y
.
Note that both
C
and
Δ
are
M
×
(
N
+ 1) matrices.
The result of the progression of equations above says that we are
looking for a
Δ
(of minimal size) such that there is a vector
x

1
in the nullspace of
C
+
Δ
. Since
v
∈
Null(
C
+
Δ
)
⇔
α
v
∈
Null(
C
+
Δ
)
for all
α
∈
R
,
and
x
in arbitrary, we are really just asking that
C
+
Δ
has a
nullspace; as long as there is at least one vector in the nullspace
70
Georgia Tech ECE 6250 Fall 2019; Notes by J. Romberg and M. Davenport. Last updated 23:01, November 5, 2019
Subscribe to view the full document.
whose last entry is nonzero, we can find a vector of the required form
just by normalizing. In short, this means that our task is to find
Δ
such that the
M
×
(
N
+ 1) matrix
C
+
Δ
is
rank deficient
, that
is rank(
C
+
Δ
)
< N
+ 1.
Put another way, we want to solve the optimization program
minimize
Δ
k
Δ
k
2
F
subject to
rank(
C
+
Δ
) =
N.
Making the substitution
X
=
C
+
Δ
, this is equivalent to solving
minimize
X
k
C

X
k
2
F
subject to
rank(
X
) =
N,
and then taking
c
Δ
=
c
X

C
.
This is a lowrank approximation problem.
1
We now consider how
to solve problems of this form (which arise in many other important
contexts).
The SVD and Matrix Approximation
Let
A
be an
M
×
N
matrix with rank
R
. We are often interested in
determining the closest matrix to
A
that has rank
r
2
? More precisely,
we would like to solve
minimize
X
k
A

X
k
2
F
subject to
rank(
X
) =
r.
(1)
The functional above is standard leastsquares, but the constraint
set (the set of rank
r
matrices) is a complicated entity. Neverthe
less, as with many things in this class, the SVD reveals the solution
immediately.
1
Or at least a “lower rank” approximation problem.
2
We will assume that
r < R
, as for
r
=
R
the answer is easy, and for
R < r
≤
min(
M, N
) the question is not wellposed.
71
Georgia Tech ECE 6250 Fall 2019; Notes by J. Romberg and M. Davenport. Last updated 23:01, November 5, 2019
Lowrank approximation.
Let
A
be a matrix with SVD
A
=
U
Σ
V
T
=
R
X
p
=1
σ
p
u
p
v
T
p
.
Then (
1
) is solved simply by truncating the SVD:
c
X
=
r
X
p
=1
σ
p
u
p
v
T
p
=
U
r
Σ
r
V
T
r
,
where
U
r
contains the first
r
columns of
U
,
V
r
contains the first
r
columns of
V
, and
Σ
r
is the first
r
columns and
r
rows of
Σ
.
The framed result above, known as the EckartYoung theorem, is an
immediate consequence of the following lemma.
Subspace Approximation Lemma.
For fixed
A
with SVD
A
=
U
Σ
V
T
, the optimization program
minimize
Q
:
M
×
r
Θ
:
r
×
N
k
A

Q
Θ
k
2
F
subject to
Q
T
Q
=
I
,
(2)
has solution
b
Q
=
U
r
,
b
Θ
=
U
T
r
A
,
where
U
r
=
u
1
u
2
· · ·
u
r
contains the first
r
columns of
U
.
We prove this lemma in the technical details section at the end of
the notes. To see how it implies the EckartYoung theorem, we can
72
Georgia Tech ECE 6250 Fall 2019; Notes by J. Romberg and M. Davenport. Last updated 23:01, November 5, 2019
Subscribe to view the full document.
interpret the search over
M
×
r
matrices
Q
with orthonormal columns
as a search over all possible
column spaces
of dimension
r
. Then the
search over
Θ
 Fall '08
 Staff