Since
U
is an
M
×
R
matrix, it is possible when
R < M
that the
reconstruction error is zero. This happens when
e
is orthogonal to
every column of
U
, i.e.
U
T
e
=
0
. Putting this together with the
work above means
0
≤
1
σ
2
1
k
U
T
e
k
2
2
≤ k
ˆ
x
ls

x
pinv
k
2
2
≤
1
σ
2
R
k
U
T
e
k
2
2
≤
1
σ
2
R
k
e
k
2
2
.
Notice that if
σ
R
is small, the worst case reconstruction error can be
very bad
.
51
Georgia Tech ECE 6250 Fall 2019; Notes by J. Romberg and M. Davenport. Last updated 21:13, November 3, 2019
Subscribe to view the full document.
We can also relate the “average case” error to the singular values. Say
that
e
is additive Gaussian white noise, that is each entry
e
[
m
] is a
random variable independent of all the other entries, and distributed
e
[
m
]
∼
Normal(0
, ν
2
)
.
Then, as we have argued before, the average measurement error is
E[
k
e
k
2
2
] =
Mν
2
,
and the average reconstruction error
1
is
E
h
k
A
†
e
k
2
2
i
=
ν
2
·
trace(
A
†
T
A
†
) =
ν
2
·
1
σ
2
1
+
1
σ
2
2
+
· · ·
+
1
σ
2
R
=
1
M
1
σ
2
1
+
1
σ
2
2
+
· · ·
+
1
σ
2
R
·
E[
k
e
k
2
2
]
.
Again, if
σ
R
is tiny, 1
/σ
2
R
will dominate the sum above, and the
average reconstruction error will be quite large.
Exercise:
Let
D
be a diagonal
R
×
R
matrix whose diagonal
elements are positive. Show that the maximizer
ˆ
β
to
maximize
β
∈
R
R
k
Dβ
k
2
2
subject to
k
β
k
2
= 1
has a 1 in the entry corresponding to the largest diagonal element of
D
, and is 0 elsewhere.
1
We are using the fact that if
e
is vector of iid Gaussian random vari
ables,
e
∼
Normal(
0
, ν
2
I
),
then for any matrix
M
,
E[
k
Me
k
2
2
]
=
ν
2
trace(
M
T
M
). We leave this as an exercise (or maybe a homework!)
52
Georgia Tech ECE 6250 Fall 2019; Notes by J. Romberg and M. Davenport. Last updated 21:13, November 3, 2019
Stable Reconstruction with the Truncated SVD
We have seen that if
A
has very small singular values and we apply
the pseudoinverse in the presence of noise, the results can be disas
trous. But it doesn’t have to be this way. There are several ways
to stabilize the pseudoinverse. We start be discussing the simplest
one, where we simply “cut out” the part of the reconstruction which
is causing the problems.
As before, we are given noisy indirect observations of a vector
x
through a
M
×
N
matrix
A
:
y
=
Ax
+
e
.
(3)
The matrix
A
has SVD
A
=
U
Σ
V
T
, and pseudoinverse
A
†
=
V
Σ

1
U
T
. We can rewrite
A
as a sum of rank1 matrices:
A
=
R
X
r
=1
σ
r
u
r
v
T
r
,
where
R
is the rank of
A
, the
σ
r
are the singular values, and
u
r
∈
R
M
and
v
r
∈
R
N
are columns of
U
and
V
, respectively. Similarly, we
can write the pseudoinverse as
A
†
=
R
X
r
=1
1
σ
r
v
r
u
T
r
.
Given
y
as above, we can write the leastsquares estimate of
x
from
the noisy measurements as
ˆ
x
ls
=
A
†
y
=
R
X
r
=1
1
σ
r
h
y
,
u
r
i
v
r
.
(4)
53
Georgia Tech ECE 6250 Fall 2019; Notes by J. Romberg and M. Davenport. Last updated 21:13, November 3, 2019
Subscribe to view the full document.
As we can see (and have seen before) if any one of the
σ
r
are very
small, the leastsquares reconstruction can be a disaster.
 Fall '08
 Staff