As seen before, if the error
e
is random, this bound is a bit pes
simistic. Specifically, if each entry of
e
is an independent identically
distributed Normal random variable with mean zero and variance
ν
2
,
then the expected noise error in the reconstruction will be
E[
k
Noise error
k
2
2
] =
1
M
1
σ
2
1
+
1
σ
2
2
+
· · ·
+
1
σ
2
R
0
·
E[
k
e
k
2
2
]
.
56
Georgia Tech ECE 6250 Fall 2019; Notes by J. Romberg and M. Davenport. Last updated 21:13, November 3, 2019
Subscribe to view the full document.
Stable Reconstruction using Tikhonov
Regularization
Tikhonov
2
regularization is another way to stabilize the leastsquares
recovery. It has the nice features that: 1) it can be interpreted using
optimization, and 2) it can be computed without direct knowledge
of the SVD of
A
.
Recall that we motivated the pseudoinverse by showing that ˆ
x
LS
=
A
†
y
is a solution to
minimize
x
∈
R
N
k
y

Ax
k
2
2
.
(5)
When
A
has full column rank, ˆ
x
LS
is the unique solution, otherwise
it is the solution with smallest energy.
When
A
has full column
rank but has singular values which are very small, huge variations
in
x
(in directions of the singular vectors
v
k
corresponding to the
tiny
σ
k
) can have very little effect on the residual
k
y

Ax
k
2
2
. As
such, the solution to (
5
) can have wildly inaccurate components in
the presence of even mild noise.
One way to counteract this problem is to modify (
5
) with a
regu
larization
term that penalizes the size of the solution
k
x
k
2
2
as well
as the residual error
k
y

Ax
k
2
2
:
minimize
x
∈
R
N
k
y

Ax
k
2
2
+
δ
k
x
k
2
2
.
(6)
The parameter
δ >
0 gives us a tradeoff between accuracy and
regularization; we want to choose
δ
small enough so that the residual
2
Andrey Tikhonov (19061993) was a 20
th
century Russian mathematician.
57
Georgia Tech ECE 6250 Fall 2019; Notes by J. Romberg and M. Davenport. Last updated 21:13, November 3, 2019
for the solution of (
6
) is close to that of (
5
), and large enough so that
the problem is wellconditioned.
Just as with (
5
), which is solved by applying the pseudoinverse to
y
, we can write the solution to (
6
) in closed form. To see this, recall
that we can decompose any
x
∈
R
N
as
x
=
V α
+
V
0
α
0
,
where
V
is the
N
×
R
matrix (with orthonormal columns) used in
the SVD of
A
, and
V
0
is a
N
×
N

R
matrix whose columns are
an orthogonal basis for the null space of
A
.
This means that the
columns of
V
0
are orthogonal to each other and all of the columns
of
V
. Similarly, we can decompose
y
as
y
=
Uβ
+
U
0
β
0
,
where
U
is the
M
×
R
matrix used in the SVD of
A
, and the columns
of
U
0
are an orthogonal basis for the left null space of
A
(everything
in
R
M
that is not in the range of
A
).
For any
x
, we can write
y

Ax
=
Uβ
+
U
0
β

U
Σ
V
T
(
V α
+
V
0
α
0
)
=
U
(
β

Σ
α
) +
U
0
β
.
Since the columns of
U
are orthonormal,
U
T
U
=
I
, and also
U
T
0
U
0
=
I
, and
U
T
U
0
=
0
, we have
k
y

Ax
k
2
2
=
h
U
(
β

Σ
α
) +
U
0
β
0
,
U
(
β

Σ
α
) +
U
0
β
0
i
=
k
β

Σ
α
k
2
2
+
k
β
0
k
2
2
,
and
k
x
k
2
2
=
k
α
k
2
2
+
k
α
0
k
2
2
.
58
Georgia Tech ECE 6250 Fall 2019; Notes by J. Romberg and M. Davenport. Last updated 21:13, November 3, 2019
Subscribe to view the full document.
Using these facts, we can write the functional in (
6
) as
k
y

Ax
k
2
2
+
δ
k
x
k
2
=
k
β

Σ
α
k
2
2
+
δ
k
α
k
2
2
+
δ
k
α
0
k
2
2
.
(7)
We want to choose
α
and
α
0
that minimize (
7
). It is clear that, just
as in the standard leastsquares problem, we need
α
0
=
0
. The part
of the functional that depends on
α
 Fall '08
 Staff