can be rewritten as
k
β

Σ
α
k
2
2
+
δ
k
α
k
2
2
=
R
X
k
=1
(
β
[
k
]

σ
k
α
[
k
])
2
+
δα
[
k
]
2
.
(8)
We can minimize this sum simply by minimizing each term indepen
dently. Since
d
dα
[
k
]
(
β
[
k
]

σ
k
α
[
k
])
2
+
δα
[
k
]
2
=

2
β
[
k
]
σ
k
+ 2
σ
2
k
α
[
k
] + 2
δα
[
k
]
,
we need
α
[
k
] =
σ
k
σ
2
k
+
δ
β
[
k
]
.
Putting this back in vector form, (
8
) is minimized by
ˆ
α
tik
= (
Σ
2
+
δ
I
)

1
Σ
β
,
and so the minimizer to (
6
) is
ˆ
x
tik
=
V
ˆ
α
tik
=
V
(
Σ
2
+
δ
I
)

1
Σ
U
T
y
.
(9)
We can get a better feel for what Tikhonov regularization is doing
by comparing it directly to the pseudoinverse.
The leastsquares
59
Georgia Tech ECE 6250 Fall 2019; Notes by J. Romberg and M. Davenport. Last updated 21:13, November 3, 2019
Subscribe to view the full document.
reconstruction ˆ
x
ls
can be written as
ˆ
x
ls
=
V
Σ

1
U
T
y
=
R
X
r
=1
1
σ
r
h
y
,
u
r
i
v
r
,
while the Tikhonov reconstruction ˆ
x
tik
derived above is
ˆ
x
tik
=
R
X
r
=1
σ
r
σ
2
r
+
δ
h
y
,
u
r
i
v
r
.
(10)
Notice that when
σ
r
is much larger than
δ
,
σ
r
σ
2
r
+
δ
≈
1
σ
r
,
σ
r
δ,
but when
σ
r
is small
σ
r
σ
2
r
+
δ
≈
0
,
σ
r
δ.
Thus the Tikhonov reconstruction modifies the important parts (com
ponents where the
σ
r
are large) of the pseudoinverse very little, while
ensuring that the unimportant parts (components where the
σ
r
are
small) affect the solution only by a very small amount. This
damp
ing
of the singular values, is illustrated below.
σ
r
σ
2
r
+
δ
0
0.5
1
1.5
2
2.5
3
0
1
2
3
4
5
6
σ
r
60
Georgia Tech ECE 6250 Fall 2019; Notes by J. Romberg and M. Davenport. Last updated 21:13, November 3, 2019
Above, we see the damped multipliers
σ
r
/
(
σ
2
r
+
δ
) versus
σ
r
for
δ
= 0
.
1 (blue),
δ
= 0
.
05 (red), and
δ
= 0
.
01 (green).
The black
dotted line is 1
/σ
r
, the leastsquares multiplier. Notice that for large
σ
r
(
σ
r
>
2
√
δ
, say), the damping has almost no effect.
This damping makes the Tikhonov reconstruction exceptionally sta
ble; large multipliers never appear in the reconstruction (
10
). In fact
it is easy to check that
σ
r
σ
2
r
+
δ
≤
1
2
√
δ
no matter the value of
σ
r
.
Tikhonov Error Analysis
Given noisy observations
y
=
Ax
0
+
e
, how well will Tikhonov reg
ularization work? The answer to this questions depends on multiple
factors including the choice of
δ
, the nature of the perturbation
e
,
and how well
x
0
can be approximated using a linear combination
of the singular vectors
v
r
corresponding to the large (relative to
δ
)
singular values. Since a closedform expression for the solution to (
6
)
exists, we can quantify these tradeoffs precisely.
We compare the Tikhonov reconstruction to the reconstruction we
would obtain if we used standard leastsquares on perfectly noise
free observations
y
clean
=
Ax
0
. This noisefree reconstruction can
61
Georgia Tech ECE 6250 Fall 2019; Notes by J. Romberg and M. Davenport. Last updated 21:13, November 3, 2019
Subscribe to view the full document.
be written as
x
pinv
=
A
†
y
clean
=
A
†
Ax
0
=
V
Σ

1
U
T
U
Σ
V
T
x
0
=
V V
T
x
0
=
R
X
r
=1
h
x
0
,
v
r
i
v
r
.
The vector
x
pinv
is the orthogonal projection of
x
0
onto the row space
(everything orthogonal to the null space) of
A
. If
A
has full column
rank, then
x
pinv
=
x
0
. If not, then the application of
A
destroys
the part of
x
0
that is not in
x
pinv
, and so we only attempt to recover
the “visible” components.
In some sense,
x
pinv
contains all of the
components of
x
0
that we could ever hope to recover, and has them
preserved perfectly.
 Fall '08
 Staff