45
Georgia Tech ECE 6250 Fall 2019; Notes by J. Romberg and M. Davenport. Last updated 9:10, October 30, 2019
Subscribe to view the full document.
Right inverse
If we reapply
A
to our solution ˆ
x
=
A
†
y
, we would
like it to be as close as possible to our observations
y
. That is,
we would like
AA
†
to be as close to the identity as possible.
Again, achieving this goal exactly is not always possible, espe
cially if
A
has more rows that columns. But we can attempt
to find the “best” right inverse, in the leastsquares sense, by
solving
minimize
H
∈
R
N
×
M
k
AH

I
k
2
F
.
(11)
The solution
ˆ
H
to (
11
) (see the exercise below) must obey
A
T
A
ˆ
H
=
A
T
.
(12)
Again, we show that
A
†
satisfies (
12
), and hence is a minimizer
to (
11
):
A
T
AA
†
=
V
Σ
2
V
T
V
Σ

1
U
T
=
V
Σ
U
T
=
A
T
.
Moral:
A
†
=
V
Σ

1
U
T
is as close (in the leastsquares sense)
to an inverse of
A
as you could possibly have
.
Exercise:
Show that the minimizer
ˆ
H
to (
9
) must obey (
10
). Do this by using
the fact that the derivative of the functional
k
HA

I
k
2
F
with respect
to an entry
H
[
k, ‘
] in
H
must obey
∂
k
HA

I
k
2
F
∂H
[
k, ‘
]
= 0
,
for all 1
≤
k
≤
N,
1
≤
‘
≤
M,
to be a solution to (
9
). Do the same for (
11
) and (
12
).
46
Georgia Tech ECE 6250 Fall 2019; Notes by J. Romberg and M. Davenport. Last updated 9:10, October 30, 2019
Technical Details: Existence of the SVD
In this section we will prove that any
M
×
N
matrix
A
with rank(
A
) =
R
can be written as
A
=
U
Σ
V
T
where
U
,
Σ
,
V
have the five properties listed at the beginning of the
last section.
Since
A
T
A
is symmetric positive semidefinite, we can write:
A
T
A
=
N
X
n
=1
λ
n
v
n
v
T
n
,
where the
v
n
are orthonormal and the
λ
n
are real and nonnegative.
Since rank(
A
) =
R
, we also have rank(
A
T
A
) =
R
, and so
λ
1
, . . . , λ
R
are all strictly positive above, and
λ
R
+1
=
· · ·
=
λ
N
= 0.
Set
u
m
=
1
√
λ
m
Av
m
,
for
m
= 1
, . . . , R,
U
=
u
1
· · ·
u
R
.
Notice that these
u
m
are orthonormal, as
h
u
m
,
u
‘
i
=
1
√
λ
m
λ
‘
v
T
‘
A
T
Av
m
=
s
λ
m
λ
‘
v
T
‘
v
m
=
(
1
,
m
=
‘,
0
,
m
6
=
‘.
These
u
m
also happen to be eigenvectors of
AA
T
, as
AA
T
u
m
=
1
√
λ
m
AA
T
Av
m
=
p
λ
m
Av
m
=
λ
m
u
m
.
Now let
u
R
+1
, . . . ,
u
M
be an orthobasis for the null space of
U
T
—
concatenating these two sets into
u
1
, . . . ,
u
M
forms an orthobasis for
all of
R
M
.
47
Georgia Tech ECE 6250 Fall 2019; Notes by J. Romberg and M. Davenport. Last updated 9:10, October 30, 2019
Subscribe to view the full document.
Let
V
=
v
1
v
2
· · ·
v
R
. In addition, let
V
0
=
v
R
+1
v
R
+2
· · ·
v
N
,
V
full
=
V
V
0
and
U
0
=
u
R
+1
u
R
+2
· · ·
u
M
,
U
full
=
U
U
0
.
It should be clear that
V
full
is an
N
×
N
orthonormal matrix and
U
full
is a
M
×
M
orthonormal matrix. Consider the
M
×
N
matrix
U
T
full
AV
full
— the entry in the
m
th
rows and
n
th
column of this
matrix is
(
U
T
full
AV
full
)[
m, n
] =
u
T
m
Av
n
=
(
√
λ
n
u
T
m
u
n
n
= 1
, . . . , R
0
,
n
=
R
+ 1
, . . . , N.
=
(
√
λ
n
,
m
=
n
= 1
, . . . , R
0
,
otherwise
.
Thus
U
T
full
AV
full
=
Σ
full
where
Σ
full
[
m, n
] =
(
√
λ
n
,
m
=
n
= 1
, . . . , R
0
,
otherwise
.
 Fall '08
 Staff