When a is full rank r min m n then we can calculate

When A is full rank ( R = min( M, N )), then we can calculate the
pseudo-inverse without using the SVD. There are three cases:
• When A is square and invertible ( R = M = N ), then
A
†
= A
- 1
.
This is easy to check, as here
A = U Σ V
T
where both U , V are N × N,
42
Georgia Tech ECE 6250 Fall 2019; Notes by J. Romberg and M. Davenport. Last updated 9:10, October 30, 2019

and since in this case V V
T
= V
T
V = I and UU
T
= U
T
U =
I ,
A
†
A = V Σ
- 1
U
T
U Σ V
T
= V Σ
- 1
Σ V
T
= V V
T
= I .
Similarly, AA
†
= I , and so A
†
is both a left and right inverse
of A , and thus A
†
= A
- 1
.
• When A more rows than columns and has full column rank
( R = N ≤ M ), then A
T
A is invertible, and
A
†
= ( A
T
A )
- 1
A
T
. (7)
This type of A is “tall and skinny”




A




,
and its columns are linearly independent. To verify equation
( 7 ), recall that
A
T
A = V Σ U
T
U Σ V
T
= V Σ
2
V
T
,
and so
( A
T
A )
- 1
A
T
= V Σ
- 2
V
T
V Σ U
T
= V Σ
- 1
U
T
,
which is exactly the content of ( 6 ).
43
Georgia Tech ECE 6250 Fall 2019; Notes by J. Romberg and M. Davenport. Last updated 9:10, October 30, 2019

• When A has more columns than rows and has full row rank
( R = M ≤ N ), then AA
T
is invertible, and
A
†
= A
T
( AA
T
)
- 1
. (8)
This occurs when A is “short and fat”


A


,
and its rows are linearly independent. To verify equation ( 8 ),
recall that
AA
T
= U Σ V
T
V Σ U
T
= U Σ
2
U
T
,
and so
A
T
( AA
T
)
- 1
= V Σ U
T
U Σ
- 2
U
T
= V Σ
- 1
U
T
,
which again is exactly ( 6 ).
A
†
is as close to an inverse of A as possible
As discussed in above, when A is square and invertible, A
†
is exactly
the inverse of A . When A is not square, we can ask if there is a
better right or left inverse. We will argue that there is not.
Left inverse Given y = Ax , we would like A
†
y = A
†
Ax = x
for any x . That is, we would like A
†
to be a left inverse of
A : A
†
A = I . Of course, this is not always possible, especially
when A has more columns than rows, M < N . But we can
ask if any other matrix H comes closer to being a left inverse
44
Georgia Tech ECE 6250 Fall 2019; Notes by J. Romberg and M. Davenport. Last updated 9:10, October 30, 2019

than A
†
. To find the “best” left-inverse, we look for the matrix
which minimizes
min
H ∈ R N × M
k HA - I k
2
F
. (9)
Here, k · k
F
is the Frobenius norm , defined for an N × M
matrix Q as the sum of the squares of the entries: 3
k Q k
2
F
=
M X
n =1
N X
n =1
| Q [ m, n ] |
2
With ( 9 ), we are finding H such that HA is as close to the
identity as possible in the least-squares sense.
The pseudo-inverse A
†
minimizes ( 9 ). To see this, recognize
(see the exercise below) that the solution
ˆ
H to ( 9 ) must obey
AA
T
ˆ
H
T
= A . (10)
We can see that this is indeed true for
ˆ
H = A
†
:
AA
T
A
†
T
= U Σ V
T
V Σ U
T
U Σ
- 1
V
T
= U Σ V
T
= A .
So there is no N × M matrix that is closer to being a left
inverse than A
†
.
3 It is also true that k Q k 2
F
is the sum of the squares of the singular values of
Q : k Q k
2
F
= λ
2
1
+ · · · + λ
2
p
. This is something that you will prove on the
next homework.