These types of systems are prevalent in array signal processing and
machine learning. We will see that if
R
N
, this system can be
solved in (much) faster than
O
(
N
3
) time.
Note that while
BB
T
is not at all invertible (since it is rank defi
cient),
γ
I
+
BB
T
will be. To see this, set
z
=
B
T
x
,
z
∈
R
R
.
14
Georgia Tech ECE 6250 Fall 2019; Notes by J. Romberg and M. Davenport. Last updated 13:22, November 13, 2019
Subscribe to view the full document.
Then we can solve the system by jointly solving for
x
and
z
:
γ
x
+
Bz
=
b
(5)
B
T
x

z
=
0
.
(6)
Solving the first equations (
5
) yields
x
=
γ

1
(
b

Bz
)
,
and then plugging this into (
6
) gives us
γ

1
B
T
(
b

Bz
)

z
=
0
⇒
(
γ
I
+
B
T
B
)
z
=
B
T
b
and so
z
= (
γ
I
+
B
T
B
)

1
B
T
b
.
But notice that this is an
R
×
R
system of equations.
So it takes
O
(
NR
2
) to construct
γ
I
+
B
T
B
,
then
O
(
R
3
) to solve for
z
,
then
O
(
NR
) to calculate
Bz
(and hence find
x
).
The dominant cost in all of this is
O
(
NR
2
), which is much less than
O
(
N
3
) if
R
N
.
Circulant systems
.
A circulant matrix has the form
H
=
h
0
h
N

1
h
N

2
· · ·
h
1
h
1
h
0
h
N

1
· · ·
h
2
h
2
h
1
h
0
.
.
.
.
.
.
.
.
.
.
.
.
h
N

1
h
N

1
h
1
h
0
15
Georgia Tech ECE 6250 Fall 2019; Notes by J. Romberg and M. Davenport. Last updated 13:22, November 13, 2019
For
H
symmetric, we have
h
k
=
h
N

k
for
k
= 1
, . . . , N

1, although
symmetry does not play too big a role in exploiting this structure.
Circulant matrices have two very nice properties:
•
We know their eigenvectors already — they are the discrete
harmonic sinusoids (i.e. the columns of the
N
×
N
DFT ma
trix).
•
Transforming into the eigenbasis is
fast
thanks to the FFT
(which is
O
(
N
log
N
)).
We can write
H
=
F
Λ
F
H
,
F
[
m, n
] =
1
√
N
e
j
2
πmn/N
and
H

1
=
F
Λ

1
F
H
,
so
H

1
b
=
F
{z}
FFT,
O
(
N
log
N
);
Λ

1
{z}
diagonal weighting,
O
(
N
);
F
H
b

{z
}
FFT,
O
(
N
log
N
)
⇒
solving an
N
×
N
system of equations can be done in
O
(
N
log
N
)
time!
This is
fast
compared to
O
(
N
3
) — on my computer, I can solve a
system like this in
N
= 20 000 in 800
μ
s (compare to 24 seconds for
the general case).
16
Georgia Tech ECE 6250 Fall 2019; Notes by J. Romberg and M. Davenport. Last updated 13:22, November 13, 2019
Subscribe to view the full document.
Toeplitz systems
.
Toeplitz matrices, which are matrices that are constant along their
diagonals, arise in many different signal processing applications, as
they are fundamental in describing the action of linear timeinvariant
systems. For example, suppose we observe the discrete convolution of
an unknown signal
x
of length
N
and a known sequence
2
a
0
, . . . , a
L

1
of length
L
. We can write the corresponding matrix equation as
a
0
0
· · ·
0
a
1
a
0
0
· · ·
0
a
2
a
1
a
0
· · ·
0
.
.
.
a
N

1
a
N

2
· · ·
a
0
.
.
.
a
L

1
a
L

2
· · ·
a
L

N
0
a
L

1
· · ·
a
L

N
+1
.
.
.
0
0
· · ·
· · ·
a
L

1
x
[0]
x
[1]
.
.
.
x
[
N

1]
=
y
[0]
y
[1]
y
[2]
.
 Fall '08
 Staff