.
.
y
[
N

1]
.
.
.
y
[
L

1]
y
[
L
]
.
.
.
y
[
L
+
N

2]
If we recover
x
from
y
using leastsquares,
b
x
=
A
†
y
= (
A
T
A
)

1
A
T
y
,
then the
N
×
N
system we need to invert,
H
=
A
T
A
is also Toeplitz
(and is of course symmetric and nonnegative definite):
H
=
h
0
h
1
· · ·
h
N

1
h
1
h
0
h
1
· · ·
h
N

2
h
2
h
1
h
0
· · ·
h
N

3
.
.
.
h
N

1
· · ·
· · ·
h
0
.
2
We are going to save some space in this section by using subscript notation
to index signals that are going in a matrix; i.e.
a
k
instead of
a
[
k
].
17
Georgia Tech ECE 6250 Fall 2019; Notes by J. Romberg and M. Davenport. Last updated 13:22, November 13, 2019
Subscribe to view the full document.
Here is a quick example in MATLAB:
>> A = toeplitz([1; randn(5,1); zeros(3,1)], [1 zeros(1,3)])
A =
1.0000
0
0
0
0.4336
1.0000
0
0
0.3426
0.4336
1.0000
0
3.5784
0.3426
0.4336
1.0000
2.7694
3.5784
0.3426
0.4336
1.3499
2.7694
3.5784
0.3426
0
1.3499
2.7694
3.5784
0
0
1.3499
2.7694
0
0
0
1.3499
>> H = A’*A
H =
23.6023
6.8156
5.0905
1.9151
6.8156
23.6023
6.8156
5.0905
5.0905
6.8156
23.6023
6.8156
1.9151
5.0905
6.8156
23.6023
Symmetric Toeplitz systems appear frequently in linear prediction,
array processing, adaptive filtering, and other areas of statistical sig
nal processing. An
N
×
N
Toeplitz system
H
can be inverted in
O
(
N
2
) time using the
LevinsonDurbin
algorithm.
The algo
rithm is relatively easy to derive, and even easier to implement. The
increase in efficiency it offers is significant, as the difference between
O
(
N
3
), the cost of solving the system using a general linear solver,
and
O
(
N
2
) is enormous even for moderate
N
.
18
Georgia Tech ECE 6250 Fall 2019; Notes by J. Romberg and M. Davenport. Last updated 13:22, November 13, 2019
Technical Details:
LU
, Cholesky, and Symmetric
QR
Factorizations
Example of
LU
factorization
Start with the original matrix
A
=
[
r
]2
1

1

3

1
2

2
1
2
Eliminate the lowerleft term. In this case we can add the first row
to the third row, i.e.
[
r
] 1
0
0
0
1
0
1
0
1
[
r
]2
1

1

3

1
2

2
1
2
=
[
r
] 2
1

1

3

1
2
0
2
1
L
1
A
=
A
1
Eliminate the term in (row,column) = (2
,
1) by adding 3
/
2 the top
row to the second row:
[
r
]1
0
0
1
.
5
1
0
0
0
1
[
r
] 2
1

1

3

1
2
0
2
1
=
[
r
] 2
1

1
0
0
.
5
0
.
5
0
2
1
L
2
A
1
=
A
2
Finally, we eliminate (3
,
2) by subtracting 4 times the second row
from the third row:
[
r
]1
0
0
0
1
0
0

4
1
[
r
] 2
1

1
0
0
.
5
0
.
5
0
2
1
=
[
r
] 2
1

1
0
0
.
5
0
.
5
0
0

1
L
3
A
2
=
U
.
19
Georgia Tech ECE 6250 Fall 2019; Notes by J. Romberg and M. Davenport. Last updated 13:22, November 13, 2019
Subscribe to view the full document.
So we have:
L
3
L
2
L
1
A
=
U
,
where
U
is upper triangular and the
L
i
are all lower triangular with
1 along the diagonal and exactly 1 nonzero offdiagonal term. Using
the facts that
1. the inverses of
L
i
are also lower triangular with 1 down the
diagonal and exactly one nonzero offdiagonal term (show this
at home!), and
2. the product of two lower triangular matrices with 1 down the
diagonal is again lowertriangular with 1 down the diagonal
(show this at home!),
we have
A
= (
L

1
1
L

1
2
L

1
3
)
U
=
LU
,
where
L
is lower diagonal with 1 down the diagonal. In the example
above, we have
L
=
L

1
1
L

1
2
L

1
3
=
[
r
] 1
0
0
0
1
0

1
0
1
[
r
]1
0
0

1
.
5
1
0
0
0
1
[
r
]1
0
0
0
1
0
0
4
1
=
[
r
]1
0
0

1
.
5
1
0

1
4
1
 Fall '08
 Staff