Power iterations for computing
v
1
To start, let’s consider the simpler problem of computing the largest
eigenvalue
λ
1
and corresponding eigenvector
v
1
of
A
.
We do this
with the following iteration.
Let
q
0
be an arbitrary vector in
R
N
with unit norm,
k
q
0
k
2
= 1.
Then for
k
= 1
,
2
, . . .
compute
z
k
=
Aq
k

1
q
k
=
z
k
k
z
k
k
2
γ
k
=
q
T
k
Aq
k
Then, as long as
q
0
is not orthogonal to
v
1
,
h
q
0
,
v
1
i 6
= 0, and
λ
1
>
λ
2
, as
k
gets large
q
k
→
v
1
and
γ
k
→
λ
1
.
A detailed proof of this, including rates of convergence, can be found
in Section 8.2 of the Golub/van Loan book. But we can see roughly
why it works with a simple calculation.
9
Georgia Tech ECE 6250 Fall 2019; Notes by J. Romberg and M. Davenport. Last updated 13:22, November 13, 2019
Subscribe to view the full document.
The vector
q
k
above can be written as
q
k
=
A
k
q
0
k
A
k
q
0
k
2
.
Since the eigenvectors of
A
,
v
1
,
v
2
, . . . ,
v
N
form an orthobasis for
R
N
, we can write
q
0
=
α
1
v
1
+
α
2
v
2
+
· · ·
+
α
N
v
N
,
for
α
n
=
h
q
0
,
v
n
i
. Then the expression for
q
k
above becomes
q
k
=
α
1
λ
k
1
v
1
+
α
2
λ
k
2
v
2
+
· · ·
+
α
N
λ
k
N
v
N
p
α
2
1
λ
2
k
1
+
α
2
2
λ
2
k
2
+
· · ·
+
α
2
N
λ
2
k
N
.
If
α
1
6
= 0 and
λ
1
> λ
2
≥ · · · ≥
λ
N
, then as
k
gets large, the first
term in each of the sums above will dominate. Thus for large
k
,
q
k
≈
α
1
λ
k
1
v
1
p
α
2
1
λ
2
k
1
=
v
1
.
Since
v
T
1
Av
1
=
λ
1
and
q
k
≈
v
1
, we also have that
q
T
k
Aq
k
≈
λ
1
.
QR iterations
Now consider the problem of computing all the eigenvectors and
eigenvalues of
A
.
We might be tempted to extend the the power
method by starting with an entire orthobasis
1
Q
0
, then take
Z
k
=
AQ
k

1
,
1
So
Q
0
is
N
×
N
and satisfies
Q
T
0
Q
0
=
I
.
10
Georgia Tech ECE 6250 Fall 2019; Notes by J. Romberg and M. Davenport. Last updated 13:22, November 13, 2019
and then renormalize the columns of
Z
k
to get
Q
k
. The problem
with this is that all of the columns of
Z
k
will converge to
v
1
— this
is just running the power method with
N
different starting points.
What we do instead is “orthonormalize” the columns of
Z
k
, we make
them orthogonal to each other at every iteration as well as unit norm.
This gives us the following iteration:
Let
Q
0
be any orthonormal matrix. For
k
= 1
,
2
, . . . ,
take
Z
k
=
AQ
k

1
(1)
[
Q
k
,
R
k
] = qr(
Z
k
)
(so
Z
k
=
Q
k
R
k
)
.
(2)
Using arguments not too different than for the power method, you
can show (again, see Golub/van Loan Section 8.2 for details) that
Q
k
→
V
,
and
Γ
k
=
Q
T
k
AQ
k
→
Λ
.
A more popular way to state the iteration (
1
)–(
2
) above, and the one
you will see in almost every textbook on numerical linear algebra, is
the following.
Set
Γ
0
=
A
. Then for
k
= 1
,
2
, . . . ,
take
[
U
k
,
R
k
] = qr(
Γ
k

1
)
(so
Γ
k

1
=
U
k
R
k
)
(3)
Γ
k
=
R
k
U
k
(4)
So at each iteration, we are computing a QR factorization, then
reversing it (cute!). This gives us the relation
R
k

1
U
k

1
=
U
k
R
k
.
To see the relationship between version 1 in (
1
)–(
2
) and version 2 in
11
Georgia Tech ECE 6250 Fall 2019; Notes by J. Romberg and M. Davenport. Last updated 13:22, November 13, 2019
Subscribe to view the full document.
(
3
)–(
4
), notice that if we initialize version 1 with
Q
0
=
I
, then
A
=
Q
1
R
1
⇒
Q
1
=
AR

1
1
AQ
1
=
Q
2
R
2
⇒
Q
2
=
A
2
R

1
1
R

1
2
.
 Fall '08
 Staff