(Fall EE236A 2007-08)
Lecture 13 Convergence analysis of the barrier method
complexity analysis of the barrier method convergence analysis of Newtons method choice of update parameter bound on the total number of Newton iterations initialization
131
Complexity analysis
well analyze the method of page 1221 with update t+ = t starting point x(t(0)) on the central path O( m log((0)/)) caveats: methods with good worst-case complexity dont necessarily work better in practice were not interested in the numerical values for the boundonly in the exponent of m and n doesnt include initialization insights obtained from analysis are more valuable than the bound itself
Convergence analysis of the barrier method 132
main result: #Newton iters is bounded by (where (0) = m/t(0))
Outline
1. convergence analysis of Newtons method for
m
(x) = tc x
T
i=1
log(bi aT x) i
(will give us a bound on the number of Newton steps per outer iteration) 2. eect of on total number of Newton iterations to compute x(t) from x(t) 3. combine 1 and 2 to obtain the total number of Newton steps, starting at x(t(0))
Convergence analysis of the barrier method
133
The Newton decrement
Newton step at x: v = 2(x)1(x)
= (AT diag(d)2A)1(tc + AT d)
where d = (1/(b1 aT x), . . . , 1/(bm aT x)) m 1 Newton decrement at x: (x) = = =
i=1
(x)T 2(x)1(x) v T 2(x)v
m 1/2
aT v i bi aT x i
2
=
diag(d)Av
Convergence analysis of the barrier method
134
theorem. if = (x) < 1, then is bounded below and (x) (x(t)) log(1 )
4 3.5 3 2.5 2 1.5 1 0.5 0 0
0.2
0.4
0.6
0.8
1
if 0.81, then (x) (x(t)) + useful as stopping criterion for Newtons method
Convergence analysis of the barrier method 135
proof: w.l.o.g. assume b Ax = 1; let x = x(t), z = 1 + Av = Av < 1 2(x)v = AT Av = (x) = tc AT 1 = z = 1 + Av 0 = AT z = tc
m
m
tc x
T
i=1
log(bi
aT x) i
= z Ax z T Ax +
T
i=1 m i=1 T
log(bi aT x) i log zi z T (b Ax) + m
m
= (1 + Ax) z + = tcT x +
i
log zi + m
i=1
(aT v + log(1 + aT v )) i i
tcT x + + log(1 )
Convergence analysis of the barrier method 136
inequalities follow from: 1. log y log z + zy 1 for y, z > 0 2.
m i=1 (yi
log(1 + yi)) y log(1 y ) if y < 1
Convergence analysis of the barrier method
137
Local convergence analysis
x+ = x 2(x)1(x) theorem: if < 1, then Ax+ < b and + 2 ( is Newton decrement at x; + is Newton decrement at x+) gives bound on number of iterations: suppose we start at x(0) with (0) 0.5, then (x) (x(t)) < after fewer than log2 log2(1/ ) iterations called region of quadratic convergence
practical rule of thumb: 56 iterations
Convergence analysis of the barrier method
138
proof. 1. 2 =
m T2 i=1 (ai v ) /(bi
aT x)2 < 1 implies aT (x + v ) < bi i i
2. assume b Ax+ = 1; let w = 1 d diag(d)2Av (+)2 = Av +
2
= =
Av +
m
2
w + Av + Av + (1 di)4 (diaT v )4 i
2(Av +)T (w + Av +)
2
(1)
(2)
i=1 m
=
i=1
diag(d)Av
4
= 4
(1) uses AT w = tc + AT 1, AT Av + = tc AT 1 (2) uses Av = Ax+ b Ax + b = 1 + diag(d)11, therefore diaT v = 1 di and wi = (1 di)2 i
Convergence analysis of the barrier method 139
Global analysis of Newtons method
damped Newton algorithm: x+ = x + sv , v = 2(x)1(x) step size to the boundary: s = 1 where = max aT v i bi aT x i aT v > 0 i ( = 0 if Av 0)
theorem. for = s 1/(1 + ), (x + sv ) (x) ( log(1 + )) very simple expression for step size if 0.5, same bound if s is determined by an exact line search (x + (1 + )1v ) (x) 0.09 (hence, convergence)
Convergence analysis of the barrier method 1310
proof. dene f (s) = (x + sv ) for 0 s < 1/ f (s) = v T (x + sv ), by integrating the upper bound
m
f (s) = v T 2(x + sv )T v
for Newton direction v : f (0) = f (0) = 2 aT v i bi aT x saT v i i
2
f (s) =
i=1
f (0) (1 s)2
twice, we obtain f (s) f (0) + sf (0) f (0) (s + log(1 s)) 2
upper bound is minimized by s = f (0)/(f (0) f (0)) = 1/(1 + ) f (s) f (0) f (0) ( log(1 + )) 2 f (0) ( log(1 + )) (since ))
1311
Convergence analysis of the barrier method
Summary
given x with Ax < b, tolerance (0, 0.5) repeat 1. Compute Newton step at x: v = 2(x)1(x)
2. Compute Newton decrement: = (v T 2(x)v )1/2 3. If , return(x) 4. Update x: If 0.5, x := x + (1 + )1v where = max{0, maxi aT v/(bi aT x)} i i else, x := x + v
upper bound on #iterations, starting at x: log2 log2(1/ ) + 11 ((x) (x(t))) usually very pessimistic; good measure in practice: 0 + 1 ((x) (x(t))) with empirically determined i (0 5, 1 11)
Convergence analysis of the barrier method 1312
#Newton steps per outer iteration
#Newton steps to minimize (x) = t+cT x theorem. if z > 0, AT z + c = 0, then
m m i=1 log(bi
aT x) i
(x (t )) t b z +
+
+T
log zi + m(1 + log t+)
i=1
in particular, for t+ = t, zi = 1/t(bi aT x(t)): i (x(t+)) (x(t)) m( 1 log ) yields estimates for #Newton steps to minimize starting at x(t): 0 + 1m( 1 log ) is an upper bound for 0 = log2 log2(1/ ), 1 = 11 is a good measure in practice for empirically determined 0, 1
Convergence analysis of the barrier method 1313
proof. if z > 0, AT z + c = 0, then
m
(x) = t c x t+cT x +
+T
i=1 m
log(bi aT x) i log zi t+z T (b Ax) + m(1 + log t+) log zi + m(1 + log t+)
i=1 m
= t+bT z +
i=1
for zi = 1/(t(bi aT x(t)), t+ = t, this yields i (x(t+)) (x(t)) m( 1 log )
Convergence analysis of the barrier method
1314
Bound on total #Newton iters
suppose we start on central path with t = t(0) number of outer iterations: log((0)/) #outer iters = log (0) = m/t(0): initial duality gap (0)/: reduction in duality gap upper bound on total #Newton steps: log((0)/) (0 + 1m( 1 log )) log 0 = log2 log2(1/ ), 1 = 11 can use empirical values for i to estimate average-case behavior
Convergence analysis of the barrier method 1315
Strategies for choosing
independent of m: #Newton steps per outer iter O(m) total #Newton steps O(m log((0)/))) = 1 + / m with independent of m #Newton steps per outer iter O(1) total #Newton steps O( m log((0)/)))
follows from: m( 1 log ) 2/2, because x x2/2 log(1 + x) for x > 0 log(1 + / m) log(1 + )/ m for m 1
Convergence analysis of the barrier method
1316
Choice of initial t
rule of thumb: given estimate p of p, choose m/t cT x p (since m/t is duality gap) via complexity theory (c.f. page 1312) given dual feasible z , #Newton steps in rst iteration is bounded by an ane function of
m
t(c x + b z ) + (x)
T
T
i=1
log zi m(1 + logt)
= t(cT x + bT z ) m log t + const. choose t to minimize bound; yields m/t = cT x + bT z there are many other ways to choose t
Convergence analysis of the barrier method 1317