(Fall EE236A 2007-08)
Lecture 12 The barrier method
brief history of interior-point methods Newtons method for smooth unconstrained minimization logarithmic barrier function central points, the central path the barrier method
121
The ellipsoid method
1972: ellipsoid method for (nonlinear) convex nondierentiable optimization (Nemirovsky, Yudin, Shor) 1979: Khachiyan proves that the ellipsoid method applied to LP has polynomial worst-case complexity
much slower in practice than simplex very dierent approach from simplex method; extends gracefully to nonlinear convex problems solved important open theoretical problem (polynomial-time algorithm for LP)
The barrier method
122
Interior-point methods
early methods (1950s1960s) methods for solving convex optimization problems via sequence of smooth unconstrained problems logarithmic barrier method (Frisch), sequential unconstrained minimization (Fiacco & McCormick), ane scaling method (Dikin), method of centers (Huard & Lieu) no worst-case complexity theory; (often) worked well in practice fell out of favor in 1970s new methods (1984) 1984 Karmarkar: new polynomial-time method for LP (projective algorithm) later recognized as closely related to earlier interior-point methods many variations since 1984; widely believed to be faster than simplex for very large problems (over 10,000 variables/constraints)
The barrier method 123
Gradient and Hessian
dierentiable function f : Rn R gradient and Hessian (evaluated at x):
f (x) x1 f (x) x2
f (x) =
. .
f (x) xn
,
2 f (x) =
2 f (x) x2 1 2 f (x) x2 x1
. .
2 f (x) x1 x2 2 f (x) x2 2
...
. .
2 f (x) x1 xn 2 f (x) x2 xn
. .
2 f (x) xn x1
2 f (x) xn x2
2 f (x) x2 n
2nd order Taylor series expansion around x: 1 f (y ) f (x) + f (x)T (y x) + (y x)T 2f (x)(y x) 2
The barrier method
124
Positive semidenite matrices
a quadratic form is a function f : Rn R with
n
f (x) = x Ax =
i,j =1
T
Aij xixj
may as well assume A = AT since xT Ax = xT ((A + AT )/2)x A = AT is positive semidenite if xT Ax 0 for all x A = AT is positive denite if xT Ax > 0 for all x = 0
The barrier method
125
Convex dierentiable functions
f : Rn R is convex if for all x and y 01 = f (x + (1 )y ) f (x) + (1 )f (y )
f : Rn R is strictly convex if for all x and y 0<<1 = f (x + (1 )y ) < f (x) + (1 )f (y )
for dierentiable f : 2f (x) positive semidenite 2f (x) positive denite for convex dierentiable f : f (x) = 0 x = argmin f = f convex
f strictly convex
f strictly convex argmin f is unique (if it exists)
The barrier method 126
Pure Newton method
algorithm for minimizing convex dierentiable f : x+ = x 2f (x)1f (x) x+ minimizes 2nd order expansion of f (y ) at x: 1 f (x) + f (x)T (y x) + (y x)T 2f (x)(y x) 2 2nd-order approx. of f d
d d
fd x
The barrier method
d d
x+
127
x+ solves linearized optimality condition: 0 = f (x) + 2f (x)(y x)
1st-order approx. of f
d d d
0 f x x+
intepretations suggest method works very well near optimum
The barrier method
128
Global behavior
pure Newton method can diverge example: f (x) = log(ex + ex), start at x(0) = 1.1
f (x)
3.5
1
f (x)
0.8 0.6
3
2.5
0.4 0.2
2
0 0.2
1.5
0.4 0.6
1
0.8 1
0.5 3
2
1
x
0
1
2
3
3
2
1
x
0
1
2
3
k 1 2 3 4 5
The barrier method
x(k) 1.129 100 1.234 100 1.695 100 5.715 100 2.302 104
f (x(k)) f 5.120 101 5.349 101 6.223 101 1.035 100 2.302 104
129
Newton method with exact line search
given starting point x repeat 1. Compute Newton direction v = 2f (x)1f (x) 2. Line search. Choose a step size t t = argmint>0 f (x + tv ) 3. Update. x := x + tv until stopping criterion is satised
globally convergent very fast local convergence (more later)
The barrier method
1210
Logarithmic barrier function
minimize cT x subject to aT x bi, i = 1, . . . , m i assume strictly feasible: {x | Ax < b} = m log(bi aT x) Ax < b i dene logarithmic barrier (x) = i=1 + otherwise
as x approaches boundary of {x | Ax < b}
The barrier method 1211
Derivatives of barrier function
m
(x) =
i=0 m
1 a = AT d Tx i bi ai
2(x) =
1 aiaT = AT diag(d)2A (bi aT x)2 i i i=1
where d = (1/(b1 aT x), . . . , 1/(bm aT x)) m 1 is smooth on C = {x | Ax < b} is convex on C : for all y Rn, y T 2(x)y = y T A diag(d)2Ay = diag(d)Ay strictly convex if rank A = n
The barrier method 1212
2
0
The analytic center
argmin (if it exists) is called analytic center of inequalities optimality conditions:
m
(x) =
i=1
1 = ai 0 bi aT x i
exists if and only if C = {x | Ax < b} is bounded unique if A has rank n dierent descriptions of the same polyhedron may have dierent analytic centers (e.g., adding redundant inequalities moves analytic center) eciently computed via Newtons method (given strictly feasible starting point)
The barrier method 1213
Force eld interpretation
associate with constraint aT x bi, at point x, the force i Fi = ai bi aT x i
Fi points away from constraint plane Fi = 1/dist(x, constraint plane) is potential of 1/r force eld associated with each constraint plane
Fi
forces balance at analytic center
The barrier method 1214
Central path
x(t) = argmin(tcT x + (x)) for t > 0
x
(we assume minimizer exists and is unique) curve x(t) for t 0 called central path can compute x(t) by solving smooth unconstrained minimization problem (given a strictly feasible starting point) t gives relative weight of objective and barrier barrier traps x(t) in strictly feasible set intuition suggests x(t) converges to optimal as t x(t) characterized by tc + 1 ai = 0 bi aT x(t) i i=1
m
The barrier method
1215
example
c t = 100 t = 10
t=1 rxac
The barrier method
1216
Force eld interpretation
imagine a particle in C , subject to forces ith constraint generates constraint force eld Fi(x) = 1 ai bi aT x i
is potential associated with constraint forces constraint forces push particle away from boundary of feasible set superimpose objective force eld F0(x) = tc pulls particle toward small cT x t scales objective force at x(t), constraint forces balance objective force; as t increases, particle is pulled towards optimal point, trapped in C by barrier potential
The barrier method 1217
Central points and duality
recall x = x(t) satises
m
c+
i=1
ziai = 0,
zi =
1 >0 t(bi aT x) i
i.e., z is dual feasible and p bT z = cT x +
i
zi(aT x bi) = cT x m/t i
summary: a point on central path yields dual feasible point and lower bound: cT x(t) p cT x(t) m/t (which proves x(t) becomes optimal as t )
The barrier method 1218
Central path and complementary slackness
optimality conditions: x optimal Ax b and z s.t. z 0, AT z + c = 0, zi(bi aT x) = 0 i
centrality conditions: x is on central path Ax < b and z , t > 0 s.t. z 0, AT z + c = 0, zi(bi aT x) = 1/t i
for t large, x(t) almost satises complementary slackness central path is continuous deformation of complementary slackness condition
The barrier method
1219
Unconstrained minimization method
given strictly feasible x, desired accuracy > 0 1. t := m/ 2. compute x(t) starting from x 3. x := x(t)
computes -suboptimal point on central path (and dual feasible z ) solve constrained problem via Newtons method works, but can be slow
The barrier method
1220
Barrier method
given strictly feasible x, t > 0, tolerance > 0 repeat 1. compute x(t) starting from x, using Newtons method 2. x := x(t) 3. if m/t , return(x) 4. increase t also known as SUMT (Sequential Unconstrained Minimization Technique) generates sequence of points on central path simple updating rule for t: t+ = t (typical values 10 100) steps 14 above called outer iteration; step 1 involves inner iterations (e.g., Newton steps) tradeo: small = few inner iters to compute x(k+1) from x(k), but more outer iters
The barrier method 1221
Example
minimize cT x subject to Ax b A R10050, Newton with exact line search
1000
100
10
duality gap
width of steps shows #Nt. iters per outer iter; height of steps shows reduction in dual. gap (1/) gap reduced by 105 in few tens of Newton iters gap decreases geometrically
10 20
1
0.1
0.01
0.001
0.0001 0
= 50 = 180 = 3
30 40 50 60
can see trade-o in choice of
70
total # Newton iters
The barrier method
1222
example continued . . . trade-o in choice of : #Newton iters required to reduce duality gap by 106
90
total # Newton iters
80
70
60
50
40
30 0
20
40
60
80
100
120
140
160
180
200
works very well for wide range of
The barrier method
1223
Phase I
to compute strictly feasible point (or determine none exists) set up auxiliary problem: minimize w subject to Ax b + w1 easy to nd strictly feasible point (hence barrier method can be used) can use stopping criterion with target value 0 if we include constraint on cT x, minimize w subject to Ax b + w1 cT x M phase I method yields point on central path of original problem many other methods for nding initial primal (& dual) strictly feasible points
The barrier method 1224