22: Lecture Subgradient Methods
April 11, 2007
Lecture 22
Outline
Methods with Dynamic Stepsize Rule Directional Derivatives and Subgradients -Subgradients and -Subdifferentals
Convex Optimization
1
Lecture 22
Method with Dynamic Stepsize with Known f
General Assumption The function f (x) is convex and domf = Rn
The set X is nonempty closed and convex
Theorem Assume that General Assumption holds and X is nonempty. Then, {xk } generated by the subgradient method with the stepsize k = f (xk )-f converges to an optimal solution. s 2
k Convex Optimization 2
Lecture 22
Proof
We use the Basic Iterate Relation: for any y X and any k 0,
xk+1 - y
2
xk - y
2
- 2k (f (xk ) - f (y)) + 2 sk k
2
Letting y = x for any x X , we obtain
xk+1 - x
2
= =
- 2k (f (xk ) - f ) + 2 sk 2 k f (xk ) - f 2 2 xk - x - k sk - k 2 2 sk
2
xk - x
(f (xk ) - f ) xk - x 2 - sk 2
2
The sequence {xk } is bounded, thus subgradient norms sk are uniformly bounded by some scalar L. Therefore, for all k and any x X ,
xk+1 - x
2
xk - x
2
(f (xk ) - f ) - L2
2
(1)
Convex Optimization
3
Lecture 22
Eq. (1) implies that
(f (xk )-f )2 L2
xk - x
2
- xk+1 - x
2
2
, and
2 (f (xk ) - f ) L2 x0 - x k=0
Hence f (xk ) f . Since {xk } X is bounded, it has limit points that belong to X by closedness of X . By continuity of f and the fact f (xk ) f , it follows that all limit points are optimal. We now show that {xk } has a unique limit point. To arrive at a contradiction, suppose there are at least two: say x and x with x = x. Let {xm}M ~ ^ ~ ^ and {xk }K be subsequences converging to x and x, respectively. By Eq. ~ ^ (1), it follows that for all m, k with m k,
xm - x < xk - x for all x X Let x = x, and m M and k K. Then by taking limits, we obtain ^
m mM
lim xm - x lim xk - x ^ ^
k kK
implying that x - x = 0 - a contradiction. ~ ^
Convex Optimization 4
Lecture 22
Convergence Rate
The convergence rate of the method with dynamic stepsize using f is linear at best For a function f with sharp minima, i.e., such that for some > 0 f (x) - f dist(x, X ) The rate is linear xk - x c k x0 - x ~ ~
for all x
for all k 0
where x X is the limit point of {xk } and ~
2 c= 1- 2 L and L is an upper bound on the subgradient norms sk The rate is important for general understanding of the method It is rare that we can take advantage of this result in practice
Convex Optimization 5
Lecture 22
Method with Dynamic Stepsize with Unknown f
xk+1 = PX [xk - k sk ]
lev fk = min f (xi) -
0ik
lev f (xk ) - fk k = sk 2
with > 0
Interpretation of the method is similar to that of the preceding version: the figures (below) illustrate the behavior when the feasible level set lev Lk = {x X | f (x) fk } is nonempty and is empty
Convex Optimization
6
Lecture 22
Convergence Result
Theorem Let General Assumption hold and f be finite. Then, for the lev lev method using k = [f (xk ) - fk ]/ sk 2, fk = min0ik f (xi) - with > 0, we have
k 0ik
lim min f (xi) f +
Proof: Assume the contrary. Then, there is an
f (xi) 2 + f +
> 0 such that
for all i
Let y X be such that + f f (y). Thus,
0ik
min f (xi) f (y) +
for all k
Convex Optimization
7
Lecture 22
From the Basic Iterate Relation: for y X and any k 0,
xk+1 - y
2
= =
xk - y xk - y xk - y xk - y
sk 2
2 2 2 2
- 2k (f (xk ) - f (y)) + 2 sk k
2
lev - k 2(f (xk ) - f (y)) - f (xk ) + fk
- k f (xk ) - f (y) + min f (xi) - f (y) -
0ik
- k (f (xk ) - f (y))
Note that k
for all k and also f (xk ) - f (y) . Hence 2 2 2 xk+1 - y xk - y - sk 2 Thus, {xk } is bounded, and the subgradients {sk } are bounded. Let sk L for all k. We then have 2 2 2 2 2 xk+1 - y xk - y - 2 x0 - y - (k + 1) 2 , L L
which cannot hold for sufficiently large k.
Convex Optimization 8
Lecture 22
Convergence Rate
Assuming that the optimal set X is nonempty, the smallest number K of iterations needed to have
0ik
min f (xi) f +
is given by
dist2(x0, X )L2 K= 2
Key observation used in the proof: (similar to the preceding proof) the 2 distance dist2(xk , X ) is reduced at least by L2 whenever f (xk ) > f + lev NOTE For the method using dynamic step with an estimate fk
Its behavior is similar to the method with a constant step k = Basic difference: it does not require subgradient boundedness for convergence, while the method with a constant step does
Convex Optimization
9
Lecture 22
Comments
Subgradient methods so considered far: Use any subgradient that is available at a given iterate Simple for implementation Convergence rate is at best linear Useful in large-scale and decentralized computations Main criticism: There are no general stopping rules A specific critera have been designed within a particular application and/or the "origin" of the convex problem Alternative methods exist: Bundle methods Use a carefully selected subgradient at a given iterate More sophisticated for implementation There is a general stopping criteria
Convex Optimization 10
Lecture 22
Slopes of Convex Scalar Function
Here, f : (a, b) R is (scalar) convex function over an interval (a, b), and x, y, z (a, b) are such that x < y < z Key property (implied by convexity of f ): Slopes are nondecreasing
f (y) - f (x) f (z) - f (x) f (z) - f (y) y-x z-x z-y
for all x < y < z
Convex Optimization
11
Lecture 22
z-y z + z-x x and by convexity of f , we have y-x z-y f (y) f (z) + f (x) z-x z-x Writing y - x = z - x - (z - y), we have z-y z-y f (y) f (z) - f (z) + f (x) z-x z-x implying that
Indeed, we have y =
y-x z-x
(2)
z-y [f (z) - f (x)] f (z) - f (y) z-x
f (z) - f (x) f (z) - f (y) z-x z-y
Similarly, by writing z - y = z - x - (y - x), from Eq. (2) we obtain y-x y-x f (y) f (z) + f (x) - f (x) z-x z-x Hence
f (y) f (x)+ y-x [f (z) - f (x)] z-x f (y) - f (x) f (z) - f (x) y-x z-x
Convex Optimization
12
Lecture 22
Right and Left Derivatives of Scalar Function
Def. Right derivative of f at x, denoted by f +(x), is given by f (x + ) - f (x) f +(x) = lim 0 Lemma For a convex f , we have f +(x) = inf >0 f (x+)-f (x) Proof: From nondecreasing slope poperty, where y = x+ and z = x+ for 0 < < , we obtain f (x + ) - f (x) f (x + ) - f (x) Thus, the ratio f (x+)-f (x) is nondecreasing in , and therefore the limit of the ratio as 0 is the same as the infimum of the ratio over > 0 Def. Left derivative of f at x, denoted by f -(x), is given by f (x) - f (x - ) f -(x) = lim 0 Lemma For a convex f , we have f -(x) = sup>0 f (x)-f (x-)
Convex Optimization 13
Lecture 22
Directional Derivative
Def. Directional derivative of a function f : Rn R at x in (nonzero) direction d, denoted by f (x; d), is given by f (x + d) - f (x) f (x; d) = lim 0 Lemma For a convex f , we have f (x; d) = inf >0 f (x+d)-f (x) Proof: The result from the right derivative property for a convex scalar function () = f (x + d) and the relation
f (x + d) - f (x) () - (0) =
Convex Optimization
14
Lecture 22
Directional Derivative and Subdiferential
Theorem Let f be convex with dom f = Rn. We then have for any x Rn and any d Rn,
f (x; d) = max sT d
sf (x)
Proof have
Let x and d = 0 be arbitary. From subgradient inequality, we
f (x + d) f (x) + sT d
for any s f (x)
Thus,
f (x + d) - f (x) max sT d f (x; d) max sT d sf (x) sf (x) To show that equality holds, we consider the interior of the epigraph of the function f , i.e., {(x, w) | f (x) < w} and the half-line C in Rn+1 given by C = {(z, v) | z = x + d, v = f (x) + f (x; d), > 0}
Convex Optimization
15
Lecture 22
The interior of the epigraph is convex. The half-line C is convex. We show that they are disjoint. Suppose they are not, so that there is (z, w) such that f (z) < w and z = x + d with w = f (x) + f (x, d) for some ^ ^ . By directional derivative property f (x; d) = inf >0 f (x+d)-f (x) , it ^ follows that
^ f (x + d) < f (x) + f (x, d) f (x) + f (x + d) - f (x) ^ ^
- a contradiction. Hence, int(epi f ) and C are disjoint. By the Separating Hyperplane Theorem, there is a hyperplane separating C and int(epi f ). Therefore, there is a vector (a, ) Rn+1, (a, ) = 0, such that
aT (x + d) + [f (x) + f (x; d)] aT z + w
for all > 0, f (x) < w, x Rn. We must have 0, since w can be arbitrarily large. If = 0, then it follows that a = 0, which cannot hold since (a, ) = 0.
Convex Optimization 16
Lecture 22
Thus, we can divide by and by letting ~ = a/ , we have a
a ~T (x + d) + f (x) + f (x; d) ~T z + w a
(3)
for all > 0, f (x) < w, x Rn. Letting 0 and w f (z), we obtain
~T (x - z) + f (x) f (z) a
implying that -~ f (x). By setting z = x and = 1 in Eq. (3), we a also obtain
f (x; d) -~T d a
which together with the inequality established earlier
f (x; d) max sT d
sf (x)
implies that
f (x; d) = max sT d
sf (x)
and the maximum is attained at -~. a
Convex Optimization 17