Lecture 23: <a href="/keyword/steepest-descent/" >steepest descent</a> Subgradient Methods April 16, 2007 Lecture 23 Outline <a href="/keyword/directional-derivative/" >directional derivative</a> s and Subgradients More Subgradient Properties <a href="/keyword/steepest-descent/" >steepest descent</a> Subgradient Bundle-Type Methods using &quot;<a href="/keyword/steepest-descent/" >steepest descent</a> Idea&quot; -Subgradients and -Subdifferentals Convex Optimization 1 Lecture 23 From Subdifferential to <a href="/keyword/directional-derivative/" >directional derivative</a> Theorem Let f be convex with dom f = Rn. We then have for any x Rn and any d Rn, f (x; d) = max sT d sf (x) If we know the whole subdifferential f (x), then for any direction d, the <a href="/keyword/directional-derivative/" >directional derivative</a> at x is obtained by maximizing dT s over s f (x) There is a relation in the other direction: having the <a href="/keyword/directional-derivative/" >directional derivative</a> s f (x; d) for all d, one can recover the subdifferential f (x) Convex Optimization 2 Lecture 23 From <a href="/keyword/directional-derivative/" >directional derivative</a> to Subdifferential Theorem Let f be convex with dom f = Rn. We then have for any x Rn and any d Rn, f (x) = {s | f (x; d) sT d for all d Rn} (= K) Proof: We have [from the definitions of subgradient and f (x; d)] that for any s f (x), f (x + d) - f (x) f (x; d) = lim 0 sT (x + d - x) lim 0 = sT d Hence f (x) K . Suppose now s K , so that sT d f (x; d) for all d Rn. Thus, we have for any d Rn: f (x + d) - f (x) sT d inf f (x + d) - f (x) &gt;0 By letting d = z - x for any z Rn, it follows that s f (x) Convex Optimization 3 Lecture 23 <a href="/keyword/directional-derivative/" >directional derivative</a> and Optimality Theorem Optimality Condition Let f be convex with dom f = Rn and let X Rn be closed and convex. The vector x is a minimizer of f over X if and only if f (x; (x - x)) 0 for all x X Proof: The result follows from the following two facts x is optimal if and only if there is s f (x) such that sT (x - x) 0 for all x X f (x; d) = maxsf (x) sT d for all d Rn [so we let d = x - x for any x X ] Note: When x is not optimal there exists a feasible direction d such that f (x; d) &lt; 0 Convex Optimization 4 Lecture 23 Subdifferential over Bounded Set Theorem Uniform boundedness over bounded set Let f be convex with dom f = Rn and let X Rn be bounded. Then, the set xX f (x) is bounded. Proof: Assume the contrary. Let {xk } X be such that dk f (xk ) is unbounded. Define yk = dk . We then have by the subgradient inequality d k f (xk + yk ) - f (xk ) dT yk = dk k Both sequences {xk } and {yk } are bounded, so they have limit points. By taking arbitrary limit points x and y , and taking appropriate subsequence ^ ^ K, it follows by continuity of f that f (^ + y ) - f (^) lim dk = x ^ x kK -a contradiction Convex Optimization 5 Lecture 23 Continuity Properties of Subdifferential Def. A mapping that maps points in Rn into subsets of Rn [(x) Rn] is referred to as set valued mapping Def. A set valued mapping : Rn P(Rn) is upper-semicontinuous ^ ^ when for xk x and dk (xk ) with dk d, we have that d (^) ^ x Convex Optimization 6 Lecture 23 Theorem Upper-semicontinuity of Subdifferential Mapping Let f be convex with dom f = Rn. Then, the subdifferential (set valued) mapping x f (x) is upper-semicontinuous: for xk x and sk f (xk ) with sk ^, we have ^ f (^) ^ s s x Proof: Straight from the subgradient inequality and continuity of f . Specifically, let xk x and sk f (xk ) with sk ^. By subgradient inequality, ^ s we have for every k, f (z) f (xk ) + sT (z - xk ) k for any z Rn Letting k and using the continuity of f , we obtain f (z) f (^) + ^T (z - x) x s ^ for any z Rn Thus, ^ f (^) s x Convex Optimization 7 Lecture 23 <a href="/keyword/steepest-descent/" >steepest descent</a> Direction for Non-Differentiable Convex Function Let f be convex with dom f (x) = Rn. Let x Rn be a point that does not minimize f over Rn Which direction provides the <a href="/keyword/steepest-descent/" >steepest descent</a> at the given x? Can be modeled as: minimize f (x + d) - f (x) over d Rn Since f (x; d) f (x + d) - f (x), we can model it as: minimize f (x; d) over d Rn The optimal value to of the preceding problem may not be finite [unbounded constraint set] Determining the <a href="/keyword/steepest-descent/" >steepest descent</a> direction is modeled as: minimize f (x; d) subject to d 1 Convex Optimization 8 Lecture 23 Minimum Norm Subgradient - <a href="/keyword/steepest-descent/" >steepest descent</a> Using the relation f (x; d) = maxsf (x) sT d, the <a href="/keyword/steepest-descent/" >steepest descent</a> direction problem becomes: min max sT d d 1 sf (x) The constraint sets for d and s are compact, so the &quot;max&quot; and &quot;min&quot; can exchange the order, and the problem is equivalent to: max sf (x) min sT d d 1 Note that for any vector s Rn and d with d 1, we have s with &quot;=&quot; holding for d = sT d - s s Thus max sf (x) min sT d = max (- s ) = - min d 1 sf (x) sf (x) s The <a href="/keyword/steepest-descent/" >steepest descent</a> direction at x is provided by the smallest norm subgradient in the subdifferential f (x) Convex Optimization 9 Lecture 23 <a href="/keyword/steepest-descent/" >steepest descent</a> Method Let f be convex with dom f = Rn Consider the problem minimize f (x) subject to x Rn Consider the <a href="/keyword/steepest-descent/" >steepest descent</a> method xk+1 = xk - k dk dk is the smallest norm subgradient: dk = minsf (xk ) s k is the exact line-search step: minimizes f (xk - dk ) over &gt; 0 Aside from being impractical, the method is not sound Wolfe provided an example when the method fails to converge [1975] The failure is attributed to the &quot;lack of continuity&quot; of the subdifferential mapping A set valued mapping is continuous at x when it is upper- and lower-semicontinuous at x The subdifferential mapping is not &quot;lower-semicontinuous&quot; Convex Optimization 10 Lecture 23 Bundle-Type Methods Basically, there are two types: Descent type methods that at each iteration Generate a new bundle of &quot;subgradients&quot; to approximate the subdifferential set Use a concept of &quot;approximate subgradient&quot;, also known as - subgradient These methods are memoryless The methods that each iteration Maintain and update a &quot;global&quot; bundle of subgradients [from all past iterates] to approximate the function Uses cutting-plane idea These methods have memory Convex Optimization 11 Lecture 23 Descent Type &quot;Approximate&quot; Subgradient Methods Let &gt; 0 be a desired level of accuracy for minimizing f over Rn Let xk be the current iterate, and &gt; 0 is a given stepsize and is a factor for decreasing the stepsize Suppose that the current bundle Bk consists of approximate subgradients g1, . . . , gk-1 of f at xk : Bk = {g1, . . . , gk-1} <a href="/keyword/steepest-descent/" >steepest descent</a> [in the bundle]: Set dk = mingconv(Bk ) g Loop Until f (xk - dk ) f (xk ) - or becomes &quot;small enough&quot; keep reducing the step = If the loop is exited with f (xk - dk ) f (xk ) - go to the next iteration: set xk+1 = xk - dk [and start building the bundle at xk+1] If the loop is exited with small enough enlarge the bundle: Bk = Bk {gk } where gk is a subgradient of f at xk - dk [an approx. sgd. of f at xk ], and go to <a href="/keyword/steepest-descent/" >steepest descent</a> step Convex Optimization 12 Lecture 23 At any iteration, the bundle Bk is initialized by computing g1 f (xk ) The first type of such descent subgradient method was proposed in the 70's [Bertsekas 1973, Lemarechal 1974] One can use k that changes at each iteration Main criticism: No guidance on &quot;adjusting k &quot; If k large, the accuracy of the solution is low If k is small, the high accuracy is required for &quot;the approximation&quot; of the subdifferential Algorithm takes too many sub-iterations to build the bundle without making any progress in function value reduction Convex Optimization 13 Lecture 23 -Subgradient and -Subdifferential An -subgradient is also referred to as &quot;approximate subgradient&quot; An -subgradient provides a linearization underestimating perturbed function f + instead of f Def. For a given &gt; 0, a vector s is an -subgradient of f at x when f (x) + sT (z - x) f (z) + for all z dom f The -subdifferential of f at x is the set of all -subgradients of f at x, denoted by f (x) Convex Optimization 14