Coursehero >> Pennsylvania >> Lehigh >> IE 417

Course Hero has millions of student submitted documents similar to the one below including study guides, homework solutions, papers, exam answer keys and textbook solutions.

Time: Last Conjugate Gradient Solving Ax = b IE417: Nonlinear Programming: Lecture 10 Jeff Linderoth Department of Industrial and Systems Engineering Lehigh University Or n n min (x) = 1/2 xT Ax - bT x, with A S++ def Conjugate Gradient Algorithm 1 2 3 Choose x0 .r0 = Ax0 - b, d0 = -d0 , k = 0 k = T -rk dk T Ad dk k 16th February 2006 xk+1 = xk + k dk k+1 = T rk+1 Adk dT Adk k 4 5 6 lehigh-logo T dk+1 = -rk+1 + k+1 dk If rk = 0, stop. Else k = k + 1, Go to 2. Jeff Linderoth IE417:Lecture 10 lehigh-logo Jeff Linderoth IE417:Lecture 10 CG for Unconstrained Optimization xRn Today: Quasi-Newton Recall: mk (d) = f (xk ) + f (xk )T dk + 1/2dT Bk d Minimizer of this quadratic function is -1 dk = -Bk f (xk ) min f (x) Fletcher-Reeves Conjugate Gradient 1 2 3 Given x0 . d0 = - f (x0 ), k = 0 Compute k . xk+1 = xk + k dk Compute f (xk+1 ), If k+1 = f (xk+1 ) = 0, stop. Else f (xk+1 )T f (xk+1 ) f (xk )T f (xk ) Step: xk+1 = xk + k dk The Question What can B do for you? Given the gradient information that you have recently seen, how would you like your model to behave? Make mk+1 ( ) match f ( ) at the last two points xk+1 = xk + dk , so mk+1 (0) = mk+1 (-k dk ) = Jeff Linderoth 4 Compute new direction: dk+1 = - f (xk+1 ) + k+1 dk Fletcher Reeves method is globally convergence as long as lehigh-logo your k satisfies the strong Wolfe conditions (with c2 < 1/2) Jeff Linderoth IE417:Lecture 10 f (xk+1 ) lehigh-logo f (xk ) IE417:Lecture 10 The Secant Condition If Bk+1 is to be symmetric, we have mk+1 (d) = So mk+1 (0) = f (xk+1 ) + Bk+1 d Secant Condition We want Bk+1 to be a positive definite matrix mapping sk into yk , so surely we need Curvature Condition sT yk > 0 to hold k Problem 8.1: If f (x) is strongly convex, then this conditions holds. Further, the curvature condition holds if we impose the Wolfe conditions on the line search, since Second Wolfe Condition: f (xk + k dk )T dk c2 f (xk )T dk -1 -1 Or k f (xk+1 )T sk k c2 f (xk )T sk So f (xk+1 )T sk - Or f (xk )T sk + (1 - c2 ) f (xk )T sk 0 f (xk+1 ) automatically To get the condition: mk+1 (-k dk ) = f (xk+1 ) - Bk+1 dk , we need Bk+1 k dk = f (xk+1 ) - f (xk ) Define displacement: sk = xk+1 - xk Define Gradient Change: yk = Secant Equation Bk+1 sk = yk Jeff Linderoth IE417:Lecture 10 lehigh-logo f (xk+1 ) - f (xk ) T yk sk (c2 - 1)k f (xk )T dk > 0 lehigh-logo since dk is a descent direction and c2 < 1 Jeff Linderoth IE417:Lecture 10 Freedom! We have some freedom in how we choose Bk+1 Let's do a rank one update Bk+1 = Bk + vv T We also want Bk+1 sk = yk , i.e. (Bk + vv T )sk = yk A solution to this is v = yk - Bk sk = (sT (yk k - Bk sk )) -1 Sherman Morrison Formula Sherman-Morrison-Woodbury ^ If A = A + uv T is a rank-one update of A, then the inverse can be updated as A-1 abT A-1 ^ A-1 = A-1 - 1 + bT A-1 a ^ Also if A = A + uV T for U, V, Rn p , then ^ A-1 = A-1 - A-1 U (I + V T A-1 U )-1 V T A-1 -1 Let Gk+1 = Bk+1 You can also just update the inverse Plug it in if you don't believe me, so the SR1 update is Bk+1 = Bk + (yk - Bk sk )(yk - Bk sk )T (yk - Bk sk )T sk lehigh-logo Gk+1 = Gk + Homework: Derive ?? In fact, if you want Bk+1 to be symmetric, is this the only solution Jeff Linderoth IE417:Lecture 10 (sk - Gk yk )(sk - Gk yk )T (sk - Gk yk )T yk IE417:Lecture 10 (1) lehigh-logo Jeff Linderoth SR1 Method Bummer!: Bk+1 Not always positive definite. So Gk+1 might not even exist! Not Bummer!: It usually works well! It is a good approximation of the Hessian, and in fact sometimes imposing that the model has no negative curvature can be bad when in fact it does have negative curvature When combined with a trust-region, the SR1 method is really quite good. What should we do if the denominator of the update is close to zero? Just Skip It Apply Hessian update (or inverse Hessian update) only if |sT (yk - Bk sk )| k 10-8 sk yk - Bk sk Jeff Linderoth IE417:Lecture 10 lehigh-logo BLAH How Good Is It!? For f (x) = 1/2xT Ax - bT x, is sT (yk - Bk sk ) = 0 k, then you k will converge to the optimal solution in n steps. lehigh-logo Jeff Linderoth IE417:Lecture 10 We Have Freedom Why don't we choose a matrix that is "close" to our current matrix. That is Bk+1 = arg min B - Bk s.t. B = B T , Bsk = yk B More on DFP This is called a rank 2 update, since it adds two symmetric rank one matrices to Bk n n n n It "just happens" that if Bk S++ , then Bk+1 S++ ! Nice! (2) With an appropriate matrix norm, one can show that DFP T T Bk+1 = (I - k yk sT )Bk (I - k sk yk ) + yk yk k T with k = (yk sk )-1 is the unique solution If you solve ?? without the condition B = B T , you get Bk+1 = Bk + (y - Bs)cT cT s This is the Davidon, Fletcher, Powell update You can also "Sherman-Morrison-Moodbury" to get an update for the inverse Gk+1 lehigh-logo for some c Rn such that cT s = 0. Note relation to Symmetric Rank 1 update lehigh-logo Jeff Linderoth IE417:Lecture 10 Jeff Linderoth IE417:Lecture 10 A Dual Perspectiver BFGS: Broyen Fletcher Goldfarb Shanno Instead of Bk+1 sk = yk , look for matrix Gk+1 yk = sk (i.e. -1 Gk+1 = Bk+1 . That is... Gk+1 = arg min G - Gk s.t. G = GT , sk = Gyk G Final Stuff Broyden Method: Take a Convex Combination of DFP and BFGS: BFGS DFP Bk+1 = (1 - )Bk+1 + k Bk+1 BFGS with line search that satiesfies the Wolfe condition is globally convergent BFGS method convergence rate is superlinear Implication of Theorem 3.5 If dk is a Quasi-Newton Search direction, and lim (Bk - 2 f (x ))d k Not surprisingly, the answer is T Gk+1 = (I - k sk yk )Gk (I - k yk sT ) + sk sT k k Why not surprising: because of mapping BH Again, you can SMW and get BFGS Bk+1 = Bk - Jeff Linderoth ys then Bk sk sT Bk k sT Bk sk k IE417:Lecture 10 k dk = 0, + T yk yk T yk sk 1 lehigh-logo 2 k = 1 will eventually be an admissible step length {xk } x superlinearly. Jeff Linderoth IE417:Lecture 10 lehigh-logo The End The Paper to Get If you really want to know about this stuff, get the paper J. E. Dennis and J. J. More, "Quasi-Newton Methods, Motivation and Theory," SIAM Review, 1977. Available from JSTOR. Homework: Turn in: 8.1. Try 8.2, 8.3, 8.4, 8.6. Matlab Hmwk: Implement BFGS, DFP, and SR1. Run on cute set and tell me which is the winner. NLP Talk Alert: Andreas W chter, IBM TJ Watson Research. a Tomorrow!, 2:30 4, Mohler Lab. lehigh-logo Jeff Linderoth IE417:Lecture 10

Find millions of documents here - Study Guides, Homework Solutions, Papers, Exam Answer Keys and more. Course Hero has millions of course related materials that will enable you to learn better, faster and get an A in all your courses.
Below is a small sample set of documents: