What remains is to show how we can actually solve an optimization problem of

# What remains is to show how we can actually solve an

• Notes
• 16

This preview shows page 3 - 5 out of 16 pages.

What remains is to show how we can actually solve an optimization problem of the form ( 2 ) without directly solving the system Hx = b . Here we will describe iterative methods — most prominently steepest descent — that do exactly this. Steepest descent Say you have an unconstrained optimization program minimize x R N f ( x ) where f ( x ) : R N R is convex. One simple way to solve this program is to simply “roll downhill”. If we are sitting at a point x 0 , then f ( · ) decreases the fastest if we move in the direction of the negative gradient -∇ f ( x ) | x = x 0 . From a starting point x 0 , we move to x 1 = x 0 - α 0 f ( x ) | x = x 0 then to x 2 = x 1 - α 1 f ( x ) | x = x 1 . . . x k = x k - 1 - α k - 1 f ( x ) | x = x k - 1 , where the α 0 , α 1 , . . . are appropriately chosen step sizes . 33 Georgia Tech ECE 6250 Fall 2019; Notes by J. Romberg and M. Davenport. Last updated 2:06, November 18, 2019

Subscribe to view the full document.

8 Jonathan Richard Shewchuk -4 -2 2 4 6 -6 -4 -2 2 4 1 2 0 Figure 8: Here, the method of Steepest Descent starts at 2 2 and converges at 2 2 . Putting it all together, the method of Steepest Descent is: (10) (11) 1 (12) The example is run until it converges in Figure 8. Note the zigzag path, which appears because each gradient is orthogonal to the previous gradient. The algorithm, as written above, requires two matrix-vector multiplications per iteration. The computa- tional cost of Steepest Descent is dominated by matrix-vector products; fortunately, one can be eliminated. By premultiplying both sides of Equation 12 by and adding , we have 1 (13) Although Equation 10 is still needed to compute 0 , Equation 13 can be used for every iteration thereafter. The product , which occurs in both Equations 11 and 13, need only be computed once. The disadvantage of using this recurrence is that the sequence defined by Equation 13 is generated without any feedback from the value of , so that accumulation of floating point roundoff error may cause to converge to some point near . This effect can be avoided by periodically using Equation 10 to recompute the correct residual. Before analyzing the convergence of Steepest Descent, I must digress to ensure that you have a solid understanding of eigenvectors. (from Shewchuk, “... without the agonizing pain”) For our particular optimization problem minimize x 1 2 x T Hx - x T b , we can explicitly compute both the gradient and the best choice of step size. The (negative) gradient is what we call the residual , the difference between b and H applied to the current iterate: - ∇ 1 2 x T Hx - x T b x = x k = b - Hx k =: r k . The steepest descent iteration can be written as x k +1 = x k + α k r k . There is a nifty way to choose an optimal value for the step size α k . We want to choose α k so that f ( x k +1 ) is as small as possible. It is not hard to show that f ( x k + α r k ) is convex as a function of α for α 0. Thus we can choose the value of α that makes the derivative of this function zero; we want d d α f ( x k + α r k ) = 0 .
• Fall '08
• Staff

### What students are saying

• As a current student on this bumpy collegiate pathway, I stumbled upon Course Hero, where I can find study resources for nearly all my courses, get online help from tutors 24/7, and even share my old projects, papers, and lecture notes with other students.

Kiran Temple University Fox School of Business ‘17, Course Hero Intern

• I cannot even describe how much Course Hero helped me this summer. It’s truly become something I can always rely on and help me. In the end, I was not only able to survive summer classes, but I was able to thrive thanks to Course Hero.

Dana University of Pennsylvania ‘17, Course Hero Intern

• The ability to access any university’s resources through Course Hero proved invaluable in my case. I was behind on Tulane coursework and actually used UCLA’s materials to help me move forward and get everything together on time.

Jill Tulane University ‘16, Course Hero Intern