This preview has intentionally blurred sections. Sign up to view the full version.
View Full Document
Unformatted text preview: 5. ITERATIVE METHODS FOR OPTIMIZATION Our first step in solving an optimization problem is to use some form of necessary condition to identify candidates for optimality. For unconstrained problems, this means finding the critical points of the objective function, which in turn requires solving ∇ f ( x ) = 0 for x . In general, a nonlinear system of equations can be difficult (or impossible) to solve exactly, so it is useful to have some scheme for approximating solutions. This chapter informally presents some of the standard methods for unconstrained minimization. However, the theory and numerical analysis of these methods go well beyond the scope of this course; further details can be found in the text of Dennis and Schnabel. 1 5.1 Descent Methods Suppose we want to minimize f ( x ) over all x ∈ R n and ¯ x is our current best “guess.” In the onedimensional case ( n = 1), there is a simple rule for attempting to improve upon ¯ x : • if f (¯ x ) > 0, we can decrease f by moving to the left of ¯ x ; • if f (¯ x ) < 0, we can decrease f by moving to the right of ¯ x . In higher dimensions, we can do the same thing in each coordinate: • if ( ∂f/∂x i )(¯ x ) > 0, we can decrease f by decreasing x i ; • if ( ∂f/∂x i )(¯ x ) < 0, we can decrease f by increasing x i . Methods based on this idea are known as coordinatedescent methods. Here is a more general notion of descent. Definition 5.1.1 (Descent direction) . A vector v is called a descent direction for f at ¯ x if there exists δ > 0 so that f (¯ x + tv ) < f (¯ x ) for all t ∈ (0 ,δ ). Method 5.1.2 (General framework for descent methods) . Given an initial guess x (0) , generate a sequence of iterates { x ( k ) } according to the following procedure: 1. [Search direction] Calculate a descent direction d ( k ) for f at x ( k ) . 2. [Stepsize] Choose a value t ( k ) so that f ( x ( k ) + t ( k ) d ( k ) ) provides an “acceptable de crease” below the value f ( x ( k ) ). 3. [Update] Set x ( k +1) = x ( k ) + t ( k ) d ( k ) . 4. [Check for termination] Stop if some prespecified criterion has been met; otherwise, replace k by k + 1 and go to step 1. 1 J.E. Dennis and R.B. Schnabel, Numerical Methods for Unconstrained Optimization and Nonlinear Equations , Prentice Hall, Englewood Cliffs, NJ, 1983. 50 There’s a lot of leeway in steps 1, 2, and 4 of the above framework. Different choices allow one to customize the procedure and perhaps prove convergence of the sequence { x ( k ) } to a point ¯ x with desirable properties, such as the necessary conditions for optimality. For step 1, the simplest way to identify a descent direction is to consider the derivative of the vrestriction ϕ v ( t ) = f (¯ x + tv ) of f at ¯ x in the direction v . Note that v is a descent direction for f at ¯ x if ϕ v (0) < 0, and recall that ϕ v ( t ) = ∇ f (¯ x + tv ) · v ....
View
Full
Document
This note was uploaded on 03/18/2012 for the course MTH 432 taught by Professor Douglasward during the Spring '12 term at Miami University.
 Spring '12
 DouglasWard
 Critical Point

Click to edit the document details