chapter5

# chapter5 - 5 ITERATIVE METHODS FOR OPTIMIZATION Our first...

This preview shows pages 1–3. Sign up to view the full content.

This preview has intentionally blurred sections. Sign up to view the full version.

View Full Document
This is the end of the preview. Sign up to access the rest of the document.

Unformatted text preview: 5. ITERATIVE METHODS FOR OPTIMIZATION Our first step in solving an optimization problem is to use some form of necessary condition to identify candidates for optimality. For unconstrained problems, this means finding the critical points of the objective function, which in turn requires solving ∇ f ( x ) = 0 for x . In general, a nonlinear system of equations can be difficult (or impossible) to solve exactly, so it is useful to have some scheme for approximating solutions. This chapter informally presents some of the standard methods for unconstrained minimization. However, the theory and numerical analysis of these methods go well beyond the scope of this course; further details can be found in the text of Dennis and Schnabel. 1 5.1 Descent Methods Suppose we want to minimize f ( x ) over all x ∈ R n and ¯ x is our current best “guess.” In the one-dimensional case ( n = 1), there is a simple rule for attempting to improve upon ¯ x : • if f (¯ x ) > 0, we can decrease f by moving to the left of ¯ x ; • if f (¯ x ) < 0, we can decrease f by moving to the right of ¯ x . In higher dimensions, we can do the same thing in each coordinate: • if ( ∂f/∂x i )(¯ x ) > 0, we can decrease f by decreasing x i ; • if ( ∂f/∂x i )(¯ x ) < 0, we can decrease f by increasing x i . Methods based on this idea are known as coordinate-descent methods. Here is a more general notion of descent. Definition 5.1.1 (Descent direction) . A vector v is called a descent direction for f at ¯ x if there exists δ > 0 so that f (¯ x + tv ) < f (¯ x ) for all t ∈ (0 ,δ ). Method 5.1.2 (General framework for descent methods) . Given an initial guess x (0) , generate a sequence of iterates { x ( k ) } according to the following procedure: 1. [Search direction] Calculate a descent direction d ( k ) for f at x ( k ) . 2. [Step-size] Choose a value t ( k ) so that f ( x ( k ) + t ( k ) d ( k ) ) provides an “acceptable de- crease” below the value f ( x ( k ) ). 3. [Update] Set x ( k +1) = x ( k ) + t ( k ) d ( k ) . 4. [Check for termination] Stop if some pre-specified criterion has been met; otherwise, replace k by k + 1 and go to step 1. 1 J.E. Dennis and R.B. Schnabel, Numerical Methods for Unconstrained Optimization and Nonlinear Equations , Prentice Hall, Englewood Cliffs, NJ, 1983. 50 There’s a lot of leeway in steps 1, 2, and 4 of the above framework. Different choices allow one to customize the procedure and perhaps prove convergence of the sequence { x ( k ) } to a point ¯ x with desirable properties, such as the necessary conditions for optimality. For step 1, the simplest way to identify a descent direction is to consider the derivative of the v-restriction ϕ v ( t ) = f (¯ x + tv ) of f at ¯ x in the direction v . Note that v is a descent direction for f at ¯ x if ϕ v (0) < 0, and recall that ϕ v ( t ) = ∇ f (¯ x + tv ) · v ....
View Full Document

## This note was uploaded on 03/18/2012 for the course MTH 432 taught by Professor Douglasward during the Spring '12 term at Miami University.

### Page1 / 10

chapter5 - 5 ITERATIVE METHODS FOR OPTIMIZATION Our first...

This preview shows document pages 1 - 3. Sign up to view the full document.

View Full Document
Ask a homework question - tutors are online