Optimization

# In particular the newton method has second order

This preview shows page 1. Sign up to view the full content.

This is the end of the preview. Sign up to access the rest of the document.

Unformatted text preview: a solution is faster for the Newton method. In particular, the Newton method has second-order convergence (or quadratic convergence) in a local neighborhood of the solution x * , such that for all k sufficiently large it holds that x k +1 - x * £ C xk - x * 2 for some constant C&gt;0 Steepest descend, on the other hand, has first-order convergence (or linear convergence) in a local neighborhood of the solution x * , which means that for all k sufficiently large it holds that x k +1 - x * £ c x k - x * for some constant 0&lt;c&lt;1 [For you: So if the rate of convergence is higher for the Newton method, why wouldn't you always use it?] RISK AND PORTFOLIO MANAGEMENT WITH ECONOMETRICS, VER. 11/21/2012. © P. KOLM. 37 Remarks: The main advantage of the standard Newton method is its fast local convergence Local convergence means that if we are sufficiently close to a solution the method guarantees finding it Although the method of steepest descend converges slower than the Newton it always guarantees to decrease the value of the objective function.10 Therefore, steepest descend and Newton type of methods are sometimes combined and used together in the same optimization routine making them one of the most efficient tools for smoothed unconstrained minimization The main drawback of the Newton-type methods is their relatively high computational cost [For you: Why does the Newton method have higher computational cost?] So-called Modified Newton, Quasi-Newton, and conjugate gradient methods are often computationally more efficient for large problems and converge faster than the method of steepest descent RISK AND PORTFOLIO MANAGEMENT WITH ECONOMETRICS, VER. 11/21/2012. © P. KOLM. 38 Modified and Quasi-Newton Methods For modified Newton and Quasi-Newton methods the search direction is chosen to be pk = -Bk-1fk where Bk is a positive definite approximation of the true Hessian ( 2 f (x k ) ) In one of the most successful and widely used general-purpose Quasi-Newton methods known as BFGS (Broyden, Fletcher, Goldfarb, and Shanno) the approximations are calculated according to Bk +1 = Bk + qkqk¢ B¢ s¢ s B - k k k k , B0 = I qk¢ sk sk¢ Bksk where I is the N×N identity matrix, and sk = x k +1 - x k qk = f (x k +q ) - f (x k ) RISK AND PORTFOLIO MANAGEMENT WITH ECONOMETRICS, VER. 11/21/2012. © P. KOLM. 39 Solution Techniques for Nonlinear Programming Problems with Constraints Karush-Kuhn-Tucker optimality conditions for a nonlinear program takes the form of a system of nonlinear equations Therefore, in order to solve the optimization problem, the majority of algorithms apply either some variant of the Newton method to this system of equations or solve a sequence of approximations of this system (like in “the sequential quadratic programming approach” described next) RISK AND PORTFOLIO MANAGEMENT WITH ECONOMETRICS, VER. 11/21/2012. © P. KOLM. 40 The Sequential Quadratic Programming Approach Recall min f (x ) x s.t. gi (x ) £ 0 i = 1,..., I h j (x ) = 0 j = 1,..., J where f, gi , and h j are smooth functions of the N dimensional variable x RISK AND POR...
View Full Document

## This document was uploaded on 02/17/2014 for the course COURANT G63.2751.0 at NYU.

Ask a homework question - tutors are online