This preview shows page 1. Sign up to view the full content.
Unformatted text preview: a solution is faster for the Newton method. In particular, the Newton method has secondorder convergence (or
quadratic convergence) in a local neighborhood of the solution x * , such that for
all k sufficiently large it holds that x k +1  x * £ C xk  x * 2 for some constant C>0 Steepest descend, on the other hand, has firstorder convergence (or linear convergence) in a local neighborhood of the solution x * , which means that for
all k sufficiently large it holds that x k +1  x * £ c x k  x * for some constant
0<c<1
[For you: So if the rate of convergence is higher for the Newton method, why
wouldn't you always use it?] RISK AND PORTFOLIO MANAGEMENT WITH ECONOMETRICS, VER. 11/21/2012. © P. KOLM. 37 Remarks: The main advantage of the standard Newton method is its fast local convergence Local convergence means that if we are sufficiently close to a solution the method guarantees finding it Although the method of steepest descend converges slower than the Newton it always guarantees to decrease the value of the objective function.10 Therefore,
steepest descend and Newton type of methods are sometimes combined and
used together in the same optimization routine making them one of the most
efficient tools for smoothed unconstrained minimization The main drawback of the Newtontype methods is their relatively high computational cost [For you: Why does the Newton method have higher
computational cost?] Socalled Modified Newton, QuasiNewton, and conjugate gradient methods are often computationally more efficient for large problems and converge faster
than the method of steepest descent RISK AND PORTFOLIO MANAGEMENT WITH ECONOMETRICS, VER. 11/21/2012. © P. KOLM. 38 Modified and QuasiNewton Methods For modified Newton and QuasiNewton methods the search direction is chosen
to be pk = Bk1fk
where Bk is a positive definite approximation of the true Hessian ( 2 f (x k ) )
In one of the most successful and widely used generalpurpose QuasiNewton
methods known as BFGS (Broyden, Fletcher, Goldfarb, and Shanno) the
approximations are calculated according to Bk +1 = Bk + qkqk¢
B¢ s¢ s B
 k k k k , B0 = I
qk¢ sk
sk¢ Bksk where I is the N×N identity matrix, and
sk = x k +1  x k
qk = f (x k +q )  f (x k ) RISK AND PORTFOLIO MANAGEMENT WITH ECONOMETRICS, VER. 11/21/2012. © P. KOLM. 39 Solution Techniques for Nonlinear Programming Problems with Constraints KarushKuhnTucker optimality conditions for a nonlinear program takes the form of a system of nonlinear equations Therefore, in order to solve the optimization problem, the majority of algorithms apply either some variant of the Newton method to this system of
equations or solve a sequence of approximations of this system (like in “the
sequential quadratic programming approach” described next) RISK AND PORTFOLIO MANAGEMENT WITH ECONOMETRICS, VER. 11/21/2012. © P. KOLM. 40 The Sequential Quadratic Programming Approach Recall min f (x )
x s.t. gi (x ) £ 0 i = 1,..., I
h j (x ) = 0 j = 1,..., J
where f, gi , and h j are smooth functions of the N dimensional variable x RISK AND POR...
View
Full
Document
This document was uploaded on 02/17/2014 for the course COURANT G63.2751.0 at NYU.
 Fall '14

Click to edit the document details